Resources Numerical accuracy Calculation policy

Calc sheets

Unit-aware formulas saved with the file: spans, loads, moments, pass or fail checks.

How the precision is kept

Formulas are evaluated in 64-bit with dimensions tracked, so adding a length to a force is an error and not a number. Results are shown to 6 significant digits; the value kept in the sheet and handed to the model has all of them.

Open item. Show up to 15 significant digits when asked.

Calculations under this policy

3 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

calc.js Shared shell, every app 64-bit 3 calculations Reviewed

Tests bin/test-calc.mjs

Add, subtract, multiply, divide

64-bit

The four operations on values that carry units.

a + b and a − b need the same dimensions; a × b and a ÷ b combine them

One 64-bit operation each, on values already in base units. Division by zero is an error, never infinity.

  • calc.js
  • lines 343 to 347
case '+': case '-':
  if (!same(a.d, b.d)) fail('unit_mismatch', 'Cannot ' + (n.op === '+' ? 'add' : 'subtract') + ' ' + describe(a.d) + ' and ' + describe(b.d) + '.', 'Both sides need the same kind of unit.');
  return { v: n.op === '+' ? a.v + b.v : a.v - b.v, d: a.d };
case '*': return { v: a.v * b.v, d: dmul(a.d, b.d) };
case '/': if (b.v === 0) fail('domain', 'Division by zero.', 'Check the divisor.'); return { v: a.v / b.v, d: ddiv(a.d, b.d) };

Comparisons

64-bit

Pass or fail checks such as M <= 40 kN m.

a ≤ b is true when a ≤ b + 1e-12 × max(1, |a|, |b|)

The allowance is about four thousand times the rounding of a 64-bit number: enough that a result equal to its limit passes, far too small to change an engineering answer.

  • calc.js
  • lines 356 to 358
const e = 1e-12 * Math.max(1, Math.abs(a.v), Math.abs(b.v));
const r = { '<': a.v < b.v - e, '<=': a.v <= b.v + e, '>': a.v > b.v + e, '>=': a.v >= b.v - e, '==': Math.abs(a.v - b.v) <= e, '!=': Math.abs(a.v - b.v) > e }[n.op];
return { v: r ? 1 : 0, d: D0, bool: true };

Number shown

64-bit

A result as text in the sheet.

6 significant digits; exponent form from 1e9 up and below 1e-4

This rounds the text only. The number kept in the sheet is unchanged. Showing fewer digits than the number holds is the open item on this policy.

  • calc.js
  • lines 227 to 234
// ---- numbers → text ----
function sig(x, digits = 6) {
  if (!Number.isFinite(x)) return String(x);
  if (x === 0) return '0';
  const a = Math.abs(x);
  if (a >= 1e9 || a < 1e-4) return x.toExponential(digits - 1).replace(/\.?0+e/, 'e').replace('e+', '×10^').replace('e-', '×10^-');
  return String(+x.toPrecision(digits));
}

In detail

Method
Each value carries a number and its dimensions. Every operator checks that the dimensions agree, and a conversion checks that the target unit is the same kind.
Number types
64-bit numbers. Lengths, areas, volumes and angles use the shared unit table; other unit factors are decimal constants from the definitions (0.45359237 kg in a pound).
Tolerances
Comparisons allow a difference of 1e-12 relative to the larger value, so a result that equals its limit passes.
Checked by
The calc test evaluates formulas, unit conversions, dimension errors and cycles.
Limits
Results are shown to 6 significant digits, in exponent form from 1e9 up and below 1e-4.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach