Resources Numerical accuracy Calculation policy

Calc sheet units, functions and driven parameters

The unit factors, powers, functions and conversions of a calc sheet, and the values a sheet writes into the app's parameters.

How the precision is kept

Every value is a 64-bit number in SI base units with its dimensions beside it. Unit factors are decimal constants; most are the exact definitions, and three are written with fewer digits than 64-bit holds. A conversion divides by the target unit's factor once. A row that drives a parameter passes the full number, but skips the update when the change is below 1e-9 of the value.

Open item. Write psi, ksi and psf from their exact definitions (lbf ÷ in² and lbf ÷ ft²), and update a driven parameter on any change.

Calculations under this policy

13 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

calc.js Shared shell, every app 64-bit 13 calculations Reviewed

Tests bin/test-calc.mjs

Built-in unit factors

64-bit

The factor from each built-in unit of mass, time, force, pressure, energy, power, current, voltage and volume to SI.

lb = 0.45359237 kg; lbf = 4.4482216152605 N; psi = 6894.757293168 Pa; psf = 47.88025898 Pa

lb, oz, lbf, kip, kgf, gal, BTU (1055.05585262 J) and the metric multiples are the exact defined values. psi and ksi stop at 13 significant digits (the definition gives 6894.757293168361…) and psf at 10 (47.88025898033584…). hp is written to 14 digits and differs from 550 ft·lbf/s by 3 parts in 10¹⁶.

  • calc.js
  • lines 50 to 59
const SI = {
  kg: [1, dim(0, 1)], g: [1e-3, dim(0, 1)], mg: [1e-6, dim(0, 1)], t: [1000, dim(0, 1)], tonne: [1000, dim(0, 1)], lb: [0.45359237, dim(0, 1)], lbs: [0.45359237, dim(0, 1)], oz: [0.028349523125, dim(0, 1)],
  s: [1, dim(0, 0, 1)], ms: [1e-3, dim(0, 0, 1)], min: [60, dim(0, 0, 1)], h: [3600, dim(0, 0, 1)], hr: [3600, dim(0, 0, 1)], d: [86400, dim(0, 0, 1)], day: [86400, dim(0, 0, 1)], wk: [604800, dim(0, 0, 1)],
  N: [1, dim(1, 1, -2)], kN: [1e3, dim(1, 1, -2)], MN: [1e6, dim(1, 1, -2)], lbf: [4.4482216152605, dim(1, 1, -2)], kip: [4448.2216152605, dim(1, 1, -2)], kgf: [9.80665, dim(1, 1, -2)],
  Pa: [1, dim(-1, 1, -2)], kPa: [1e3, dim(-1, 1, -2)], MPa: [1e6, dim(-1, 1, -2)], GPa: [1e9, dim(-1, 1, -2)], bar: [1e5, dim(-1, 1, -2)], psi: [6894.757293168, dim(-1, 1, -2)], ksi: [6894757.293168, dim(-1, 1, -2)], psf: [47.88025898, dim(-1, 1, -2)],
  J: [1, dim(2, 1, -2)], kJ: [1e3, dim(2, 1, -2)], MJ: [1e6, dim(2, 1, -2)], Wh: [3600, dim(2, 1, -2)], kWh: [3.6e6, dim(2, 1, -2)], BTU: [1055.05585262, dim(2, 1, -2)],
  W: [1, dim(2, 1, -3)], kW: [1e3, dim(2, 1, -3)], MW: [1e6, dim(2, 1, -3)], hp: [745.69987158227, dim(2, 1, -3)],
  A: [1, dim(0, 0, 0, 1)], mA: [1e-3, dim(0, 0, 0, 1)], kA: [1e3, dim(0, 0, 0, 1)], V: [1, dim(2, 1, -3, -1)], kV: [1e3, dim(2, 1, -3, -1)], mV: [1e-3, dim(2, 1, -3, -1)],
  ohm: [1, dim(2, 1, -3, -2)], K: [1, dim(0, 0, 0, 0, 1)], L: [1e-3, dim(3)], lit: [1e-3, dim(3)], liter: [1e-3, dim(3)], litre: [1e-3, dim(3)], gal: [3.785411784e-3, dim(3)],
};

Constants and fallback lengths

64-bit

pi, e, standard gravity g0, and the length units used when no unit table is given.

g0 = 9.80665 m/s²; in = 0.0254 m; ft = 0.3048 m; yd = 0.9144 m

g0, in, ft and yd are the exact defined values. pi and e are Math.PI and Math.E, the nearest 64-bit numbers.

  • calc.js
  • lines 60 to 62
const CONST = { pi: [Math.PI, D0], e: [Math.E, D0], g0: [9.80665, dim(1, 0, -2)] };
// Length fallback when no units table is given.
const LEN = { m: 1, mm: 1e-3, cm: 1e-2, km: 1e3, in: 0.0254, ft: 0.3048, yd: 0.9144 };

Unit lookup

64-bit

The SI factor and dimension of a unit name.

deg = π ÷ 180; rad = 1

Built-in units come from the table above; lengths, areas, volumes and angles come from the shared unit table's factor(). A degree is Math.PI ÷ 180, one rounded division.

  • calc.js
  • lines 249 to 256
function unitValue(id) {
  if (Object.prototype.hasOwnProperty.call(SI, id)) return { v: SI[id][0], d: SI[id][1] };
  if (lib) for (const [kind, d] of KINDS) { const u = lib.unitOfMark(id, kind); if (u) return { v: lib.factor(u), d, id: u }; }
  else if (Object.prototype.hasOwnProperty.call(LEN, id)) return { v: LEN[id], d: dim(1) };
  if (id === 'deg' || id === '°') return { v: Math.PI / 180, d: D0 };
  if (id === 'rad') return { v: 1, d: D0 };
  return null;
}

Dimension comparison

64-bit

Whether two values have the same dimensions, so they can be added, compared or converted.

|aᵢ − bᵢ| < 1e-9 for each of the five exponents

Exponents are small numbers such as 1, 2 or 0.5, compared with an absolute allowance of 1e-9. The allowance absorbs rounding in fractional powers; it does not touch the value itself.

  • calc.js
  • line 34
const same = (a, b) => a.every((x, i) => Math.abs(x - b[i]) < 1e-9);

Power

64-bit

a ^ b for a value with units.

value = aᵇ; dimensions × b

One Math.pow call. The exponent must have no unit, and a result that is not finite is an error. Whole-number powers are not expanded into multiplications, so the result is whatever Math.pow returns.

  • calc.js
  • lines 348 to 353
case '^': {
  if (!isZero(b.d)) fail('unit_mismatch', 'An exponent cannot have a unit (' + describe(b.d) + ').', '');
  const r = Math.pow(a.v, b.v);
  if (!Number.isFinite(r)) fail('domain', 'That power is not a real number.', '');
  return { v: r, d: dpow(a.d, b.v) };
}

Conversion with to

64-bit

M to kN m: checks the dimension and records the unit to show the answer in.

shown = value ÷ factor of the target unit

The value stays in SI; only the factor of the target unit is recorded. The division happens once, when the row is shown.

  • calc.js
  • lines 335 to 339
case 'to': {
  const a = evalTree(n.a, env), u = evalTree(n.u, env);
  if (!same(a.d, u.d)) fail('unit_mismatch', 'Cannot show ' + describe(a.d) + ' in ' + describe(u.d) + '.', 'Pick a unit of ' + describe(a.d) + '.');
  return { ...a, show: n.u, showFactor: u.v };
}

Shown amount

64-bit

The number and unit a row displays, and the amount passed to other modules.

amount = value ÷ factor

One division by the display unit's factor. The amount returned keeps all digits; the text beside it goes through sig() and has 6. With the shared unit table present, a length with no unit asked for is shown in meters.

  • calc.js
  • lines 401 to 415
/** The text of a result in a unit expression ("kN m") or the natural unit. */
function show(val, unitText, tree) {
  const d = val.d;
  if (val.bool) return { text: val.v ? 'true' : 'false', unit: '' };
  let factor = 1, label = '';
  if (unitText) {
    const u = evalTree(parse(unitText, isUnit), { vars: {}, params: new Set(), get: () => null, sheetRef: () => null });
    if (!same(u.d, d)) fail('unit_mismatch', 'The answer is ' + describe(d) + ', which cannot be shown in "' + unitText + '" (' + describe(u.d) + ').', 'Use a unit of ' + describe(d) + ', or clear the unit.');
    factor = u.v; label = unitText;
  } else if (val.show) { factor = val.showFactor; label = unparse(val.show); }
  else if (lib && same(d, dim(1)) && lib.format) { return { text: sig(val.v) + ' m', unit: 'm', amount: val.v }; }
  else { label = DEFAULT_UNIT[NAME_OF(d)] || (isZero(d) ? '' : dimText(d)); }
  const amount = val.v / factor;
  return { text: sig(amount) + (label ? ' ' + label : ''), unit: label, amount };
}

Square root and absolute value

64-bit

sqrt(a) and abs(a) for a value with units.

√a halves the dimensions

Math.sqrt on the SI value; a negative argument is an error and not NaN. The dimension exponents are multiplied by 0.5.

  • calc.js
  • lines 380 to 381
case 'sqrt': need(1); if (a[0].v < 0) fail('domain', 'The square root of a negative number.', ''); return { v: Math.sqrt(a[0].v), d: dpow(a[0].d, 0.5) };
case 'abs': need(1); return { v: Math.abs(a[0].v), d: a[0].d };

Rounding to a step

64-bit

round, floor and ceil, with an optional step such as round(x, 5 mm).

round(a ÷ s) × s

With a step the value is divided by the step, rounded and multiplied back, so the result is a multiple of the step only to within one rounding of that product. Math.round sends halves toward +∞. A zero step is an error.

  • calc.js
  • lines 382 to 386
case 'round': case 'floor': case 'ceil': {
  if (a.length < 1 || a.length > 2) fail('syntax', f + ' takes a value and, for round, an optional step.', '');
  if (a.length === 2) { alike(); const s = a[1].v; if (!s) fail('domain', 'The step cannot be zero.', ''); return { v: Math[f](a[0].v / s) * s, d: a[0].d }; }
  return { v: Math[f](a[0].v), d: a[0].d };
}

Minimum, maximum, sum and average

64-bit

min, max, sum and avg of values of one kind.

sum = Σ aᵢ in order; avg = sum ÷ n

All values must have the same dimensions. The sum adds the terms left to right with no compensation, so many terms of mixed size lose low digits in the usual way.

  • calc.js
  • lines 387 to 391
case 'min': case 'max': case 'sum': case 'avg': {
  if (!a.length) fail('syntax', f + ' needs at least one value.', ''); alike();
  const vs = a.map((x) => x.v);
  return { v: f === 'min' ? Math.min(...vs) : f === 'max' ? Math.max(...vs) : vs.reduce((s, x) => s + x, 0) / (f === 'avg' ? vs.length : 1), d: a[0].d };
}

Trigonometry, logarithms and exponent

64-bit

sin, cos, tan, their inverses, atan2, ln, log10, exp and pow.

sin, cos, tan of radians; asin, acos need |x| ≤ 1; ln, log10 need x > 0

Arguments must be plain numbers or angles, which are held in radians. asin and acos reject a value outside [−1, 1] instead of clamping it, so a result that overshoots 1 by rounding is an error. The pow() function, unlike the ^ operator, does not check for a result that is not finite.

  • calc.js
  • lines 392 to 396
case 'sin': case 'cos': case 'tan': need(1); plain(); return { v: Math[f](a[0].v), d: D0 };
case 'asin': case 'acos': case 'atan': need(1); plain(); if ((f !== 'atan') && Math.abs(a[0].v) > 1) fail('domain', f + ' needs a value between -1 and 1.', ''); return { v: Math[f](a[0].v), d: D0 };
case 'atan2': need(2); alike(); return { v: Math.atan2(a[0].v, a[1].v), d: D0 };
case 'ln': case 'log10': case 'exp': need(1); plain(); if (f !== 'exp' && a[0].v <= 0) fail('domain', 'The logarithm of a number that is not positive.', ''); return { v: Math[f === 'ln' ? 'log' : f](a[0].v), d: D0 };
case 'pow': need(2); if (!isZero(a[1].d)) fail('unit_mismatch', 'An exponent cannot have a unit.', ''); return { v: Math.pow(a[0].v, a[1].v), d: dpow(a[0].d, a[1].v) };

Reading an app parameter

64-bit

@Name: the app's parameter as a value in SI.

value = parameter value × factor of its unit

One multiplication by the unit's factor. A parameter given as a bare number is taken as a plain number with no unit.

  • calc.js
  • lines 307 to 319
// ---- evaluation ----
function paramValue(name) {
  if (!params || typeof params.get !== 'function') fail('param_missing', 'This app has not handed its parameters over, so @' + name + ' cannot be read.', 'Use a number instead, or open a document that has parameters.');
  const g = params.get(name);
  if (g === null || g === undefined) fail('param_missing', 'There is no parameter "' + name + '" in this document.', 'Check the name' + (typeof params.list === 'function' ? ' (parameters: ' + params.list().slice(0, 12).join(', ') + ')' : '') + '.');
  if (typeof g === 'number') return { v: g, d: D0 };
  if (g && typeof g.value === 'number') {
    if (!g.unit) return { v: g.value, d: D0 };
    const u = unitValue(g.unit); if (!u) fail('unknown_unit', 'The parameter "' + name + '" is in a unit this sheet does not know: ' + g.unit + '.', '');
    return { v: g.value * u.v, d: u.d };
  }
  fail('param_missing', 'The parameter "' + name + '" has no number.', '');
}

Values written to driven parameters

64-bit

What each drive would set in the app: the row's answer in the parameter's unit.

value = row value ÷ factor of the unit; change when |current − value| > 1e-9 × max(1, |value|)

The value handed to the app is the full 64-bit quotient. The change test is relative at 1e-9 with a floor of 1e-9 absolute, so a parameter is left as it is when the new value differs only in the tenth digit or beyond.

  • calc.js
  • lines 547 to 561
/** What the drives would set now: [{ param, row, value, unit, current, change }] and the ones that cannot. */
function plan() {
  const res = evaluate(), set = [], skipped = [];
  for (const d of doc.drives) {
    const h = find(d.row), r = h && res.get(h.sheet.id + '.' + h.row.name);
    if (!r || !r.ok) { skipped.push({ param: d.param, row: d.row, why: r && r.error ? r.error.message : 'The row is gone.' }); continue; }
    let value = r.value, unit = d.unit || '';
    if (unit) { const u = evalTree(parse(unit, isUnit), { vars: {}, params: new Set(), get: () => null, sheetRef: () => null }); if (!same(u.d, r.dim)) { skipped.push({ param: d.param, row: d.row, why: 'The row is ' + describe(r.dim) + ' and the parameter is set in ' + unit + '.' }); continue; } value = r.value / u.v; }
    else if (!isZero(r.dim)) { const g = params && params.get ? params.get(d.param) : null; const gu = g && typeof g === 'object' && g.unit ? g.unit : ''; if (gu) { const u = unitValue(gu); if (u && same(u.d, r.dim)) { value = r.value / u.v; unit = gu; } } }
    const cur = params && params.get ? params.get(d.param) : null;
    const curNum = cur && typeof cur === 'object' ? cur.value : cur;
    set.push({ param: d.param, row: d.row, value, unit, current: typeof curNum === 'number' ? curNum : null, change: typeof curNum !== 'number' || Math.abs(curNum - value) > 1e-9 * Math.max(1, Math.abs(value)) });
  }
  return { set, skipped };
}

In detail

Method
Each unit is a factor to SI and a dimension vector. Functions check dimensions, then call the matching Math function. A drive divides the row's SI value by the factor of the parameter's unit.
Number types
64-bit numbers. Dimension exponents are 64-bit and may be fractions (a square root halves them).
Tolerances
Dimension exponents are compared within 1e-9 (absolute). A driven parameter is updated only when |current − new| > 1e-9 × max(1, |new|).
Limits
psi and ksi are written to 13 significant digits and psf to 10, so results in those units agree with the exact definitions to about 5 parts in 10¹⁴ and 7 parts in 10¹². A driven parameter can stay up to 1e-9 (relative) away from the sheet's value.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach