Resources Numerical accuracy Calculation policy

Check report values

The required and found values written in the Problems panel and in the check report exported as CSV.

How the precision is kept

A problem keeps its required and found values as 64-bit numbers. When they are written as text, in the panel and in the CSV export, the default formatter rounds each to 2 decimal places in its own unit. The report object the export is built from still holds the unrounded numbers.

Open item. Write the Required and Found columns of the CSV export with all the digits the numbers hold.

Calculations under this policy

1 calculation in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

problems.js Shared shell, every app 64-bit 1 calculation Reviewed

Tests bin/test-problems.mjs

Required and found values as text

64-bit

A limit or a measured value in words, such as ≤ 196.85 mm or 863.6–965.2 mm.

text = round(x × 100) ÷ 100, then the unit

The default formatter rounds to 2 decimals in the value's own unit, so a value such as 0.004 m is written as 0 m. The CSV writer in the same file calls this function with the default formatter for the Required and Found columns, so the exported file has the rounded text. A caller can pass its own formatter.

  • problems.js
  • lines 469 to 477
/** A requirement or a found value in words: "≤ 196.85 mm", "863.6–965.2 mm". fmt(value, unit) writes one number. */
export function valueText(v, fmt = (x, u) => String(Math.round(x * 100) / 100) + (u && u !== 'pct' ? ' ' + u : u === 'pct' ? ' %' : '')) {
  if (!v) return '';
  if (v.op) {
    if (v.op === 'between') return fmt(v.value[0], v.unit).replace(/\s*\S+$/, '') + '–' + fmt(v.value[1], v.unit);
    return v.op.replace('<=', '≤').replace('>=', '≥') + ' ' + fmt(v.value, v.unit);
  }
  return v.value === null ? '–' : fmt(v.value, v.unit);
}

In detail

Method
Math.round(x × 100) ÷ 100, then the unit; a range is written as two rounded numbers.
Number types
64-bit numbers.
Checked by
The Problems test compares the text of a range with the expected 863.6–965.2 mm.
Limits
The CSV columns Required and Found carry 2 decimal places at most, whatever the unit.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach