Resources Numerical accuracy Calculation policy
How the simulation window rounds the numbers it prints and puts into input boxes.
Results are rounded when they are turned into text, to 4 significant digits (sig) or to a fixed number of decimals (fmt). The numbers behind them are not changed. A value shown in an input box is cut to 6 decimals, and that text is what is read back if the person edits the box.
2 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.
Tests None found by the review
Prints a result with 4 significant digits.
toPrecision(4); toExponential(3) when |v| ≥ 1e5 or 0 < |v| < 1e-3
The text is rounded to 4 digits and converted back with a unary plus to drop trailing zeros. The value is not stored.
/** 4 significant figures, plain. */
export const sig = (v) => (Number.isFinite(v) ? (Math.abs(v) >= 1e5 || (Math.abs(v) < 1e-3 && v !== 0) ? v.toExponential(3) : String(+v.toPrecision(4))) : '');Prints a number with a set number of decimals, and fills number boxes with 6 decimals.
toFixed(dp), trailing zeros removed
A number box is filled with fmt(value, 6), so an input value with more than 6 decimals is shown cut, and the cut text is what is read when the person changes that box.
export const fmt = (v, dp = 3) => (Number.isFinite(v) ? String(+Number(v).toFixed(dp)) : '');The arithmetic is 64-bit, but the method or the data sets the accuracy: a solver tolerance, a mesh, a step size, a datum shift, a table. The limit is stated where the result is shown.