Round to 3 decimals
64-bitThe rounding applied to every coordinate in the package.
round(v × 1000) ÷ 1000
One 64-bit multiply, round and divide. The exported coordinate keeps 3 decimals only.
const r3 = (v) => Math.round(v * 1000) / 1000;Resources Numerical accuracy Calculation policy
Redline package coordinates
The positions written into a redline package that lists each markup on its sheet.
How the precision is kept
Each markup's box and points are written twice, in points and as fractions of the sheet, and every value is rounded to 3 decimals.
Calculations under this policy
2 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.
redline.js MarkupCraft 0.8.1
Tests None found by the review
Position as a fraction of the sheet
64-bitA markup's box and points as fractions of the sheet's width and height.
fx = (x − x₀) ÷ w; fy = (y − y₀) ÷ h
The subtraction and division are 64-bit and the result goes through r3. w and h come from the sheet view box and there is no check for a zero size.
const w = s.view[2] - s.view[0], h = s.view[3] - s.view[1];
const f = (x, y) => [r3((x - s.view[0]) / w), r3((y - s.view[1]) / h)];
const lo = f(b[0], b[1]), hi = f(b[2], b[3]);In detail
- Method
- Fraction = (coordinate − sheet origin) ÷ sheet size; both forms rounded with round(v × 1000) ÷ 1000.
- Number types
- 64-bit numbers.
- Tolerances
- Rounding step 0.001: 0.001 pt for points and sheet size, and 0.1 percent of the sheet width or height for fractions.
- Limits
- On a large sheet a fraction step of 0.001 is several points. The sheet width and height are not guarded against zero.
Approach: 64-bit, stated limit
The arithmetic is 64-bit, but the method or the data sets the accuracy: a solver tolerance, a mesh, a step size, a datum shift, a table. The limit is stated where the result is shown.