Numbers written into edited PDF page content

How the precision is kept

Every number the editor writes into page content passes through one function that keeps at most 4 decimals and no exponent. The page coordinates it writes are therefore rounded in the saved file.

1 calculation in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

/** A number as a content stream writes it (no exponent, at most 4 decimals). */
export function num(v) {
  if (!Number.isFinite(v)) return '0';
  const s = (Math.abs(v) < 5e-5 ? 0 : v).toFixed(4).replace(/\.?0+$/, '');
  return s === '-0' ? '0' : s;
}

In detail

Method

Number.prototype.toFixed(4), trailing zeros removed.

Number types

64-bit numbers in, text out.

Tolerances

Values with an absolute value under 5e-5 are written as 0. The step is 0.0001 in PDF user-space units (points on a default page).

Limits

Rounding is absolute, not relative to the size of the number. Non-finite values are written as 0.

Approach: 64-bit, stated limit

The arithmetic is 64-bit, but the method or the data sets the accuracy: a solver tolerance, a mesh, a step size, a datum shift, a table. The limit is stated where the result is shown.

Every policy that takes this approach