Unit conversion

How the precision is kept

Each unit is an exact fraction of its base unit, written from its legal definition (the international foot is 3048/10000 m, the US survey foot 1200/3937 m). A conversion combines the two fractions as whole numbers before it touches the value, so the factor adds no rounding of its own: 3 yd is exactly 9 ft. Degrees and gons carry π, held as the nearest 64-bit value. A test re-derives every fraction in exact arithmetic.

4 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

/** A value from one unit to another of the same kind (slopes: through the grade). NaN when they do not go together. */
export function convert(v, from, to) {
  const a = BY_ID.get(from), b = BY_ID.get(to);
  if (!a || !b || a.kind !== b.kind || !Number.isFinite(v)) return NaN;
  if (a.kind === 'slope') return fromBase(toBase(v, from), to);
  if (a === b || a.factor === b.factor) return v;
  if (a.pi === b.pi) return v * (a.n * b.d) / (a.d * b.n);   // one rounding: the fraction first
  return v * a.factor / b.factor;
}

/** parse(), with the value in unit `to` (same kind) and one rounding: parseTo('3 yd', 'length', 'ft') → value 9 exactly. */
export function parseTo(text, kind, to, opts) {
  const r = parse(text, kind, opts);
  if (!r.ok) return r;
  return { ...r, value: kind === 'slope' ? fromBase(r.value, to) : convert(r.amount, r.per, to) };
}

/** A number in a unit → the base (slopes: H:V 3 → 1/3, percent 2 → 0.02). */
export function toBase(v, id) {
  const u = BY_ID.get(id);
  if (!u || !Number.isFinite(v)) return NaN;
  if (u.id === 'hv') return v === 0 ? Infinity : 1 / v;
  return u.pi ? v * Math.PI * u.n / u.d : v * u.n / u.d;
}

/** The base → a number in a unit. */
export function fromBase(v, id) {
  const u = BY_ID.get(id);
  if (!u || Number.isNaN(v)) return NaN;
  if (u.id === 'hv') return v === 0 ? Infinity : 1 / v;
  return u.pi ? v / Math.PI * u.d / u.n : v * u.d / u.n;
}

In detail

Method

Each unit is stored as a numerator and a denominator: one unit is n/d of the base unit (meter, square meter, cubic meter, radian, or rise over run for slopes). A conversion combines the two fractions as whole numbers and then applies them to the value.

Number types

Whole-number numerators and denominators, kept below 2^53 so each fraction is exact. Values are 64-bit.

Tolerances

None in the conversion. Rounding for display happens only when a value is written out: a number of decimals per unit, feet and inches to the nearest 1/16 inch unless another denominator is chosen, angles to whole seconds.

Checked by

The unit test re-derives every fraction from the legal definition of the unit in exact arithmetic.

Limits

Degrees and gons include π, which no binary number holds exactly. It is the nearest 64-bit value.

Approach: Exact values

Constants and factors are held as exact fractions from their definitions. The arithmetic is ordered so that no rounded factor is ever used.

Every policy that takes this approach