Resources Numerical accuracy Calculation policy

Rounding for display and for money

Rounds a number to a fixed count of decimals, and writes a length in feet and inches to the nearest fraction of an inch.

How the precision is kept

Rounding is half away from zero, with a small nudge so that a value that should sit exactly on a half but is stored a hair below it still rounds up. The same roundTo is used for labels and for money (rates and extended amounts). Inputs and outputs are 64-bit numbers; the feet-inches text is built from integers.

Calculations under this policy

4 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

units.js TakeoffCraft 0.8.1 64-bit 4 calculations Reviewed

Tests bin/test-takeoffcraft.mjs

Round half away from zero

64-bit

Rounds a number to a given count of decimals.

r = ⌊|v|·10ᵈ + 0.5 + 1e-9·max(1, |v|·10ᵈ)⌋ ÷ 10ᵈ

y = |v| × 10ᵈ is rounded as floor(y + 0.5 + 1e-9 × max(1, y)). The nudge is relative, so 1.005 (stored as 1.00499999999999989) rounds to 1.01; so does any value within 1e-9 of the boundary from below. The final ÷ 10ᵈ is one rounded division.

  • units.js
  • lines 195 to 201
  • TakeoffCraft 0.8.1
/** Half away from zero, tolerant of binary noise (1.005 → 1.01, 2.675 → 2.68). */
export function roundTo(v, digits = 0) {
  const f = 10 ** digits;
  const y = Math.abs(v) * f;
  const r = Math.floor(y + 0.5 + 1e-9 * Math.max(1, y));
  return (v < 0 ? -r : r) / f;
}

Grouped fixed-point text

64-bit

Writes a number with a fixed count of decimals and thousands separators.

The value is rounded by roundTo first and then printed with toFixed, so toFixed's own rounding never decides a digit. This is text for labels, the status line and the legend in a marked PDF; it is not read back.

  • units.js
  • lines 187 to 193
  • TakeoffCraft 0.8.1
/** Grouped fixed-point number: 1234.5 → "1,234.50". */
export function fixed(v, digits = 2) {
  if (!Number.isFinite(v)) return '';
  const r = roundTo(v, digits);
  const [i, f] = Math.abs(r).toFixed(digits).split('.');
  return (r < 0 ? '-' : '') + i.replace(/\B(?=(\d{3})+(?!\d))/g, ',') + (f ? '.' + f : '');
}

Feet and inches to the nearest fraction

64-bit

Writes a length in feet as feet, inches and a fraction of an inch.

n = round(|ft|·12·denom + 1e-9); feet = ⌊n ÷ (12·denom)⌋; inches = ⌊rem ÷ denom⌋

n = round(|ft| × 12 × denom + 1e-9) is a whole count of fractions; feet, inches and the fraction then come from integer subtraction and division, and the fraction is reduced by halving. The 1e-9 nudge is absolute in units of one fraction.

  • units.js
  • lines 159 to 172
  • TakeoffCraft 0.8.1
/** Feet-inches to the nearest 1/denom inch: 12'-6 1/2", 0'-3/4" is written 0'-0 3/4". */
export function formatFtIn(ft, denom = 16) {
  if (!Number.isFinite(ft)) return '';
  const neg = ft < 0;
  const n = Math.round(Math.abs(ft) * 12 * denom + 1e-9);   // whole 1/denom inches
  const perFoot = 12 * denom;
  const feet = Math.floor(n / perFoot);
  const rem = n - feet * perFoot;
  const inch = Math.floor(rem / denom);
  let num = rem - inch * denom, den = denom;
  while (num && num % 2 === 0 && den % 2 === 0) { num /= 2; den /= 2; }
  const zero = feet === 0 && inch === 0 && num === 0;
  return (neg && !zero ? '-' : '') + feet + "'-" + inch + (num ? ' ' + num + '/' + den : '') + '"';
}

Length in a chosen display unit

64-bit

Writes a length in feet-inches, decimal feet, inches, meters or millimeters.

m = ft × 0.3048; mm = ft × 0.3048 × 1000

Meters are feet × 0.3048 shown to 3 decimals; millimeters are feet × 0.3048 × 1000 shown to 0 decimals. Display only.

  • units.js
  • lines 174 to 185
  • TakeoffCraft 0.8.1
/** A length for display in a chosen unit: 'ftin', 'ft' (decimal), 'in', 'm', 'mm'. */
export function formatLength(ft, unit = 'ftin', digits = 2) {
  if (!Number.isFinite(ft)) return '';
  switch (unit) {
    case 'ftin': return formatFtIn(ft);
    case 'ft': return fixed(ft, digits) + "'";
    case 'in': return fixed(ft * 12, digits) + '"';
    case 'm': return fixed(ft * M_PER_FT, 3) + ' m';
    case 'mm': return fixed(ft * M_PER_FT * 1000, 0) + ' mm';
    default: return fixed(ft, digits);
  }
}

In detail

Method
roundTo: floor(|v|·10ᵈ + 0.5 + nudge) ÷ 10ᵈ with the sign put back. formatFtIn: the length as a whole count of 1 ÷ denom inches, then integer division into feet, inches and a reduced fraction.
Number types
64-bit numbers; integers once the count of fractions is formed.
Tolerances
roundTo nudge: 1e-9 × max(1, |v|·10ᵈ), relative. formatFtIn nudge: 1e-9 of a 1 ÷ denom inch, absolute.
Limits
Any value up to 1e-9 (relative) below a rounding boundary rounds up, whether or not it was meant to be on the boundary. The feet-inches nudge stops having any effect above 2²⁴ fractions (about 87,000 ft at sixteenths), where it is less than half the spacing of 64-bit numbers.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach