Resources Numerical accuracy Calculation policy

TIFF scan resolution

Reads the resolution of a scanned drawing from a TIFF file as dots per inch.

How the precision is kept

A TIFF resolution is a ratio of two 32-bit unsigned integers. The reader divides them in 64-bit and, for files in centimeters, multiplies by 2.54. The pixel data is decoded to 8-bit samples, which this entry does not cover.

Calculations under this policy

2 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

raster/tiff.js CadCraft 0.8.2 64-bit 2 calculations Reviewed

Tests bin/test-raster.mjs

Rational tag value

64-bit

A TIFF RATIONAL (two 32-bit integers) as one number.

value = numerator ÷ denominator

Both integers are read as unsigned 32-bit and divided once in 64-bit, so the quotient is the nearest 64-bit number to the ratio. A denominator of 0 is replaced by 1.

  • raster/tiff.js
  • lines 42 to 46
  • CadCraft 0.8.2
const cap = Math.min(count, 1 << 20);
for (let k = 0; k < cap; k++) {
  const p = at + k * TYPE_SIZE[type];
  vals.push(type === 3 ? u16(p) : type === 4 || type === 13 ? u32(p) : type === 1 || type === 7 ? b[p] : type === 5 ? u32(p) / (u32(p + 4) || 1) : type === 8 ? dv.getInt16(p, le) : type === 9 ? dv.getInt32(p, le) : u32(p));
}

Resolution in dots per inch

64-bit

The horizontal and vertical resolution of a page, in dots per inch.

dpi = resolution × 2.54 (centimeter files); dpi = resolution (inch files)

One multiplication by 2.54, the exact number of centimeters in an inch. No dpi is returned when either resolution is not above zero or the unit tag is 1 (no absolute unit).

  • raster/tiff.js
  • lines 58 to 63
  • CadCraft 0.8.2
function describe(t) {
  const w = one(t, 256, 0), h = one(t, 257, 0), bits = (t.get(258) || [1])[0], samples = one(t, 277, 1), comp = one(t, 259, 1);
  const unit = one(t, 296, 2), xr = one(t, 282, 0), yr = one(t, 283, 0);
  const dpi = xr > 0 && yr > 0 && unit !== 1 ? { x: unit === 3 ? xr * 2.54 : xr, y: unit === 3 ? yr * 2.54 : yr } : null;
  return { w, h, bits, samples, compression: comp, photometric: one(t, 262, 1), ...(dpi ? { dpi } : {}) };
}

In detail

Method
RATIONAL value = numerator ÷ denominator; dpi = value × 2.54 when the resolution unit is the centimeter, the value itself when it is the inch, and no dpi when the file says there is no absolute unit.
Number types
Unsigned 32-bit integers from the file; the quotient and the dpi are 64-bit numbers.
Limits
A zero denominator is read as 1. A file with no resolution unit tag is read as inches.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach