Resources Numerical accuracy

Calculations

Every calculation under our calculation policies: what it works out, its formula and the precision it runs at. Open one to see its code as it is in the released app.

  • 6,917 calculations in 1,248 source files
  • 143 exact
  • 6,286 in 64-bit
  • 335 with a 32-bit step
  • Last reviewed

Calculations

  • Feet and inches

    Reads 12'6", 12'-6 1/2" or 6" as inches.

    inches = feet × 12 + inches

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Degrees, minutes and seconds

    Reads 45°30'15" or 45d30m15s as degrees.

    degrees + minutes ÷ 60 + seconds ÷ 3600

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Slopes

    Reads 2%, 3H:1V, 1:3 or 6/12 as a grade (rise over run).

    percent ÷ 100; H:V → V ÷ H; rise:run → rise ÷ run

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Writing feet and inches

    Writes a length as feet, inches and a fraction of an inch.

    n = round(|inches| × den); feet = ⌊n ÷ 12den⌋; inches = ⌊rest ÷ den⌋; fraction = rest ÷ den, reduced

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Writing degrees, minutes and seconds

    Writes an angle as degrees, minutes and seconds.

    t = round(|degrees| × 3600 × 10^dec); split by 3600 and 60

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Writing decimals

    Writes a number with a fixed count of decimals.

    text = v.toFixed(dec)

    Unit table: reading and writing values unit-table.js Every app
    64-bit
  • Writing a length

    A length as text in one of the five formats.

    engineering: r = round(|v| × 10^p) ÷ 10^p; feet = ⌊r ÷ 12 + 1e-12⌋; architectural: whole inches and a fraction of 1⁄2^p

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Whole part and fraction

    Splits a value into a whole number and a reduced fraction of a power of two.

    n = round(|v| × den); whole = ⌊n ÷ den⌋; numerator = n − whole × den

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Writing decimals

    A number with up to prec decimals, trailing zeros dropped.

    |v| < 0.5 × 10^−prec → 0; else v.toFixed(prec)

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Writing an area

    An area as text: square drawing units, or square inches and square feet in the feet-and-inches formats.

    sq ft = sq in ÷ 144

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Numbers and fractions typed

    Reads 6, 6.5, 6-1/2, 6 1/2 or 1/2 as a number.

    whole + numerator ÷ denominator

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • One typed length

    Reads a single length such as 12'6" or 6-1/2 into drawing units.

    feet × 12 + inches, with the sign applied last

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Millimeters per drawing unit

    The size of one drawing unit in millimeters, from the drawing's units.

    inch 25.4; foot 304.8; yard 914.4; mile 1 609 344; micron 0.001

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    Exact
  • A length typed with a unit

    Converts a number with a unit mark, such as 450mm or 6in, into drawing units.

    mm, cm, m, in, ft: value × mm per typed unit ÷ mm per drawing unit; other marks: parseTo into the drawing's unit

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Writing an angle

    An angle in degrees as the drawing's angle format writes it.

    decimal: deg.toFixed(4)

    Lengths as text: formats and typed lengths units.js
    • CadCraft
    64-bit
  • Lengths with units inside an entry

    Replaces 900mm, 2ft or 3'6" in the typed text by the same length in drawing units.

    mm = feet × 304.8 + inches × 25.4, or value × mm per unit; text = (mm ÷ mm per drawing unit) to 12 significant digits

    Values typed in the Properties panel propexpr.js
    • CadCraft
    64-bit
  • Degrees, minutes and seconds inside an expression

    Replaces 45°30'15" in an angle expression by decimal degrees.

    degrees + minutes ÷ 60 + seconds ÷ 3600, written with 12 significant digits

    Values typed in the Properties panel propexpr.js
    • CadCraft
    64-bit
  • Full-precision text of a value

    The text a number field shows while it is edited.

    v.toPrecision(12)

    Values typed in the Properties panel propexpr.js
    • CadCraft
    64-bit
  • Millimeters per drawing unit

    The size of one drawing unit, or of the sheet's paper unit, in millimeters.

    inch 25.4; foot 304.8; yard 914.4; mile 1 609 344

    Values typed in the Properties panel propexpr.js
    • CadCraft
    Exact
  • Limits on a typed value

    Checks a result against the field's kind and limits.

    whole number: |v − round(v)| ≤ 1e-9; v ≥ min − 1e-12; v ≤ max + 1e-12

    Values typed in the Properties panel propexpr.js
    • CadCraft
    64-bit
  • Angles typed in radians, grads or degrees-minutes-seconds

    Reads 0.5r, 100g or 45°30'15" as degrees.

    radians × (180 ÷ π); grads × 0.9; d + m ÷ 60 + s ÷ 3600

    Values typed in the Properties panel propexpr.js
    • CadCraft
    64-bit
  • Length, area and perimeter of an object

    The measures the panel shows for one object.

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Centroid of a closed polyline

    The point a closed polyline is scaled about when its area is typed.

    a = Σ (xᵢyⱼ − xⱼyᵢ); c = Σ (pᵢ + pⱼ)(xᵢyⱼ − xⱼyᵢ) ÷ 3a

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Scaling a polyline to a typed area

    Resizes a closed polyline about its centroid so its area becomes the value typed.

    k = √(A ÷ A₀); p′ = c + (p − c)·k

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Line: length and angle

    Shows a line's length, angle and deltas, and moves its end point when length or angle is typed.

    L = |p₂ − p₁|; new end = p₁ + L·(cos a, sin a)

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Circle: diameter, circumference and area

    Shows a circle's diameter, circumference and area, and sets the radius when one of them is typed.

    d = 2r; C = 2πr; A = πr²; r = d ÷ 2, C ÷ 2π, √(A ÷ π)

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Arc: sweep, length, chord and segment area

    Shows an arc's angles, sweep, length, chord and segment area, and changes the arc when one is typed.

    θ = (a₁ − a₀) mod 2π; length = rθ; chord = 2r·sin(θ ÷ 2); area = r²(θ − sin θ) ÷ 2

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Ellipse: axes and angles

    Shows an ellipse's major and minor axis, ratio, axis angle and start and end angles, and changes them when typed.

    major = |m|; minor = |m| × ratio; ratio = minor ÷ major

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Ellipse parameter and true angle

    Converts between an ellipse's stored parameter and the angle seen from its center.

    angle = atan2(ratio·sin t, cos t); t = atan2(sin a ÷ ratio, cos a)

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Do several objects share a value

    Decides whether a field shows one value or 'varies' for a selection.

    |a − b| ≤ 1e-9 × max(1, |a|, |b|)

    Properties panel: measures and geometry edits propdefs.js
    • CadCraft
    64-bit
  • Text held by a number field

    Chooses whether a field holds the formatted text (12'-6") or the plain number.

    formatted text kept when |read(text) − value| ≤ 1e-9 × max(1, |value|)

    Properties panel: measures and geometry edits props.js
    • CadCraft
    64-bit
  • Rigid fit from point pairs

    The rotation and shift that best take the picked points of one scan onto their targets, with the error left at each pair.

    R from the top eigenvector (a quaternion) of Horn's 4 × 4 matrix; t = c_dst − R·c_src; rms = √(Σ rᵢ² ÷ n)

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Eigenvalues of a symmetric matrix

    Eigenvalues and eigenvectors of the 3 × 3 and 4 × 4 symmetric matrices the fit and the normals use.

    Jacobi rotations: θ = (a_qq − a_pp) ÷ 2a_pq; t = sign(θ) ÷ (|θ| + √(θ² + 1))

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Combining transforms

    The product of two 4 × 4 transforms, used to add each iteration's step to the running transform.

    o = a · b, each entry a sum of four products

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Inverse of a rigid transform

    The transform that undoes a rotation and shift.

    R⁻¹ = Rᵀ; t′ = −Rᵀ·t

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Transform in shifted coordinates

    Rewrites a transform for coordinates measured from the working origin, and back.

    t_local = t + R·o − o; back: t = t_local − R·o + o

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Is a matrix a rigid transform

    Checks that a matrix is a pure rotation and shift before it is accepted.

    |rᵢ·rⱼ − δᵢⱼ| ≤ tol; |det − 1| ≤ tol; last row 0 0 0 1 within tol

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Rotation about an axis

    The 4 × 4 matrix for a turn by an angle about an axis through a center.

    R = c·I + s·[axis]ₓ + (1 − c)·axis·axisᵀ; t = center − R·center

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Angle and shift of a transform in words

    The rotation angle in degrees and the length of the shift that a result is reported with.

    angle = acos((trace − 1) ÷ 2); shift = |t|

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Working points of a scan

    The thinned copy of a scan's points, measured from the session origin, that registration works on.

    work = pos + cloud origin − session origin, every stride-th point

    Scan registration register.js
    • CadCraft
    • BimCraft
    32-bit in part
  • Nearest point in the fixed scans

    For a moved point, the closest point of the fixed scans within the match distance.

    d² = Δx² + Δy² + Δz², smallest over the grid cells searched ring by ring

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Surface normals

    The direction square to the surface at each fixed point, from its 10 nearest neighbors.

    normal = eigenvector of the smallest eigenvalue of the neighbors' covariance

    Scan registration register.js
    • CadCraft
    • BimCraft
    32-bit in part
  • Matching and the error of a step

    Pairs each moving point with its nearest fixed point and reports the root-mean-square distance, and the distance along the normals.

    rms = √(Σ d² ÷ matched); plane rms = √(Σ ((p − q)·n)² ÷ matched)

    Scan registration register.js
    • CadCraft
    • BimCraft
    32-bit in part
  • Point-to-plane step

    The small turn and shift that reduce the distances along the normals in one iteration.

    minimize Σ ((ω × p + t + p − q)·n)²: 6 × 6 normal equations for (ω, t); R = rotation by |ω| about ω

    Scan registration register.js
    • CadCraft
    • BimCraft
    32-bit in part
  • Solving the 6 × 6 system

    Solves the linear equations of the point-to-plane step.

    Gaussian elimination with partial pivoting, then back substitution

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • When the iteration stops

    Decides when a phase has settled and narrows the match distance for the next step.

    stop when |previous − error| ≤ tol × max(error, 1e-12) or error < 1e-12; cut = min(start, max(2.5 × rms, 1e-6 × diagonal))

    Scan registration register.js
    • CadCraft
    • BimCraft
    64-bit
  • Pixels to drawing coordinates and back

    Maps a pixel of a placed picture to the drawing, a drawing point back to a pixel, and gives the size of one pixel.

    block = p + u·px + v·(h − py); drawing = M·block; pixel size = |M·u|

    Scanned drawings: size, scale, align, crop and trace scan.js
    • CadCraft
    64-bit
  • Arc through three points

    The counterclockwise arc through a start, a middle and an end point.

    d = 2(aₓ(b_y − c_y) + bₓ(c_y − a_y) + cₓ(a_y − b_y)); center from |a|², |b|², |c|² ÷ d; r = |a − center|

    Scanned drawings: size, scale, align, crop and trace scan.js
    • CadCraft
    64-bit
  • Traced lines and arcs as drawing objects

    Turns a trace result in pixels into lines, arcs and circles in drawing coordinates.

    circle r = r(px) × pixel size; arc through start, middle and end

    Scanned drawings: size, scale, align, crop and trace scan.js
    • CadCraft
    64-bit
  • First size of an attached scan

    The size of one pixel when a scan is attached.

    per pixel = 25.4 ÷ dpi when the file states it and the drawing has units, else 1; typed width ÷ pixel width

    Scanned drawings: size, scale, align, crop and trace scan-cmd.js
    • CadCraft
    64-bit

A calculation is added only after its source was read. Each statement can be checked in the code shown on its policy's page; the app repositories are public. When a release changes a calculation, its page says so until it has been read again.