Resources Numerical accuracy

Calculations

Every calculation under our calculation policies: what it works out, its formula and the precision it runs at. Open one to see its code as it is in the released app.

  • 6,917 calculations in 1,248 source files
  • 143 exact
  • 6,286 in 64-bit
  • 335 with a 32-bit step
  • Last reviewed

Calculations

  • Significand width of a number

    How many bits of a number this device keeps.

    bits = the first n where 1 + 2⁻ⁿ = 1

    Device number check accuracy.js Every app
    Exact
  • Digits of a graphics float

    The number of significant digits stated for the graphics processor's float type.

    bits < 23: digits = max(1, ⌊(bits + 1) × log₁₀ 2⌋); bits ≥ 23: the fixed text "about 7"

    Device number check accuracy.js Every app
    64-bit
  • Size and position text

    A dimension, port size or port position as text in millimeters.

    text = round(x × 10⁴) ÷ 10⁴

    Library configurator sizes library-configurator.js Every app
    64-bit
  • Option value used for matching

    The text an option value is compared by when the page picks the variant that matches the selected options.

    numbers: round(v × 10⁴) ÷ 10⁴ as text; other values: trimmed text

    Library configurator sizes library-configurator.js Every app
    64-bit
  • Mesh buffers

    The item's box or mesh as geometry for drawing, turned from z up to y up.

    Library 3D preview library-3d.js Every app
    32-bit, display only
  • Camera position

    Where the camera sits as the view is turned and zoomed.

    position = target + dist × (sin φ sin θ, cos φ, sin φ cos θ)

    Library 3D preview library-3d.js Every app
    32-bit, display only
  • Variant body buffers

    The chosen variant's box or mesh as geometry for drawing.

    Library 3D preview library-configurator.js Every app
    32-bit, display only
  • Required and found values as text

    A limit or a measured value in words, such as ≤ 196.85 mm or 863.6–965.2 mm.

    text = round(x × 100) ÷ 100, then the unit

    Check report values problems.js Every app
    64-bit
  • Built-in unit factors

    The factor from each built-in unit of mass, time, force, pressure, energy, power, current, voltage and volume to SI.

    lb = 0.45359237 kg; lbf = 4.4482216152605 N; psi = 6894.757293168 Pa; psf = 47.88025898 Pa

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Constants and fallback lengths

    pi, e, standard gravity g0, and the length units used when no unit table is given.

    g0 = 9.80665 m/s²; in = 0.0254 m; ft = 0.3048 m; yd = 0.9144 m

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Unit lookup

    The SI factor and dimension of a unit name.

    deg = π ÷ 180; rad = 1

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Dimension comparison

    Whether two values have the same dimensions, so they can be added, compared or converted.

    |aᵢ − bᵢ| < 1e-9 for each of the five exponents

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Power

    a ^ b for a value with units.

    value = aᵇ; dimensions × b

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Conversion with to

    M to kN m: checks the dimension and records the unit to show the answer in.

    shown = value ÷ factor of the target unit

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Shown amount

    The number and unit a row displays, and the amount passed to other modules.

    amount = value ÷ factor

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Square root and absolute value

    sqrt(a) and abs(a) for a value with units.

    √a halves the dimensions

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Rounding to a step

    round, floor and ceil, with an optional step such as round(x, 5 mm).

    round(a ÷ s) × s

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Minimum, maximum, sum and average

    min, max, sum and avg of values of one kind.

    sum = Σ aᵢ in order; avg = sum ÷ n

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Trigonometry, logarithms and exponent

    sin, cos, tan, their inverses, atan2, ln, log10, exp and pow.

    sin, cos, tan of radians; asin, acos need |x| ≤ 1; ln, log10 need x > 0

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Reading an app parameter

    @Name: the app's parameter as a value in SI.

    value = parameter value × factor of its unit

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Values written to driven parameters

    What each drive would set in the app: the row's answer in the parameter's unit.

    value = row value ÷ factor of the unit; change when |current − value| > 1e-9 × max(1, |value|)

    Calc sheet units, functions and driven parameters calc.js Every app
    64-bit
  • Time-weighted average

    The average of a value that changes at events: queue length, units busy, work in process.

    average = (Σ vᵢ × Δtᵢ) ÷ (t − t₀)

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Mean, deviation and percentiles of observations

    Waiting time, service time and time in system: mean, standard deviation, minimum, maximum, P50 and P95.

    m ← m + (x − m) ÷ n; s ← s + (x − m_old)(x − m_new); sd = √(s ÷ (n − 1)); P = sorted[⌊p × k⌋]

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Event time

    When a scheduled event happens.

    t_event = t_now + delay

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Conveyor slots and timing

    How many items fit on a belt, the gap between items and the travel time.

    slots = max(1, ⌊length ÷ pitch + 1e-9⌋); headway = pitch ÷ speed; travel = length ÷ speed

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Shift edges

    When a server goes on and off shift.

    x = t − ⌊t ÷ period⌋ × period; on shift when a ≤ x < b

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Server utilization

    The share of time a server's units were busy, blocked, down and idle.

    utilization = average busy units ÷ units; idle = max(0, 1 − busy − blocked − down)

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Throughput

    Finished entities per unit of time at a sink.

    throughput = count ÷ (t − t₀)

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Resource utilization

    The share of a shared resource's capacity that was in use.

    utilization = average in use ÷ average capacity

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Vehicle trip times

    How long a vehicle takes to reach the pick station and to carry the load to the drop station.

    to pick = distance ÷ speed + load time; carry = distance ÷ speed + unload time

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Student t factor

    The factor for a 95 % confidence range.

    df ≤ 30: table value; df ≤ 60: 2.0; df ≤ 120: 1.98; above: 1.96

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Replication mean and confidence half-width

    The mean of each result over independent runs and its 95 % range.

    mean = Σx ÷ n; sd = √(Σ(x − mean)² ÷ (n − 1)); half = t × sd ÷ √n

    Flow simulation (discrete events) engine.js Every app
    64-bit
  • Uniform and normal draws

    The random numbers every arrival, service time and route choice is drawn from.

    u = (k + 0.5) ÷ 2³², k a 32-bit integer; z = √(−2 ln u) × cos(2π v)

    Flow simulation (discrete events) rng.js Every app
    64-bit
  • Draw from a distribution

    One value from an exponential, uniform, triangular, normal, lognormal, Erlang, Weibull or empirical distribution.

    exp: −mean × ln u; tri: a + √(u (b − a)(c − a)) or b − √((1 − u)(b − a)(b − c)); weibull: scale × (−ln u)^(1 ÷ shape)

    Flow simulation (discrete events) rng.js Every app
    64-bit
  • Mean of a distribution

    The mean used for load estimates and for routing to the least loaded target.

    tri: (a + m + b) ÷ 3; weibull: scale × Γ(1 + 1 ÷ shape)

    Flow simulation (discrete events) rng.js Every app
    64-bit
  • Gamma function

    Γ(x), used for the mean of a Weibull distribution.

    Lanczos approximation, g = 7, 9 coefficients

    Flow simulation (discrete events) rng.js Every app
    64-bit
  • Thermal voltage

    kT/q, the voltage scale of every diode and transistor junction.

    VT = 0.025852 V

    Circuit solver circuit.js Every app
    64-bit
  • Value with an engineering suffix

    4.7k, 100n, 1meg: a component value read from text.

    value = number × factor of the suffix (t, g, meg, k, m, u, n, p, f); mil = 25.4e-6

    Circuit solver circuit.js Every app
    64-bit
  • Source waveform value

    The value of a DC, sine or pulse source at a time.

    sin: off + amp × e^(−damp (t − delay)) × sin(2π f (t − delay) + phase); pulse: linear edges over tr and tf

    Circuit solver circuit.js Every app
    64-bit
  • Diode current and conductance

    The current through a diode and its slope at a junction voltage.

    i = Is (e^(v ÷ (n VT)) − 1); g = Is e^(v ÷ (n VT)) ÷ (n VT)

    Circuit solver circuit.js Every app
    64-bit
  • Transistor currents

    Collector and base current of a bipolar transistor.

    ic = If − Ir − Ir ÷ βr; ib = If ÷ βf + Ir ÷ βr; If = Is (e^(vbe ÷ VT) − 1)

    Circuit solver circuit.js Every app
    64-bit
  • MOSFET level 1 current

    Drain current of a MOSFET in the triode and saturation regions.

    triode: id = kp (vov vds − vds² ÷ 2)(1 + λ vds); saturation: id = kp vov² (1 + λ vds) ÷ 2

    Circuit solver circuit.js Every app
    64-bit
  • Switch conductance

    The conductance of a voltage-controlled switch.

    ln g = ln(1 ÷ roff) + a (ln(1 ÷ ron) − ln(1 ÷ roff)); a = 1 ÷ (1 + e^(−(v − vt) ÷ vh))

    Circuit solver circuit.js Every app
    64-bit
  • Circuit equations

    The matrix and right side for one Newton step, at DC or at one time step.

    capacitor: geq = C ÷ h (backward Euler) or 2C ÷ h (trapezoid); inductor: r = L ÷ h or 2L ÷ h

    Circuit solver circuit.js Every app
    64-bit
  • Newton iteration

    Repeats the linear solve until the voltages and currents stop changing.

    stop when |Δxᵢ| ≤ 1e-3 × max(|xᵢ new|, |xᵢ old|) + 1e-6 V (or 1e-12 A) for every unknown

    Circuit solver circuit.js Every app
    64-bit
  • Operating point

    The DC voltages and currents of the circuit.

    direct Newton, then gmin stepping, then source stepping

    Circuit solver circuit.js Every app
    64-bit
  • DC sweep

    The circuit solved for each value of a swept source.

    Circuit solver circuit.js Every app
    64-bit
  • AC small-signal response

    Complex node voltages and branch currents at each frequency.

    capacitor: jωC; inductor: −jωL on its branch row; ω = 2π f

    Circuit solver circuit.js Every app
    64-bit
  • Transient

    Voltages and currents over time.

    trapezoidal rule after a first backward-Euler step; output at t = k × tstep

    Circuit solver circuit.js Every app
    64-bit
  • Junction voltage limiting

    Keeps a diode or transistor voltage from jumping too far in one iteration.

    v = v_old + vt ln(1 + (v_new − v_old) ÷ vt) when the step exceeds 2 vt above vcrit

    Circuit solver circuit.js Every app
    64-bit

A calculation is added only after its source was read. Each statement can be checked in the code shown on its policy's page; the app repositories are public. When a release changes a calculation, its page says so until it has been read again.