Resources Numerical accuracy

Calculations

Every calculation under our calculation policies: what it works out, its formula and the precision it runs at. Open one to see its code as it is in the released app.

  • 6,917 calculations in 1,248 source files
  • 143 exact
  • 6,286 in 64-bit
  • 335 with a 32-bit step
  • Last reviewed

Calculations

  • Dense linear solve

    Solves the nodal equations of the pipe network and the load-flow Jacobian.

    Gaussian elimination with partial pivoting

    Pipe, duct and gas networks numeric.js Every app
    64-bit
  • Bus admittance matrix

    The admittances that link every pair of buses.

    y = 1 ÷ (r + jx); Yff = (y + jb ÷ 2) ÷ |a|²; Yft = −y ÷ conj(a); Ytt = y + jb ÷ 2

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Power injected at each bus

    Real and reactive power from the voltages.

    Pᵢ = Σ Vᵢ Vⱼ (G cos δ + B sin δ); Qᵢ = Σ Vᵢ Vⱼ (G sin δ − B cos δ)

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Load flow iteration

    Corrects angles and voltages until the power balances.

    J Δx = (P_sched − P, Q_sched − Q); stop when max |mismatch| < 1e-9 pu

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Reactive limits of generator buses

    Switches a voltage-controlled bus to a fixed reactive power when it passes its limit.

    switch when Q_g > Qmax + 1e-9 or Q_g < Qmin − 1e-9

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Branch flows and losses

    Power at both ends of each branch and the loss in it.

    S_f = V_f conj(I_f); loss = P_f + P_t

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Per-unit base

    Ohms to per unit on a voltage and power base.

    Z_base = kV² ÷ MVA; z_pu = z ÷ Z_base

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Feeder voltage drop

    The three-phase voltage drop of a feeder.

    ΔV ≈ √3 I (R cos φ + X sin φ), φ = acos(pf)

    Load flow and short-circuit currents power.js Every app
    64-bit
  • Sequence impedance matrices

    The impedance seen from every bus in the positive, negative and zero sequence.

    Z = Y⁻¹ for each sequence

    Load flow and short-circuit currents shortcircuit.js Every app
    64-bit
  • Sequence to phase values

    Phase currents and voltages from the sequence components.

    I_a = I₀ + I₁ + I₂; I_b = I₀ + a² I₁ + a I₂; I_c = I₀ + a I₁ + a² I₂, a = −½ + j√3 ÷ 2

    Load flow and short-circuit currents shortcircuit.js Every app
    64-bit
  • Fault current

    Initial and peak fault current, fault power, and the voltages and branch currents during the fault.

    3-phase: I₁ = c ÷ (Z₁ + Z_f); line to ground: I₁ = c ÷ (Z₁ + Z₂ + Z₀ + 3Z_f); i_p = κ √2 I_k, κ = 1.02 + 0.98 e^(−3R/X)

    Load flow and short-circuit currents shortcircuit.js Every app
    64-bit
  • Complex division

    a ÷ b for the impedances and currents of the fault study.

    a ÷ b = a conj(b) ÷ (b_re² + b_im²)

    Load flow and short-circuit currents numeric.js Every app
    64-bit
  • Complex matrix inverse

    Inverts a sequence admittance matrix.

    Gauss–Jordan elimination with pivoting by magnitude

    Load flow and short-circuit currents numeric.js Every app
    64-bit
  • Curve constants

    The constants of the IEC and IEEE inverse-time curves and the cable factors.

    IEC normal inverse k = 0.14, α = 0.02; IEEE very inverse A = 19.61, B = 0.491, p = 2; cable k: copper PVC 115

    Protection curves and grading protection.js Every app
    64-bit
  • Relay operating time

    How long a relay takes to trip at a current.

    IEC: t = TMS × k ÷ (M^α − 1); IEEE: t = TD ÷ divisor × (A ÷ (M^p − 1) + B), M = I ÷ pickup

    Protection curves and grading protection.js Every app
    64-bit
  • Fuse or breaker curve from points

    The operating time of a device given as a table of current and time.

    ln t = ln t₀ + (ln I − ln I₀) ÷ (ln I₁ − ln I₀) × (ln t₁ − ln t₀)

    Protection curves and grading protection.js Every app
    64-bit
  • Cable damage time

    How long a cable can carry a fault current.

    t = (k S ÷ I)²

    Protection curves and grading protection.js Every app
    64-bit
  • Grading margin

    The time between a downstream and an upstream device at each fault current.

    margin = t_up − t_down; ok when margin ≥ the grading interval (0.3 s by default)

    Protection curves and grading protection.js Every app
    64-bit
  • Pickup and cable checks

    Pickup against load and the smallest fault, and clearing time against the cable.

    pickup ≥ 1.05 × load; pickup ≤ smallest fault ÷ 1.5; clearing time ≤ (k S ÷ I)²

    Protection curves and grading protection.js Every app
    64-bit
  • Coordinate and aperture format

    How a position and an aperture size are written in a Gerber file.

    X, Y = round(mm × 10⁶); size = 6 decimals

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Pad as a flash or a region

    A pad, or its grown copy for the solder mask, in the file.

    axis-aligned: aperture (w + 2 × grow) by (h + 2 × grow); else a 4-corner region rotated by the pad angle

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Drill sizes and positions

    The tool table and hole positions of the drill files.

    tool size = 3 decimals; X, Y = 4 decimals of a millimeter

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Board size in the job file

    The board's width and height stated in the job file.

    size = extent of the outline, 4 decimals

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Coordinate scale when reading

    The factor that turns the file's integers into coordinates.

    scale = 10^(−decimals of X)

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Coordinates when reading

    Each X and Y of a Gerber file as a number.

    x = integer × scale

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Drill file when reading

    Tool diameters and hole positions read back from a drill file.

    decimal text to the nearest 64-bit number

    Gerber and drill file numbers gerber.js Every app
    64-bit
  • Real number when reading

    An 8-byte GDSII real as a number.

    value = sign × (mantissa ÷ 2⁵⁶) × 16^(exponent − 64)

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Real number when writing

    A number as an 8-byte GDSII real.

    mantissa = round(v × 2⁵⁶) after scaling v into [1/16, 1) by powers of 16

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    Exact
  • Database unit

    User units per database unit and meters per database unit, read from the file.

    two 8-byte reals from the UNITS record

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Integers written to the stream

    Coordinates, widths, layers and array counts as the file stores them.

    coordinates and widths: 32-bit signed; layers and counts: 16-bit signed

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Placement transform

    A referenced cell's points reflected, magnified, rotated and moved.

    (x, y) → mag × (x, ±y), rotated by the angle in degrees, then translated

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Nested placements

    The combined transform of a cell placed inside a placed cell.

    mag = mag_parent × mag_child; angle = angle_parent ± angle_child; reflection toggles

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Array placement

    The position of each copy in a rows-by-columns array.

    position(i, j) = origin + i × (c − o) ÷ cols + j × (r − o) ÷ rows

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Path outline

    A path with a width turned into a polygon.

    offset = ± half width along the corner bisector ÷ max(0.2, cos of the half angle)

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Database units to user units

    Flattened coordinates in the file's user unit, usually micrometers.

    x_user = x_db × userPerDb

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Shapes and area per layer

    How many shapes each layer has and their total area.

    area = Σ |shoelace area|

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Width, spacing and area checks

    Shapes that are narrower, closer or smaller than a rule allows.

    area < minArea − 1e-12; 0 < distance < minSpace − 1e-12; facing-edge width < minWidth − 1e-12

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Narrowest width of a shape

    The smallest distance between two edges that face each other across the inside.

    distance from one edge's line to the other, for antiparallel edges that overlap

    GDSII layout: reals, flattening, areas and checks gds.js Every app
    64-bit
  • Half-perimeter wire length

    The estimate of total connection length reported before and after placement.

    Σ over nets of (x_max − x_min) + (y_max − y_min)

    Board placement and routing place.js Every app
    64-bit
  • Overlap and outline penalties

    The cost added when courtyards overlap or reach outside the board.

    cost = 50 × (overlap area + area outside the margin)

    Board placement and routing place.js Every app
    64-bit
  • Annealing schedule

    How the search cools from large moves to small ones.

    T₀ = max(1, 0.15 (W + H)); T_end = 0.002 (W + H); T ← T × (T_end ÷ T₀)^(1 ÷ iterations)

    Board placement and routing place.js Every app
    64-bit
  • Final part positions

    The positions stored on the board when placement ends.

    x, y = round(value × 1000) ÷ 1000

    Board placement and routing place.js Every app
    64-bit
  • Routing grid

    The grid of points tracks may pass through.

    pitch = (track width + clearance) ÷ 2 by default; x = ox + i × pitch

    Board placement and routing route.js Every app
    64-bit
  • Obstacle map

    Which grid points a track or a via of another net may not use.

    blocked when distance to copper < clearance + track width ÷ 2 + 0.25 × pitch

    Board placement and routing route.js Every app
    64-bit
  • Track points

    The corner points of a routed track.

    drop a point when |(b − a) × (c − b)| ≤ 1e-9

    Board placement and routing route.js Every app
    64-bit
  • Probability of a basic event

    The probability of an event from a fixed value, a failure rate and time, a test interval, or repair data.

    failed by t: 1 − e^(−λt); tested every τ: 1 − (1 − e^(−λτ)) ÷ (λτ), or λτ ÷ 2 below 1e-9; repairable: MTTR ÷ (MTTF + MTTR)

    Fault trees and reliability block diagrams logic.js Every app
    64-bit
  • At least k of n events

    The probability that k or more of a set of independent events occur.

    the distribution of the count, built one event at a time, summed from k up

    Fault trees and reliability block diagrams logic.js Every app
    64-bit
  • AND, OR and voting gates

    The probability of a gate from its inputs when no event is repeated.

    AND: Π pᵢ; OR: 1 − Π (1 − pᵢ)

    Fault trees and reliability block diagrams logic.js Every app
    64-bit
  • Top event with repeated events

    The probability of the top event when the same event appears in several places.

    P = p_x × P(tree | x occurs) + (1 − p_x) × P(tree | x does not)

    Fault trees and reliability block diagrams logic.js Every app
    64-bit

A calculation is added only after its source was read. Each statement can be checked in the code shown on its policy's page; the app repositories are public. When a release changes a calculation, its page says so until it has been read again.