Resources Numerical accuracy Calculation policy

Design study variants and ranking

Makes the list of variants a design study runs, checks each result against its limits, finds the trade-off front and ranks the variants.

How the precision is kept

Parameter values are 64-bit and are rounded to 12 significant digits when the variants are made. Limits and the trade-off front use plain comparisons with no tolerance, and the ranking score is a weighted sum of values scaled between the smallest and largest feasible result.

Open item. Keep the variant parameter values at full 64-bit precision instead of rounding them to 12 significant digits.

Calculations under this policy

6 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

study/study.js CadCraft 0.8.2 64-bit 6 calculations Reviewed

Tests tools/study-test/study.mjs, tools/process-test/test-link.mjs, tools/process-test/test-chemmount5.mjs

Parameter levels

64-bit

The evenly spaced values of a parameter between its two bounds.

vᵢ = from + (to − from) × i ÷ (steps − 1)

Each level is computed from its own index, so no error accumulates along the range. The last level is from + (to − from), which can differ from `to` in the last binary digit.

  • study/study.js
  • line 86
  • CadCraft 0.8.2
const levels = (p) => (p.values ? p.values.slice() : Array.from({ length: p.steps }, (_, i) => (p.steps === 1 ? p.from : p.from + (p.to - p.from) * i / (p.steps - 1))));

Rounding of parameter values

64-bit

Rounds a parameter value before it is put in a variant.

v → v to 12 significant digits

Every grid level and every random value passes through toPrecision(12) before it is stored in the variant and handed to the solver, so a variant's inputs carry 12 significant digits, not 15 to 17.

  • study/study.js
  • line 87
  • CadCraft 0.8.2
const clean = (x) => (typeof x === 'number' ? +x.toPrecision(12) : x);

Variant list

64-bit

Every combination of parameter levels, or a seeded random sample between the bounds.

random: v = from + (to − from) × u, u = 32-bit integer ÷ 2³²

The random fraction comes from a 32-bit generator, so it has 2³² possible values; the same seed gives the same variants. A grid larger than 500 variants is refused. Choosing from a list of values uses Math.floor of the fraction times the list length.

  • study/study.js
  • lines 89 to 113
  • CadCraft 0.8.2
/** The variants a study asks for: [{ id: 'v1', label, set: { param: value } }] (configs: set = { $config: name }). */
export function expandVariants(study) {
  const s = study.source;
  let list;
  if (s.kind === 'configs') list = s.names.map((n) => ({ label: n, set: { $config: n } }));
  else if (s.kind === 'list') list = s.variants.map((v) => ({ label: v.label, set: { ...v.set } }));
  else if (s.mode === 'random') {
    const r = rng(s.seed === undefined ? 1 : s.seed);
    list = Array.from({ length: s.count }, () => ({ set: Object.fromEntries(s.params.map((p) => {
      if (p.values) return [p.name, p.values[Math.floor(r() * p.values.length)]];
      return [p.name, clean(p.from + (p.to - p.from) * r())];
    })) }));
  } else {
    const ls = s.params.map((p) => levels(p).map(clean));
    const total = ls.reduce((a, l) => a * l.length, 1);
    if (total > LIMITS.variants) bad('That grid is ' + total + ' variants; the most one study runs is ' + LIMITS.variants + '. Use fewer steps or a random study.');   // i18n-ignore
    list = [];
    const idx = new Array(ls.length).fill(0);
    for (let n = 0; n < total; n++) {
      list.push({ set: Object.fromEntries(s.params.map((p, i) => [p.name, ls[i][idx[i]]])) });
      for (let i = ls.length - 1; i >= 0; i--) { if (++idx[i] < ls[i].length) break; idx[i] = 0; }
    }
  }
  return list.map((v, i) => ({ id: 'v' + (i + 1), label: v.label || Object.entries(v.set).map(([k, x]) => k + '=' + x).join(', '), set: v.set }));
}

Meeting the limits

64-bit

Whether a variant's results meet every limit of the study.

min goal: value ≤ limit; max goal: value ≥ limit

Plain 64-bit comparisons with no tolerance: a value exactly on the limit passes, and one that exceeds it by rounding alone fails.

  • study/study.js
  • lines 142 to 146
  • CadCraft 0.8.2
/** A row meets every metric limit (min: value ≤ limit; max: value ≥ limit) and has all its numbers. */
export function feasible(study, row) {
  if (!row || row.error !== undefined || row.missing || !row.values) return false;
  return study.metrics.every((m) => m.limit === undefined || (m.goal === 'min' ? row.values[m.key] <= m.limit : row.values[m.key] >= m.limit));
}

Trade-off front

64-bit

The variants that no other variant beats on every goal at once.

a dominates b when a is no worse on every goal and better on at least one

Dominance is decided by the sign of 64-bit differences with no tolerance, so two results that differ in the last digit are ordered by that digit.

  • study/study.js
  • lines 148 to 157
  • CadCraft 0.8.2
const dom = (study, a, b) => {   // a is no worse than b on every goal and better on one
  let better = false;
  for (const m of study.metrics) {
    if (m.goal === 'none') continue;
    const d = m.goal === 'min' ? b.values[m.key] - a.values[m.key] : a.values[m.key] - b.values[m.key];
    if (d < 0) return false;
    if (d > 0) better = true;
  }
  return better;
};

Weighted ranking

64-bit

A score for each feasible variant from its goals and the weights a person gives.

x = (value − lo) ÷ (hi − lo); score = Σ w × (x for max goals, 1 − x for min goals)

Each goal is scaled between the smallest and largest feasible value; when they are equal the scaled value is taken as 1. The score is a short weighted sum, and ties are broken by the variant id.

  • study/study.js
  • lines 165 to 171
  • CadCraft 0.8.2
/** Rank feasible rows by a weighted sum of goals, each metric scaled 0 (worst) to 1 (best) across the feasible rows. */
export function rankRows(study, rows, weights = {}) {
  const ok = rows.filter((r) => feasible(study, r)), goals = study.metrics.filter((m) => m.goal !== 'none');
  const span = Object.fromEntries(goals.map((m) => { const v = ok.map((r) => r.values[m.key]); return [m.key, [Math.min(...v), Math.max(...v)]]; }));
  const score = (r) => goals.reduce((s, m) => { const [lo, hi] = span[m.key], w = weights[m.key] === undefined ? 1 : weights[m.key], x = hi === lo ? 1 : (r.values[m.key] - lo) / (hi - lo); return s + w * (m.goal === 'min' ? 1 - x : x); }, 0);
  return ok.map((r) => ({ id: r.id, label: r.label, score: score(r) })).sort((a, b) => b.score - a.score || (a.id < b.id ? -1 : 1));
}

In detail

Method
Grid or seeded random sampling of parameters; feasibility by comparison with limits; Pareto dominance; weighted sum of min-max scaled goals.
Number types
64-bit numbers. The random generator is a 32-bit integer generator (mulberry32) scaled to 0–1.
Tolerances
Parameter values rounded to 12 significant digits. No tolerance in limit or dominance comparisons.
Limits
A random parameter value has at most 2³² distinct positions between its bounds. Two results that differ only by rounding are treated as different on the trade-off front.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach