Sketch photo to drawing: sheet, perspective and scale
Finds the sheet of paper in a photo, straightens it, and places the traced lines and arcs in the drawing at a stated scale.
How the precision is kept
The arithmetic is 64-bit, but the result is set by the photo: sheet corners are found on a copy reduced to 400 pixels, the sheet is resampled to 1800 pixels on its long side, and lines are traced to a pixel tolerance. The scale assumes a paper size; with paper left on auto the long side is taken as 297 mm.
Open item. When paper is auto and the detected ratio matched a standard sheet, use that sheet's long side instead of 297 mm.
Calculations under this policy
7 calculations in 2 source files. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.
corners = the pixels of the largest bright region with the smallest and largest x + y and x − y; photo position = (pixel + 0.5) × reduction factor
The search runs on a copy reduced so its long side is at most 400 pixels, and each corner is reported at the center of a reduced pixel. A corner is therefore located to about one reduced pixel, several pixels of a large photo.
photosketch/photosketch.js
lines 46 to 84
ModelCraft 0.8.1
/** The largest bright patch that fills most of its four-cornered outline: the paper. Photo pixels, TL TR BR BL (y down). */
export function findSheet(img, { minArea = 0.15, minFill = 0.82 } = {}) {
const g = toGrey(img), s = downsample(g, 400), n = s.w * s.h;
if (s.w < 8 || s.h < 8) return null;
const lvl = otsuLevel(s.data);
const seen = new Uint8Array(n), stack = new Int32Array(n);
let best = null;
for (let i0 = 0; i0 < n; i0++) {
if (seen[i0] || s.data[i0] <= lvl) continue;
let sp = 0, cnt = 0; stack[sp++] = i0; seen[i0] = 1;
let tl = null, br = null, tr = null, bl = null, tlv = Infinity, brv = -Infinity, trv = -Infinity, blv = Infinity;
while (sp) {
const i = stack[--sp], x = i % s.w, y = (i - x) / s.w; cnt++;
const a = x + y, b = x - y;
if (a < tlv) { tlv = a; tl = [x, y]; } if (a > brv) { brv = a; br = [x, y]; } if (b > trv) { trv = b; tr = [x, y]; } if (b < blv) { blv = b; bl = [x, y]; }
if (x > 0 && !seen[i - 1] && s.data[i - 1] > lvl) { seen[i - 1] = 1; stack[sp++] = i - 1; }
if (x < s.w - 1 && !seen[i + 1] && s.data[i + 1] > lvl) { seen[i + 1] = 1; stack[sp++] = i + 1; }
if (y > 0 && !seen[i - s.w] && s.data[i - s.w] > lvl) { seen[i - s.w] = 1; stack[sp++] = i - s.w; }
if (y < s.h - 1 && !seen[i + s.w] && s.data[i + s.w] > lvl) { seen[i + s.w] = 1; stack[sp++] = i + s.w; }
}
if (!best || cnt > best.cnt) best = { cnt, tl, tr, br, bl };
}
if (!best || best.cnt < minArea * n) return null;
const q = [best.tl, best.tr, best.br, best.bl];
let a = 0; for (let i = 0; i < 4; i++) { const p = q[i], r = q[(i + 1) % 4]; a += p[0] * r[1] - r[0] * p[1]; }
const quad = Math.abs(a) / 2, sign = a >= 0 ? 1 : -1;
// ink that closes a figure cuts the paper in pieces: count every bright pixel inside the outline, not only the biggest piece
let inside = 0;
const x0 = Math.min(...q.map((p) => p[0])), x1 = Math.max(...q.map((p) => p[0])), y0 = Math.min(...q.map((p) => p[1])), y1 = Math.max(...q.map((p) => p[1]));
for (let y = y0; y <= y1; y++) for (let x = x0; x <= x1; x++) {
if (s.data[y * s.w + x] <= lvl) continue;
let in4 = true;
for (let i = 0; i < 4 && in4; i++) { const p = q[i], r = q[(i + 1) % 4]; if (sign * ((r[0] - p[0]) * (y - p[1]) - (r[1] - p[1]) * (x - p[0])) < 0) in4 = false; }
if (in4) inside++;
}
const fill = quad > 0 ? inside / quad : 0;
if (fill < minFill || fill > 1.15) return null;
return { corners: q.map(([x, y]) => ({ x: (x + 0.5) * s.k, y: (y + 0.5) * s.k })), fill: Math.min(1, fill), area: inside / n };
}
Perspective transform from four points
64-bit
The 3 × 3 matrix that takes four points to four others.
8 equations in h₁…h₈ with h₉ = 1, solved by Gauss–Jordan elimination
Partial pivoting on an 8 × 9 array of 64-bit numbers; a pivot below 1e-12 in absolute value returns null. The entries mix pixel coordinates and their products, so the system's conditioning depends on the photo's size in pixels.
photosketch/photosketch.js
lines 88 to 102
ModelCraft 0.8.1
/** 3×3 matrix (row-major, h33 = 1) with H·src[i] = dst[i] for four point pairs. */
export function solveHomography(src, dst) {
const A = [];
for (let i = 0; i < 4; i++) {
const { x, y } = src[i], { x: u, y: v } = dst[i];
A.push([x, y, 1, 0, 0, 0, -u * x, -u * y, u], [0, 0, 0, x, y, 1, -v * x, -v * y, v]);
}
for (let c = 0; c < 8; c++) { // Gauss–Jordan with partial pivoting
let p = c; for (let r = c + 1; r < 8; r++) if (Math.abs(A[r][c]) > Math.abs(A[p][c])) p = r;
if (Math.abs(A[p][c]) < 1e-12) return null;
[A[c], A[p]] = [A[p], A[c]];
for (let r = 0; r < 8; r++) if (r !== c) { const f = A[r][c] / A[c][c]; for (let k = c; k < 9; k++) A[r][k] -= f * A[c][k]; }
}
return [...Array.from({ length: 8 }, (_, i) => A[i][8] / A[i][i]), 1];
}
export const applyH = (H, x, y) => { const w = H[6] * x + H[7] * y + H[8]; return { x: (H[0] * x + H[1] * y + H[2]) / w, y: (H[3] * x + H[4] * y + H[5]) / w }; };
Sheet proportions
64-bit
The long-to-short ratio of the straightened sheet.
w = mean of top and bottom lengths, h = mean of left and right; r = max ÷ min; replaced by the named paper's ratio, or by the nearest standard ratio within 8 %
The measured ratio comes from the photo's corner distances, which perspective distorts; it is used as measured only when no paper is named and no standard ratio lies within 8 %.
photosketch/photosketch.js
lines 106 to 117
ModelCraft 0.8.1
/** The straight sheet's long/short ratio: the named paper's, else the nearest standard one within 8 %, else as measured. */
function sheetRatio(corners, paper) {
const d = (a, b) => Math.hypot(a.x - b.x, a.y - b.y);
const [tl, tr, br, bl] = corners;
const w = (d(tl, tr) + d(bl, br)) / 2, h = (d(tl, bl) + d(tr, br)) / 2;
const landscape = w >= h;
let r = Math.max(w, h) / Math.min(w, h);
const p = paper && paper !== 'auto' ? PAPERS.find((x) => x.key === paper) : null;
if (p) r = p.w / p.h;
else { const near = RATIOS.reduce((a, b) => (Math.abs(b - r) < Math.abs(a - r) ? b : a)); if (Math.abs(near - r) / near <= 0.08) r = near; }
return { ratio: r, landscape };
}
Drawing units per pixel
64-bit
The size of one pixel of the straightened sheet in drawing units, and the mapping from pixels to drawing coordinates.
mm per px = long side in mm ÷ long side in px; units per px = mm per px × scale ÷ unit in mm; x = origin_x + px·k, y = origin_y + (height − py)·k
With a named paper the long side is its size times an exact unit factor. With paper on auto the long side is 297 mm whatever ratio was found, so a sketch on any other sheet comes out at the wrong size unless the scale is corrected.
Converts each traced arc to drawing units as a circle or as an arc through its start, middle and end.
r = r_px × units per px; circle when a₁ − a₀ ≥ 2π − 1e-6; else arc through start, middle (at (a₀ + a₁) ÷ 2) and end
An arc is rebuilt with arcThrough from its start, middle and end after they are converted to drawing coordinates, so its center and radius are recomputed from those three points and not scaled from the traced values. A full circle keeps its traced center and radius, scaled.
The center, radius and end angles of the arc through three points.
d = 2·(a_x(b_y − c_y) + b_x(c_y − a_y) + c_x(a_y − b_y)); center from the sums of |p|²·(differences) ÷ d; r = |a − center|
The center formula uses the squared coordinates of the points themselves, so its terms nearly cancel when the points are far from the origin compared with the radius. Points count as collinear when |d| is below 1e-12, an absolute value in squared drawing units.
Otsu threshold and flood fill for the sheet; a four-point homography solved by Gauss–Jordan elimination with partial pivoting; bilinear resampling; drawing units per pixel = paper length in mm ÷ pixels × scale ÷ unit in mm; arcs rebuilt through three points.
Number types
64-bit numbers for geometry; 8-bit images. Unit factors in paper.js: inches 25.4, feet 304.8, yards 914.4, miles 1 609 344 mm, which are the exact definitions.
Tolerances
Homography pivot below 1e-12 (absolute) is degenerate. A standard paper ratio is adopted when within 8 % of the measured one. Trace tolerance 1.6 pixels by default. An arc is a circle when its sweep is within 1e-6 rad of 2π. Three points are collinear when the determinant is below 1e-12 (absolute).
Limits
Lengths in the drawing are right only if the sheet is the paper size assumed. Corner positions are good to about one pixel of the reduced image.
Approach: 64-bit, stated limit
The arithmetic is 64-bit, but the method or the data sets the accuracy: a solver tolerance, a mesh, a step size, a datum shift, a table. The limit is stated where the result is shown.