Resources Numerical accuracy Calculation policy

Map layers: web Mercator, tile grids and world files

Converts longitude and latitude to web Mercator and back, lays out the tiles of a map service, reads a picture's world file, and writes the box of a map request.

How the precision is kept

Longitude and latitude go through the spherical web Mercator formulas with R = 6378137 m in 64-bit. Tile bounds are the grid origin plus whole tile counts times the tile size. A world file's six numbers are read as 64-bit and shifted by half a pixel to pixel corners. The box written into a WMS or map-server export request is rounded to 3 decimals (values of 10,000 or more) or to 12 significant digits.

Open item. Draw the returned map picture at the same rounded box that was sent in the request, or write the box with 17 significant digits.

Calculations under this policy

10 calculations in 1 source file. The code is read from the released app when this page loads. It is the calculation people run, not a copy kept for this page.

basemap.js CadCraft 0.8.2 64-bit 10 calculations Reviewed

Tests bin/test-civilcraft-maps.mjs, bin/test-cadcraft.mjs

Longitude and latitude to web Mercator

64-bit

The web Mercator x and y in meters for a longitude and latitude in degrees.

x = R·λ·π ÷ 180; y = R·ln(tan(π ÷ 4 + φ·π ÷ 360)), R = 6378137

Latitude is clamped to ±85.0511287798066° first. Close to the equator π ÷ 4 + φ·π ÷ 360 is a large number plus a small one and the tangent is close to 1, so y keeps an absolute error of about R × 10⁻¹⁶, near 10⁻⁹ m, rather than 15 relative digits: evaluating the formula for φ = 10⁻⁹ degrees gives 0.00011132000 m against an exact 0.00011131949 m.

  • basemap.js
  • lines 42 to 46
  • CadCraft 0.8.2
// ------------------------------------------------------------------ web Mercator
export function lonLatToMerc(lon, lat) {
  const l = Math.max(-85.0511287798066, Math.min(85.0511287798066, lat));
  return [R_MERC * lon * Math.PI / 180, R_MERC * Math.log(Math.tan(Math.PI / 4 + l * Math.PI / 360))];
}

Web Mercator to longitude and latitude

64-bit

The longitude and latitude in degrees for a web Mercator x and y in meters.

λ = x ÷ R × 180 ÷ π; φ = (2·atan(e^(y ÷ R)) − π ÷ 2) × 180 ÷ π

For small y the latitude is the difference of 2·atan(e^(y ÷ R)) and π ÷ 2, two nearly equal numbers, so it carries an absolute error of a few 10⁻¹⁵ degrees (under 10⁻⁹ m on the ground) however small it is. The longitude is two divisions and one multiplication.

  • basemap.js
  • lines 47 to 49
  • CadCraft 0.8.2
export function mercToLonLat(x, y) {
  return [x / R_MERC * 180 / Math.PI, (2 * Math.atan(Math.exp(y / R_MERC)) - Math.PI / 2) * 180 / Math.PI];
}

Standard tile grid

64-bit

The origin and the resolution of each zoom level of the usual 256-pixel web Mercator tiles.

origin = (−πR, πR); res(z) = 2πR ÷ 256 ÷ 2^z, z = 0…22

πR is Math.PI × 6378137 in 64-bit. Each zoom's resolution divides by 256 and by a power of two, both exact, so every level carries only the rounding of π × R.

  • basemap.js
  • lines 50 to 54
  • CadCraft 0.8.2
/** The standard XYZ ("Google") tile grid: 256 px tiles, zoom 0 = one tile for the world. */
export const XYZ_GRID = Object.freeze({
  code: 3857, origin: [-HALF_MERC, HALF_MERC],
  matrices: Array.from({ length: 23 }, (_, z) => Object.freeze({ id: String(z), res: (2 * HALF_MERC) / 256 / 2 ** z, tw: 256, th: 256, mw: 2 ** z, mh: 2 ** z })),
});

Tile that holds a place

64-bit

The tile column and row at a zoom level for a longitude and latitude.

col = ⌊(x + πR) ÷ (res × 256)⌋; row = ⌊(πR − y) ÷ (res × 256)⌋

Whole tile numbers by Math.floor of a 64-bit quotient with no tolerance, so a place within rounding of a tile edge can fall in either neighbor.

  • basemap.js
  • lines 55 to 59
  • CadCraft 0.8.2
/** Tile column/row of a lon/lat at zoom z (whole numbers). */
export function tileAt(lon, lat, z) {
  const [x, y] = lonLatToMerc(lon, lat), m = XYZ_GRID.matrices[z];
  return [Math.floor((x + HALF_MERC) / (m.res * 256)), Math.floor((HALF_MERC - y) / (m.res * 256))];
}

Tiles covering a box

64-bit

The tiles of a grid that cover a box, each with its bounds in the grid's units.

bounds = [ox + c·tw, oy − (r + 1)·th, ox + (c + 1)·tw, oy − r·th], tw = res × tile width

Column and row ranges are Math.floor of (box − origin) ÷ tile size, clipped to the matrix. Each tile's bounds are the origin plus a whole count times the tile size, worked out separately for every tile, so no error builds up across a row. The list is cut at the cap (64 by default) and a coarser level is used when more would be needed.

  • basemap.js
  • lines 70 to 95
  • CadCraft 0.8.2
/** Tiles of `grid` covering bbox [minX, minY, maxX, maxY] (grid units) at screen resolution `res`; a coarser matrix when
 *  more than `cap` tiles would be needed. */
export function gridTiles(grid, bbox, res, { cap = 64 } = {}) {
  const ms = [...grid.matrices].sort((a, b) => b.res - a.res);
  let i = ms.indexOf(pickMatrix(grid, res));
  for (; i >= 0; i--) {
    const m = ms[i], tw = m.res * m.tw, th = m.res * m.th;
    let c0 = Math.floor((bbox[0] - grid.origin[0]) / tw), c1 = Math.floor((bbox[2] - grid.origin[0]) / tw);
    let r0 = Math.floor((grid.origin[1] - bbox[3]) / th), r1 = Math.floor((grid.origin[1] - bbox[1]) / th);
    c0 = Math.max(0, c0); r0 = Math.max(0, r0);
    if (Number.isFinite(m.mw)) c1 = Math.min(m.mw - 1, c1);
    if (Number.isFinite(m.mh)) r1 = Math.min(m.mh - 1, r1);
    if (!(c1 >= c0 && r1 >= r0)) return { matrix: m, tiles: [] };
    if ((c1 - c0 + 1) * (r1 - r0 + 1) > cap && i > 0) continue;
    const tiles = [];
    for (let r = r0; r <= r1; r++) for (let c = c0; c <= c1; c++) {
      if (tiles.length >= cap) break;
      tiles.push({ col: c, row: r, bounds: [grid.origin[0] + c * tw, grid.origin[1] - (r + 1) * th, grid.origin[0] + (c + 1) * tw, grid.origin[1] - r * th] });
    }
    // the middle first: what the eye looks at arrives first
    const cx = (bbox[0] + bbox[2]) / 2, cy = (bbox[1] + bbox[3]) / 2;
    tiles.sort((a, b) => Math.hypot((a.bounds[0] + a.bounds[2]) / 2 - cx, (a.bounds[1] + a.bounds[3]) / 2 - cy) - Math.hypot((b.bounds[0] + b.bounds[2]) / 2 - cx, (b.bounds[1] + b.bounds[3]) / 2 - cy));
    return { matrix: m, tiles };
  }
  return { matrix: ms[0], tiles: [] };
}

Meters per unit of a coordinate system

64-bit

The length in meters of one degree, foot, US survey foot or meter, for turning a WMTS scale into a resolution.

degree: 2π × 6378137 ÷ 360; foot: 0.3048; US survey foot: 1200 ÷ 3937; otherwise 1

0.3048 and 1200 ÷ 3937 are the exact definitions, the second evaluated as one 64-bit quotient. Meters per degree is 2π × 6378137 ÷ 360, the equatorial radius used as a sphere.

  • basemap.js
  • lines 260 to 261
  • CadCraft 0.8.2
/** Metres per unit of a system for WMTS scale denominators (0.28 mm pixels). Geographic: metres per degree. */
export const metresPerUnit = (unit) => (unit === 'deg' ? 2 * Math.PI * 6378137 / 360 : unit === 'ft' ? 0.3048 : unit === 'ftUS' ? 1200 / 3937 : 1);

Tile grid of a WMTS tile matrix set

64-bit

The resolution and top-left corner of each level of a WMTS tile matrix set.

res = scale denominator × 0.00028 ÷ meters per unit

0.00028 is the 0.28 mm pixel written as a 64-bit number; it has no exact binary form, so each resolution is rounded in the product and again in the division. The top-left corner is used as written in the capabilities, with its two numbers swapped when the system is latitude-first or when only the swapped corner's matrix holds the middle of the layer's box.

  • basemap.js
  • lines 262 to 280
  • CadCraft 0.8.2
/** A WMTS tile matrix set as a grid with x/y corners. `toSet(lon, lat)` maps WGS 84 to the set's system and `bbox84`
 *  (the layer's) decides the axis order of TopLeftCorner when the set's own CRS does not say it clearly. */
export function wmtsGrid(set, unit, { toSet = null, bbox84 = null } = {}) {
  const k = metresPerUnit(unit);
  let swap = set.latFirst;
  if (!set.crs84 && !set.latFirst && toSet && bbox84) {
    // projected systems with northing-first axes (NZTM…) write TopLeftCorner northing first: try both, keep the one whose
    // matrix holds the layer's middle
    try {
      const [mx, my] = toSet((bbox84[0] + bbox84[2]) / 2, (bbox84[1] + bbox84[3]) / 2);
      const m = set.matrices[0], w = m.scale * 0.00028 / k * m.tw * m.mw, h = m.scale * 0.00028 / k * m.th * m.mh;
      const holds = ([x, y]) => mx >= x && mx <= x + w && my <= y && my >= y - h;
      if (!holds(m.tl) && holds([m.tl[1], m.tl[0]])) swap = true;
    } catch { /* keep the CRS's own order */ }
  }
  const matrices = set.matrices.map((m) => ({ id: m.id, res: m.scale * 0.00028 / k, tw: m.tw, th: m.th, mw: m.mw, mh: m.mh, origin: swap ? [m.tl[1], m.tl[0]] : [m.tl[0], m.tl[1]] }));
  // one origin per grid; WMTS allows one per matrix — keep each matrix's own
  return { code: set.code, origin: matrices[0].origin, matrices };
}

World file to a pixel-corner transform

64-bit

The transform from pixel corners to coordinates for a picture, from the six numbers of its world file.

x₀ = C − A ÷ 2 − B ÷ 2; y₀ = F − D ÷ 2 − E ÷ 2

The six numbers are read from the text with Number. The shift from pixel center to pixel corner halves A, B, D and E (exact) and subtracts them from C and F. A file whose determinant A·E − B·D is under 1e-30 in size (absolute) is refused.

  • basemap.js
  • lines 398 to 408
  • CadCraft 0.8.2
/** A world file (.pgw .jgw .tfw .wld …): six numbers A D B E C F — x = A·col + B·row + C, y = D·col + E·row + F, for the
 *  CENTRE of pixel (col, row). → GDAL geotransform [x0, dxCol, dxRow, y0, dyCol, dyRow] for pixel CORNERS. */
export function parseWorldFile(text) {
  const s = String(text || '');
  if (s.length > 10000) throw new Error(t('basemap.world.long'));
  const v = s.trim().split(/\s+/).map(Number);
  if (v.length !== 6 || !v.every(Number.isFinite)) throw new Error(t('basemap.world.six'));
  const [A, D, B, E, C, F] = v;
  if (Math.abs(A * E - B * D) < 1e-30) throw new Error(t('basemap.world.zero'));
  return [C - A / 2 - B / 2, A, B, F - D / 2 - E / 2, D, E];
}

Pixel to coordinates

64-bit

The coordinates of a pixel position in a picture's own system.

x = g₀ + u·g₁ + v·g₂; y = g₃ + u·g₄ + v·g₅

One expression per coordinate: two products added to the origin, in 64-bit.

  • basemap.js
  • line 417
  • CadCraft 0.8.2
export const pixelToSystem = (gt, u, v) => [gt[0] + u * gt[1] + v * gt[2], gt[3] + u * gt[4] + v * gt[5]];

Numbers in a map request

64-bit

How the corners of the requested box are written into a WMS or map-server export address.

|v| ≥ 10,000: v.toFixed(3); otherwise v.toPrecision(12)

The box sent to the server is rounded to 3 decimals for values of 10,000 or more and to 12 significant digits otherwise. maplayers.js then draws the returned picture at the unrounded box, so the picture can sit up to 0.0005 units, or half a unit in the 12th digit, from where it was requested.

  • basemap.js
  • line 206
  • CadCraft 0.8.2
const fmtNum = (v) => (Math.abs(v) >= 1e4 ? v.toFixed(3) : String(+v.toPrecision(12)));

In detail

Method
Spherical web Mercator: x = R·λ, y = R·ln tan(π ÷ 4 + φ ÷ 2). WMTS resolution = scale denominator × 0.00028 ÷ meters per unit (the 0.28 mm pixel named in the code's comment). World file to a geotransform for pixel corners by a half-pixel shift.
Number types
64-bit numbers. Tile widths and heights read from a saved grid pass through | 0; they are pixel counts, not measurements.
Tolerances
Latitude is clamped to ±85.0511287798066°. A world file or a stored picture transform with |A·E − B·D| under 1e-30 (absolute) is refused.
Limits
The request box is rounded as described, while maplayers.js draws the returned picture at the unrounded box. Near the equator the Mercator y and its inverse are good to about 10⁻⁹ m absolute, not to 15 relative digits. Meters per degree uses the radius 6378137 m as a sphere for every geographic system.

Approach: 64-bit, direct

Every value is a 64-bit number from input to result. Formulas are closed form or a direct solve, and every comparison uses a stated tolerance.

Every policy that takes this approach