The Data Pipeline

A site with no server-side code has to settle at build time everything it cannot compute at runtime. This page follows what that settling involves, in order. In what form do the positions, brightnesses and colours of some 38,000 stars reach the browser? The catalogue carries no colour at all, so where does a star get its blue-white or orange tint? How is a heavy catalogue split across two loading stages? How are the constellation lines and the city list produced, and what discipline binds them? And finally, how are the provenance and licences of all this data handled?

One rule: decide it at build time

Every dataset is finalised at build time. Build-only scripts read the raw catalogues and dumps and write their products into public/data/. The browser repacks those bytes into the layout the GPU wants and scales the directions onto the celestial sphere, but it computes no new values: it only loads and draws.

That rule splits the code in two. One half runs in the browser, and the astronomy maths lives in its pure computation layer. The other half runs only at build time, on Node.js, free to use CSV parsers and image tooling — and in exchange it never imports a single line of app source. The first prohibition keeps heavy tooling out of the browser; the second keeps the build from being dragged along by the app. They point in opposite directions, but both exist to stop code that ships to the browser from mixing with code that only ever runs at build time.

The consequence is deliberate duplication of the few helpers both sides need. The conversion from right ascension and declination into a unit vector exists once in the app and once, separately, in the build. Keeping the build free of any dependency on app code was preferred over avoiding the duplication: a build script is run on its own from Node, as npx tsx scripts/build-stars.ts, and rebuilding the catalogue should not be dragged to a halt by a change in how the app arranges its files or in what the browser needs. The guard against drift is simple: each copy has its own test written against the same expected values, the axes of the J2000 equatorial coordinates written as unit vectors — RA 0h, Dec 0 degrees is the +X axis, and Dec 90 degrees, the north celestial pole, is +Z. Change one copy alone and that copy's test fails.

The star catalogue binary

Positions and brightnesses come from the AT-HYG v4.0 "HYGLike" subset — a star catalogue by David Nash / astronexus, cut to the column layout of the long-standing HYG catalogue, distributed as a gzipped CSV of a dozen-odd megabytes and holding 119,670 stars before any magnitude filtering. The build script reads it as a stream, discarding the Sun's row and any row whose magnitude or position is not a number. The rest is written out as two catalogue tiers, and the two are nested: the bright tier sits, unchanged, in the leading part of the full tier.

Three magnitudes are involved: 8.0 is the faint end of what goes into the catalogue, 6.8 the faint end of what is drawn on screen, and 6.5 the cut between the two tiers. A larger magnitude means a fainter star, so all three mark a limit on the faint side — and they link up into one chain. 6.8 is the figure usually quoted for what the naked eye reaches under a sky free of light pollution (raise the light pollution and that end moves back towards the bright side; how it was chosen is in the section "Rendering the Atmosphere and the Stars"). 6.5 is the usual rule of thumb for the naked-eye limit. And 8.0 is that drawing end of 6.8 plus 1.2 magnitudes of headroom. That headroom has no assigned use today; it is simply enough slack that the drawing end could later move a little fainter without rebuilding the catalogue.

Conversely, a setting that loads only the bright tier shows nothing fainter than 6.5, so the full 6.8 is only ever drawn when the full catalogue has been loaded — which settings load which file is covered under "Loading in two stages", later on this page. The current output is 8,037 stars brighter than magnitude 6.5 in the bright catalogue tier, and 38,168 brighter than magnitude 8 in the full one.

The format is not JSON but a small hand-rolled binary of fixed-length records, little-endian throughout. The first 16 bytes are a header of four 32-bit unsigned integers — a magic number, a version, the star count, and a fourth word left empty for future extensions (currently zero) — followed by one 28-byte record per star, seven 32-bit floats each. A record carries direction, brightness and colour: the direction as a unit vector in the equatorial frame of the J2000 reference epoch, the brightness as the apparent V magnitude (V being the visual band, green to yellow) exactly as the catalogue gives it, and the colour as linear RGB baked from the B−V colour index. The directions ship in that year-2000 frame and stay there. Bringing them to the sky of the current date happens in the browser, and no star is moved individually: the whole celestial sphere is turned by a single matrix that carries the precession from the 2000 frame to the frame of the day, as the section "Astronomy: Where the Stars Are" describes. The files are 225,052 bytes for the bright catalogue tier and 1,068,720 bytes for the full one.

export const MAGIC = 0x41535452 // "ASTR"
export const VERSION = 1
const HEADER_BYTES = 16
const RECORD_FLOATS = 7
const RECORD_BYTES = RECORD_FLOATS * 4 // 28

JSON was rejected for two reasons. First, the full catalogue as JSON allocates one object per star to parse, 38,168 of them, and every one of those costs something to create and something more to reclaim later through garbage collection. With the binary, the bytes that arrive are never copied at all: the same memory is simply viewed as an array of floats. Exactly one copy follows, and it is the one that splits that single run of numbers into the separate position, brightness and colour attributes the GPU wants. Second, fixed-length records make the offset of star i a single multiplication. The difference on the wire, by contrast, is small: rewrite the bright catalogue tier of 8,037 stars as the equivalent JSON and gzip both, since compressed bytes are what the browser actually downloads. After gzip the binary comes to about 154 KB and the JSON to about 163 KB, some 6 per cent apart. The guess here is that what the binary really buys is not bytes transferred but allocations after arrival — and it stays a guess, because neither the parse time nor the cost of those allocations has been measured.

The ordering is part of the design too: records are sorted by magnitude ascending, brightest first. So "draw only the 6,000 brightest" reduces to setting a draw range over the leading 6,000 elements of the buffer already on the GPU. Star count becomes a cheap knob — no re-upload — which is what makes cutting quality cheap when frames get expensive.

Names live outside the binary, in a separate JSON file that only pairs a record index with a proper name, a Bayer designation (Greek letter plus constellation abbreviation, e.g. Alp CMa for Sirius in Canis Major) and a HIP number from the Hipparcos catalogue. Positions and magnitudes are needed every frame; names only when a label is drawn. That difference in frequency is exactly what the file split encodes. Japanese names are not in there either: the translation table keyed by English name or abbreviation lives in the app, so data and language never mix.

One practical correction sits around names. AT-HYG v4.0 gives each member of a multiple star its own row, so companions such as "Capella B" appear next to their primaries. Among the companions in this catalogue that carry a proper name, the separation from the primary is at most about a hundredth of a degree: 0.0093 degrees for Capella B. Acrux B is the degenerate case, separation zero — the catalogue gives it exactly the same position as its primary. Either way, labelling both produces two overlapping, unreadable strings. The build therefore looks for proper names shaped "base name, space, a single letter B, C or D" and drops one only when that base name is itself present in the same file. "Capella B" goes because "Capella" is there. The letters stop at D because the catalogue holds essentially nothing past D. In numbers: 69 proper names end in a space and a single capital, of which 60 end in B, seven in C, one in D and one in A, and nothing beyond D occurs at all. That single A, "Struve 2398 A", has no bare "Struve 2398" in the catalogue, so widening the rule to A would change nothing — and for the same reason its partner "Struve 2398 B" survives too.

The loading path in the app applies the same rule a second time. That looks redundant, but the two passes differ in job: the build pass takes the duplicates out of the shipped file itself, the app pass is insurance. It guards three cases: a future rebuild of the catalogue reintroducing names of the same shape, the name file being hand-edited, and an older copy being rolled back and shipped. In each of them the fix the build baked in is missing from the file that actually reaches the browser. Filtering again at load time means a duplicate label never reaches the screen. Hand-editing is possible at all because the name file is a shipped artefact with nothing standing between it and an editor. The city names later in this section work the other way round: if a city ends up without a label a Japanese reader can read, the build itself stops, so patching the output by hand is never an available escape.

Colour: from B−V to linear RGB

The catalogue carries no colour. What it carries is the B−V colour index, that difference between a star's brightness through a blue filter and through a yellow one. The build converts it into colour in three steps and bakes the result into each record.

• B−V to surface temperature, via the empirical Ballesteros (2012) relation.
• Temperature to sRGB, the standard display colour space, by linear interpolation over Mitchell Charity's published blackbody colour table.
• sRGB to linear RGB, because the shaders work in linear light and the renderer applies its own tone mapping later.

Stars whose row carries no B−V — 58 of the 38,168 brighter than magnitude 8 — get the colour of a cool F-type star, a little hotter and whiter than the Sun and near the F/G boundary, around B−V 0.58 or 6,000 K. All three steps approximate blackbody radiation, where a heated body glows in a colour set by its temperature alone, rather than reproducing true stellar spectra; the judgement is that it is good enough for the tints on screen. In return, no colour maths at all is left to run in the browser.

Loading in two stages

The catalogue loads in two stages. The bright catalogue tier sits on the same origin as the page itself and is always fetched for the first frame. The full catalogue is fetched only when two conditions meet: the quality tier asks for it, and the browser has gone idle. Note that these catalogue tiers are different from the quality tiers: a catalogue tier is a way of splitting the data across files, a quality tier is a setting for how expensive the rendering may be, and the latter decides which file gets loaded. Of the four quality tiers, only the top two, high and ultra, ask for the full catalogue; low and medium draw from the bright catalogue tier alone, so nothing fainter than magnitude 6.5 appears at all. The full-tier file also comes from somewhere else: R2, the object storage used for large files. What R2 does for the site is in the section "Delivery and the Standalone Pages". The first frame never waits on the faint stars.

// Only the 'full' tiers pull the big catalogue, and only when idle.
if (tierConfig(tier).starCount !== 'full' || !handlers.onUpgrade) return
// …
if (typeof ric === 'function') {
  idleHandle = ric(runUpgrade, { timeout: 2500 })
} else {
  idleHandle = setTimeout(runUpgrade, 1500) as unknown as number
}

In the excerpt, ric is a local alias for requestIdleCallback. The 2,500 ms cap handed to it is not decoration: with the 3D canvas rendering continuously the browser may never consider itself idle at all, and without the cap the upgrade would simply never fire. A browser without requestIdleCallback cannot report idleness at all, so that branch gives up and uses a plain 1,500 ms timer.

What the second stage fetches is the whole catalogue, not a delta against what the browser already holds. The bright catalogue tier is its leading part, so the same 8,037 stars arrive a second time — refetching the full 1 MB catalogue was judged simpler than building and applying deltas. The swap itself is quiet: the star component rebuilds its vertex buffers, the per-vertex arrays it hands the GPU, from the new data, and it is not remounted, so the shaders survive untouched. Because no shader is rebuilt, none has to be recompiled for the GPU either.

Constellation lines and cities

The raw constellation data is a set of polylines joining star to star with straight segments. The build subdivides any segment wider than five degrees into roughly two-degree steps along the great circle — the circle a plane through the sphere's centre traces on its surface, which is the shortest path between two points on a sphere — before shipping it. The intermediate points come from spherical linear interpolation, and the renderer is left with nothing to do but join the points it is given.

What the subdivision buys is not shape but distance — the value the depth buffer uses to decide which of two things is drawn in front. The observer's camera always sits at the centre of the celestial sphere, so a straight line between two endpoints points the same way as the arc, and the figures would not look deformed without it. The stars sit on a celestial sphere of radius 1000, and the lines just inside it, at radius 995 — a 0.5 per cent gap that keeps the lines and the bright-star billboards from swapping places in depth. The gap is kept no wider than that so the lines still read as sitting on the stars: 0.5 per cent of radius 1000 is 5 units; with the observer at the centre of the sphere that gap is invisible on screen and matters only to the depth test.

The raw data really does contain long segments: 345 of its 743 segments span more than five degrees, and the longest is about 26 degrees. Join the ends of that longest one with a straight line and its midpoint drops about 2.5 per cent inward (half of 26 degrees is 13, and 1 − cos 13° is that sag). Inward is towards the observer at the centre of the sphere, so sagging inward is moving forwards, and this sag is five times the gap: the line comes out in front of the stars. Subdivided, every vertex lies on the same shell, and a two-degree hop sags only 0.015 per cent of the radius at its midpoint (1 − cos 1°), about a thirtieth of the gap. The line therefore stays behind the stars along its whole length. The five-degree threshold follows from the same gap: a five-degree chord dips about 0.1 per cent of the radius at its midpoint (1 − cos 2.5°), comfortably inside the 0.5 per cent.

/** Segments wider than this (great-circle degrees) get densified. */
const MAX_STEP_DEG = 5
/** Target step size for the densified sub-segments. */
const SUBDIV_STEP_DEG = 2

Two things about counting the output. A constellation is not always one polyline — wherever the figure breaks, it gets another — and the number of entries does not match the number of constellations. The entries differ because Serpens is split into a head and a tail, two separate pieces of sky, each its own entry. In numbers: the current output is 150 polylines and 2,169 points, 76,757 bytes uncompressed; there are 89 entries (one JSON record per constellation) against 88 constellations, and 35 of those entries hold more than one polyline. Here too no Japanese is emitted, only the three-letter IAU (International Astronomical Union) abbreviations and English names.

The city list behind the observer picker is built from the dump of settlements over 15,000 people published by GeoNames, an open database of place names carrying coordinates, populations and alternate names in many languages. Selection is the union of four rules: every national capital, the Japanese prefectural capitals plus Tokyo, the thousand most populous cities (a round figure chosen to size the list), and a hand-picked set that population alone would miss but that the map needs as reference points. That currently yields 1,154 cities.

One rule here is unusually strict. Because the interface is Japanese-first, every selected city must carry a label a Japanese reader can read. The build tries four fallbacks in order: the Japanese alternate name from GeoNames, the Chinese Han name for cities in China, Taiwan, Hong Kong and Macau, the Korean Hangul name for Korea, and finally a hand-curated katakana table. Han characters work as a substitute because they are read as Japanese characters as they stand. Hangul is the compromise of the set: the name is left in Hangul, in the knowledge that most Japanese readers cannot read it. If none of the four hits, the build does not warn and carry on: it throws an exception, so the city file is never written at all. The exception message is formatted as lines ready to paste straight into that katakana table. Because the build guarantees that a usable label always exists, no city-label code needs a "fall back to English" branch.