Data bundles

The contract

A bundle is a directory apps/site/public/data/bundles/<id>/ containing manifest.json and its files. The manifest is validated by bundleSchema (src/content.config.ts) and read through the bundles content collection; the files are served as-is at /data/bundles/<id>/<file>.

Private bundles (D11) use the same layout under apps/site/private/data/bundles/<id>/. They are read and served only by the internal build (pnpm dev:internal / pnpm build:internal), badged Internal — not for distribution on /data/, and a public report that tries to read one fails the build and pnpm check. Make one public with git mv into public/data/bundles/.

{
  "id": "noaa-mlo-co2", "version": "2026.08",
  "title": "…", "description": "…", "publisher": "NOAA Global Monitoring Laboratory",
  "provenance": "Downloaded from … on 2026-08-17; unmodified except …",
  "sourceUrl": "https://gml.noaa.gov/ccgg/trends/", "licence": "Public domain (US Government work)", "licenceUrl": "…",
  "citation": "Lan, X., Tans, P. and K.W. Thoning: … https://doi.org/10.15138/9N0H-ZH07",
  "updated": "2026-08-05", "temporalCoverage": "1958-03/2026-07", "spatialCoverage": "Mauna Loa Observatory …",
  "schema": { "co2_mm_mlo.csv": { "year": "integer", "average": "number (ppm)" } },
  "files": [ { "path": "co2_mm_mlo.csv", "format": "csv", "bytes": 38688, "sha256": "…", "description": "…" } ],
  "keywords": ["carbon dioxide"]
}

<DataDownload bundle="id" /> renders it; /data/ lists all bundles.

Size

Cloudflare static assets are capped at 25 MiB per file. Larger files go to R2: set baseUrl (or per-file url) in the manifest and keep only the manifest in the repo. Loaders can read from a local mirror or the URL.

Loaders (build time)

import { loadCsv, loadParquet, loadJson, fetchRows } from '@data/loaders';
const rows = await loadParquet('noaa-mlo-co2', 'co2_mm_mlo.parquet');   // Row[] — plain objects
const annual = await loadCsv('noaa-mlo-co2', 'co2_annmean_mlo.csv');     // '#' comment lines stripped, types inferred

Rows are memoised per build and cached on disk (.cache/data/) by file size + mtime, so unchanged files cost nothing on rebuild. Rows never go through the content-layer store (which is JSON in memory) — only manifests do.

fetchRows(url) is the runtime twin with the same row shape, for islands and tools that read the bundle in the browser or call the CSDR data API. A page can move from build-time to live data without changing how it consumes rows.

Analysis cells

Keep derived quantities in one memoised module per topic (src/components/analyses/<topic>/<topic>-data.ts) and expose them through small .astro components: charts, a table, inline values. The report imports the components; it never touches rows. See src/components/analyses/co2/.

Adding a bundle

/new-bundle (skill) or by hand: create the directory, drop the files, write manifest.json (the skill computes bytes and sha256), reference the id in the report’s data_bundles.