[danohu]

Stacked on #19 (and so #16). Review only the last 3 commits; I'll rebase onto main once #19 merges.

Adds export.write_tile, which writes a pipeline step's per-tile raster output under EXPORT_ROOT, as a COG or a zarr. It is a Python API only: nothing calls it automatically, and a workflow opts in.

export.write_tile(dset=emit.workflow(tile_id=tile_id), format=export.Format.COG, name="emit", tile_id=tile_id)
export.write_tile(dset=statistical.get_downscaled_luc_emissions(tile_id=tile_id), format=export.Format.ZARR, name="statistical-emissions", tile_id=tile_id)

Builds on #19 (jdluc/export.py, EXPORT_ROOT) and #16 (the .complete marker on zarrs), so it merges after both.

What it writes

• Path: {output_root or EXPORT_ROOT}[/{version}]/{name}/{tile_id}.{tif|zarr}.
  • version is optional, for example a run id or a release label. Blank, the default, writes directly under the export root, so a rerun writes over the last one.
  • A set version puts the whole export under {root}/{version}/ and is stamped as cornerstone-export-version. It must be one path segment.
  • write_table takes the same version=, so the tiles and tables of one export sit together. trace.export_workflow doesn't pass one yet.
  • Provenance is stamped by get_metadata either way.
• Variables: variables defaults to every data variable, in the dataset's order. A variable that isn't in dset, or a bare str in place of a list, raises before anything is written. dset is lazy, and only the selected variables are computed, as they are written.
• COG: every variable becomes float32 2-D bands with NaN as no-data.
  • A variable with dimensions beyond (y, x) is flattened to one band per combination of coordinate values, named variable:value[:value…]. This is the : convention emit already uses for its band names.
  • Bands are chunked with geo.get_chunk_size.
  • The file is staged locally as a GeoTIFF, then band names are set, it is validated, converted to a COG and uploaded whole. write_dask_dataset_to_cog and stack_bands are added to storage, next to the zarr writer, and the COG compute runs under COMPUTE_LOCK, as the zarr write does.
• Zarr: the variables are written as they are, keeping their dtypes and extra dimensions. String coordinates become object, because zarr v3 has no fixed-length strings. String data variables are left as they are, because converting a lazy one would load it whole. Metadata goes in the dataset attributes.
  • Exports are meant to be re-runnable, so an export writes over an existing zarr. This differs from the cache, which refuses to overwrite.
  • Write order: remove {uri}.complete, write with mode="w", then touch {uri}.complete.
  • write_dask_dataset_to_zarr gains mode=. It defaults to "w-", so the cache is unchanged.
  • The marker URI is now storage.get_zarr_complete_uri, shared with ZarrCacher.

utils.get_git_version now returns the short commit SHA on a detached HEAD, rather than raising. This is how CI checks out a pull request, and every export stamps the version. A checkout with no .git still fails, as before.

docs/data.md now points to write_tile as the way to write the raster outputs under EXPORT_ROOT.

Not in scope

Default deliverables or band selections, changes to emit or statistical, a CLI, orchestration, and mosaics.

Testing

• jdluc/__tests__/export_test.py runs on synthetic data and needs no .env. It covers:
  • default and subset variables, a missing variable, and a bare str;
  • COG band names and order, flattening of a 3-D variable, float32 with NaN no-data, and metadata tags;
  • zarr dtypes and extra dimensions, the .complete marker, rewriting an export, and removal of a stale marker when a write fails;
  • the output URI with and without output_root and version, and a version that isn't one path segment;
  • the cornerstone-export-version tag, present only when a version is set.
• pytest jdluc orchestration and pre-commit run --all-files pass.
• Checked on a real emit tile: a 4096×4096 window with all 20 variables, written as a COG and as a zarr. Band names, dtypes, tags and .complete were as expected, including after a rewrite.