Repository navigation
Conversation
For zarr files, set up a .complete marker file to check when data is fully written. This replaces relying on zarr.json -- that doesn't work because zarr.json is the first to be written. This might happen from a failed or aborted calculation, or because multiple processes are trying to write the same file (e.g. running 2 countries which share a tile) A partially-written zarr might be not cause any warning signs later, since it might be techncially valid in spite of missing data. Now an empty .complete marker is written beside the zarr after its last chunk, and the entry counts as cached only once the marker exists. A zarr found without its marker is a write that was killed part-way or is still running: serialize raises FileExistsError and leaves it for someone to resolve by hand, usually by deleting the zarr, rather than writing over it and risking a race with a live writer. Entries written before this change have no marker, so each raises once until deleted.
An autouse fixture in jdluc/__tests__/conftest.py stubs Config.from_dot_env for every test not marked integration. Each field gets its type's empty value, with two dask workers, so a unit test that reaches for a storage root fails instead of touching a bucket. Integration tests still read the real .env. This replaces the fixture storage_test carried for itself, so the next unit test that writes a zarr needs no copy of it, and a new Config field needs no edit to the tests.
An optional `orchestration` package with one subcommand per workflow task (plan, ingest-tile, compute, trace), and an example Argo WorkflowTemplate that runs each tile's ingest -> compute as a DAG of its own, so a failed tile stops only itself. jdluc runs unmodified in the pods; every cache-key argument passes straight through, and an import-linter contract keeps jdluc from importing orchestration. trace writes the emissions-factor table to a new EXPORT_ROOT, through a small `export.write_table` utility. The package is built to be wrapped rather than edited: `build_parser` and `run` let a wrapper add subcommands, `plan.get_task_to_items` returns the lists for it to add to before `write_task_to_items`, and the template's tasks can be called by templateRef. See docs/orchestration.md. Also adds the Dockerfile the workflow's image is built from.
get_name no longer mentions mosaics, get_task_to_items describes the flat lists without a flat workflow to point at, and trace's --help is a whole sentence.
A Python API only: nothing calls it, a workflow opts in. Writes the chosen
variables of a lazy dataset to {EXPORT_ROOT}/{name}/{tile_id}.{tif|zarr},
provenance from get_metadata.
- COG: every variable as float32 2-D bands, NaN no-data; extra dims are
flattened to one band per combination of values, named
variable:value[:value...]. Staged locally and uploaded whole, through
storage.write_dask_dataset_to_cog (moved into storage with stack_bands).
- zarr: variables as they are, dtypes and extra dims kept. An export is
re-runnable, so it writes over an existing zarr (write_dask_dataset_to_zarr
gains mode=, default "w-" so the cache is unchanged), dropping the
.complete marker first and touching it once the write finishes.
…on}/
write_tile and write_table take version=, such as a run id or a release
label. Blank, the default, writes directly under EXPORT_ROOT as before; set,
the whole export goes under {EXPORT_ROOT}/{version}/ and the version is
stamped as cornerstone-export-version. A version must be one path segment.
Also from review:
- write_tile rejects a bare str as variables, rather than reading its letters
- the COG compute runs under COMPUTE_LOCK, as the zarr write does
- iter_bands notes how datetime and float coords name their bands
- to_zarr_dataset says why only coords become variable-length strings
A detached HEAD, as CI checks out a pull request, has no active branch, so get_git_version raised and with it every export that stamps provenance. It now returns the short commit SHA. No .git still fails, as before.
… derive emit writes nine `cie-` bands beside its output: the conversion source and year, and each carbon pool the conversion puts at risk, before any destination gate, destination-specific soil factor or discount. `derive_from_cie` rebuilds every existing band from them alone, and a test checks it does on a grid covering each event, destination and soil case. Additive: the existing bands, and every consumer of them, are unchanged. `CIE_BANDS` names the bands for `export.write_tile`. docs/crop_independent_emissions.md sets the two side by side; tools/compare-emit-layers.py checks the identities on a real tile.
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
emitnow writes ninecie-(Crop-Independent Emissions) bands beside its existing output. They describe the same conversions and carbon pools, but stop before anything that depends on what the land became:emit.derive_from_cierebuilds every existing band from thecie-bands alone.test_derive_from_cie_reproduces_every_band_it_replaceschecks this on a grid covering every event, destination and soil case.Additive only. The existing bands are unchanged, and so is every consumer:
jurisdictional_directandstatisticalstill read them directly. Switching consumers toderive_from_cieis left for a follow-up.Changes
jdluc/emit.py:ConversionSource,ConversionRecord.sourceandget_crop_independent_emissions.get_cropland_soil_loss_fraction,derive_from_cieandget_name_to_legacy_darray.workflowsplit into harmonize +get_output_dset.CIE_BANDS, forexport.write_tile(variables=emit.CIE_BANDS, ...).docs/crop_independent_emissions.md: the existing and CIE bands side by side, and how each existing band derives from CIE.docs/data.md: thecie-bands in the emit schema.jdluc/__tests__/emit_test.py:test_cie_*and the derivation equivalence test.tools/compare-emit-layers.py: checks the identities on a real tile, and totals carbon by source and by whether a conversion fired.Cost
Each emit tile goes from 16 bands to 25, so the emit cache grows by about 56%.
Not included
derive_from_cie.Testing
uv run pytest jdluc orchestration: 1363 passed.uv run pre-commit run --all-files: passes.mainand this branch, and ofderive_from_cieagainst them withtools/compare-emit-layers.py. The tile's inputs weren't cached, so the run was stopped during ingest. Results will be added here.