Repository navigation
Persist ngmix epoch-cut counts; tile cleaning no longer needs module logs - #955
Merged
Merged
Conversation
Capture each ngmix summary before temp cleanup, aggregate the full chunk set at make-cat, and publish the manifest beside final_cat. Refuse reclamation without validated durable counts. This changes completeness params across stages and the make-cat/clean shell fingerprints; deploy between campaigns, not as an unreviewed resume. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
Keep clean_tiles enabled by default, remove the stale false-default comment, and document the ngmix summary prerequisite, persistent schema, backfill procedure and campaign-boundary deployment. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
Use frozen run defaults, require persisted final_cat and complete successful chunk evidence, publish counts atomically, then reuse clean_tile.reclaim. Dry-run uses the same whitelist traversal and never follows links. smk-b01 dry-run: 25 tiles, 53,290,513,588 logical bytes (49.631 GiB), no skips or refusals; nothing written or deleted. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
…kfill Cover missing/ambiguous summaries, full chunk coverage, zero counts, duplicate log copies, stable manifest mtimes, cleanup refusal, read-only byte accounting, publication-before-cleaning and idempotent retries. Include the exact PR #922 producer line. Container verification after restacking: 199 passed, 1 skipped across targeted, structural and ngmix producer suites; ASTRA validation passes. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
force-pushed
the
feat/epoch-cuts-persist
branch
from
October 25, 2026 22:06
d731900 to
d6bb5fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We preserve ngmix's epoch-cut counts in persistent tile manifests, so tile cleaning can reclaim scratch without retaining module logs.
These are chunk/tile diagnostics, including rejected epochs and objects that never reach the final catalogue; no scientific cuts or catalogue values change.
Durable counts and cleanup
epoch cuts:summary before temporary output disappears. Missing, malformed or ambiguous summaries fail the check rather than becoming zero counts.tile_make_cataggregates the full configured chunk set and publishestile_make_cat.jsonbesidefinal_cat-<tile>.hdf5. Itsepoch_cutsrecord holdsschema_version,n_chunks, per-chunk counts and tile totals forconsidered,masked_fraction,central_veto,failedandobjects_emptied.clean_tilerequires a valid persistent catalogue and count manifest before deleting scratch. Existing survivor checks, tombstone records and symlink-safe deletion remain in force.Counts represent sequential vetoes, not independent cut incidences.
objects_emptiedcounts objects whose considered epochs were all removed before fitting.Keeping these totals in a manifest preserves their scope and chunk coverage without changing the catalogue schema.
Deployment
Deploy at a campaign boundary.
The shared completeness/helper hash changes rule parameters across stages; make-cat and cleanup dependencies and shell commands also change.
Resuming an existing campaign against this workflow can schedule science stages again.
For an idle existing campaign with retained logs, use
workflow/scripts/backfill_epoch_cuts.py --run-dir <run> --products-dir <products> --n-chunks <N> --dry-runfirst.The dry run writes and deletes nothing.
Without
--dry-run, the tool atomically publishes the count manifest before reclaiming scratch through the same cleaner.Don't run it alongside campaign jobs or another cleaner.
Historical FITS and HDF5 catalogues are supported; missing catalogues are skipped and ambiguous catalogues or incomplete evidence are refused.
Production backfill has not been exercised by this replay.
workflow/README.mddocuments the schema and usage.tests/unit/test_epoch_cuts_persist.pycovers parsing, chunk coverage, durable publication, cleanup refusal and backfill safety.— Codex on behalf of Cail.