Skip to content

Pairing check warns and records DR6-only losses; empty tiles produce empty products - #981

Merged
cailmdaley merged 7 commits into
developfrom
fix/match-guard-warn
Oct 30, 2026
Merged

cailmdaley merged 7 commits into
developfrom
fix/match-guard-warn

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

External DR6-only pairing losses now warn and record instead of rejecting a tile. Zero-object tiles also have a valid workflow partition, so the scheduled stages can publish empty products.

Sparse edge tiles 058.220 and 207.238 still exceeded the external pairing budget after #975.
The SExtractor-side wrong-pixels check remains a hard failure.
External-only objects never gain a shape.

  • Record for downstream cuts: every newly joined tile's published tile_make_cat.json carries catalogue_match: n_sextractor, n_external, n_paired, n_external_only, n_dropped, external_paired_fraction, and external_warning_budget.
    Join the manifest's unit, converted to float, to final-catalogue TILE_ID.
  • Warning threshold: retain max(20, 0.02 N + 50), without the 50 when no pairs exist.
    The external allowance argument/config key is removed; the threshold no longer controls acceptance.
  • Empty products: explicit empty stamp shapes, empty EPOCH handling, and safe make-cat reductions support zero-row catalogues.
    The workflow retains its configured chunk count and gives every empty chunk the bounded interval (2, 1), avoiding ngmix's unbounded upper-limit sentinel.
    Ordinary chunk, merge, completeness, and publication paths then receive all expected empty files.
  • Decision record: detection.tile_detection states that no pipeline cut applies to external-only losses. Whether the catalogue cuts on external_paired_fraction is open in Decide whether the catalogue cuts on ShapePipe's external_paired_fraction sp_validation#444. The record gives the reason each alternative fails, and pins the removed MATCH_EXTERNAL_UNPAIRED_ALLOWANCE as absent: a config that sets it again fails the decision tests, checked by injecting it.

Nonzero partitions and catalogue measurements are unchanged.
A before/after probe found byte-identical partition documents for 512 nonzero cases.

Tests at c94fbd06: 1153 not-slow and 51 workflow tests pass on a clean checkout in the campaign SIF, inside a four-core Slurm allocation.
The workflow regression resolves the production DAG, executes its partition writer and all 16 chunk readers, and feeds their bounds through the real module-to-publication path.
Restoring zero-object rejection or the unbounded empty range each makes that workflow regression fail.
The matching, durable-record, SExtractor-failure, and empty-module regressions remain covered.
Later commits change only comments and the decision record. The 77 decision-contract tests pass at the head, 9b1d898.

A wholly empty batch still fails the separate external checker's science-selection requirement; this PR does not change that gate policy.

Refs #974.

— OpenAI coding agent on behalf of Cail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley merged commit 6060c6d into develop Oct 30, 2026
3 checks passed
@cailmdaley
cailmdaley deleted the fix/match-guard-warn branch October 30, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant