Skip to content

Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample - #897

Merged
cailmdaley merged 25 commits into
developfrom
feat/unions-catalogue-tile-detection
Sep 29, 2026
Merged

Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample#897
cailmdaley merged 25 commits into
developfrom
feat/unions-catalogue-tile-detection

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

For real data, we now measure shapes for the objects in Stephen Gwyn's UNIONS tile catalogue (DR6), instead of detecting our own on each tile with SExtractor. Measuring shapes on that catalogue lets shape, photometry and redshift catalogues share one object list, so we can match them by ID rather than by position (#832). Image simulations have no DR6 catalogue, so they keep running SExtractor on the tile. PSF stars come from our own exposure-level SExtractor run in both cases.

One object list, one ID

Every final catalogue carries TILE_UNIQUE_ID = tile_id·10⁶ + NUMBER. Here tile_id = RRR·1000 + DDD identifies the tile and NUMBER is the object's number in that tile's catalogue. Together they make an ID that is unique across the survey, so shape, photometry and redshift catalogues can be joined directly. The ID is defined in one place (cfis.get_tile_unique_id), which raises if NUMBER reaches 10⁶ and the ID would collide.

From catalogue to the stamps ngmix needs

The rest of the pipeline expects what a SExtractor run produces: a FITS-LDAC catalogue with a postage stamp (VIGNET) per object, cut from the tile image. We fetch the DR6 catalogue for each tile and convert it into exactly that form, keeping Stephen's NUMBER. From there on, both detection modes run the same chain.

A SExtractor run also marks neighbours. In each object's stamp it writes −1e30 on pixels belonging to other detections, and ngmix gives those pixels zero weight and fills them with noise. A catalogue alone carries no such information. Without it, neighbour light enters the shape fit: on our first 25-tile test run in catalogue mode, 6.1% of objects came out more than a magnitude brighter than their catalogue magnitude, against 2.7% for the same tiles in SExtractor mode. So we also fetch Stephen's segmentation map for the tile. The map has the tile's pixel grid, but its labels are not the catalogue's NUMBERs. We give each object the footprint under its centre and mark every other footprint in its stamp, as SExtractor would. On tile 202.301, all but one of the 36,065 objects find their own footprint, no two share one, and the marking covers every neighbour pixel and none of an object's own. The fraction of stamps dropped as more than a third masked is 0.58%, against 0.55% in SExtractor mode.

Technical notes

  • Settings. tile_detection is unions_catalogue for data and sextractor for image simulations. Both are defaults per input type in workflow/config.yaml, which a run config can override. The DR6 files come from vos:cfis/tiles_DR6.
  • ngmix chunks select catalogue rows rather than NUMBER ranges, so the chunks cover every object exactly once however NUMBER is ordered or spaced.
  • Merged catalogues. DR6 lacks nine SExtractor-only columns (Questions and uncertainties about Stephen Gwyn's official DR6 SExtractor run #924). In catalogue mode, the final-catalogue merge drops those nine and stays strict on all the others. It requires TILE_UNIQUE_ID, so final catalogues made with and without this PR can't be merged together.
  • Campaign boundary. The workflow's params change: tile_detect runs the converter, and tile_get_catalogue is a new rule. Resuming a campaign started before this PR reruns its tile units.
  • Decision record. astra.yaml records the choice as detection.tile_detection and the neighbour marking as detection.catalogue_neighbour_marking.
  • Tested on the 25-tile Nibi pilot run in catalogue mode. It exposed the missing columns and the missing neighbour masking, both fixed here. A rerun of those tiles, compared against the SExtractor-mode run, is the check on the neighbour masking.

Closes #832.

— Claude (Opus) on behalf of Cail

🤖 Generated with Claude Code

cailmdaley and others added 6 commits September 17, 2026 13:11
@cailmdaley @claude
Add the UNIONS tile-catalogue inis to the workflow config chain
598a87d 
Move the two v2.0-era configs (fetch Steven Gwyn's per-tile .cat, convert it
with read_ext_sexcat_runner) into workflow/config/cfis and adapt them to the
workflow's conventions: fixed run names, NUMBER_LIST from SP_UNIT_NUM,
explicit chained INPUT_DIRs, and the catalogue source and retrieve mode
taken from the environment the rules export. The converter writes into
run_sp_tile_Sx so that the downstream tile chain reads one path in either
detection mode. get_images_runner expands RETRIEVE so it can be set per run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Add tile_detection: unions_catalogue to the workflow
8702483 
A run config key, validated at parse time, that picks where the tile's galaxy
sample comes from. `sextractor` is the existing SExtractor rule, byte for byte.
`unions_catalogue` replaces it with two rules: tile_get_catalogue fetches the
UNIONS per-tile .cat from inputs.catalogues (a local mirror or a vos: URL),
and tile_detect converts it with read_ext_sexcat_runner into run_sp_tile_Sx,
linking the module output as sextractor_runner so that tile_vignets, ngmix
and make_cat read the one path they always have; the manifest is
tile_detect.json in both modes. completeness.py checks tile_detect per mode
through SP_TILE_DETECTION, exported on that rule alone so a SExtractor run's
prologue is unchanged; run_report lists the fetch stage only for catalogue
runs. The catalogue's per-tile unique object ID reaches the final catalogue
as TILE_UNIQUE_ID through make_cat's copy of every sexcat column.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Test the UNIONS catalogue conversion and its workflow wiring
379be48 
read_ext_sexcat on a synthetic ASCII catalogue and image: LDAC layout, the
tile header in LDAC_IMHEAD, the SExtractor aliases, zero-padded VIGNET stamps
cut at the right pixels, TILE_UNIQUE_ID = tile_id * 1e6 + NUMBER, and that
column surviving make_cat.save_sextractor_data into the final catalogue. On
the workflow side: the two detection inis share the stage's run dir, the
fetch ini feeds the converter, the tile_detect count table is selected per
mode with sextractor as the unset default, and run_report lists the fetch
stage only for catalogue runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Fix TILE_UNIQUE_ID merge, NUMBER contiguity, and FITS int64 dtype test
affcdc0 
final_cat.param lists TILE_UNIQUE_ID again; merge_final_cat.py and
create_final_cat.py now drop a requested column that is absent from a
given catalogue (logging once) instead of failing, so sextractor-mode
runs (without the column) and unions_catalogue-mode runs (with it)
both merge cleanly.

read_ext_sexcat's NUMBER passthrough is not guaranteed contiguous or
in order in the external catalogue, which ngmix_range requires;
TILE_UNIQUE_ID already preserves the original NUMBER, so the converter
now renumbers NUMBER to 1..n_obj in output row order and says so in
its docstring.

test_read_ext_sexcat's TILE_UNIQUE_ID dtype check compared against
np.int64 directly; a FITS 'K' column reads back big-endian, so the
check now compares dtype.kind and itemsize instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Build the UberSeg segmentation map for the UNIONS catalogue path
930b5af 
UberSeg identifies a stamp's central object BY LABEL: uberseg_weight keeps
the pixels whose nearest segmentation footprint carries the object's own
catalogue NUMBER. The UNIONS per-tile catalogue carries no segmentation map,
and a SExtractor run's labels are its own detection numbers, so the two
numberings have to be bridged before ngmix can mask a neighbour.

`blend_handling:` (noisefill | uberseg) joins `tile_detection:` in the run
config. The pair unions_catalogue + uberseg adds tile_segmentation: one
SExtractor run on the tile image whose only product is the SEGMENTATION check
image (config_tile_Sg.ini), followed by seg_relabel.py, which gives each
catalogue object the footprint its position falls in, relabels it with that
object's NUMBER, marks every unclaimed footprint a neighbour, and paints a
small disc for an object that falls on sky or on a footprint already claimed.
The result lands beside the sexcat, where vignetmaker's segmentation run looks
whichever mode wrote the catalogue. Steven's catalogue stays the sample;
SExtractor runs once, for the one product the catalogue cannot give.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Default tile_detection to the UNIONS catalogue
57b7525 
The per-tile catalogue is the sample; SExtractor detection is the explicit
alternative. `inputs.catalogues` travels with it -- the Snakefile refuses
unions_catalogue without one -- and is committed as TBD for the run to set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley

cailmdaley commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

can this be split into separate (perhaps sequential) PRs on ubserseg and per-tile catalogs?

@martinkilbinger
martinkilbinger self-requested a review September 28, 2026 13:12
@cailmdaley

cailmdaley commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

let's split out uberseg from this by scoping to only noise fill, and tackle uberseg along with #915, #916, etc.

cailmdaley and others added 6 commits September 28, 2026 16:11
@cailmdaley @claude
Build TILE_UNIQUE_ID once, in make_cat, from one helper
36e96b3 
cfis.get_tile_unique_id defines the survey-wide object ID,
tile_id * 10**6 + NUMBER with tile_id = RRR * 1000 + DDD, and raises when
NUMBER (or tile_id) falls outside [0, 10**6). cfis.get_tile_id parses the
tile, split_tile_unique_id inverts the encoding, and get_tile_number now
refuses tile components with more than three digits.

make_cat writes TILE_UNIQUE_ID next to TILE_ID for every final catalogue,
whatever produced the detection catalogue. read_ext_sexcat is a plain
format converter again: it copies NUMBER unchanged and adds no ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Select ngmix chunks by catalogue row, not NUMBER value
4e76ccc 
ID_OBJ_MIN/MAX are now 1-based closed row ranges (ngmix.chunk_rows), so the
partition ngmix_range.py writes covers every object once however NUMBER is
ordered or spaced. ngmix_range emits NGMIX_ROW_MIN/MAX, and checks that the
EPOCH extensions are row-aligned instead of requiring NUMBER = 1..N.
NUMBER stays the object's identity: vignet/PSF store keys, the uberseg
label check and position seeding are unchanged. fake_psf keys its PSF
store by NUMBER rather than row + 1.

For SExtractor catalogues (NUMBER = row) chunk contents are unchanged, but
ngmix_range.py's hash changes, so a resume across this commit re-measures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge develop into feat/unions-catalogue-tile-detection
ba332f4 
Conflicts resolved by intent:
- config.yaml: develop's machines table; blend_handling stays top level,
  tile_detection and inputs.catalogues move into the table for input_type
  data (tile_detection joins run_config.MACHINE_KEYS), so image sims fall
  back to sextractor.
- bin/sp: develop's launcher, plus SP_TILE_DETECTION/SP_BLEND_HANDLING for
  run_report.
- get_images_runner.py, README.md: develop's text, plus the tile_detection
  notes.
The Snakefile treats inputs.catalogues: TBD as unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge refactor/tile-unique-id: TILE_UNIQUE_ID in make_cat, ngmix row …
7c91679 
…chunks

read_ext_sexcat keeps the catalogue's NUMBER and builds no ID (the
refactor side of the one conflict); make_cat builds TILE_UNIQUE_ID for both
detection modes and ngmix chunks select rows, so the renumbering is no
longer needed. test_read_ext_sexcat.py, seg_relabel's docstring, tile.smk
and the merge scripts' comments follow. TILE_UNIQUE_ID joins the cfis and
cfis_image_sims final_cat.param lists; the missing-column filter keeps
older final catalogues mergeable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Segmentation run uses the tile detection's MegaPipe filter
e34b275 
config_tile_Sg.ini takes DOT_CONV_FILE from config_tile_Sx.ini (#896), so
the uberseg footprints are the ones SExtractor mode would draw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge origin/develop into feat/unions-catalogue-tile-detection
bfd2f87 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley mentioned this pull request Sep 28, 2026
Merged
cailmdaley and others added 4 commits September 28, 2026 17:18
@cailmdaley @claude
Move UberSeg segmentation to its own branch
2e09027 
The tile_segmentation rule, seg_relabel.py, config_tile_Sg.ini and the
blend_handling setting move to feat/uberseg-seg-map, which feeds the
consolidated blend/defect PR. This branch keeps the Gwyn catalogue path,
TILE_UNIQUE_ID and row-based ngmix chunking; the catalogue path has no
segmentation map, so uberseg needs the SExtractor mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Default tile_detection and psf_model per input type, not per machine
5511597 
Both follow from the kind of input (the UNIONS catalogue and PSFEx for
data, SExtractor and the true simulation PSF for image sims), so they move
out of the machine entries into a top-level input_types: table. run_config
layers config.yaml < input_types[input_type] < machines[machine][input_type]
< run config, and expands $variables in every resolved key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Record tile_detection in astra.yaml
9d1e023 
Real data now takes its tile sample from Gwyn's UNIONS catalogue. The
detection sub-analysis gains a tile_detection decision (committed:
unions_catalogue), pinned to the input_types table, and notes that its
SExtractor decisions govern only the sextractor path. The converter's
multi-epoch post-processing is tagged under epoch_membership_ccd_bounds,
which applies to both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Point inputs.catalogues at vos:cfis/tiles_DR6 on both machines
21e7c4e 
Stephen Gwyn's per-tile catalogues live at vos:cfis/tiles_DR6 (the path
develop's config_tile_Git_cat_vos.ini already uses). tile_get_catalogue
fetches them with vcp inside the job, using ~/.ssl/cadcproxy.pem; that works
from a candide compute node in the runtime container.

The unset/placeholder check moves from the Snakefile into
run_config.catalogue_source, so the parse-time refusal is unit-tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley changed the title Use Steven Gwyn's UNIONS tile catalogue as the tile galaxy sample (default); segmentation map for UberSeg Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample Sep 28, 2026
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley @claude
Merge origin/feat/uberseg-seg-map into feat/masked-pixels
23e823e 
Brings the tile segmentation map for UberSeg into #922, so the masked-pixels
work sits in one place. feat/uberseg-seg-map is built on #897
(feat/unions-catalogue-tile-detection), so this provisionally carries #897's
content too (UNIONS catalogue detection, TILE_UNIQUE_ID, ngmix row chunks)
until #897 lands on develop.

No textual conflicts: ngmix.py's row-selection chunking (#897) and the epoch
cuts, defect fill and constants (#915/#916) touch disjoint hunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley
Merge origin/feat/unions-catalogue-tile-detection into feat/masked-pi…
6bf7aa7 
…xels

Keeps #897's content here current (inputs.catalogues set to vos:cfis/tiles_DR6,
parse-time check in run_config.catalogue_source).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	workflow/config.yaml
@cailmdaley

cailmdaley commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

@martinkilbinger this is converging and ready for review!

@cailmdaley
cailmdaley marked this pull request as ready for review September 28, 2026 16:48
cailmdaley and others added 2 commits September 28, 2026 20:38
@cailmdaley @claude
Merge origin/develop into feat/unions-catalogue-tile-detection
0a0396a 
Brings in #879 (campaign merges), #911 (spread model removed), #854,
#905 and #923.

- run_config: input_types:/machines: table resolver kept; develop's
  required `run:` and unresolved-$ check folded in (now over every
  non-table key).
- create_final_cat.read_data keeps develop's raise on a missing column;
  the branch's skip-missing path is dropped (final_cat_merge is the one
  producer of final_cat_<run>.hdf5).
- final_cat.param carries both NUMBER and TILE_UNIQUE_ID.
- DAG tests follow the committed tile_detection default (data plans
  tile_get_catalogue); params pin regenerated for the tile_detect switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DksuyF9YrZAwHQeAMXsxp4
@cailmdaley @claude
Leave the retired standalone merge_final_cat.py to develop
9f034f8 
The workflow merges final catalogues through final_cat_merge; the
standalone script is retired, so this branch no longer patches it or
tests filter_available_columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DksuyF9YrZAwHQeAMXsxp4
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley
Merge #897 (feat/unions-catalogue-tile-detection) into integration/v2…
e57a1ce 
…-bigrun
martinkilbinger
martinkilbinger reviewed Sep 29, 2026
View reviewed changes
Comment thread workflow/config.yaml Outdated
martinkilbinger
martinkilbinger reviewed Sep 29, 2026
View reviewed changes

@martinkilbinger martinkilbinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good to me!

Just a small change to de-claudify the config file comment string.

Comment thread workflow/config.yaml Outdated
@cailmdaley @claude
final_cat_merge: request only the columns the tile's detection mode c…
2905a88 
…an carry

Under tile_detection: unions_catalogue the detection columns are the UNIONS
DR6 catalogue's, which lacks nine SExtractor columns final_cat.param names
(MAG_WIN, MAGERR_WIN, FLUX_AUTO, FLUXERR_AUTO, FLUX_APER, FLUXERR_APER,
SNR_WIN, FWHM_IMAGE, FWHM_WORLD), so every catalogue-mode merge raised.
merge_final_cat.py now takes --tile-detection and subtracts exactly
SEXTRACTOR_ONLY_COLUMNS in that mode; the merge stays strict on the rest.
tests/module/test_final_cat_columns.py runs the converter on the DR6 header
and asserts that list is exactly the requested SExtractor columns DR6 lacks.
The params pin moves for final_cat_merge only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ
cailmdaley added a commit that referenced this pull request Sep 29, 2026
@cailmdaley @claude
tests: key the retired-key ordering check on either defaults call
2e12ccd 
#897 renames run_config.apply_machine_defaults to apply_defaults; the
ordering assertion now holds on both sides of that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley mentioned this pull request Sep 29, 2026
Open
cailmdaley and others added 6 commits September 29, 2026 16:16
@cailmdaley @claude
Mask neighbours in catalogue-mode VIGNETs from the DR6 segmentation map
8da837b 
tile_detection: unions_catalogue cut VIGNET straight from the tile image, so
ngmix (which masks where the tile VIGNET is -1e30) masked no neighbours: in
smk-g10 NGMIX_MAG_NOSHEAR ran >1 mag bright of MAG_AUTO for 6.1% of objects,
against 2.7% with SExtractor.

tile_get_catalogue now also fetches CFIS.<tile>.r.seg.fits.fz. The converter
relabels it to the catalogue's NUMBER by the footprint under each object's
centre pixel (relabel_seg: unclaimed footprints -1, a small disc for objects
without a footprint), writes it beside the sexcat as seg-<tile>.fits for
UberSeg, and sets VIGNET pixels on other objects' footprints to -1e30.
Off-image VIGNET pixels are -1e30 too, as SExtractor writes them.

Records detection.catalogue_neighbour_marking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ
@cailmdaley @claude
completeness: expect the seg-stamp vignetmaker run under blend_handli…
a4da900 
…ng: uberseg

tile_vignets gains a third vignetmaker run, the segmentation stamps ngmix's
SEG_VIGNET_PATH reads, when SP_BLEND_HANDLING=uberseg. Noise-fill runs
export nothing and expect exactly what they did before. Landing this here
keeps SCRIPT_HASH(completeness.py) at one campaign boundary with #897.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Revert "completeness: expect the seg-stamp vignetmaker run under blen…
97c11a7 
…d_handling: uberseg"

UberSeg's seg stamps will ride the tile catalogue as a SEG_VIGNET column, so
there is no extra vignetmaker run to expect. completeness.py is again
byte-identical to 8da837b, keeping SCRIPT_HASH at the value smk-g11 runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
config.yaml: plainer input_types comment (review suggestion)
a8facf5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Drop the relabelled seg-map file output
bced5a3 
Its only consumer was an UberSeg seg run that no longer exists; the relabelled map is still used in memory for VIGNET neighbour marking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
completeness: the catalogue converter writes one file
0ce4060 
The relabelled seg map is no longer written, so expect=2 failed every catalogue-mode tile_detect; a runner-driven test now ties the count to the converter's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley merged commit f477c41 into develop Sep 29, 2026
3 checks passed
@cailmdaley
cailmdaley deleted the feat/unions-catalogue-tile-detection branch September 29, 2026 23:56
This was referenced Sep 30, 2026
Open
Merged
Merged
cailmdaley added a commit that referenced this pull request Oct 1, 2026
@cailmdaley @claude
Record: the decision record matches the pipeline after #897 (#930)
ec492ff 
* Clarify detection decisions after catalogue switch

* Correct masking decision scope

* Clarify astrometry and epoch provenance record

* Record the MCCD workflow gate

* Scope PSF likelihood noise and sentinel tags

* Update catalogue and completeness decision record

* Record that the tile SExtractor config reproduces DR6 detection (#929)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley added a commit that referenced this pull request Oct 2, 2026
@cailmdaley @claude
Detect tiles with SExtractor on data too; take NUMBER from the UNIONS…
e3ef252 
… catalogue (#933)

* Measure windowed centroids for UNIONS-catalogue tiles

Since #897 the data path takes its tile sample from the UNIONS DR6
catalogue, which has only isophotal barycentres, and read_ext_sexcat
copied them into XWIN_*: every tile-side position (stamp centres and
OFFSET, the ngmix centroid prior, tile PSF interpolation, epoch
membership, the catalogue RA/Dec) was the barycentre, while image
simulations get SExtractor's windowed centroid.

read_ext_sexcat now measures XWIN_IMAGE/YWIN_IMAGE at every catalogue
object: a vectorised numpy port of SExtractor 2.25's compute_winpos
(Gaussian window of sigma 2 FLUX_RADIUS / 2.35, 4-sigma aperture with
11x11 edge subsampling, step 2x the weighted mean offset, at most 16
steps, stop below 1e-4 px), on the tile less a SExtractor-like
BACK_SIZE 512 / BACK_FILTERSIZE 9 background, with neighbours'
segmentation footprints replaced by their mirror image as MASK_TYPE
CORRECT does. XWIN_WORLD/YWIN_WORLD come from the tile WCS, 1-based.
SExtractor's fallbacks to the barycentre (FLAGS_WIN 1, 2, 4, 8) are
reproduced; 16 marks an unmeasurable object and 32 a centroid still
moving after 16 steps. FLAGS_WIN joins final_cat.param, and epoch
membership reads XWIN_WORLD as in the SExtractor path.

On tile 186.307 it reproduces SExtractor's XWIN_IMAGE exactly when
SExtractor does not mask neighbours (0 of 41,425 objects off by more
than 0.01 px in windows free of zero-weight pixels), and for 97% of
objects on the DR6 catalogue path; the barycentre it replaces is off by
0.16 px at the median and 1.4 px at the 99th percentile. astra.yaml's
object_position_columns and background_model say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki

* Windowed centroid: mirror no-data pixels, guard NaN and wild FLUX_RADIUS

Review of #933 and the 202.301 run on Nibi:

- Pixels that are 0 or not finite hold no data in a tile. They were left
  out of the background but read as signal in the window. They now take
  their mirror image through the centre, as SExtractor's MASK_TYPE
  CORRECT treats pixels under the weight threshold, and a mirror that is
  itself invalid counts as 0.
- Non-finite pixels never enter the background or the sums: samples
  outside the aperture are zeroed before the product, and a non-finite
  sum gives FLAGS_WIN 16 at the barycentre.
- The second moments are normalised as SExtractor's are after its loop,
  so a negative source is FLAGS_WIN 4, not 6.
- FLUX_RADIUS <= 0 or NaN (DR6 carries values down to -1.2e6) is an
  empty window, FLAGS_WIN 4; windows wider than MAX_HALF = 256 px are
  FLAGS_WIN 16; and the working box never exceeds the image. The -12402
  radius on 202.301 had built an 84k-px box and been killed at 16 GB.

On 186.307 the agreement with SExtractor is unchanged (2.8% of objects
off by more than 0.01 px on the DR5 tile, 3.4% on the DR6 path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki

* Revert the windowed-centroid port

The port re-implemented SExtractor's windowed centroid in numpy for
UNIONS-catalogue tiles. Running SExtractor itself on the DR6 image and
joining its rows to the DR6 catalogue replaces it (next commits).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config values expand ${VAR:-default}

A value with ${VAR:-default} expands to default when VAR is unset or
empty, as in the shell; $VAR and ${VAR} stay strict. The same change as
feat/uberseg-seg-vignet's, so the two merge cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sextractor_runner: join detections to an external catalogue

MATCH_CATALOGUE (an ASCII SExtractor catalogue of the same image) pairs
each SExtractor row with its mutual nearest neighbour within MATCH_RADIUS
pixels; paired rows take its NUMBER, unpaired rows leave the catalogue, and
a SEG_VIGNET column, when present, is relabelled to the new numbering. Fewer
than MATCH_MIN_FRACTION of the rows pairing stops the run: that is what a
catalogue measured on other pixels (a DR5 image against DR6) looks like.
The join runs before the post-processing keys epochs on NUMBER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Detect tiles with SExtractor on data too; drop the catalogue converter

Data and image simulations both run tile_detect as SExtractor
(config_tile_Sx.ini), so XWIN, FLAGS_WIN, the VIGNET neighbour markers and
every SExtractor column are measured one way. Under tile_detection:
unions_catalogue, tile_get_catalogue fetches only the UNIONS .cat and
tile_detect joins its detections to it (SP_MATCH_CATALOGUE), keeping the
shared NUMBER and TILE_UNIQUE_ID. On DR6 186.307 all 41,201 detections pair
(median separation 4e-5 px); 45 catalogue objects, faint deblended children,
have no detection.

Gone: read_ext_sexcat and its tests, config_tile_Uc.ini, the runner link,
the segmentation-map fetch, merge_final_cat's SEXTRACTOR_ONLY_COLUMNS and
--tile-detection, and the detection.catalogue_neighbour_marking decision.
completeness.py changes, so this lands at a campaign boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sextractor_runner: single-image SExtractor without a detection image

Without DETECTION_IMAGE the runner called SExtractor in dual-image mode
with the same image and weight twice, which changes the background,
MAG_AUTO and FLUX_RADIUS of a fifth of a tile's objects against the
single-image call and the UNIONS catalogue. Dual-image mode stays for a
real detection image.

Closes #936

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* match_catalogue: two-sided pairing guard; record the eight-tile validation

The run also stops when fewer than MATCH_MIN_FRACTION of the catalogue
objects pair, which catches a detection configuration that finds fewer
objects than the catalogue. Tests cover the radius bound and both sides
of the guard. astra.yaml records the eight-tile numbers, single-image
detection, and the MAG_AUTO differences beyond a pixel-stack overflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config expansion rejects malformed ${ forms

${VAR-x}, ${VAR:=x} and ${ VAR } passed through as literal paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tile_detect: 1 core, 4 GB, 20 min from eight measured tiles

Peak RSS 1.75 GiB and wall time 143 s at most; SExtractor is built
without threads. One retry doubles memory and runtime. Image
simulations export SP_MATCH_CATALOGUE empty. Stale comments on the
catalogue fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update the params pin for this campaign boundary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tile SExtractor: MEMORY_PIXSTACK 3000000, so no object is truncated

300000 overflowed on four of eight DR6 tiles (a cD galaxy, a bright
star's halo, 13 times on Alioth's tile); 3000000 clears all eight.
Peak RSS rises to 1.84 GiB and wall time to 160 s on the worst tile;
tile_detect's 4000 MB stays at 2.1x. The exposure config never
overflows on the 40 CCDs checked and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* match_catalogue: guard at 2% beyond 20 unpaired per side

The guard catches other pixels (DR5 pairs ~95%), not deblending near
large objects (DR6 tiles pair at least 99.2%) or a few children on a
near-empty edge tile. New key MATCH_TOLERATED_UNPAIRED = 20;
MATCH_MIN_FRACTION = 0.98.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley mentioned this pull request Oct 6, 2026
Open
cailmdaley added a commit that referenced this pull request Oct 25, 2026
@cailmdaley @claude
UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile…
7440b01 
… SExtractor run) (#925)

* test(ngmix): specify defect fill, epoch cuts and shear recovery near defects

Defects (nonzero flag, zero weight, invalid RMS) must be zero-weighted and
noise-filled under every BLEND_HANDLING, with neighbour pixels left raw
under uberseg. The masked-fraction cut must count that same raw set, and an
epoch with a defect closer than EPOCH_CENTRAL_DEFECT_RADIUS to the stamp
centre must be dropped, at a strict boundary and at the configured radius.
The runner must pass both cut options on, the HSM centroid must ignore raw
defect values, each epoch must keep its own OFFSET, and every tile must log
its epoch-cut tally.

The science test recovers the full 2x2 response for columns, 3-px bleeds,
single pixels and edge bands at the veto radius, on 0.3" and 0.5" galaxies
through round and elliptical 0.7" PSFs, and requires |c1|, |c2| < 5e-4 and
|m11|, |m22| < 1%. Flagged pixels in the simulation hold a hot value, so an
unfilled defect shows up. A column two pixels inside the radius is the
positive control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

* feat(ngmix): noise-fill defects under every BLEND_HANDLING, veto central ones

Metacal shears the whole stamp without reading the weights, so a defect
left raw under uberseg reached the fit. Every defect (nonzero flag, zero
weight, invalid RMS) is now zero-weighted and noise-filled whatever
BLEND_HANDLING is; uberseg only zeroes the weight of neighbour-side pixels
and keeps their light. The option that meant "no neighbour treatment" is
renamed from noisefill to none, and the retired name is rejected.

The defect set is not symmetrized: at the veto radius the one-sided fill
leaves |c| <= 3e-4, while a four-fold mask would quadruple m (-2.7% against
-0.64% for a 3-px bleed at 10 px on a 0.5" galaxy) and, through an elliptical
PSF, still leave c1 = 7.3e-4.

An epoch is dropped when a defect lies closer than
EPOCH_CENTRAL_DEFECT_RADIUS (default 10 px) to the stamp centre, the
smallest radius at which columns, 3-px bleeds, single pixels and edge bands
give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3" and 0.5" galaxies
through a 0.7" PSF. The masked-fraction cut counts the same raw defect set;
EPOCH_MASKED_FRACTION_CUT (default 1/3) makes it configurable. Each tile
logs "epoch cuts: considered=N masked_fraction=A central_veto=B
objects_emptied=C". The HSM centroid is measured on the filled image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

* ngmix: read each epoch's OFFSET from the object dict in hand

prepare_postage_stamps already holds the object's galaxy vignet dict, so the
per-epoch OFFSET is read from it instead of a second sqlitedict lookup. The
pixel scale sets only the centroid-prior width; the comments that also
credited it with a noise window are corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

* tests: name the none-mode uberseg test for none, not noisefill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ngmix): specify DEFECT_FILL = interpolate and its shear recovery

The interpolation fills only short defect runs (at most 3 px along a row or
column, with clean pixels at both ends); anything wider, and runs reaching
the stamp border, stays noise-filled. The interpolant must reproduce linear
planes without reading defect values, commute with quarter turns of the
stamp, and apply one operator to the science and metacal noise images. The
weight is zero on the defects and on the quarter-turn orbit of the
interpolated pixels, whose light stays. Each defect is vetoed at the radius
of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS for interpolated pixels,
EPOCH_CENTRAL_DEFECT_RADIUS for noise-filled ones. The runner and Ngmix pass
DEFECT_FILL to the epoch cuts and to metacal; noise stays the default.

The science test recovers the full response matrix for interpolated columns,
full and finite 3-px bleeds and single pixels at the interpolated radius, on
0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs and on a 0.7"
galaxy through a 0.9" PSF, and for noise-filled wide defects at the noise
radius. A bleed three pixels inside the radius is the positive control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

* feat(ngmix): DEFECT_FILL = interpolate, with a fill-dependent central veto

DEFECT_FILL = interpolate fills short defect runs (at most 3 px along a row
or column, with clean pixels at both ends: columns, 3-px bleeds, single
pixels) with a Clough-Tocher interpolant of the clean pixels within 4 px.
The interpolant is averaged over the four quarter turns of the stamp, and
each turn's triangulation serves the science image and the metacal noise
image alike. Wider holes and runs reaching the stamp border keep the noise
fill. noise stays the default and is unchanged.

Interpolation restores the object's light, so the remaining bias is the
hole in the likelihood: a one-sided zero-weight column 8 px from a 0.5"
galaxy still gives c1 = -1.3e-3. The weight is therefore also zero on the
quarter-turn orbit of the interpolated pixels (their light stays), which
brings that column to c1 = +2e-6. Symmetrizing the fill instead would trade
true light for interpolated light over four times the area.

Each defect pixel is vetoed at the radius of its own fill:
EPOCH_INTERPOLATED_DEFECT_RADIUS (default 7 px) for interpolated pixels and
EPOCH_CENTRAL_DEFECT_RADIUS (10 px) for noise-filled ones. 7 px is the
smallest radius at which interpolated columns, 3-px bleeds and single pixels
give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3", 0.5" and 0.7" galaxies,
round and elliptical PSFs; it does not scale with galaxy size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

* ngmix: restore the BLEND_HANDLING name noisefill

The neighbour option keeps develop's name, noisefill, in the code, the runner
config key, tests, astra.yaml and the committed universe; the name none is
dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ngmix): specify neighbour markers apart from defects

SExtractor's -1e30 markers in the tile VIGNET form their own per-epoch mask
(stamp.neighbours): a footprint 3 px from the centre, or covering 45% of the
stamp, drops no epoch; noisefill zero-weights and noise-fills exactly the
marked pixels, bit for bit as develop does; uberseg leaves them raw and
weighted; a flagged column near the centre is still vetoed; and
do_ngmix_metacal hands each epoch's mask to make_ngmix_observation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ngmix): keep SExtractor's neighbour markers out of the defect set

prepare_postage_stamps wrote the tile VIGNET's -1e30 markers (other
detections' footprints) into each epoch's flag stamp as 2**10, so
defect_mask counted them: they fed the masked-fraction cut and the central
veto. Every epoch shares the tile VIGNET, so a neighbour within
EPOCH_CENTRAL_DEFECT_RADIUS dropped every epoch of the object.

The markers are now their own per-epoch mask, stamp.neighbours
(MegaCam-flipped with the epoch), threaded through do_ngmix_metacal and
make_ngmix_observation to prepare_ngmix_weights. The epoch cuts count
genuine defects only. Under noisefill the marked pixels get weight 0 and
the defects' noise realisation, the output noisefill gave when they arrived
as flags; under uberseg they are ignored and the seg-based weight handles
neighbours. Defects are treated identically under both. Docstrings, @sc
contracts, runner comments and the astra.yaml decisions say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ngmix): off-tile pixels are defects, not neighbours

SExtractor writes -1e30 into the tile VIGNET beyond the tile's edge as well
as on other detections' footprints. Beyond the edge the epoch holds the
object's own light, cut off, so treating those pixels as neighbours exempted
edge objects from both epoch cuts and measured them with a noise-filled band
near their centre.

split_tile_markers classifies the markers geometrically on the flipped tile
VIGNET: the union of stamp rows and columns that are entirely -1e30 (the
off-image part of a rectangle clip) is off-tile, the rest is the neighbour
mask. Off-tile pixels are flagged OFF_TILE_FLAG (2**10) in the epoch's flag
stamp, so under both blend handlings they are zero-weighted, filled, and
counted by the masked-fraction cut and the central veto. Docstrings, @sc
contracts, runner comments and astra.yaml say so.

Tests: an object 3 px from the tile edge loses every epoch; a band 12 px
away passes the default cuts and is vetoed at radius 13; a corner's L-shaped
off-tile region is flagged exactly and a border-touching neighbour stays a
neighbour; off-tile pixels are filled under both blend handlings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ngmix): interpolate defects from the pixels the image keeps

Under DEFECT_FILL = interpolate and BLEND_HANDLING = noisefill, the
Clough-Tocher support included marked neighbour pixels that noisefill
replaces with noise, so a defect beside a bright neighbour was filled with
light the image no longer contains. The support now excludes ~clean (the
defects and the removed neighbour pixels); the NaN fallback to the noise
fill covers degenerate supports. Under uberseg the neighbour light stays in
the image and the support is unchanged. interpolate_defects names its
second argument `excluded`.

Test: a flagged pixel next to a 1e4 marked neighbour interpolates to the
kept-pixel interpolant (1.4, against 3102 with the neighbour in the
support) under noisefill, and from the raw neighbour light under uberseg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ngmix): the defect fill keeps the stamp's dtype

fill_defects used np.where, which promotes a float32 stamp to float64;
develop's noisefill kept the galaxy stamp's dtype. The filled image is cast
back to the input dtype, so noisefill stays bit-identical to develop on
float32 stamps. The develop-equivalence test gains a float32 case and
checks dtypes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ngmix): off-tile pixels are the marked rows and columns at the stamp border

Every entirely -1e30 stamp row or column counted as off-tile. In an edge
stamp, a neighbour footprint that covers the on-image part of a row (a
wide DR6 footprint running from the tile edge) completes that row, so the
neighbour's pixels became OFF_TILE_FLAG defects: counted by the epoch
cuts, interpolated under DEFECT_FILL = interpolate, filled under uberseg.
Only the runs of marked rows and columns that start at a stamp border are
off-tile now.

Tests build tile VIGNETs with the catalogue-mode converter: on the real
202.301 DR6 patch every stamp splits exactly into off-image pixels and
seg-map neighbour pixels, and the edge case above keeps its footprint in
the neighbour mask through prepare_postage_stamps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

* docs: -1e30 tile-VIGNET markers come from SExtractor or the catalogue converter

The neighbour-marking rationale and the converter's BIG comment describe
the markers as ngmix neighbours under flag 2**10; ngmix splits them into
off-tile defects and a neighbour mask. The masking text names the tile
VIGNET, not SExtractor, as the markers' carrier, since catalogue mode
paints them from the DR6 segmentation map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

* Expand ${VAR:-default} in config values

A committed ini can then name a workflow knob with its default, and the
workflow exports the variable only when the knob leaves its default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Write SEG_VIGNET from the catalogue-mode converter

Under SEG_VIGNET = True the converter adds one int32 stamp per object of
the relabelled segmentation map, cut with the window and centres VIGNET
is cut with, so the two register pixel for pixel. The stamp cutter moves
to sextractor_script.cut_stamps for the SExtractor path to share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Write SEG_VIGNET from the tile SExtractor run

Under SEG_VIGNET = True, sextractor_runner cuts the SEGMENTATION check
image on each object's VIGNET grid and adds it to the sexcat. VIGNET is
centred on the pixel nearest SExtractor's double barycentre; the float32
X_IMAGE/Y_IMAGE round to it except on exact half pixels, which are
resolved by matching VIGNET against image minus BACKGROUND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ngmix: read UberSeg's seg stamps from the tile catalogue's SEG_VIGNET

Tile_cat takes the stamps from the catalogue it already reads, and
BLEND_HANDLING = uberseg fails before the object loop when the column is
missing. SEG_VIGNET_PATH, its separate catalogue and its checks go. The
stamp columns stay views into the loaded table, so it is held once, and
BLEND_HANDLING is environment-expanded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Carry blend_handling to tile_detect and ngmix as two prologue exports

blend_handling: uberseg (config.yaml, default noisefill) exports
SP_SEG_VIGNET=True to tile_detect and SP_BLEND_HANDLING=uberseg to
tile_ngmix, and adds the SEG_VIGNET column's memory to each ngmix chunk.
The committed inis read them as ${SP_SEG_VIGNET:-False} and
${SP_BLEND_HANDLING:-noisefill}, so noisefill plans the campaign it
planned without the knob and the params pin does not move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* make_cat: leave SEG_VIGNET out of final_cat, with VIGNET

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* astra.yaml: UberSeg reads the tile catalogue's SEG_VIGNET

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Drop inherited blend variables before any rule

A noisefill campaign launched from a shell that exports SP_SEG_VIGNET or
SP_BLEND_HANDLING (bare, or in the APPTAINERENV_/SINGULARITYENV_ forms
that survive --cleanenv) would hand them to every job through the slurm
executor's --export=ALL. The Snakefile now pops them at parse time, so
only blend_env's exports under uberseg set them. Prologues and shells are
unchanged; the DAG harness gains a launch_env to test it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Centre SExtractor-mode SEG_VIGNET on X/Y_IMAGE_DBL

Under SEG_VIGNET, sextractor_runner runs SExtractor with a copy of the
param file that adds X_IMAGE_DBL and Y_IMAGE_DBL; add_seg_vignet centres
each stamp at rint(DBL - 1), as SExtractor centres VIGNET, and drops the
two columns, so the catalogue is noisefill's plus SEG_VIGNET. The
half-pixel tie matching against image minus background goes, and with it
the BACKGROUND check image dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Point the stale SEG_VIGNET_PATH message and astra note at SEG_VIGNET

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Make UberSeg the default blend handling

blend_handling defaults to uberseg in config.yaml and the Snakefile, so a
campaign without the key writes SEG_VIGNET in both detection modes and runs
ngmix with BLEND_HANDLING = uberseg. The committed inis keep the modules'
own noise-fill defaults and the workflow exports the uberseg values, so
blend_handling: noisefill plans exactly the previous noise-fill campaign:
params_pin_noisefill.json is the previous pin, and a noise-fill run dir
resumes with the key set explicitly. The default pin moves in tile_detect
and tile_ngmix only.

astra's committed blend_handling becomes uberseg, with noisefill the
comparison arm. test_committed_blend_handling_fills_defects checks the
committed blend_handling x defect_fill pair against prepare_ngmix_weights:
the workflow default is the committed blend handling, and under it every
defect is zero-weighted and filled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

* Measure windowed centroids for UNIONS-catalogue tiles

Since #897 the data path takes its tile sample from the UNIONS DR6
catalogue, which has only isophotal barycentres, and read_ext_sexcat
copied them into XWIN_*: every tile-side position (stamp centres and
OFFSET, the ngmix centroid prior, tile PSF interpolation, epoch
membership, the catalogue RA/Dec) was the barycentre, while image
simulations get SExtractor's windowed centroid.

read_ext_sexcat now measures XWIN_IMAGE/YWIN_IMAGE at every catalogue
object: a vectorised numpy port of SExtractor 2.25's compute_winpos
(Gaussian window of sigma 2 FLUX_RADIUS / 2.35, 4-sigma aperture with
11x11 edge subsampling, step 2x the weighted mean offset, at most 16
steps, stop below 1e-4 px), on the tile less a SExtractor-like
BACK_SIZE 512 / BACK_FILTERSIZE 9 background, with neighbours'
segmentation footprints replaced by their mirror image as MASK_TYPE
CORRECT does. XWIN_WORLD/YWIN_WORLD come from the tile WCS, 1-based.
SExtractor's fallbacks to the barycentre (FLAGS_WIN 1, 2, 4, 8) are
reproduced; 16 marks an unmeasurable object and 32 a centroid still
moving after 16 steps. FLAGS_WIN joins final_cat.param, and epoch
membership reads XWIN_WORLD as in the SExtractor path.

On tile 186.307 it reproduces SExtractor's XWIN_IMAGE exactly when
SExtractor does not mask neighbours (0 of 41,425 objects off by more
than 0.01 px in windows free of zero-weight pixels), and for 97% of
objects on the DR6 catalogue path; the barycentre it replaces is off by
0.16 px at the median and 1.4 px at the 99th percentile. astra.yaml's
object_position_columns and background_model say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki

* Windowed centroid: mirror no-data pixels, guard NaN and wild FLUX_RADIUS

Review of #933 and the 202.301 run on Nibi:

- Pixels that are 0 or not finite hold no data in a tile. They were left
  out of the background but read as signal in the window. They now take
  their mirror image through the centre, as SExtractor's MASK_TYPE
  CORRECT treats pixels under the weight threshold, and a mirror that is
  itself invalid counts as 0.
- Non-finite pixels never enter the background or the sums: samples
  outside the aperture are zeroed before the product, and a non-finite
  sum gives FLAGS_WIN 16 at the barycentre.
- The second moments are normalised as SExtractor's are after its loop,
  so a negative source is FLAGS_WIN 4, not 6.
- FLUX_RADIUS <= 0 or NaN (DR6 carries values down to -1.2e6) is an
  empty window, FLAGS_WIN 4; windows wider than MAX_HALF = 256 px are
  FLAGS_WIN 16; and the working box never exceeds the image. The -12402
  radius on 202.301 had built an 84k-px box and been killed at 16 GB.

On 186.307 the agreement with SExtractor is unchanged (2.8% of objects
off by more than 0.01 px on the DR5 tile, 3.4% on the DR6 path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki

* Revert the windowed-centroid port

The port re-implemented SExtractor's windowed centroid in numpy for
UNIONS-catalogue tiles. Running SExtractor itself on the DR6 image and
joining its rows to the DR6 catalogue replaces it (next commits).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config values expand ${VAR:-default}

A value with ${VAR:-default} expands to default when VAR is unset or
empty, as in the shell; $VAR and ${VAR} stay strict. The same change as
feat/uberseg-seg-vignet's, so the two merge cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sextractor_runner: join detections to an external catalogue

MATCH_CATALOGUE (an ASCII SExtractor catalogue of the same image) pairs
each SExtractor row with its mutual nearest neighbour within MATCH_RADIUS
pixels; paired rows take its NUMBER, unpaired rows leave the catalogue, and
a SEG_VIGNET column, when present, is relabelled to the new numbering. Fewer
than MATCH_MIN_FRACTION of the rows pairing stops the run: that is what a
catalogue measured on other pixels (a DR5 image against DR6) looks like.
The join runs before the post-processing keys epochs on NUMBER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Detect tiles with SExtractor on data too; drop the catalogue converter

Data and image simulations both run tile_detect as SExtractor
(config_tile_Sx.ini), so XWIN, FLAGS_WIN, the VIGNET neighbour markers and
every SExtractor column are measured one way. Under tile_detection:
unions_catalogue, tile_get_catalogue fetches only the UNIONS .cat and
tile_detect joins its detections to it (SP_MATCH_CATALOGUE), keeping the
shared NUMBER and TILE_UNIQUE_ID. On DR6 186.307 all 41,201 detections pair
(median separation 4e-5 px); 45 catalogue objects, faint deblended children,
have no detection.

Gone: read_ext_sexcat and its tests, config_tile_Uc.ini, the runner link,
the segmentation-map fetch, merge_final_cat's SEXTRACTOR_ONLY_COLUMNS and
--tile-detection, and the detection.catalogue_neighbour_marking decision.
completeness.py changes, so this lands at a campaign boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sextractor_runner: single-image SExtractor without a detection image

Without DETECTION_IMAGE the runner called SExtractor in dual-image mode
with the same image and weight twice, which changes the background,
MAG_AUTO and FLUX_RADIUS of a fifth of a tile's objects against the
single-image call and the UNIONS catalogue. Dual-image mode stays for a
real detection image.

Closes #936

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* match_catalogue: two-sided pairing guard; record the eight-tile validation

The run also stops when fewer than MATCH_MIN_FRACTION of the catalogue
objects pair, which catches a detection configuration that finds fewer
objects than the catalogue. Tests cover the radius bound and both sides
of the guard. astra.yaml records the eight-tile numbers, single-image
detection, and the MAG_AUTO differences beyond a pixel-stack overflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Config expansion rejects malformed ${ forms

${VAR-x}, ${VAR:=x} and ${ VAR } passed through as literal paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tile_detect: 1 core, 4 GB, 20 min from eight measured tiles

Peak RSS 1.75 GiB and wall time 143 s at most; SExtractor is built
without threads. One retry doubles memory and runtime. Image
simulations export SP_MATCH_CATALOGUE empty. Stale comments on the
catalogue fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update the params pin for this campaign boundary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tile SExtractor: MEMORY_PIXSTACK 3000000, so no object is truncated

300000 overflowed on four of eight DR6 tiles (a cD galaxy, a bright
star's halo, 13 times on Alioth's tile); 3000000 clears all eight.
Peak RSS rises to 1.84 GiB and wall time to 160 s on the worst tile;
tile_detect's 4000 MB stays at 2.1x. The exposure config never
overflows on the 40 CCDs checked and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* match_catalogue: guard at 2% beyond 20 unpaired per side

The guard catches other pixels (DR5 pairs ~95%), not deblending near
large objects (DR6 tiles pair at least 99.2%) or a few children on a
near-empty edge tile. New key MATCH_TOLERATED_UNPAIRED = 20;
MATCH_MIN_FRACTION = 0.98.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* SEG_VIGNET keeps own-vs-other through the DR6 join

match_catalogue relabels every SEG_VIGNET label through the whole
SExtractor -> DR6 map, unpaired footprints to UNMATCHED_LABEL (-1). That
separates each row's own footprint from every other one only if the
catalogue's NUMBERs are unique and positive, so the join now refuses a
catalogue where they are not.

Tests, through sextractor_runner with SExtractor's outputs stubbed: on a
crowd where each paired row's DR6 NUMBER is another object's SExtractor
NUMBER (one of them the dropped detection's), each own footprint carries
the final NUMBER and nothing else does, the dropped footprint is -1, and
uberseg_weight and seg_has_neighbour see the stamps as they saw the
SExtractor-labelled ones. Sims (no join) keep the check image's stamps. A
negative control shows a partial relabel collides. make_cat's exclusion of
SEG_VIGNET moves here from the deleted converter tests. Swapping the join
before add_seg_vignet, relabelling only paired labels, or dropping the
NUMBER guard each fails a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* tile_detect: 3000 MB more under uberseg for SEG_VIGNET

Measured on DR6 tile 202.301 with sextractor_runner as tile_detect runs it
(post-processing off): peak RSS 1.53 GiB under noisefill, 2.93 GiB under
uberseg. add_seg_vignet reads the 400 MB check image and adds an int32
column the size of VIGNET, which the join holds twice. Scaled to #933's
1.84 GiB worst tile, uberseg needs ~3.5 GiB, above the 4000 MB noisefill
request; 7000 MB is 2x that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* Update the params pins for UberSeg on #933's detection

Regenerated with --update-params-pin in shapepipe-develop-240b37e4.sif
with host snakemake 9.23.1. The noisefill pin is #933's params_pin.json
byte for byte, so blend_handling: noisefill plans exactly #933's chain;
the uberseg default moves in tile_detect and tile_ngmix only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* tile.smk: keep the catalogue path on one line, as the tile-detection test reads it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* SEG_VIGNET centres as SExtractor centres VIGNET: (int)(mx + 0.49999)

SExtractor 2.25.0 (src/analyse.c, ix=(int)(obj->mx+0.49999); the SIF's
Debian 2.25.0+ds-3 does not patch it) truncates the 0-based barycentre
plus 0.49999. np.rint differs at exact halves with an odd floor and for
fractional parts in (0.5, 0.50001); vignet_centre now mirrors SExtractor.

The tests' expected centres are written out by hand from that rule and
their stamps cut by plain slicing, not by cut_stamps; they cover exact
halves and fractions either side of 0.50001, and an object only the double
positions resolve. Reverting to np.rint fails five of them. On DR6 tile
202.301 no object falls where the two rules disagree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* match_catalogue: refuse NUMBERs the int32 columns cannot hold

The guard took unique positive NUMBERs but not their type or range: a
non-integer, or a value past 2**31 - 1, narrows on the way into the int32
NUMBER and SEG_VIGNET columns (4294967295 becomes -1, UNMATCHED_LABEL). The
join now requires integers in [1, MAX_NUMBER]. Boundary tests cover
repeats, 0, negatives, 5.5, 5.0, 2**31, 2**32 - 1 and accept 2**31 - 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* ngmix: each chunk holds only its own rows' tile stamps

Tile_cat memory-maps the tile catalogue (and the segmentation vignet
catalogue) and copies the stamps of the chunk's rows only, into a
ChunkStamps indexed by tile-catalogue row; rows outside the chunk raise
IndexError. The per-object columns (obj_id, ra, dec, flux) stay full
length. Ngmix.process passes its ID_OBJ_MIN/ID_OBJ_MAX bounds to
Tile_cat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

* Masked-pixel figures for #922: pixel classes on a synthetic and a real DR6 blend, and a defect-type gallery

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* Defect gallery from real DR6 epochs; dr6_epochs.py scans an exposure for every defect type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: one weight-symmetrization step, SYMMETRIZE_WEIGHTS names the classes it covers

prepare_ngmix_weights sorts each stamp pixel into clean, light-replaced
(noise-filled defects, noisefill neighbours) and light-kept-at-zero-weight
(interpolated defects, the uberseg neighbour side). One step zeroes the
weight on the quarter-turn copies, about the stamp centre, of the
light-kept holes that SYMMETRIZE_WEIGHTS names:

- interpolated (default): the interpolated defects, as before;
- interpolated_and_neighbours: also the uberseg neighbour side (#815);
- none.

Light-replaced pixels are never symmetrized. Under noisefill + noise
every value is a no-op, so the defaults are unchanged.

uberseg_weight becomes uberseg_mask, returning the neighbour-side pixels,
so the weight map is built once from the pixel classes.

Measured with defect_response (6 seeds, 0.5" galaxy, 0.7" PSF, neighbour
at 10 px): a lightless neighbour footprint gives c1 = -7.8e-3 under
uberseg and 2e-6 symmetrized, but with neighbour light (0.1x flux) c1
goes from +1.1e-3 to +1.2e-2, because the one-sided hole partly offsets
the neighbour light in the target's cell. Symmetrizing also keeps 117 of
1580 weighted pixels (97% of S/N) at 10 px and 25 (71%) at 7 px. Hence
the neighbour side stays opt-in. tests/science/test_weight_symmetrization.py
guards both results; defect_response gains a neighbour and the fit S/N.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: interpolation is the only defect fill, and the veto reads the fill's interpolated set

Short defect runs are always interpolated and the rest noise-filled;
DEFECT_FILL, SYMMETRIZE_WEIGHTS and the three epoch-cut keys are gone,
and the radii and the masked fraction are module constants.
central_defect_vetoes takes the interpolated set prepare_ngmix_weights
uses, removed noisefill neighbours included, so a run that ends on a
neighbour is vetoed at the noise-fill radius it gets. A non-finite
interpolant raises instead of falling back to noise. Off-tile pixels
get zero weight instead of a flag sentinel. Only interpolated holes are
weight-symmetrized; the #815 neighbour-side option is dropped.

The defect science tests merge into tests/science/test_defect_recovery.py,
and the module tests lose their option plumbing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* CI: run tests/science once, in its own step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: one-orientation defect interpolant; interpolated_defects names the set

interpolate_defects fits one Clough-Tocher interpolant instead of averaging
four quarter-turned ones. For the defects the veto keeps the average moves
c by at most 5e-5 and m by at most 4e-4 (test_defect_recovery cases), and
the quarter-turn weight copies already make the likelihood symmetric.

interpolated_defects(defect, neighbour, blend_handling) is the one rule
for which defect pixels are interpolated; prepare_postage_stamps (the
central veto) and prepare_ngmix_weights (the fill) both call it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* masked-pixel figures: one defect treatment, off-tile as zero weight

pixel_classes compares noisefill and uberseg; defect_gallery shows raw,
image and weight under the default. Both take the interpolated set from
interpolated_defects, and violet marks only pixels zeroed by the
quarter-turn symmetrization. dr6_epochs gives off-tile pixels zero weight
and stores the off_tile mask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* Bias-vs-distance figure for one defect: production treatment per defect type, noise fill as counterfactual

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: epoch masked-fraction cut at 10%, matching DES

EPOCH_MASKED_FRACTION_CUT = 0.10 (was 1/3). The central veto is the bias
control; far-field defects bias nothing measured up to 58% of the stamp,
so the fraction cut is set for DES Y3/Y6 comparability and as a margin
against real-data effects the sims do not model. On tile 202.301 it drops
5.08% of epochs (3.38% at 1/3) and empties 0.55% of objects (0.35%); the
extra drops are all CCD-edge or off-tile bands 10-20 px from the object.

astra.yaml: ten_percent is the default, one_third excluded with the
reason; universes/committed.yaml follows. Tests: the defect-set cut test
uses 9.8%/11.8% epochs; corner and edge-neighbour classification tests use
geometries under 10%; the off-tile central-veto test lifts the fraction
cut to isolate the veto; the far-field recovery cases no longer assert
against the cut and are documented as the evidence it is not a bias
control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bias_vs_distance: legend names the masked-fraction cut from its constant

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: noise-fill central veto at 13 px, covering the 0.7" galaxy through a 0.9" PSF

A noise-filled 4-column cluster (8% of the stamp, kept by the 10% cut)
on that galaxy gives m11 = -6.6% at 10 px and first passes at 13 px.
The science test checks the cluster and the edge band there; the
bias-vs-distance script gains the cluster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: DEFECT_WEIGHTING option; noise image interpolated in fixnoise's frame

One function, defect_weighting_masks, decides how interpolated defects enter
the image and the weight map: fourfold_zero (the PR's design, still the
default), des_y6 (pizza-cutter: mask OR its k=1 quarter turn, interpolated at
full weight), hole and full. The option reaches ngmix from the run config:
`defect_weighting` -> SP_DEFECT_WEIGHTING (tile_ngmix) -> DEFECT_WEIGHTING in
config_tile_Ng_template.ini, validated at parse time against the module's
DEFECT_WEIGHTINGS; tile_vignets carries it too so a change reruns the whole
shape group. Params pin updated (tile_ngmix only): a campaign boundary.

Under every option the metacal noise image is now interpolated in the frame
ngmix's fixnoise shears it in (rot90 k=1, back with k=3), by the science
image's operator. Interpolated in the detector frame, its interpolated pixels
sat a quarter turn from the science image's and left an S/N-dependent
additive bias (c1 = 2.7e-3 for a 3-px bleed at 7 px on the 0.7"/0.9" galaxy
at S/N 30). A test pins ngmix's k=1/k=3 behaviourally.

astra: weight_symmetrization becomes shape_measurement.defect_weighting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* ngmix: per-object defect diagnostics in the shape catalogue

Over the epochs handed to the fit, each object now carries n_epoch_interp
(epochs with an interpolated defect pixel), min_dist_interp and
min_dist_noisefill (smallest distance in pixels from the stamp centre to an
interpolated / noise-filled defect pixel; -1 when there is none). They read
only masks, so distance-split null tests are catalogue queries. make_cat
writes them as NGMIX_N_EPOCH_INTERP, NGMIX_MIN_DIST_INTERP and
NGMIX_MIN_DIST_NOISEFILL (never-fit rows: 0, -1, -1), and both final_cat.param
files list them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

* Plain-language defect_fill rationale; name DEFECT_WEIGHTING's default

The defect_fill rationale in astra.yaml now says what a defect is, why its
image value matters under metacal, what the fill does, and why. The ngmix
template and config.yaml name the default defect weighting, fourfold_zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

* des_y6: interpolate a quarter-turn copy only where the interpolant reaches it

A copy wedged in a corner of the clean region (an edge band beside the
stamp edge) lies outside the support's hull, so the fill came back NaN and
the finiteness guard dropped every epoch of the object. Such copies now
keep their light, like copies on already-replaced pixels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

* ngmix: a failed epoch drops that epoch, not the object

An exception while building one epoch's ngmix observation (for example the
non-finite interpolant guard in prepare_ngmix_weights) used to reach the
per-object except, losing every epoch of the object. do_ngmix_metacal now
catches it per epoch, drops that epoch and fits the object on the rest; the
existing epoch logic (N_EPOCH, masked-fraction cut) decides the rest. An
object whose every epoch fails still raises.

The drop is kept: Postage_stamp.epoch_failures holds (epoch name, exception),
process() logs each at warning level with tile, object and epoch, adds them to
the "epoch cuts" tally as failed=, and writes the count as a new per-object
column n_epoch_failed -> NGMIX_N_EPOCH_FAILED (final_cat.param for cfis and
cfis_image_sims; never-fit rows 0). astra.yaml epoch_masked_fraction_cut
records the rule; contract failed-epoch-dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

* ngmix: neighbours are markers on another object's segmentation footprint

With MASK_TYPE CORRECT, SExtractor writes -1e30 into an object's tile VIGNET
over every above-threshold pixel outside its pre-CLEAN footprint: other
detections, but also sub-MINAREA sky blobs and fragments CLEAN merged into the
object itself. split_tile_markers took every interior marker as a neighbour,
so noisefill noise-filled the object's own light (15% of NEIGHBOUR_FLAG=0
objects differed noisefill vs uberseg on 25 tiles).

split_tile_markers(tile_vign, shape, seg, object_number) now keeps as
neighbours only markers on a seg label other than 0 and the object's own
(the join's -1, a detection that left the catalogue, counts). Off-tile
detection is unchanged. The other interior markers become nothing: -1e30 is a
flag in the tile VIGNET only, and the epoch stamps hold real light there (the
object's own or sky), so they keep weight and image and are neither
interpolated nor noise-filled. The narrower neighbour mask feeds the noisefill
fill and interpolated_defects consistently; uberseg reads no marker, so its
output is bit-identical (tested). Without a seg stamp every interior marker
stays a neighbour, and ngmix warns once.

tile_detect therefore writes SEG_VIGNET under both blend handlings
(SP_SEG_VIGNET exported always; DETECT_SEG_MEM_MB / NGMIX_SEG_MEM_MB
unconditional). blend_handling now only moves tile_ngmix's prologue: the
uberseg params pin is unchanged and the noisefill pin moves (campaign
boundary for noisefill only). astra.yaml blend_handling, defect_fill and
pixel-mask text updated; contract neighbours-are-other-footprints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

* ngmix: DEFECT_FILL run-config knob (interpolate | noise)

defect_fill joins defect_weighting on the same path: run config
`defect_fill` -> Snakefile (validated against ngmix.DEFECT_FILLS) ->
tile_ngmix exports SP_DEFECT_FILL -> [NGMIX_RUNNER] DEFECT_FILL ->
Ngmix(defect_fill=...) -> prepare_postage_stamps / prepare_ngmix_weights.

interpolate (default) is unchanged. noise makes interpolated_defects
return no pixel, so every defect is noise-filled, and the central veto,
which reads the same set, holds every defect at the 13 px noise-fill
radius; defect_weighting then has nothing to act on.

astra: defect_fill loses [HARDCODED], noise becomes a selectable
(non-default) option, Values pins ngmix.py::DEFECT_FILL. The workflow
wiring test covers both defect options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J2y26yR4NvPvSDJtcbwL3M

* tile_detect: build the catalogue in memory, run SExtractor on node-local disk (#947)

* make_post_process: write the tile catalogue once

make_post_process appended each EPOCH HDU through FITSCatalogue.save_as_fits,
which flushed and reopened the ~670 MB uberseg catalogue per exposure, then
rebuilt LDAC_OBJECTS for N_EPOCH with add_col, another whole-file write. It
now reads the catalogue with astropy, as add_seg_vignet and match_catalogue
do, builds the EPOCH HDUs and N_EPOCH in memory, and writes the file once.

The columns are laid out by FITSCatalogue.fits_column (add_cols' column
builder, now a public static method), so the file is byte-identical to the
per-step route: a new test builds the reference through save_as_fits and
add_col and compares bytes, and on DR6 191.307 (31,736 rows, 8 epochs) the
output is byte-identical to the previous code's. Measured on that tile:
16.4 s -> 5.0 s node-local, 43.7 s -> 28.5 s on idle /scratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW

* tile_detect: run SExtractor and its post-processing node-local

On nibi tile_detect took 9-44 min per tile in single-allocation campaigns,
of which SExtractor was 1-2 min: the SEG_VIGNET cut, the DR6 join and the
post-processing each rewrite the whole catalogue (~670 MB under uberseg) on
/scratch, which a loaded node slows ~30x (191.307 in smk-g13-us: 18, 15 and
9 min). Every tile's ngmix waits on this step.

sextractor_runner takes an optional WORK_DIR: SExtractor writes its catalogue
and check images in a fresh directory under it (sp-detect<num>.*, which
tile_local's stale-store sweep also matches), the rewrites happen there, and
the files are moved to the output directory once complete. The directory is
removed on the way out, also on failure, so a failed run publishes nothing.
Without WORK_DIR the runner writes in the output directory as before.

config_tile_Sx.ini sets WORK_DIR = /local/scratch, which both profiles bind
and the node-local tile store already uses. The ini is not part of
params.pre, so the params pins do not move and running campaigns are not
invalidated; the published files and the completeness counts are unchanged.

Measured on DR6 191.307 under uberseg in one 2-core step: 36 s end to end
through WORK_DIR against 112 s in place on idle /scratch (44 min in the
loaded g13-us allocation). Catalogue, BACKGROUND and SEGMENTATION check
images are identical to the in-place run's except SExtractor's own
extraction time cards (SEXTIME, SEXELAPS) in LDAC_IMHEAD.
tests/module/test_sextractor_work_dir.py checks the staged run publishes the
in-place bytes, leaves the work directory empty, and publishes nothing when
the join refuses the catalogue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Regenerate params pins after merging develop

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X

* test_sextractor_seg_vignet: #925's final-catalogue test on #942's read_sextractor_data

(cherry picked from commit 5b0b357)
(cherry picked from commit de2b00f)

* Regenerate params pins after merging feat/masked-pixels

* params pins regenerated for feat/uberseg-seg-vignet on develop 902c32f

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@martinkilbinger martinkilbinger martinkilbinger left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Adopt Steven-Gwyn universal object IDs for v2

2 participants

@cailmdaley @martinkilbinger