[NoopDog]

Closes #724

What changed

• Which tars are read. The tar fetcher now also reads a GenomicsDB workspace's callset.json for the folders header_classifier.CALLSET_FOLDERS declares: ANVIL_T2T_CHRY's SGDP_CHM13v2/genomics_db and SGDP_GRCh38/genomics_db. An SGDP tar and a 1000 Genomes tar can share a file name, so the pipeline binds each record's dataset and folder into the new FileTypeConfig.samples_member hook. The fetcher asks that hook with the member names, as it asks kept_member.
• Fetch and cache. The member is read after the head walk, in the tar's first and last MiB (_read_kept). Its text is cached beside vcfheader.vcf as TarEvidence.callset, with a looked-for flag, callset_read.
  • A read that fails is not marked looked for, so the next run reads it again. The tar keeps its member names and classifications.
  • The hook is asked before the cached text is used. The cache is keyed by md5, so a file of the same bytes in a folder not declared gets no callset.
• Parse. callset_samples moved from external_lineage into cohort_steps, so the classifier and the Workaround for #624: link ANVIL_T2T_CHRY's 1000 Genomes workspace tars to the Schatz lab's published sample maps, checked by each tar's callset.json #710 importer share one parser. parse_tar_head parses each distinct text once per run. It gives ParsedTarHead.samples, or why there are none: no_callset, callset_unreadable or not_a_callset.
• Samples. TAR_CONFIG.samples reads through a new rule, code_rules.TAR_CALLSET_SAMPLES (tar_callset_samples). It takes all the names through rules/sample_namespaces.yaml, or none.
  • SampleReader gains two optional hooks, used only by the tar reader. applies keeps a tar whose callset was not wanted out of the counts. why_unread puts the cause on an unread reading.
  • Every producer that reads samples now writes details.unread_samples, the files per cause per dataset, beside sample_outcomes and unrecognised_samples.
• Reconcile is unchanged. A tar's samples settle as any file's do.
• Docs and tests. CLAUDE.md and the docstrings that list the sample readers are updated, and so is the contract's "What is not true yet" bullet on 2.11, which said the SGDP tars are not read. New tests cover the fetcher, the parse causes, the folder check, the producer's counts, and the cached callset withheld from a file it is not wanted for. The golden fixture gains the empty unread_samples key on the five producers that read samples.

Why

The 62,022 SGDP workspace tars in ANVIL_T2T_CHRY had no samples, so they reached no donor. The 1000 Genomes tars get theirs from the published sample maps (#710), but the SGDP maps aren't public (#624). Each tar names its own samples in callset.json.

Assumptions I made

• The scope is a path check, declared in code. The issue asked for exactly that, so it is a dict in header_classifier, not YAML. Adding the 1000 Genomes tars is Check the 1000 Genomes workspace tars' samples against their own callset.json #725.
• The raw text is cached, not the parsed names, as the issue asked, so a parser change needs no new read. That adds about 33 KB per SGDP tar to the tar cache, about 2 GB in all.
• A compressed tar's callset.json is not read. The head walk keeps only one member, and a gzip stream can't be entered at its tail. The cause is counted, and the tar's classifications are not failed. No SGDP tar is compressed (measured 0).
• The per-cause counts stay in the tar producer's own details. The reconcile report still shows a single unread total per dataset.
• Deferred to Check the 1000 Genomes workspace tars' samples against their own callset.json #725, where reading the 1000 Genomes tars makes them matter (comment):
  • one download per window for both members;
  • a compressed tar's callset;
  • one record shape per member;
  • a general "members to keep" hook in place of the folder dict;
  • the external lineage importer reading the cached callset.
• Left as they are after a no-context review, since none occurs in today's data:
  • A wanted tar whose whole fetch fails is counted only as "content unreadable", not by sample cause.
  • A compressed SGDP tar's callset is counted as callset_unreadable and asked again each run.
  • Two threads saving one md5 at once can drop a callset just read; the cost is a repeat read, and no tar md5 is shared.
  • The 2-byte gzip check assumes the server honours Range, as AWS and GCP do.
  • Duplicate sample names are dropped, as every sample reader does.
• The run's counter misreports member-only reads. "From cache" counts a tar whose cache entry existed, even when only a member was read over the network. That predates this change; vcfheader.vcf (Link T2T's GenomicsDB tars to their cohort: sample maps, the tars' own step, values read through the map #621) behaves the same.

How to verify

The run used below is output/anvil/20261021_172513:

• make classify at 60abcef ran the tar producer in about 1 h 54 min, and reconciled its own run.
• 43 callset.json reads failed transiently in that pass. make classify-tar RUN_DIR=output/anvil/20261021_172513 re-read just those 43 (2 min, all read), then make reconcile RUN_DIR=… ran.
• At the final code (4b302eb), make classify-tar RUN_DIR=… (from cache) and make reconcile RUN_DIR=… ran again. Every number below is unchanged. The run's tar file and reconciled/ are now from 4b302eb; its other 13 inference files are from 60abcef, and the later commits change no output for them. The two commits after 4b302eb change only docs and tests.

The issue's definition of done, item by item:

• make classify, then make reconcile. Run as above. To repeat it on a fresh checkout of this branch: make classify (SGDP tars come from the cache if data/evidence/anvil/tar is shared, otherwise about 2 h of range reads), then make reconcile.

• About 62,022 more files with samples. Count reconciled file records whose samples is set: 591,781 in run 20261019_225524, 653,803 here, +62,022.

  • These are exactly the 62,022 SGDP tars: 60,840 list 279 samples, and the 1,139 chrY tars list 175.
  • Against Carry each file's storage path from the snapshot's file_inventory onto the input, so files with the same name can be told apart #717's run on the same input (20261021_154039), the only change in either artifact is samples on those 62,022 tar records. donors, samples and external_lists are identical.

• About 62,022 more files reaching a donor (591,045 today). This holds only under a looser count than the one behind 591,045.

  Count	20261019_225524	this run
  files where every sample has a donor (the 591,045 measure)	591,045	591,045 (+0)
  files where at least one sample has a donor	591,333	653,355 (+62,022)

  • Every SGDP tar lists the four genomes Four SGDP genomes left without a donor by #559's sex check: S_Palestinian-2, S_Naxi-2, S_Norwegian-1, B_Australian-3 #718 left without a donor: simons_data_sample_33, _187, _273 and _304. Each chrY tar lists _273.
  • So no tar meets the "every sample" measure until Four SGDP genomes left without a donor by #559's sex check: S_Palestinian-2, S_Naxi-2, S_Norwegian-1, B_Australian-3 #718 is settled. Then all 62,022 do.
  • To check: join each reconciled record's samples.list_id to reconciled/sample_lists.ndjson.gz, and each member to a reconciled/samples.ndjson.gz record with collected_from.

• The tar producer's samples outcomes per dataset, with every tar not read counted by cause. In output/anvil/20261021_172513/tar_classifications.json, under metadata.details:

  • sample_outcomes: {"ANVIL_T2T_CHRY": {"named": 62022}}
  • unread_samples: per cause. The first pass had {"ANVIL_T2T_CHRY": {"callset_unreadable": 43}}; after the re-read it is empty, because every callset was read.
  • The run log prints the same as Header samples, … and Samples not read, … lines.
  • A tar whose callset is not wanted (1000 Genomes, ANVIL_T2T, any non-workspace tar) is not counted.

• make test-all (3,264 + 98 passed), make lint-all (Pyright 0 errors), make format-check and make validate-metadata all pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS

🐦 Built with cc-claude-tools

[Copilot]

🟡 Changes recommended

The cache-upgrade path incorrectly treats plain workspaces named .tar.gz as compressed, preventing sample extraction that succeeds on a fresh fetch.

1 open issue

• Recheck compression before reading callset in cache upgrade · New

What changed in this PR

Adds sample extraction from SGDP GenomicsDB workspace tars using each archive’s own callset.json, feeding the existing sample reconciliation process. Donor-link gaps tracked in #718 remain outside this change.

Changes:

• Fetches and caches callsets only for declared SGDP folders, with retries for failed member reads.
• Shares callset parsing with the external lineage importer and records unread-sample causes.
• Adds regression coverage and updates fixtures and documentation.

File	Description
tests/​test_fetchers.py	Tests callset fetching, caching, retries, and folder isolation.
tests/​test_external_lineage.py	Moves shared parser tests to cohort coverage.
tests/​test_cohort_steps.py	Tests parsing, folder selection, and sample reporting.
tests/​fixtures/​golden/​expected_output.json	Updates expected sample metadata.
src/​meta_disco/​sample_identity.py	Adds tar sample reading and unread-cause hooks.
src/​meta_disco/​records.py	Defines the unread-samples metadata key.
src/​meta_disco/​reconcile_samples.py	Documents additional sample sources.
src/​meta_disco/​pipeline.py	Binds folder context and aggregates unread causes.
src/​meta_disco/​header_classifier.py	Declares eligible SGDP workspace folders.
src/​meta_disco/​file_types.py	Enables tar callset and sample readers.
src/​meta_disco/​fetchers.py	Fetches and caches selected callset members.
src/​meta_disco/​external_lineage.py	Uses shared callset parsing and constants.
src/​meta_disco/​evidence.py	Adds callset text and read-state fields.
src/​meta_disco/​cohort_steps.py	Parses callsets and reports failure causes.
src/​meta_disco/​code_rules.py	Registers the tar sample rule.
scripts/​generate_rules_report.py	Updates sample-rule explanatory text.
CLAUDE.md	Documents callset caching and sample extraction.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

A failed callset-only re-walk of a cached tar named compressed can suppress classifications and lineage supported by its cached evidence.

1 open issue

• Preserve cached header when callset-only tar rewalk fails · New

1 resolved since last review

• Recheck compression before reading callset in cache upgrade

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No blocking issues remain, and the supplied regression coverage addresses scoped fetching, cache isolation, retry behavior, and sample-reader integration.

No open issues 🎉

1 resolved since last review

• Preserve cached header when callset-only tar rewalk fails

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The scoped fetch and cache changes fit the existing sample pipeline, with targeted regression coverage and no unresolved blocking findings.

No open issues 🎉

_{Review effort: Balanced}