[NoopDog]

Closes #717

What changed

• Manifests are requested by dataset id. Each deployment now declares every dataset's dataset_id beside its snapshot, in one line per dataset (deployments._datasets). azul_manifest.manifest_filters filters on datasets.dataset_id, not datasets.title. The downloader refuses a record from an Azul manifest whose title or id is not the declared one.
• The input carries each file's storage path. file_path is a new optional slot of the input contract (schema/metadata.yaml, regenerated model). Azul's compact manifest now has a files.file_path column, and record_from_compact_manifest_row reads it. The verbatim and TDR reader already copied anvil_file.file_path, and now reads an empty one as null.
• A parent several files carry by name is settled by folder.
  • records.folder_of gives a record's folder: interned, "" at the top of the dataset, None with no path.
  • Where a name matches several files, edges.matches / edges.in_folder keep the one in the child's own folder.
  • Every name lookup uses it: the index producer, checksum files, the VCF-header steps, and the tar and sample-map cohort steps.
  • The step readers take folder as a required argument that may be None. The pipeline passes ClassifierRecord.folder.
• On a GATK 4 HaplotypeCaller line the contigs still decide first. 821 ANVIL_T2T gVCFs name a CRAM that two files carry (v2 re-alignments) #620's @SQ/##contig check runs before the folder, and the folder only breaks a tie the contigs leave. It never stands in for an alignment header that can't be read.
• The index producer counts folder-settled parents, as metadata.details.settled_by_folder, per dataset.
• Comments and declarations corrected:
  • the six name rules' declarations in code_rules (they feed the rules report),
  • the index producer's docstring,
  • snapshot_input's claim that Azul drops file_path,
  • models.JOIN_KEY_FILE_PATH,
  • CLAUDE.md.

Why

• ANVIL_T2T_CHRY stores one pipeline's outputs under the same file names in a folder per cohort (1KGP/, SGDP_CHM13v2/, SGDP_GRCh38/). By name alone, 15,006 indexes (14,866 in T2T_CHRY, 140 in T2T) had two or three candidate parents and took none. Each index sits beside the file it indexes, so its folder settles it.

• While re-downloading prod on 2026-10-21, the title filters for AnVIL_HPRC_R2 and AnVIL_IGVF_Mouse_R1 returned compact manifests Azul had stored on 2026-10-17:

  Dataset	Rows returned by title	Files in the dataset
  AnVIL_HPRC_R2	11,905	16,271
  AnVIL_IGVF_Mouse_R1	3,334	6,786

  The same datasets requested by id came back complete. The id is the stable way to name a dataset. The row-count check is still what refuses a short manifest, by either filter.

Assumptions I made

• The path is a check, not a source. It only chooses among files that already match by name. It makes no claim about any dimension or sample, and it is not on output records. Putting it there is a follow-up, to be filed.
• Same folder only. A candidate counts only if it is in exactly the child's folder, not a nearby one. No current case needs anything looser.
• Content outranks the folder. On the one rule that reads content to settle a shared name (GATK 4 HaplotypeCaller), the contigs decide and the folder only breaks a remaining tie.
• The prod input is rebuilt. data/anvil/prod/ (shared across clones) was rebuilt from the 2026-10-21 manifests with this code: 708,088 records, every one with a path. The 09-17 input is kept under data/anvil/archive/prod_anvil15_20260917/.
• The verification run is marked dirty. Run output/anvil/20261021_114828 was made before the commit, so its provenance says dirty. Reports are not regenerated here; I'd regenerate them from a run on main after merge, as with Regenerate the reports from run 20261019_225524 as reconciled on main after #719 #720/Regenerate the reports from run 20261019_225524 as reconciled on main after #719 (#720) #721.
• A report to Azul about the stored short manifests is drafted for a human to send. It is not filed.

How to verify

The issue's proposed scope, mapped to steps:

• 1. Each file's path is on the input record, through every reader.
  1. Run make download, then make validate-metadata. Expect 708,088 records checked and OK — no problems.
  2. Run python -c "import json; print(sum(1 for l in open('data/anvil/prod/anvil_files_metadata.ndjson') if json.loads(l).get('file_path')))". Expect 708088.
  3. Run pytest tests/test_snapshot_input.py -k Parity. The compact and verbatim derivations agree on every file, file_path included.
• 2. A same-named parent is settled by its folder.
  1. Run make classify.
  2. In the new run's index_classifications.json, check metadata.details. Expect ambiguous_parent: 0 (it was 15,006 in 20261019_225524), matched_to_parent: 224674 (it was 209,668), and settled_by_folder: {"ANVIL_T2T": 140, "ANVIL_T2T_CHRY": 14866}.
  3. Run make reconcile (part of make classify). In reconcile_report.json, expect lineage.conflicts to be {}. The new edges name the same parents AnVIL's lineage does.
  4. Run python scripts/compare_corpus.py --artifact reconciled --old-run output/anvil/20261019_225524 --new-run <new run>. Expect no label changes on files present in both runs.
• Manifests are requested by dataset id. Run pytest tests/test_deployments.py -k DatasetId: each request names the declared id, and another dataset's manifest is refused.
• The folder rules. Run pytest tests/test_index_propagation.py tests/test_edges.py tests/test_producer_steps.py -k "folder or contigs". This covers three cohort folders, the top level, no single carrier in the folder, contigs over folder, and folder as tie-break.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
🐦 Built with cc-claude-tools

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues in input compatibility, dataset validation, or parent-resolution precedence; project tests were not executed.

No open issues 🎉

What changed in this PR

This PR addresses #717 by carrying storage paths into classification inputs and using the child’s folder to distinguish same-named parents. It also requests Azul manifests by declared dataset ID and validates returned dataset identities.

Changes:

• Add optional file_path support across input readers, schema, and runtime records.
• Apply folder-based parent resolution across producers while preserving GATK 4 contig checks as the stronger evidence.
• Declare dataset IDs, validate manifest records, and update tests and documentation.

File	Description
tests/​test_snapshot_input.py	Check storage-path parity and empty-path handling.
tests/​test_producer_steps.py	Test folder tie-breaking and contig precedence.
tests/​test_metadata_schema.py	Validate optional, non-empty storage paths.
tests/​test_index_propagation.py	Cover folder-based index parent resolution.
tests/​test_edges.py	Test checksum folder matching and folder-key isolation.
tests/​test_deployments.py	Test dataset-ID requests and identity rejection.
tests/​test_cohort_steps.py	Update cohort reader calls with folder arguments.
tests/​test_azul_manifest.py	Test ID filters and compact storage-path parsing.
src/​meta_disco/​snapshot_input.py	Normalize empty paths and correct reader documentation.
src/​meta_disco/​schema/​metadata.yaml	Declare the optional storage-path slot.
src/​meta_disco/​schema/​metadata_model.py	Add the generated storage-path field and validation.
src/​meta_disco/​records.py	Derive folders and retain them on classifier inputs.
src/​meta_disco/​producer_steps.py	Resolve shared names by folder after applicable contig checks.
src/​meta_disco/​pipeline.py	Pass each input’s folder to step readers.
src/​meta_disco/​models.py	Clarify storage-path join-key documentation.
src/​meta_disco/​file_types.py	Document the updated step-reader arguments.
src/​meta_disco/​edges.py	Centralize folder-based candidate selection.
src/​meta_disco/​deployments.py	Declare dataset IDs beside snapshot identities.
src/​meta_disco/​cohort_steps.py	Pass folders into cohort parent lookups.
src/​meta_disco/​code_rules.py	Update rule descriptions for folder resolution.
src/​meta_disco/​azul_manifest.py	Filter manifests by ID and read compact storage paths.
scripts/​download_anvil_manifest.py	Validate declared IDs and returned dataset identities.
scripts/​classify_index_files.py	Use the index’s folder to resolve shared parent names.
CLAUDE.md	Document dataset-ID checks and input-only storage paths.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Equivalent dataset-root path spellings produce different folder keys, leaving valid same-folder parents unresolved.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Normalize root-level path folder keys to avoid false ambiguity

src/​meta_disco/​records.py:150

Equivalent root-level paths get different folder keys: ./sample.bam produces '.', but sample.bam.bai produces ''. If another sample.bam exists in a subfolder, edges.in_folder retains both candidates and the index is declined as ambiguous despite having one same-folder parent. Normalize '.' to '' so both spellings use the documented dataset-root key, and add a regression case for this mixed spelling.

_{Review effort: Balanced}

[NoopDog]

Copilot's previously missed finding (records.py, root-level folder keys): fixed in 3bb6d17. folder_of now normalises the folder first and reads . as the top of the dataset, so ./sample.bam and sample.bam share the top-level folder ""; a ./x.bam case is added to test_a_record_s_folder_is_its_path_less_the_name_normalised.

[Copilot]

🟢 Approval recommended

Static review found no blocking issues, and the updated callers and regression coverage support dataset validation, folder disambiguation, and contig precedence.

No open issues 🎉

_{Review effort: Balanced}