Register raw microdata releases as source artifacts: licence/access fields, hash-only registration, first UK registrations (#221)

[MaxGhenis]

Summary

Implements chronicle#221: Chronicle registers every raw microdata release its
consumers build from, and stores the content of none of them.

Depends on #222 (the raw-microdata-identity ADR and the governance,
architecture, AGENTS.md, and non-goal amendments). This PR deliberately makes
none of those edits, so the two do not conflict — but until #222 merges, the
governance doc still forbids what this branch does, and the ADR file that four
files cite does not exist on main. Merge #222 first.

Contract is shared verbatim with microcosm#848.

Schema and validation

Manifest file entries gain two fields:

• licence — the publisher's terms, as an identifier or URL.
• access — a closed set: public | licensed | restricted.

public is inferred where absent; fetch-artifact now writes the class
explicitly onto every entry it touches. Both fields are required on a
kind: microdata_release manifest, the new manifest-level carve-out.

Only public bytes enter a Chronicle store, so the R2 key
raw/{source_id}/{package_id}/{year}/{sha256}/{filename} exists only for that
class. A licensed or restricted release is registered hash-only: the
manifest carries the checksum, size, vintage, licence, access route, and
verification timestamp, and no bytes are fetched, written, or uploaded.

A registration is identified by {source_id, package_id, year, sha256, filename} — the tuple consumers reference under chronicle_artifact.

New command

chronicle register-artifact registers a licensed or restricted release from
CLI arguments alone. It refuses --access public, a missing or malformed
checksum, a missing licence, vintage, access route, or verification timestamp,
bytes sitting beside the manifest, a publisher_table manifest, a foreign
source_id, and an unrequested reissue. Public releases keep the existing
fetch-artifact path, which gains --access, --licence, and --kind.

First registrations

15 hash-only registrations, every value read verbatim from Microcosm's reviewed
pins and cross-checked field by field:

• DWP Family Resources Survey 2023-24 — 14 tabs, access: licensed, UK Data
  Service End User Licence, study SN 9367, DOI 10.5255/UKDA-SN-9367-2.
  Microcosm carries 19 licensed_microdata references across five stages;
  they resolve to 14 distinct files (identical SHA-256 where they repeat).
• HMRC Survey of Personal Incomes Public Use Tape 2022-23 — put2223uk.tab,
  access: restricted, study SN 9422, DOI 10.5255/UKDA-SN-9422-1.

chronicle inventory-artifacts --root db/data is valid with 195 artifacts: 180
with R2 links, 15 hash-only with no local bytes.

BE-SILC 2023 is not registered. Microcosm pins it with no sha256, no
size_bytes, and no per-file filename — the locator names four file roles.
No hash is invented; the blocker and its unblock steps are recorded in
docs/data-sources/be-silc-2023-registration-blocker.md.

Tooling

scripts/register_microdata_releases.py resolves a declarative catalogue
against a read-only microcosm checkout. emit writes the hash-only manifests
byte-stably; plan prints the exact fetch-artifact --upload-r2 commands to
run from a networked machine for the public US releases, copying publisher URLs
verbatim and printing TODO_PUBLISHER_URL where Microcosm records none (CPS
basic monthly 2024, SIPP 2023). The catalogue is the only authored content —
every checksum, size, filename, and vintage is read from Microcosm.

Audit fixes

An adversarial pass over the first implementation found the byte boundary was
escapable. All six are fixed here with regression tests:

1. Licensed bytes could reach the package tree and R2. The hash-only guard
   only inspected files[year] for the requested year and only entries that
   survived a kind-gated list expansion, while the write target and R2 key are
   directory-scoped. A release registered under one vintage could be fetched
   under another, and a list under a manifest with an absent or misspelled
   kind was invisible to the guard. Both were reproduced writing licensed
   bytes into the tree. The guard now walks every vintage and expands lists
   unconditionally.
2. A fetch silently deleted an existing registration by replacing the whole
   files[year] value. It now replaces only the entry for its own filename.
3. emit stopped being re-runnable after a reissue — registration raised on
   the first filename match with a different checksum before reaching the exact
   match. Now two passes.
4. --skip-hash-only reported a contract violation as valid, disagreeing
   with inventory-artifacts about the same tree. Hash-only entries are now
   always validated; the flag turns off the refusal, not the check.
5. No way to register a public microdata release as kind: microdata_release, though the ADR puts redistributable public-use files in
   scope. fetch-artifact gains --kind, and a release manifest keeps the list
   shape so several files share one vintage (ACS household + person).
6. Every generated fetch command died at argparse — plan emitted
   --source-url, which fetch-artifact does not accept. Fixed to --url, and
   a test now parses each generated command against the real CLI parser.

Also hardened _year_mapping, which returned a list and let the caller raise an
uncaught TypeError that validate-package does not catch.

No content, anywhere

No microdata row, cell, or fact is added. No source package parses a release:
validate-package fails with microdata_release_not_parseable, and the FRS and
SPI packages hold nothing but manifest.yaml. Bucket names stay configurable
(ledger-raw default, unchanged).

Chronicle Governance

For any source package, consumer contract, schema, or source-data
boundary change:

• Approved Chronicle agent role:

  • ledger-source-ingestor — the db/data/** microdata-release manifests
  • ledger-contract-maintainer — the manifest schema (licence, access,
    kind), its validation, and the registration command

  Note both roles' allowed_paths in .github/chronicle-agents.yml predate
  this work and cover none of chronicle/registration.py,
  chronicle/artifacts.py, chronicle/harness.py, chronicle/cli.py,
  db/data/**, scripts/**, docs/**, or README.md. Flagging rather than
  quietly widening the manifest; reviewers may want a follow-up that extends
  the globs.

• Deterministic checks run:

  • uv run pytest -q — full suite
  • uv run ruff check . — passes
  • uv run ruff format --check . — 14 pre-existing failures, all in files this
    branch does not touch (verified unchanged against origin/main); the
    branch's own files pass
  • uv run chronicle inventory-artifacts --root db/data — valid, 195
    artifacts, 15 hash-only
  • scripts/register_microdata_releases.py emit — re-run is byte-stable
  • every registered checksum cross-checked verbatim against Microcosm's pins

• LLM judge verdicts:

  • ledger-source-fidelity: PASS — no fact is added. Every registered
    value (sha256, size, filename, vintage) is transcribed verbatim from the
    consumer's reviewed pin, machine-verified against Microcosm, and each entry
    records hash_source plus a note that Chronicle did not recompute the
    checksum. BE-SILC is refused rather than guessed.
  • ledger-contract: PASS — the change adds two manifest fields and one
    manifest kind, preserves provenance, and moves no Microcosm responsibility
    into Chronicle. Registration is identity only; selection, reconciliation,
    aging, and imputation are untouched.
  • ledger-boundary: PASS — no microdata row, cell, or fact enters
    Chronicle; no source package parses a release; no licensed or restricted
    bytes enter any Chronicle store, and the escapes that previously allowed
    that are closed and regression-tested.

Tests

tests/test_chronicle_microdata_registration.py — 69 tests covering the access
and kind vocabularies, every manifest-entry validation code, each command's
refusals, the six audit regressions, public microdata releases, the generated
fetch plan, and the committed FRS/SPI registrations as identity-only.

Two existing assertions in tests/test_chronicle_artifacts.py gain the new
hash_only_count and hash_only_refused_count report keys.

🤖 Generated with Claude Code

Gate round 2 (9/4): rebased onto #226 and #228; all nine round-1 findings closed

Rebased onto ops-rename-slice1 (PR #226 at fb1bc1d, which carries main at 9da0243 = #228). Conflicts in chronicle/artifacts.py were resolved so #226's stray-manifest refusal, files-block check, rename refusal, skipped publish entries and explicit-null files: handling all survive beside this PR's pre-fetch preflight.

1. Filename aliases — filenames are bare, case-folded names; every alias of a hash-only registration is refused, and manifest-wide path collisions are rejected.
2. Byte reader — _read_source_artifact_content refuses a hash-only entry before the package tree, the content-addressed cache, or a fetch is touched; SourceArtifactSpec.assert_parseable decides both carve-outs from the manifest alone; validate-package reports hash_only_artifact_not_parseable and manifest_kind_missing; build-suite validates before creating or replacing its output directory.
3. Reclassifying an archived identity — refused while the recorded R2 object exists.
4. Strict pre-fetch validation — a --kind that conflicts with the stored kind, a kindless manifest with entries, and every inventory-level manifest error are refused before the publisher is read.
5. Inference before I/O — the destination basename is inferred from the URL without a read when --filename is omitted, so a gated artifact is never downloaded to be refused.
6. _upsert_manifest — replaces the matched entry in place under Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226's recorded-identity guard, carries forward fields a fetch does not own, never turns a publisher-table mapping into a list, and always writes kind.
7. Year keys — the vintage entry is selected by either key spelling; a vintage recorded under both spellings is refused rather than silently preferring one.
8. Reviewed identity — registrations are consumer_pin (attested_by: PolicyEngine/microcosm, pinned_from path and commit, no verified_at); plan emits --publisher, --vintage, --filename, --expected-sha256/--expected-size-bytes when Microcosm's pin is of the publisher bytes, the allowlisted licence and its evidence; anything unknown is a TODO the CLI refuses. The 15 committed FRS and SPI rows are re-emitted that way with byte-identical SHA-256/size pins (verified against the microcosm blobs at 2fb2e2f8 and de7451bd).
9. Hermetic tests — tests/fixtures/microcosm holds synthetic consumer manifests in Microcosm's shapes with the reviewed pins verbatim; emit reproduces both committed manifests byte for byte, plan matches a golden command list, every planned command parses against the real CLI, and the committed pins are a golden mapping. The commit-reading tests run against a copy of the fixture in a throwaway consumer repository.

Also implemented from the #222 ADR (which this PR still does not edit): the explicit kind rule with its frozen grandfathered-manifest list (chronicle/grandfathered_manifests.py), the redistributable-licence allowlist with per-term evidence (chronicle/licences.py), artifact-bound licence_evidence, hash_source/attested_by/pinned_from/attestation_evidence, staging outside the repository for public bytes, and the no-tracked-microdata-bytes guard. Docs: docs/agent-source-package-harness.md "Hash-Only Registrations", README quick-start, storage note.

Verification on 7f9bfe6: full suite 1219 passed, 1 skipped; ruff check clean.

Merge order unchanged: #222 first.

Follow-on hardening (9/4, after the round-2 fixes)

• The byte boundary spans every manifest a package directory keeps (c0d9d74). --manifest (Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226) lets a fetch record into one of several manifests in a publisher directory; the hash-only guard read only that one, so a release registered licensed in manifest.yaml could be fetched, published, parsed, or reclassified through a sibling manifest_<package>.yaml. fetch-artifact now reads every manifest the directory keeps before the publisher is touched and refuses a name or digest any of them registers hash-only, refuses to overwrite a file a sibling records as other bytes, refuses a manifest whose source_id/package_id differ from the fetch's, and refuses an artifact named like a manifest; publish-raw and inventory-artifacts report filename_collision_across_manifests / sha256_collision_across_manifests; register-artifact takes --manifest with the same refusals; the source-package byte reader refuses a file a sibling registers hash-only. 18 tests in tests/test_chronicle_package_directory.py.
• Manifests are read strictly (7f9bfe6). Duplicate mapping keys are refused (PyYAML would silently keep the last), filename keys are NFC-normalised as well as case-folded, and validate_file_entry reports nameless or malformed entries, misspelled fields, a storage block or chronicle_fetch attestation on a hash-only entry, a consumer attestation signed by Chronicle, and evidence URLs without a host; register-artifact validates every existing entry before recording. tests/test_chronicle_manifest_reading.py.

Gate round 3 (9/4, Sol): twelve findings closed

Entry keys use a closed schema (typos such as acess are refused, never public-by-default); the source-package reader and the fetch validate the selected manifest and every sibling strictly before any boundary decision or byte read; normalized-duplicate manifest names are refused; every no-local-bytes check scans real directory entries by normalized identity (shared helper across fetch, publish, inventory, registration and the public-release tree guards); inventory counts storage.r2 only for a validated content-addressed R2 locator (public microdata releases must also bind source, package and vintage); consumer pins are checked against the recorded commit's blob with git cat-file before emission; registration refuses symlinked output paths and persists under an exclusive lock by atomic same-directory replacement (two-thread regression); default sweeps discover every supported manifest name; only YAML null reads as an empty manifest; blank identity fields are replaced or refused before persistence; FRS selectors compare all stage references. Adversarial follow-ups closed in the same lane, including publish's dry preflight opening a valid artifact before honoring a malformed-sibling error. Full suite on this head: 1,268 passed, 1 skipped; ruff check clean; UK pins and #222-owned files unchanged.

Gate round 4 (9/4): eleven findings closed

Public entries without a sha256 are hashed and collision-checked against gated siblings before any upload, cache, or parse; inventory short-circuits on manifest, sibling, or entry errors before any byte read; source parsing validates storage.r2 locators and package-wide collisions before content access; the cross-bucket key rule is reconciled with #226's historical-route compatibility (a legacy recorded route whose checksum tail matches is preserved history; a key contradicting the entry's own package/year that is not a recorded legacy shape is refused); consumer pins record the repository, commit, and path of the blob actually checked; fetch, publish, and registration share one per-package lock keyed on physical file identity; required textual fields must be non-empty strings; the stray-default guard covers every supported manifest spelling; FRS cross-stage checks compare stable byte identity. Full suite on this head: 1,295 passed, 1 skipped; ruff check clean; UK pins and #222-owned files unchanged.

Integration with PR #226 (cb4c7ef)

• Rebased by the round-3 fix lane onto Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226's then-head ba8147a with the four round-3 findings fixed (cross-vintage fetch overwrite validated before writes; archived-alias refusal against current and historical R2 objects; consumer provenance bound to the verified repository origin; staging destination validated before publisher access), then integrated with Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226's final head 3eea393 through two merge commits (2c71ace, cb4c7ef). One fixture commit (bf24aa2) declares kind: publisher_table on the six manifests Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226's newest artifact tests build, as this branch's registration contract requires.
• Both semantics sets are retained and each has a named regression: the strict loader (merge precedence, recursive-merge refusal), effective digests and the unidentified-entry rule inside this branch's collision classes, the pre-upload bytes guard, the initialization exemption alongside expected pins and the table-vintage guard, and selected-set override eligibility inside the locked publish wrapper.
• Verification on cb4c7ef: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 552 passed; full suite uv run pytest -q -p no:cacheprovider 1661 passed, 1 skipped (20:55); ruff check/format clean.
• Merge order: ADR: raw microdata in Chronicle is identity, not content #222 → Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3) #226 → this PR (then a merge of main).

Astra gate bbd833a9 round 1 (46560bc)

• Source readers bypassed shared-file identity checks (medium): the source-package reader now runs the same directory-level owner agreement as inventory and publish (effective digests via the recorded-R2 resolver; an unidentified entry's bytes are hashed against identified siblings) before returning or caching bytes. Red 4f7aea5 (6 cases), fix adb030c.
• Publish-raw staging path not validated (medium): public microdata publication resolves its staging destination through _validated_microdata_staging_destination (every component, realpath, no symlinked component, outside the repository and every package) before any byte is read or uploaded. Red a164a2b (13 cases incl. staging beneath db/data and a symlinked ancestor), fix 63c6bb9.
• Registration lacked directory-wide agreement (medium): registration validates the complete proposed package (effective digests, archived collisions) before acquiring the writer lock and again before atomic replacement; an existing sibling collision refuses an unrelated registration with no lock file and no manifest byte changed. Red 2040886 (5 cases incl. revalidation under lock), fix b820b18.
• Verification on 46560bc: full suite 1,691 passed, 1 skipped (21:20, lane); my spot-check of the artifacts + source-package + consumer-contract + env + manifest-reading + microdata-registration + microdata-staging suites 809 passed; ruff check/format clean on the six changed files.

Astra gate bbd833a9 round 2 (89737a1)

• Source reader could parse a sibling's public microdata (medium): _assert_package_identities_parseable classifies every sibling's current and archived filename and checksum and refuses before any read, fetch, cache write, or row parse when any sibling registers the selected entry's identity as microdata_release; selected declared and effective digests are checked before I/O. Red 9c8738e (32 cases), control e543cea, fix dfd3b99.
• Registration omitted effective owners and explicit locator checks (medium): both _prepare_registration_payload calls validate every current and archived R2 locator explicitly (locator errors are refusals), enforce filename/checksum agreement, and run the same owner-agreement validator publication and inventory use (effective digests, owners across vintages within one manifest), before lock creation and again under the lock before replacement, metadata-only. Reproductions: test_registration_refuses_invalid_recorded_sibling_before_mutation (5 cases), test_registration_rechecks_recorded_sibling_under_lock_before_replacement (5 cases), positive control, and a no-artifact-read control; fix 651571b.
• Verification on 89737a1: full suite 1,737 passed, 1 skipped (24:21, lane); my spot-check of the eight artifact + source-package + consumer-contract + env + manifest-reading + microdata-registration + microdata-staging + package-directory suites 902 passed; ruff check/format clean; scope audit against 46560bc: exactly the five expected files changed.

[MaxGhenis]

Review (Fable, main): approving, with the Sol gate running. Cross-checked every registered hash against Microcosm's uk/source_stages.json: 15 of 15 match, none extra, BE-SILC correctly refused rather than guessed. The refusal surface is the right shape: bytes for licensed/restricted can enter no store by any command, and the six escapes the lane found in its own first cut are the kind of thing this boundary exists for.

Sequencing: #222 merges first (its gate is running; merge-on-agreement), then this. No file overlap between the two.

Notes, none blocking:

• private_microdata → restricted is a safe over-classification (SPI PUT is a UKDS licensed study; both classes are hash-only, so nothing changes byte-wise). Fine as is; say so in the catalogue comment if it is not already.
• dataset: dwp_dwp-frs-2023-24 is a mechanical id; harmless, could read dwp_frs_2023_24 in a follow-up.
• The agent-role allowed_paths gap the body flags is real and predates this PR; follow-up to widen the globs rather than silently editing the roles file here.
• The Microcosm side (microcosm#848, in flight) must reference exactly {source_id: dwp, package_id: dwp-frs-2023-24} and the SPI tuple committed here; I will align that PR to these ids.
• The public-release fetch commands from plan will be run from a networked session after merge (two need publisher URLs Microcosm does not record: CPS basic monthly 2024, SIPP 2023).

[MaxGhenis]

Gate round 1: changes requested. The 15 registrations match the consumer pins and add no bytes or fact surfaces, but Sol found real escapes around the boundary, all of which stand: filename aliases (./adult.tab vs adult.tab) bypass the hash-only guard; the source-package byte reader checks manifest kind but never the selected entry's access, so a mapping-shaped manifest with access: licensed can still be read, cached, and parsed; register-artifact can reclassify a public identity that already has an R2 object as licensed/restricted, leaving the object in R2 while inventory looks clean; kind conflicts and missing microdata access are not rejected before I/O; with --filename omitted the artifact is downloaded before the inferred filename is refused; _upsert_manifest is destructive and shape-unsafe; year keys are looked up as int only while readers accept '2023'; the generated public-release commands drop publisher/vintage and carry the expected sha256 only as a comment (fetch-artifact needs --expected-sha256 and the metadata params); the catalogue tests hardcode a home-directory checkout and skip on failure.

Two additions from the ADR review (#222 round 2) fold into the same fix: byte custody requires an affirmative, evidence-backed redistribution signal, not merely access: public (an allowlist of redistributable licence terms enforced in code, everything else is licensed); and the prohibition vocabulary is canonicalized to records, rows, columns, row values, cells, and facts.

Fix lane queues behind #226's round-2 fixes because both rewrite chronicle/artifacts.py; it rebases onto that branch first.

[MaxGhenis]

Cross-PR convention ruling (chronicle#227 ↔ microcosm#853), so both sides land on one shape:

1. One Chronicle package per publisher release, files listed under the vintage. Chronicle's kind: microdata_release manifests hold a list under files[year] precisely so several distributed files share one registration (14 FRS tabs = one UKDS study). The consumer reference {source_id, package_id, year, sha256, filename, access} already disambiguates files inside a package by filename + sha256. So the ids are dwp / dwp-frs-2024-25 (year 2024, 14 tabs) and hmrc / hmrc-spi-public-use-tape-2022-23 (year 2022, put2223uk.tab), not one package per tab.
2. Vintage: FRS 2024-25 (UKDS SN 9563). Microcosm main moved to the 2024-25 tabs today (#832); chronicle#227 registered the 2023-24 hashes from a stale checkout, so only 1 of its 15 hashes still matches main. The Chronicle side regenerates from microcosm origin/main (emit against a fresh checkout) and adds the other licensed/private roots main pins (WAS round 8, LCFS 2023-24 and 2018-20, ETB 1977-2024, SPI PUT): 19 hashes today.
3. Access class derived from the Microcosm kind as #853 does (licensed_microdata → licensed; private_microdata/restricted_microdata → restricted; public_microdata → public), with byte custody additionally requiring the licence allowlist (ADR round 2). versioned_derived_microdata is never registered.

Chronicle's #227 fix round and a small #853 follow-up apply this.