Operational rename, slice 1: CHRONICLE_* env names, chronicle.db output, configurable buckets, cutover checklist (#143, mechanism 3)

[MaxGhenis]

Summary

First slice of #143 mechanism 3 (operational stores migrate by dual-run). Code and docs only; no infrastructure changes and no defaults flipped.

• chronicle/env.py — one shared reader for the rename window: CHRONICLE_<X> is read first, then POLICYENGINE_LEDGER_<X>, then LEDGER_<X>, with a once-per-process FutureWarning (so CLI operators actually see it) naming the preferred variable. Replaces the three ad-hoc _env helpers in db/supabase_client.py, chronicle/source_package.py, and db/pe_source_inventory.py. Names outside the window are read literally.
• R2 bucket names are configurable (CHRONICLE_R2_RAW_BUCKET, CHRONICLE_R2_DERIVED_BUCKET, plumbed through fetch-artifact, publish-raw, publish-derived, bootstrap-r2 and their --r2-bucket defaults). Defaults stay ledger-raw / ledger-derived; the flip is a follow-up PR once consumers repoint.
• chronicle.db for new suite outputs; ledger.db still read and inferred.
• Docs: docs/storage-architecture.md gains an "Environment Variable Rename Window" section (the previous sentence had the fallback direction backwards) and a "Bucket Cutover" checklist; README and the harness doc swept.

Two defects found and fixed on the way, each with a regression test that passes on main's behavior and fails on the bug:

• db/cli.py raised ImportError on every db CLI subcommand (a private helper import that pytest never exercised).
• The consumer-fact boundary guard matched the hard-coded ledger-derived: / r2://ledger-derived/ strings, so the bucket rename would have made it stop firing silently. It now matches on shape (*-derived bucket or derived/ key prefix).

Out of scope, deliberately: the ledger CLI alias, Supabase schema and mirror table names, governance role ids and concept authorities (hashed → mechanism 1), hash domains and schema ids (the epoch PR).

Note for the cutover checklist: this branch's count of manifest-declared raw objects differs from the one on #143 (the checklist tells the operator to recount rather than trust a number). The backfill performed today used the tracked manifests on main: 157 distinct keys, all now present in chronicle-raw (#225 has the integrity findings).

Review fixes

Gate round 1

Both findings are applied on this branch.

[high] fetch-artifact could attach a recorded R2 URI to new bytes. The preserve rule keyed on the bucket, so a repeated fetch that did not re-upload into the same bucket kept the recorded storage.r2 block while rewriting the entry's sha256/size_bytes. Reproduced against this branch's parent by serving two different bodies from one URL: the entry ends up declaring the fetched bytes' sha256 under a key addressed by the superseded bytes' one, both when the fetch only registers the bytes and when the bucket default has moved — the shape #225 hit when the IRS re-published the 2022 IRA tables.

The rule now keys on identity: the recorded key's {sha256}/{filename} tail against the fetched bytes.

• Identical — the recorded block is preserved exactly, whichever bucket is configured now.
• Different — SourceArtifactRevisionError, raised before the cached artifact or its manifest entry is touched and before anything is uploaded, naming recorded and fetched sha256/size_bytes and the ADR rule that same vintage plus new bytes is a new release_revision.
• --record-revision opts in: the fetched bytes get their own content-addressed key under the configured bucket — never the old key — and the superseded block moves to storage.previous_r2 with its sha256, size_bytes and fetched_at, so the earlier bytes stay addressable at the URI archived witness records pin.
• publish-raw applies the same check before treating a recorded block as history: a local file the recorded object does not hold is refused as recorded_r2_identity_mismatch with nothing uploaded.

storage.previous_r2 is a sibling key rather than a new shape for storage.r2, because every reader — inventory-artifacts, publish-raw, source_package._artifact_content, the suite's raw-R2-link acceptance check — reads storage.r2 alone, and publish-raw already spreads the rest of the storage block when it writes back, so a revision survives publication untouched. All 180 tracked manifest entries carrying a storage.r2 block are content-addressed and agree with their declared sha256 and filename, so the check never fires on tracked data.

[low] Env isolation was scoped to one module. The autouse fixture moves to tests/conftest.py and clears all three prefixes for every test. db.supabase_client resolves LEDGER_SCHEMA at import — during collection, before any fixture runs — so tests/test_chronicle_namespace.py re-imports it under the cleared environment instead of asserting the constant it bound at collection time.

New tests in tests/test_chronicle_artifacts.py (fake downloader, no network): a repeated fetch of identical bytes preserves the block; a repeated fetch of different bytes is refused on all three routes (re-uploaded, registered without an upload, after the bucket rename) with the cache, manifest and upload log untouched; --record-revision records the new key under the configured bucket and retains the previous object; a revised manifest still inventories as one R2-linked artifact; publish-raw refuses a mismatched local file and publishes a registered revision; and the CLI exits 1 with the message, then 0 with the flag. tests/test_chronicle_env.py asserts no rename-window variable reaches a test.

Verification on the fix head: uv run pytest -q green; CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q green (five tests failed this way before the shared fixture); uv run ruff check . clean.

Gate round 2

Seven findings, all applied on this branch. Each has a regression test, and each was first reproduced against the round-1 head (34d1d0f) — the pre-fix behavior quoted under each item is that run's output, not an inference.

[high] CHRONICLE_SCHEMA did not configure the primary Supabase mirror writer. chronicle/mirror.py, its harness wrapper and the --schema CLI default all defaulted to the literal ledger, so setting the variable to rehearse a cutover moved the read-side client and left the writer pointed at production. chronicle/env.py now owns default_chronicle_schema() — one home for the CHRONICLE_SCHEMA → POLICYENGINE_LEDGER_SCHEMA → LEDGER_SCHEMA → "ledger" ladder — and the loader resolves through it whenever no schema is supplied. An explicit --schema still wins, and the default is unchanged. Pre-fix: with CHRONICLE_SCHEMA=chronicle_probe, load_supabase_mirror still reports schema='ledger'.

[high] The derived-fact boundary was not rename-safe. chronicle/consumer_contract.py matched the .ledger_derived suffix literally. It now matches the whole final dot-segment against both ledger_derived and chronicle_derived, so a producer that renames its derived rows cannot walk a downstream target fact through the guard. Pre-fix: '.chronicle_derived'.endswith('.ledger_derived') is False — the guard never fired.

[high] fetch-artifact could not address a package's non-default manifest. Seven tracked packages keep a manifest_*_source_package.yaml, and three publisher directories keep two of them: db/data/irs_soi/ira_contributions/ holds the traditional manifest beside the Roth one. A fetch that always wrote manifest.yaml would write a third manifest neither package reads, so the IRA revision workflow the docs cite would never see the recorded block. --manifest <filename> selects it (default manifest.yaml); the name must be a filename inside --out-dir, not a path. Pre-fix: fetch_source_artifact() rejects manifest_filename as an unexpected keyword, and a fetch into that directory writes manifest.yaml.

[high] Revision protection vanished when the entry had no storage.r2. A fetch that only registered bytes, or one whose upload failed — the state #225 landed in — left an entry with a declared sha256 and no recorded block, and the guard skipped it entirely. A manifest entry identifies its bytes from the moment it is registered, so the comparison is now against the entry's recorded identity: the recorded key's {sha256}/{filename} once published, the declared sha256 before that. An ordinary fetch of different bytes over either is refused unless --record-revision opts in; a revision over a never-published entry supersedes nothing and gets no empty previous_r2. Pre-fix: re-fetching different bytes over a registered entry rewrote its sha256 with no refusal.

[medium] Recorded-R2 validation read the key or the uri, whichever came first. _validated_recorded_r2 now cross-checks every supplied locator field against every other — key against the URI's path, bucket against its authority, provider against its scheme — and the resulting key against the content-addressed {sha256}/{filename} shape. A contradiction raises RecordedR2LocatorError at fetch time and refuses as recorded_r2_locator_invalid at publish time, with nothing uploaded. Pre-fix: a block whose key and uri named different objects was preserved verbatim — key sha c63744a4… beside uri sha 1e9b3fdb… — so the entry kept publishing a URI for bytes it no longer described.

[medium] A malformed manifest read as an absent one. _read_manifest now refuses a document that parses as anything but a mapping, and an unparseable one, with MalformedManifestError — raised before the publisher is read at all, so a refusal costs no fetch. inventory-artifacts and publish-raw report it as a manifest error instead of crashing on .get. An absent or empty document still reads as an empty mapping. Pre-fix: a list-valued manifest.yaml was overwritten by the fetch.

[low] Schema resolution happened at import, during collection. db.supabase_client resolved both schemas into module constants when it was first imported — at collection, before any fixture — so a legacy variable could warn and seed state that no fixture could take back. Both are now functions (chronicle_schema(), targets_schema()), and tests/conftest.py additionally strips the rename window in pytest_configure, before collection imports anything. Pre-fix: importing under LEDGER_SCHEMA=zzz bound LEDGER_SCHEMA='zzz' and emitted a FutureWarning.

All four refusals share a SourceArtifactManifestError base, so fetch-artifact reports every one as exit 1 with nothing written.

New round-2 tests: tests/test_chronicle_mirror.py (the loader and its CLI write to the configured schema, under each name in the window, with an explicit --schema still winning); tests/test_chronicle_artifacts.py (a two-manifest package addressed by name and its revision refused in the right manifest, on the direct and CLI paths; a manifest name that tries to leave the package; revision protection over a registered entry and over a failed upload; every locator contradiction, at fetch and at publish time; malformed manifests refused before the fetch and reported by both sweeps); tests/test_chronicle_consumer_contract.py (the chronicle spelling rejected identically, and the marker matched as a whole segment); tests/test_chronicle_namespace.py and tests/test_chronicle_env.py (schema read per call, no constant frozen at import).

Verification on the round-2 head (ea67f0a): uv run pytest -q — 839 passed, 1 skipped; LEDGER_SCHEMA=zzz uv run pytest -q -W error::FutureWarning tests/test_chronicle_namespace.py — 6 passed; CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q — 839 passed, 1 skipped; uv run ruff check . clean; uv run ruff format --check . clean for every file this branch touches (the 13 unformatted files are pre-existing on main and none are touched here).

Scope is unchanged: no boundary change, nothing under releases/ or db/data manifests, defaults still ledger-raw / ledger-derived / ledger, and the shared functions #227 will rebase onto keep their signatures with the new arguments defaulted.

Chronicle Governance

This PR does not touch the source-data boundary: no source packages, facts, profiles, or schemas change. Storage configuration, CLI defaults, and docs only.

Tests

tests/test_chronicle_env.py (new, hermetic: chronicle-first precedence, legacy fallback with one warning per process, out-of-window names read literally), tests/test_chronicle_artifacts.py (bucket configuration, chronicle.db output with ledger.db read-back, the db/cli.py regression), tests/test_chronicle_consumer_contract.py (boundary guard by shape). CI runs the full suite on this PR; the lane that authored the branch ran out of budget before pasting its local run, so treat CI here as the verification of record.

🤖 Generated with Claude Code

Rebased onto #228 (9/4)

Rebased onto main at 9da0243 (epoch dual-domain acceptance). Five conflicts, all mechanical: import lines in chronicle/artifacts.py, chronicle/source_package.py, chronicle/suite.py, tests/test_chronicle_artifacts.py (both sides kept), and the --build-id help text in chronicle/harness.py (merged: accepted epoch prefixes, defaults inferred from reports, chronicle.db, or a legacy ledger.db). Full suite on the rebased head: 955 passed, 1 skipped; ruff check clean; the one file the rebase left unformatted is formatted in the last commit.

Gate round 3 (9/4)

Five low-severity findings, all applied: ManifestNameError (a SourceArtifactManifestError that is still a ValueError) so the CLI reports a bad --manifest name as error: … / exit 1; a files block that is not a mapping is refused before the publisher is read; identical bytes under a different filename are refused as a rename (with and without --record-revision), naming the --filename that keeps the recorded identity; publish-raw reports an entry whose recorded object in another bucket holds the local bytes as skipped (new skipped_count; sweep exits 0 after the bucket flip); fetching with the default manifest name into a directory that keeps manifest_*.yaml files and no manifest.yaml is refused (AmbiguousManifestError, closing the #225 path without the flag). Full suite on this head: 961 passed, 1 skipped.

Sol gate rounds 2–3 (9/4)

Twelve further findings closed across two Sol rounds. The documented bucket-cutover sweep now exits 0 over a copy of the tracked registry (161 manifests, 194 artifacts skipped or linked, none uploaded or failed) because recorded content-addressed keys are accepted as preserved history when their checksum/filename tail matches the bytes, while contradictions still refuse. Shared physical archives across manifest pairs (USDA, CMS, SSA) are recognised: fetch refuses changes by default and rewrites every owner consistently under --record-revision. Manifest-declared artifact paths that are absolute, parent-traversing, symlinked, or manifest-named are refused before any read; the ambiguity guard covers every supported manifest spelling and refuses creating any manifest beside an existing registry; sweep --manifest must be a literal supported filename; all manifest reads use the strict duplicate-key loader; a present non-list previous_r2 is refused; LEDGER_SCHEMA/TARGETS_SCHEMA are import-time snapshots of the environment-aware resolvers. Round 3: new registration identities must be one canonical R2 key segment (IdentitySegmentError); matching_directory_entry refuses more than one physical alias; resource_directory is contained inside the resource package (no absolute, parent, dot, or symlinked-ancestor components); manifest-named entries that are not regular files fail sweeps and discovery loudly. Full suite on this head: 1,069 passed, 1 skipped; ruff check clean; no tracked db/data/** manifest changed.

Peer round 4 (9/4, Astra): nine findings closed, then rebased onto current main

The derived-provenance boundary now derives from the configured derived bucket and prefix (is_derived_r2_route, shared by publication and the consumer guard; case-insensitive scheme, archived routes retained) instead of a -derived name heuristic. Source parsing and inventory reuse _validated_recorded_r2 before any read, so contradictory, incomplete, or non-R2 locators are refused rather than emitted as provenance, and local or fetched bytes are checked against the recorded object's digest even without a separately declared checksum. publish-derived walks the build tree with lstat first and refuses symlinks and non-regular entries before any upload. Every new-key publication path (derived, first-time raw) requires canonical identity segments; present manifest source_id/package_id declarations must be canonical strings equal to the arguments (no strip or stringify). A single case- or Unicode-normalized filename alias whose physical spelling differs from the manifest is an entry error in publish and inventory, never a read. Logical vintage duplicates (2024 beside "2024") are refused by one validate_manifest_vintages invoked from load_manifest_document, so every manifest consumer sees them. Discovery's non-regular-manifest refusal is translated to MalformedManifestError at the publish and inventory sweeps and their CLI wrappers. The source-package resolver requires regular files for both manifest and artifact entries. The branch is rebased onto current main (the OTS proof "deletions" a reviewer saw were the stack's stale base, not changes in this PR).

Full suite on the rebased head: 1,238 passed, 7 skipped (Astra lane) and, after the rebase, every test green with the cutover sweep pinned by invariants; ruff check clean.

Astra gate 6aa29716 round 1 (9681c1c)

• False shared-file collision (medium, chronicle/artifacts.py): validate_package_directory compared only the declared sha256, so a sibling manifest identifying the same file solely through its content-addressed R2 locator read as an empty digest after an identical-byte refetch recorded sha256 on the selected manifest. It now takes an entry_digest resolver (default unchanged) and the owner-agreement preflight passes _effective_recorded_digest, which resolves the digest the recorded R2 key encodes via _validated_recorded_r2 before falling back to the declared field. Regression test_identical_byte_refetch_keeps_r2_identified_shared_file_valid (both sweeps valid before and after the refetch).
• Recursive merges (low, chronicle/registration.py): the strict loader tracks the mappings whose merges are being expanded and refuses a source that is already active with ConstructorError("found a recursive merge…"), a yaml.YAMLError the readers handle. Regression test_strict_loader_refuses_recursive_merges_as_yaml_errors (direct, via sequence, via merged mapping; repeated non-cyclic source still merges).
• Verification on 9681c1c: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 485 passed; ruff check/format clean on the changed files; full suite running (result will be appended).

Astra gate 6aa29716 round 2 (cd14372)

• Unidentified siblings after selected or partial publication (medium): an entry with no recorded identity (no sha256, no locator) read as an empty digest, so identifying one manifest (selected publish, or a full sweep whose second upload failed) turned its sibling into a false filename_collision_across_manifests. Such entries are now skipped by the directory-level comparison (nothing to contradict), and the owner-agreement preflight hashes their package-local bytes against the identified owners and refuses before any upload when they differ. Regressions: test_selected_publication_of_a_shared_unidentified_file_keeps_siblings_valid, test_partial_upload_failure_of_a_shared_file_stays_retryable, test_sweeps_refuse_identifying_a_shared_file_whose_bytes_changed (all red on 9681c1c).
• Verification on cd14372: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 488 passed; ruff check/format clean; full suite running (result will be appended).

Astra gate 6aa29716 round 3 (7d9fd06)

• First fetch of a predeclared entry (medium): _manifest_file_owners refused every entry naming the artifact without a recorded identity, including the selected entry on its own first fetch, so files: {2024: {filename, source_url}} could never acquire its bytes. The fetch preflight, the sibling-bytes check, and the manifest writer now name the (manifest, vintage) entry they are initializing; it is skipped as not-yet-an-owner, while every other unidentified entry naming the file keeps the refusal. Regressions: test_first_fetch_initializes_a_predeclared_entry_without_identity[False|True] (red on cd14372) and test_fetch_still_refuses_an_unidentified_owner_in_another_manifest (retained protection).
• Verification on 7d9fd06: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 491 passed; ruff check/format clean; full suite running (result will be appended).

Astra gate 6aa29716 round 4 → fresh gate (d544ffe)

• Selected publication overrides leaked onto sibling manifests (medium): the package-wide publish preflight applied --source-id/--package-id to every sibling, so publishing one manifest with its missing identifier supplied was refused when a sibling declared a different one. The override now completes or confirms only the selected manifest; unselected siblings are preflighted with their own identifiers (inventory takes no overrides). Regression test_selected_publication_overrides_apply_only_to_the_selected_manifest[source_id|package_id] (red on 7d9fd06).
• Verification on d544ffe: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 493 passed; ruff check/format clean; full suite running (result will be appended).

Astra gate cc680cc1 round 1 (3eea393)

• Default sweeps withheld overrides from selected siblings (medium): override eligibility was decided per manifest being processed, so in a default sweep (every manifest selected) a selected sibling met through another manifest's package preflight was treated as unselected and the publication aborted with r2_identity_invalid. The selected set is now computed up front from the root sweep; selected siblings take the overrides, unselected siblings keep their own identifiers. Regression test_default_sweep_overrides_apply_to_every_selected_sibling[source_id|package_id] (red on d544ffe).
• Verification on 3eea393: focused artifacts + source-package + consumer-contract + env + manifest-reading suites 495 passed; ruff check/format clean; full suite uv run pytest -q -p no:cacheprovider tests 1281 passed, 1 skipped (20:11). Astra gate cc680cc1 round 2: approve.

[MaxGhenis]

Fable+Sol gate: changes requested, and the high finding is exactly the failure mode today's backfill hit in the wild (IRS re-published the 2022 IRA tables with different bytes under the same URL, #225):

1. [high] fetch-artifact can attach the recorded R2 URI to new bytes. When the recorded storage.r2 block is preserved on a repeated fetch, the manifest is rewritten with the newly fetched sha256 and size_bytes but keeps the old content-addressed key and URI. A publisher revision then produces a manifest whose provenance is false. Fix: compare the recorded identity (filename + sha256) with the fetched bytes before preserving anything; identical → preserve the recorded block; different → refuse, or record a new content-addressed key under the configured bucket as a new revision, never the old key. The regression test must repeat with different bytes, not identical ones.
2. [low] Env isolation is scoped to one test module. Other tests assert default buckets/schemas without clearing CHRONICLE_R2_RAW_BUCKET / CHRONICLE_SCHEMA; move isolation to a shared fixture or clear explicitly where defaults are asserted.

A fix lane is applying both on this branch; re-gate after.

[MaxGhenis]

Gate round 2: changes requested again, all agreed and being fixed on this branch (round-2 lane): CHRONICLE_SCHEMA did not reach the primary Supabase mirror writer (harness/mirror still default to the literal ledger schema); the .ledger_derived suffix check is not rename-safe; fetch-artifact cannot address a package's non-default manifest file (--manifest added), so the IRA revision workflow the docs cite would miss the recorded block; revision protection vanished when an entry lacked storage.r2; recorded-R2 locator fields must be cross-checked, not key-or-URI; malformed manifests must be rejected rather than treated as absent; collection-time schema resolution must be lazy. Re-gate follows.