diff --git a/PROGRESS.md b/PROGRESS.md index 24773009..9d1938ee 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,51 +1,750 @@ -# Lane C5 progress +# Operational rename, slice 1 (chronicle#143, mechanism 3) + +Lane C5's handoff notes previously lived here; its durable record is +`LANE_C5_REPORT.md`. This file now tracks the active lane on this branch. ## State -- Branch: `be-2025-vintages` from `origin/main` at `5c15bfd`. -- Worktree inputs are staged under `.lane-raw/` and must remain uncommitted. -- Lane C5 is complete, validated, independently reviewed, and ready for handoff. -- The requested staged C2 report is absent, but root `LANE_C2_REPORT.md` is byte-identical - to the sibling lane's staged copy (SHA-256 `4590e0dc...50f06e7`) and is the pattern used. +- Branch: `ops-rename-slice1`, cut from `origin/main` at `ff3efd3`. +- Scope: env names, R2 bucket configurability, `ledger.db` -> `chronicle.db`, + and the docs for all three. Code and docs only; no infrastructure changes. +- Out of scope and deliberately untouched: the `ledger` console-script alias, + the Supabase `"ledger"` schema and mirror table names, governance role ids and + concept authorities, hash domains and schema ids, anything under `releases/`. +- This PR does not touch the source-data boundary. No package spec, parser, + selector, manifest, or fact value changes. ## Done -- Read the repository Chronicle boundary rules in `AGENTS.md`. -- Read `.lane-raw/SOURCES.md` and confirmed all five named publisher artifacts are present. -- Confirmed the worktree is otherwise clean apart from `.lane-raw/` and the shared `.venv` link. -- Verified all five staged artifact SHA-256 pins exactly. -- Mapped FPB workbook cells: 990 facts across T01/T06/T07/T11/T17/T24, with - 2022–2025 observations and 2026–2031 `source_projection` facts. -- Confirmed PDF boundary evidence: printed page 19 calls 2026 the first projection year; - annex table units appear on printed pages 45, 48, 49, 53, 58, and 65. -- Chosen Eurostat layout: two vintage-specific source-package aliases share new manifest - entries, preserving the prior package YAMLs, raw bytes, and fact outputs unchanged. -- Reproduced the Statbel curator logic: 18 NUTS1 × sex × age-band cells totaling 11,825,551. -- Added the hash-pinned FPB workbook and publication PDF plus the - `fpb-economic-outlook-2026-2031-june-2026` package alias. -- Built 990 line-specific publisher facts (99 per year): 396 observations for - 2022–2025 and 594 `source_projection` facts for 2026–2031. -- Passed FPB `validate-package` and `build-suite`: 990 facts, full cell lineage, - zero acceptance errors, and pinned 2025 cells 320578 / 77771 / 5602 million euro. -- Re-ran the Statbel 2026 curator logic on the 2025 ZIP and added the hash-pinned - raw capture plus its deterministic 18-row curated CSV. -- Passed Statbel 2025 `validate-package` and `build-suite`: 18 facts totaling - 11,825,551, 66 constraints, full lineage, and zero acceptance errors. -- Added the Eurostat `gov_10a_taxag` 2025 and `spr_exp_func` 2024 manifest - entries plus vintage-specific package aliases, without modifying either - prior artifact or prior package specification. -- Passed both new Eurostat package validations and suite builds: 12 tax facts - and 9 ESSPROS facts, full lineage, and zero acceptance errors. -- Extended Belgium and Eurostat regressions for FPB table counts/cells and - assertion boundary, vintage non-overlap, prior-output digests, Statbel pins, - and the declared 0.25% Statbel/FPB population comparison tolerance. -- Passed 43 focused tests and the full merged-bundle regression: 157,177 facts, - 148 packages, zero aggregate-key duplicates, and expected goldens throughout. -- Recorded pins, counts, boundary evidence, curator commands, validation tails, - and consumer fact families in `LANE_C5_REPORT.md`. -- Passed independent `ledger-source-fidelity` and `ledger-boundary` reviews with - no required corrections. +- Read `AGENTS.md`, `docs/storage-architecture.md`, + `docs/agent-source-package-harness.md`, and the mechanism-3 migration spec in + the first comment of PolicyEngine/chronicle#143. +- Enumerated every ledger-named env read in tracked Python: the four real + variables (`LEDGER_SOURCE_ARTIFACT_CACHE_DIR`, `LEDGER_SOURCE_ARTIFACT_FETCH`, + `LEDGER_PE_US_DATA_ROOT`, `LEDGER_PE_UK_DATA_ROOT`) plus + `POLICYENGINE_LEDGER_SCHEMA`. `LEDGER_MIRROR_TABLES`, + `LEDGER_MIRROR_PRIMARY_KEYS`, and `LEDGER_DB_SCHEMA_VERSION` are module + constants, not env reads, and name out-of-scope surfaces. +- Added `chronicle/env.py`: one shared `env_value`/`env_flag`/`env_names` + helper reading `CHRONICLE_` first, then `POLICYENGINE_LEDGER_` and + `LEDGER_` with a once-per-process `ChronicleEnvDeprecationWarning` naming + the preferred variable. +- Replaced all three ad-hoc helpers (`db/supabase_client._env`, + `chronicle/source_package._env_value`/`_truthy_env`, + `db/pe_source_inventory._env_value`) with the shared helper. +- Made the R2 bucket names configurable via `CHRONICLE_R2_RAW_BUCKET` and + `CHRONICLE_R2_DERIVED_BUCKET`, plumbed through fetch-artifact, publish-raw, + publish-derived and bootstrap-r2. Defaults unchanged at `ledger-raw` and + `ledger-derived`. Both manifest write paths now preserve a recorded + `storage.r2` block instead of restating it under a renamed bucket. +- Emitted `chronicle.db` for new suite outputs, with `ledger.db` still accepted + on read and on derived-artifact kind inference. +- Added `tests/test_chronicle_env.py` plus artifact tests: 75 hermetic tests + covering the lookup ladder, precedence, the once-per-process warning, and every + real call site. +- Swept the docs. `docs/storage-architecture.md` gained an "Environment Variable + Rename Window" section (the old text stated the fallback direction backwards) + and a "Bucket Cutover" section; `docs/agent-source-package-harness.md` and + `README.md` follow. Verified 186 distinct `ledger-raw` objects across 154 + tracked manifest files, every key content-addressed by sha256. + +## Review fixes (gate round 1) + +The Fable+Sol gate requested changes; both findings are applied on this branch. + +- **[high] `fetch-artifact` could attach a recorded R2 URI to new bytes.** The + preserve rule keyed on the bucket, so a repeated fetch that did not re-upload + into the same bucket kept the recorded `storage.r2` block while rewriting the + entry's `sha256`/`size_bytes`. Reproduced against this branch's parent by + serving two different bodies from one URL: the entry ends up declaring the + fetched bytes' `sha256` under a key addressed by the superseded bytes' one, + both when the fetch only registers the bytes and when the bucket default has + moved. + The rule now keys on identity — the recorded key's `{sha256}/{filename}` tail + against the fetched bytes. Identical preserves the block exactly; different + raises `SourceArtifactRevisionError` before the cached artifact or its + manifest entry is touched, naming recorded and fetched `sha256`/`size_bytes` + and the ADR rule that same vintage plus new bytes is a new release revision. + `--record-revision` opts in: the new bytes get their own content-addressed key + under the configured bucket, never the old key, and the superseded block moves + to `storage.previous_r2`. `publish-raw` applies the same check before treating + a recorded block as history (`recorded_r2_identity_mismatch`, nothing + uploaded). +- **[low] Env isolation was scoped to one module.** The autouse fixture moved to + `tests/conftest.py` and now clears all three prefixes for every test. + `db.supabase_client` resolves `LEDGER_SCHEMA` at import — during collection, + before any fixture — so `tests/test_chronicle_namespace.py` re-imports it + under the cleared environment instead of asserting the constant it bound at + collection time. + +`storage.previous_r2` is a sibling key, chosen because every reader +(`inventory-artifacts`, `publish-raw`, `source_package._artifact_content`, the +suite's raw-R2-link acceptance check) reads `storage.r2` alone, and +`publish-raw` already spreads the rest of the `storage` block when it writes +back, so a revision survives publication untouched. All 180 tracked manifest +entries that carry a `storage.r2` block are content-addressed and agree with +their declared `sha256` and `filename`, so the identity check never fires on +tracked data. + +## Review fixes (gate round 2) + +The second Fable+Sol gate requested changes again. Seven findings, each fixed +with a regression test on this branch. Plan, in dependency order: + +1. **[high] `CHRONICLE_SCHEMA` does not reach the Supabase mirror writer.** + `chronicle/harness.py` and `chronicle/mirror.py` default the schema to the + literal `"ledger"`; only `db.supabase_client` reads the renamed variable. + Resolve through the shared helper whenever no explicit `--schema` is given. +2. **[high] The derived-fact boundary check is not rename-safe.** + `chronicle/consumer_contract.py` matches the `.ledger_derived` suffix only. +3. **[high] `fetch-artifact` cannot address a package's non-default manifest.** + Seven tracked packages keep a `manifest_*_source_package.yaml`; three + directories keep two. A fetch into one of them writes a third manifest and + never sees the recorded block. +4. **[high] Revision protection vanishes when the entry has no `storage.r2`.** +5. **[medium] Recorded-R2 locator fields must be cross-checked**, not read as + key-or-URI, before a block is preserved or published. +6. **[medium] `_read_manifest` must reject a malformed document**, not treat a + non-mapping YAML payload as an absent manifest. +7. **[low] Schema resolution must be lazy** so no legacy variable is read at + collection, before the autouse isolation fixture runs. + +## State (round 2) + +- Read both gate rounds on PolicyEngine/chronicle#226 and the code each finding + names. +- Scanned all 154 tracked manifest files (187 `files` entries, every one + carrying `storage.r2`): every recorded block supplies provider, bucket, key + and uri; every key is content-addressed; every declared `sha256`/`filename` + agrees with its key tail; no `uri` contradicts its `key`. Strict locator + validation therefore refuses nothing that is tracked today. +- All seven findings are applied, each with a regression test, and each + reproduced against this branch's previous head (`34d1d0f`) first. + +### What each fix does + +1. `chronicle/env.py` gains `default_chronicle_schema()`: one home for the + `CHRONICLE_SCHEMA` -> `POLICYENGINE_LEDGER_SCHEMA` -> `LEDGER_SCHEMA` -> + `"ledger"` ladder. `load_supabase_mirror`, its harness wrapper and the + `--schema` CLI default all resolve through it when no schema is supplied; + an explicit `--schema` still wins. Defaults unchanged. +2. `chronicle/consumer_contract.py` matches the whole final dot-segment of a + `source_record_id` against both `ledger_derived` and `chronicle_derived`. +3. `fetch-artifact --manifest ` selects which of a package's + manifests the entry belongs to (default `manifest.yaml`); the name must be + a filename inside `--out-dir`. +4. Revision protection now compares against the entry's recorded identity -- + the recorded key's `{sha256}/{filename}` once published, the declared + `sha256` before that -- so a registered-but-unpublished entry, or one whose + upload failed, is protected exactly like a published one. +5. `_validated_recorded_r2` cross-checks every supplied locator field against + every other and against the content-addressed key shape. A contradiction is + `RecordedR2LocatorError` at fetch time and `recorded_r2_locator_invalid` at + publish time, never a preserved block. +6. `_read_manifest` refuses a non-mapping or unparseable document + (`MalformedManifestError`) before the publisher is read at all; + `inventory-artifacts` and `publish-raw` report it instead of crashing. +7. `db.supabase_client` resolves both schemas per call rather than at import, + and `tests/conftest.py` strips the rename window in `pytest_configure`, so + no module can read or warn from an operator's shell during collection. + +All four refusals share a `SourceArtifactManifestError` base, so the +`fetch-artifact` CLI reports every one as exit 1 with nothing written. + +### Reproduced against `34d1d0f` (the round-1 head) + +Running the same operations against a checkout of the previous head: + +1. `load_supabase_mirror` default `schema='ledger'`; with + `CHRONICLE_SCHEMA=chronicle_probe` the load still reports `schema='ledger'`. +2. `'.chronicle_derived'.endswith('.ledger_derived')` is False: the boundary + never fired for the chronicle spelling. +3. `fetch_source_artifact()` rejects `manifest_filename` as an unexpected + keyword; a fetch into `ira_contributions/` writes `manifest.yaml`. +4. A fetch of different bytes over a registered (unpublished) entry was + accepted silently: the entry's `sha256` was rewritten with no refusal. +5. A block whose `key` and `uri` named different objects was preserved + verbatim, key sha `c63744a4...` beside uri sha `1e9b3fdb...`. +6. A list-valued `manifest.yaml` was overwritten by the fetch. +7. Importing `db.supabase_client` under `LEDGER_SCHEMA=zzz` bound + `LEDGER_SCHEMA='zzz'` and emitted a `FutureWarning` at collection. + +## Verification + +- `uv run pytest -q`: green. +- `uv run ruff check .`: clean. +- `uv run ruff format --check .`: clean for every file this branch touches. 13 + files are unformatted on `main` already and are byte-identical here; CI runs + `ruff check` only, so they are pre-existing and out of scope. +- CI's db CLI gate (`chronicle init` / `load all` / `stats`): passes. +- `CHRONICLE_R2_RAW_BUCKET=zzz CHRONICLE_SCHEMA=zzz uv run pytest -q`: green. + Before the shared fixture it failed five tests — four bucket-default + assertions in `tests/test_chronicle_artifacts.py` and the collection-time + schema constant in `tests/test_chronicle_namespace.py`. ## Next -- None; ready for handoff. No push was performed. +- Push and open the PR against `main`. Do not merge. +- Follow-up PR, after Max creates and backfills the new buckets: flip + `DEFAULT_R2_RAW_BUCKET` / `DEFAULT_R2_DERIVED_BUCKET` to `chronicle-raw` / + `chronicle-derived`. + +## Review fixes (Sol gate round 3) + +### State + +- Detached HEAD: `fb1bc1df`, the PR #226 head supplied for the ten-finding Sol + gate round. +- Scope: ten operational-rename findings in artifact fetch/publish/inventory, + Supabase compatibility aliases, and the README cutover procedure. No tracked + `db/data/**` manifest will be changed. +- `CLAUDE.md`, one of the requested initial reads, is absent from both this + worktree and `/Users/maxghenis/PolicyEngine/chronicle`; `AGENTS.md` and the + remaining requested guidance/code/tests are present. +- PR #227 is available read-only at + `/Users/maxghenis/PolicyEngine/_worktrees/chronicle-227-fix`. Applicable + non-microdata hunks from `daafac0` and `c0d9d74` will be ported with the same + function names and shapes. + +### Done + +- Re-established the lane state in this committed progress log before making + gate-round code or test changes. +- Read `AGENTS.md`, the Bucket Cutover and Publisher Revisions contracts in + `docs/storage-architecture.md`, the README cutover instructions, and the + named implementation/test surfaces. Confirmed Chronicle must preserve + publisher bytes and provenance, refuse unsafe fetches before I/O, and leave + schema/bucket value cutovers explicit. +- Reproduced findings 1, 4, 5, and 8 with 11 failing cases: mismatched manifest + identity reached the publisher read, quoted year keys bypassed revision + protection, duplicate year spellings and malformed entries reached I/O, and + both refetch and revision discarded entry metadata. +- Ported the non-microdata parts of #227's `_assert_manifest_identifies`, + `_select_vintage_entry`, `_FETCH_OWNED_FIELDS`, and in-place + `_upsert_manifest` flow. Fetch now validates manifest identity and entry + shape before I/O, resolves either year-key spelling while refusing both, + preserves the recorded key spelling, and carries forward every field it does + not own. The 11 focused cases now pass. +- Reproduced finding 2 with both default sweeps reporting only one of four + supported manifest names, and finding 7 with all four falsy non-mapping + `files` values producing `(inventory.valid, publish.valid) == (True, True)`. +- Ported #227's `is_manifest_filename` / `package_manifest_paths` shapes and + made default root sweeps discover `manifest.yaml`, `manifest.yml`, and both + `manifest_` extensions. Both sweeps now share `_manifest_files`, so + only `None` means absent and every other non-mapping value is reported. All + 106 artifact tests pass. +- Reproduced finding 3 as a green preserved-bucket skip for a key routed to the + wrong package/year, and finding 6 as both a fetch reaching I/O and a green + publish skip for a self-consistent `s3://` block under `storage.r2`. +- Raw publish now compares the recorded key with the canonical + source/package/year key before any bucket-change skip. Recorded R2 validation + also requires `provider='r2'` (and therefore an `r2://` effective URI). The + focused canonical-key/provider tests, including the existing URI-only and + stale-country cases, pass without uploads or manifest rewrites on refusal. +- Reproduced finding 9 with the shipped import raising `ImportError` for + `LEDGER_SCHEMA`, then restored `LEDGER_SCHEMA` and `TARGETS_SCHEMA` as + deprecated aliases of the stable defaults. Runtime queries remain on the + lazy functions and still honor post-import environment changes; 55 focused + namespace/env/client cases pass (1 skipped for absent real credentials). +- Reproduced both parts of finding 10: the README did not state that an + unqualified mirror load writes to `ledger`, and it named the absent + `supabase/migrations/20260504_chronicle_bronze.sql` file. +- README now instructs operators to create and apply the deployment migration, + states the `ledger` runtime default, and gives both supported ways to target + `chronicle`. The storage architecture and source-package harness use the same + truthful procedure; no documentation names the absent SQL file. Both README + regression tests pass. +- Adversarial review tightened the same contracts before final verification. + Mixed-case package manifests were reproduced as omissions from both root + sweeps and from the stray-default guard; default discovery now filters every + recursive filename through the case-insensitive #227 helper. Missing + `provider` and missing `uri` locators were each reproduced reaching publisher + I/O; `storage.r2` now requires explicit `provider: r2` and an `r2://` URI. + The artifact file's 112 tests pass. +- Clarified that *all* schema environment overrides, including deprecated + spellings, precede the `ledger` default. The README test now also requires + the create/apply-migration instruction, so deleting the guidance cannot pass + vacuously. All 14 mirror tests pass. +- Completed the remaining non-microdata preflight port from #227. Reproduced + four invalid explicit/inferred artifact filenames reaching `_read_artifact`, + including `filename=manifest.yaml`, which could overwrite the selected + manifest, and reproduced an undiscoverable `custom.yaml` manifest reaching + publisher I/O. Ported `is_bare_filename`, `bare_filename`, + `_infer_artifact_filename` from `daafac0`, and the manifest-like artifact + refusal plus discoverable `_manifest_path` restriction from `c0d9d74`. All + 117 artifact tests pass after the pre-I/O fix. + +### Next + +- None in this lane. All ten Sol findings, the non-microdata #227 port + completeness pass, and adversarial follow-ups are committed and verified. + The final evidence and per-finding commit map are in the runner's external + `-o out.md` report, not a repository-root file. + +### Final verification + +- `uv run ruff check .`: exit 0 (`All checks passed!`). +- `uv run ruff format --check` on all six changed Python files: exit 0 + (`6 files already formatted`). +- Full `uv run pytest -q -p no:cacheprovider`: direct exit 0, 995 passed, + 1 skipped, 18 warnings in 1539.05 seconds. +- A fresh `/tmp` copy of tracked USDA `fy69_to_current` reports two manifests + and two artifacts in inventory. Publish includes both: FY2024 is one safe + preserved-bucket skip and the known misrouted FY2025 package/year key is one + explicit failure, with zero uploads. No tracked manifest was changed. +- The external `-o out.md` report records every failing-first command and + observation, regression test, fix commit, and exact #227 port provenance. + +## Review fixes (Sol gate round 2, eight findings) + +### State + +- Detached HEAD began at `c36f3fc8`, the supplied head of PR #226; base is + `main` at `9da02431`. The worktree was clean at intake. +- Scope is the eight supplied findings: cutover compatibility, shared-file + revision ownership, artifact/manifest path safety, duplicate YAML keys, + malformed revision history, and import-time Supabase alias compatibility. +- No tracked `db/data/**` manifest will be modified. The whole-tree cutover + regression will operate on a temporary copy and use a non-writing uploader. +- PR #227 is available read-only at + `/Users/maxghenis/PolicyEngine/_worktrees/chronicle-227-fix` at `5557cb9e`. + Applicable non-microdata hunks will be ported with the same function names + and shapes so its later rebase stays straightforward. + +### Done + +- Read the existing `PROGRESS.md`, the Bucket Cutover and Publisher Revisions + sections of `docs/storage-architecture.md`, and all six requested code/test + files: `chronicle/artifacts.py`, `chronicle/harness.py`, + `chronicle/source_package.py`, `db/supabase_client.py`, + `tests/test_chronicle_artifacts.py`, and `tests/test_chronicle_env.py`. +- Read the GitNexus debugging/refactoring workflow guidance. No GitNexus MCP + graph tools are exposed in this session, so dependency tracing will use + repository search, focused tests, and the stacked PR's committed diffs. +- Confirmed the baseline gaps in the named code: `_read_manifest` still uses + `yaml.safe_load`; `_root_manifest_paths` passes a non-default value to + `rglob`; manifest-declared filenames reach direct path joins and reads; + fetch updates only its selected manifest; `_superseding_storage` converts a + non-list `previous_r2` to an empty history; and the Supabase compatibility + constants are hard-coded defaults. +- **Finding 6 reproduced and fixed.** Red command: + `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_artifacts.py::test_fetch_refuses_duplicate_manifest_keys_before_publisher_io` + exited 1 with four failures; every duplicate (`source_id`, `package_id`, + `files`, vintage) reached the publisher-read sentinel. Test-only commit: + `56f9ce1`. Ported #227 commit `7f9bfe6`'s `StrictManifestLoader` and + `load_manifest_document` verbatim into `chronicle/registration.py`, switched + artifact manifest reads to it, and shared it with source-package artifact + manifest reads. Fix commit: `77e6fda`. The same focused command now exits 0 + with 4 passed. +- **Findings 3, 4, and 5 reproduced and fixed.** Test-only commit: `264e46e`. + The finding 4 command covering creation beside `manifest.yml`, + `Manifest.yaml`, a mistyped named manifest, and a symlinked manifest exited + 1 with 4 failures, all reaching the publisher-I/O sentinel. The finding 5 + command covering parent traversal plus `*`, `?`, and character-class glob + selectors exited 1 with 4 failures because none raised `ManifestNameError`. + The finding 3 command covering absolute, parent-traversing, and symlinked + artifact paths exited 1 with 3 failures because inventory considered each + path valid (and the local, non-network publisher stub could read it). +- Artifact and manifest inputs are now resolved only as literal package-local + directory entries: unsafe declared filenames return the named + `non_canonical_filename` error, symlinks are refused before reads, sweep + selectors cannot contain separators or glob syntax, and no new manifest + spelling can be created beside an existing registry. Fix commit: `0017520`. + Ported the #227-shaped `is_bare_filename`, `bare_filename`, `filename_key`, + `matching_directory_entry`, and package-manifest helpers from `44e1f8d`, + `7f9bfe6`, and `c2b7722`'s corresponding safety changes; the generalized + registry-creation and exact sweep-selector guards are slice-1 additions. + The focused post-fix command exits 0 with 13 passed, and the complete + artifact module exits 0 with 132 passed. +- **Finding 2 reproduced and fixed.** Red command: + `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_artifacts.py::test_shared_archive_revision_is_refused_through_an_unregistered_owner + tests/test_chronicle_artifacts.py::test_record_revision_updates_every_owner_of_usda_shared_archive + tests/test_chronicle_artifacts.py::test_record_revision_updates_every_same_manifest_owner` + exited 1 with 3 failures: the empty selected vintage bypassed a sibling's + recorded identity, the USDA second manifest retained its old checksum, and + the SSA-style second key retained its old checksum. Test-only commit: + `3636395`. +- Fetch now strictly loads every manifest in the package directory before + publisher I/O, establishes one normalized byte identity for every entry + naming the physical file, and applies the revision guard to all owners. An + explicit revision rewrites every owner from payloads rendered before the + first manifest write, preserves owner-specific metadata, and archives each + owner's own R2 provenance. Ported #227's `_package_manifests` and + `_assert_siblings_record_these_bytes` names/shapes from `c0d9d74`, including + the normalized manifest-alias exclusion from `235c616`; the coordinated + all-owner rewrite is slice-1-specific. Fix/docs commit: `7da26a9`. The red + command now exits 0 with 3 passed; the artifact module exits 0 with 135 + passed. +- **Finding 1 reproduced and fixed.** The whole-tree regression copies tracked + `db/data/**` to `tmp_path`, configures `CHRONICLE_R2_RAW_BUCKET=chronicle-raw`, + and installs a successful non-writing uploader. Red command: + `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_artifacts.py::test_documented_bucket_cutover_sweep_accepts_the_tracked_registry` + exited 1 with the observed tuple `exit=1`, `valid=false`, 156 manifests, 187 + artifacts, 94 skipped, 93 failed, 94 R2-linked, zero uploaded, and five + report errors. Test-only commit: `7f0f473`. +- A syntactically valid recorded R2 locator whose content-addressed tail + matches the package-local checksum and filename is now preserved and skipped + before reconstructing today's country/source/package/year route. This + accepts 84 pre-country-prefix UK keys, eight explicit Statbel 2023 routes, + the shared USDA route, and five fully recorded Eurostat manifests that omit + package IDs, while locator contradictions and byte mismatches remain errors. + Fix commit: `25deb29`; no #227 hunk applies. The same whole-tree command now + exits 0 with 161 manifests, 194 artifacts, 194 skipped/R2-linked, zero + uploaded or failed, and no errors. The artifact module exits 0 with 136 + passed. +- **Finding 7 reproduced and fixed.** Red command: + `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_artifacts.py::test_fetch_refuses_non_list_previous_r2_before_publisher_io` + exited 1 with 3 failures: mapping, scalar, and null `previous_r2` values all + reached the publisher-I/O sentinel. Test-only commit: `e418813`. + `_validated_recorded_storage` now rejects every present non-list history + before `_read_artifact`, so `_superseding_storage` cannot replace malformed + provenance. Fix commit: `8b7ab22`; no #227 hunk applies. The focused command + now exits 0 with 3 passed (5 passed including ordinary and shared revision + history controls). +- **Finding 8 reproduced and fixed.** A fresh-process regression sets each of + `CHRONICLE_SCHEMA`, `POLICYENGINE_LEDGER_SCHEMA`, and `LEDGER_SCHEMA` before + importing the compatibility constants, alongside + `POLICYENGINE_TARGETS_SCHEMA`. Red command: + `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_env.py::test_supabase_schema_compatibility_aliases_honor_import_time_environment` + exited 1 with 3 failures; every subprocess returned `ledger` / `targets`. + Test-only commit: `e2ceedf`. +- `LEDGER_SCHEMA` and `TARGETS_SCHEMA` are now import-time snapshots of the + same lazy resolver functions runtime queries use, restoring their original + environment-backed behavior while later environment mutations remain lazy + through `chronicle_schema()` / `targets_schema()`. Fix commit: `f41ad0f`; no + #227 hunk applies. The focused regression now exits 0 with 3 passed; it and + the runtime/default namespace controls exit 0 with 11 passed. +- **Adversarial path-boundary follow-up reproduced and fixed.** Test-only + commit `465a34c` showed normalized manifest aliases and a package-local + artifact symlink reaching the publisher-read sentinel; invalid sweep names + were accepted when the root was absent and escaped the CLI as tracebacks; + manifest-named artifacts were not refused; and publish could upload a valid + first entry before discovering an invalid later entry. The same audit also + reproduced contradictory owners across sibling manifests being ignored by + both sweeps. +- Fix commit `8e766ad` ports #227 commit `235c616`'s corrected + `_package_manifests` normalized-alias guard, validates fetch destinations + before publisher I/O, rejects manifest-named artifacts in publish and + inventory, validates sweep selectors before checking the root, and reports + those refusals cleanly from both CLIs. It also corrects the object-key docs + to describe compatible recorded history. The six focused path/CLI cases now + pass. Complete-package publish preflight and sibling-owner validation remain + the next coherent fix. +- **Source-package reader follow-up reproduced and fixed.** Test-only commits + `92af48c`, `9a51d99`, `3228238`, and `578c8b2` showed that a package spec + could read an absolute or parent-traversing artifact, follow artifact and + manifest symlinks, accept a normalized filename alias or manifest-named + artifact, and select an unsafe/unsupported manifest path before artifact + I/O. `SourceArtifactSpec` now resolves both manifest and artifact resources + through the shared #227 filename-identity helpers before opening either. + The ten focused cases pass, and the full source-package module passes with + 135 tests (13 warnings). Fix/journal commit: `64fe94b`. +- **Residual side-effect ordering reproduced.** The focused command covering + `test_fetch_refuses_invalid_r2_identity_before_publisher_io` and + `test_publish_preflights_entire_root_before_any_upload` exited 1 with three + failures. Both slash-only R2 identity fields reached the publisher-read + sentinel; a root sweep uploaded and rewrote the valid `a_good` package before + reporting the unsafe filename in `z_bad`. +- Fetch now validates the source/package object-key components before reading + publisher bytes. Raw publish now completes one root-wide read/identity/entry + preflight and returns every refusal before its first uploader call or + manifest rewrite. The three red cases plus the existing entry-, sibling-, + and tracked-cutover preflight controls pass (9 tests). Fix commit: + `4991e8f`. The complete artifact module passes with 154 tests (12 warnings). + +### Next + +None. The eight findings, required #227 ports, adversarial preflight follow-ups, +and final verification are complete. The external `-o out.md` report contains +the per-finding reproduction/fix/test/commit/port map. + +### Final verification (eight-finding round) + +- The first bare `uv run` lint invocations exited 2 before Ruff started because + the sandbox denied access to `/Users/maxghenis/.cache/uv`. Re-running through + the permitted existing cache (`UV_CACHE_DIR=/tmp/chronicle-uv-cache`) gave: + `uv run ruff check .` exit 0 (`All checks passed!`) and `uv run ruff format + --check` on the eight changed Python files exit 0 (`8 files already + formatted`). +- `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p no:cacheprovider`: + direct exit 0, 1,039 passed, 7 skipped, 18 warnings in 1,323.73 seconds. +- Focused artifact module: direct exit 0, 154 passed, 12 warnings in 7.15 + seconds. Focused source-package module: direct exit 0, 135 passed, 13 warnings + in 193.89 seconds. +- `git diff c36f3fc8..HEAD -- db/data` is empty: no tracked source manifest was + modified. + +### Deliberate boundaries + +- Did not add crash-atomic multi-file transactions for an unexpected write or + process failure midway through a coordinated revision. Every deterministic + refusal is preflighted before writes/uploads and the successful path updates + all owners; true cross-file crash atomicity needs a separate staging/rollback + design and is not one of the eight findings. +- Did not port #227's `iter_directory_entries`: its exact implementation depends + on #227's microdata list-entry machinery, which this slice explicitly + excludes. Also retained the current artifact-local `bare_filename` exception + wrapper so filename refusals remain `SourceArtifactManifestError` and the + existing CLI reports them cleanly; #227's later rebase can adopt its broader + exception hierarchy together. +- Did not add a new rule for two physical artifact files whose names normalize + to the same key when the exact requested spelling sorts first. That is + defense-in-depth outside findings 3/4; there are no such collisions in the + tracked tree. Manifest-name collisions are already refused independent of + sort order. + +## Peer round 4 (nine findings) + +### State + +- Started from detached HEAD `fa98993a` with a clean worktree; no branches, + pushes, stashes, GitHub access, or tracked `db/data/**` changes are authorized. +- Scope: the nine supplied provenance, publication preflight, canonical + identity, manifest-vintage, filename, and regular-file findings. +- Preserve the shared helper names used by stacked PR #227; its worktree is + read-only and will not be modified. +- Report path: `/tmp/chronicle-226-round4/out.md` (no explicit runner `-o` + path was provided in the visible request). + +### Done + +- Read the existing journal, storage architecture, role rules, and named code + surfaces. Began reviewing the named test modules and shared registration + helpers. GitNexus debugging guidance is available, but no graph tools are + exposed; use repository search and hermetic regression tests. +- Established this committed state/done/next journal before implementation. + +### Next + +- Add and run failing regressions for each finding before its implementation. +- Fix and commit coherent steps, recording exact red commands and observations + in the external report. +- Run full Ruff lint, formatting checks for changed Python files, and the full + pytest suite with direct exit codes; record counts and final commit map. + +### Round 4 reproduction checkpoint + +- Added publication/tree/identity/alias/sibling regression coverage in + `tests/test_chronicle_artifact_peer4.py` before any associated fix. +- Red command: `UV_CACHE_DIR=/tmp/chronicle-uv-cache uv run pytest -q -p + no:cacheprovider tests/test_chronicle_artifact_peer4.py` (stdout/stderr saved + to `/tmp/chronicle-226-round4/main-red.log`) exited 1: 60 failed, 12 warnings. +- Finding 3: all 4 non-regular tree cases reached build-ID inference before + refusal. Finding 4: all 24 invalid publication identity cases reached an + artifact read or build-ID inference. Finding 5: all 10 noncanonical manifest + declarations reached the publisher read. Finding 6: all 4 single case/Unicode + aliases reached artifact reads. Finding 8: 12 explicit-selector sibling cases escaped as plain + `ValueError` through the functions and harness; 6 top-level CLI tests had + an invocation error (corrected and verified below). +- Consumer guard, source resolver, and shared vintage regressions are being + developed independently; all implementations remain gated on observed red. + +### Finding 2: source provenance reader + +- Reproduced malformed/contradictory/non-R2 locators and checksum/filename + mismatches with 10 failures and 1 passing control before implementation; + exact command and output are in the external report's source evidence. +- `SourceArtifactSpec` now reuses `_validated_recorded_r2`, checks the manifest + identity, and uses the immutable object's digest to validate local/fetched + bytes even without a separately declared checksum. Invalid metadata is + refused before artifact/cache I/O; bad fetched bytes before cache writes. +- The same focused command now exits 0: 11 passed, 12 warnings. Inventory's + half of finding 2 remains next; no source package or source data was changed. + +### Finding 5: exact manifest declarations + +- Present `source_id` and `package_id` now pass `_require_identity_segment` + without stripping or stringification and must equal the fetch argument. + Absent fields remain eligible for first registration; null/empty fields do + not impersonate absence. +- Focused regressions and existing other-package refusal controls passed: + direct exit 0, 12 passed. The red checkpoint above recorded all 10 new + declaration cases reaching publisher I/O before this fix. + +### Finding 7: shared logical vintage validation + +- Before implementation, the new manifest-vintage tests exited 1: 5 failed, + 1 passed. Duplicate keys `2024` and `"2024"` reached artifact I/O through + publish, inventory, source loading, and fetch of another selected vintage. +- `load_manifest_document` now invokes shared `validate_manifest_vintages` + over the entire manifest, so every consumer refuses a logical duplicate + through its existing controlled YAML error path. Shared names are unchanged. +- Six regressions plus existing quoted-year compatibility tests exit 0: + 8 passed, 12 warnings. Scoped Ruff lint/format checks both pass. + +### Finding 1: configured derived provenance + +- Consumer guard regressions ran red before implementation: direct exit 1, + 35 failed and 1 passed; configured bucket/prefix provenance did not produce + `derived_fact_provenance`. Exact evidence is in the external report. +- The guard resolves derived bucket/prefix configuration lazily, retains exact + archived routes, and checks raw locator fields, source-file locators, and + source URL. Added `default_r2_derived_prefix` with the standard environment + lookup ladder so publication and the guard can share it. +- Full consumer-contract module passes: direct exit 0, 95 passed, 32 warnings; + scoped Ruff checks pass. Wiring configured prefixes into publication and + refusing unrecognizable explicit routes remains a publication follow-up. + +### Finding 9: regular source resources + +- Directory and FIFO manifest/artifact regressions ran red: direct exit 1, + 4 failed, 2 passed. The resolver returned non-regular manifests and reached + artifact-read sentinels; no test opened a FIFO. +- The source resolver now requires an existing selected entry to be a regular + file after symlink and spelling checks. Missing artifact entries still use + the existing checksum-validated cache/fetch path; zip-backed resources work. +- Same focused command exits 0: 6 passed, 12 warnings. The full source-package + module is running. Exact red/green commands are in the external evidence. + +### Finding 8: controlled sibling-manifest refusals + +- `_package_manifests` now translates discovery's `ValueError` into + `MalformedManifestError`, so publish/inventory and both CLI entry points + use the shared controlled refusal path even with an explicit selector. +- The first post-fix command actually exited 1: 12 passed and 6 top-level + CLI cases failed because the test passed argv to a zero-argument entry point. + Corrected the test to set sys.argv and assert SystemExit. The corrected + command exits 0: 18 passed, 12 warnings; no uploads or rewrites occur. + The earlier premature passing-count journal entry is corrected here. + +### Additional round 4 red checkpoint + +- Added inventory locator/identity regressions, explicit raw identity overrides, + root/nested/excluded-registry derived symlinks, and derived prefix publication + integration before fixing those paths. Focused `additional-red.log` command + exited 1: 24 failed, 1 passed, 60 deselected, 14 warnings. +- Finding 2 inventory: six malformed/contradictory identity cases reached + artifact reads; absent separate checksum allowed wrong local bytes as a valid + R2 link. Valid URI-only locator control passed. +- Finding 3 additional tree cases reached upload or silently skipped the link. + Finding 4 explicit overrides reached reads. Finding 1 both-custom explicit + route reached inference; all three prefix environment names were ignored by + publication. Exact command and observations are in the external report. +- Full source-package module completed: direct exit 0, 157 passed, 13 warnings + in 192.07 seconds, including both source-reader fixes. + +### Finding 3: complete derived tree preflight + +- Derived publication now walks the root and every descendant with `lstat`, + allowing only directories and regular files. It refuses symlinks/FIFOs, + including an excluded registry filename, before build-ID inference, file + reads, uploads, or registry writes. +- Both failing-first tree groups now pass: direct exit 0, 7 passed, + 78 deselected, 12 warnings. Safe nested directories remain publishable. + +### Finding 6: exact physical artifact spelling + +- Publish and inventory now turn a single normalized alias with different + physical spelling into `artifact_spelling_mismatch`, and neither reads + bytes after alias errors. Duplicate-alias refusal also stops before reads. +- Case/Unicode regressions plus duplicate/symlink controls exit 0: 6 passed, + 12 warnings. Missing filenames retain their separate missing-name error. + +### Finding 4: canonical identities on new publication paths + +- Derived publication validates source/package identifiers before inference or + artifact reads. Raw publication preserves original argument/declaration types + until new-key validation, rejects noncanonical identifiers and contradictory + declarations before reading artifacts, and retains root-wide preflight. +- Identity regressions plus declaration controls exit 0: 44 passed, + 41 deselected, 12 warnings. Six historical-identity controls and the full + tracked-registry cutover test also pass: 7 passed, 12 warnings. Recorded + objects keep their original routes even when old manifests omit package_id. + +### Finding 2: inventory provenance + +- Inventory now reuses `_validated_recorded_r2` before artifact reads, rejects + contradictory/incomplete/non-R2 locators and mismatched checksum/filename + declarations, and checks actual bytes against the recorded digest even when + a separate checksum is absent. Invalid entries expose no R2 link. +- Seven failing-first defect cases plus the valid URI-only control now pass: + direct exit 0, 8 passed, 83 deselected, 12 warnings. Together with `d5e9952`, + both consumers named in finding 2 now enforce immutable provenance. + +### Finding 1: publication route enforcement + +- Publication and the consumer guard now share `is_derived_r2_route` and the + lazy prefix resolver. Derived key generation honors all three environment + spellings for the configured prefix. +- A both-custom explicit bucket/prefix combination must identify a configured + or archived derived route, otherwise publication refuses it before build + reads. The storage architecture documents the shared route configuration. +- Four failing-first route integration cases now pass: direct exit 0, + 4 passed, 87 deselected, 16 warnings. All nine findings are implemented; + focused integration verification and full final checks remain. + +### Final review follow-ups + +- Focused integration passed: direct exit 0, 409 passed, 40 warnings in 15.10s. + Ruff lint passed and all 8 changed Python files passed formatting checks. +- Interrupted the first full pytest run (direct exit 130) after peer review + found that date/set YAML identity refusals could fail CLI JSON serialization. + It is not a completed full-suite verification and will be restarted. +- Reproduced a remaining configured-route bypass with uppercase `R2://` in + raw URI, source_file, or source URL: 3 failed, 3 lowercase controls passed. + URI scheme matching is now case-insensitive. Full consumer module now passes: + direct exit 0, 101 passed, 32 warnings; scoped Ruff checks pass. +- Additional failing-first checks cover date/set report serialization (8 failed), + vintage namespace escapes and build-ID separators (8 failed), and fetch + vintage namespace escapes. These publication fixes precede the full restart. + +- Committed the publication follow-up regressions before their fixes. Red + commands/logs: `report-identity-red.log` (8 failed), + `namespace-followup-red.log` (8 failed, 99 deselected), and + `fetch-vintage-red.log` (3 failed); all direct exits 1. CLI refusals hit + non-JSON date/set values; malformed vintage/build segments reached reads or + upload sentinels. Exact commands will be included in the final report. + +- Raw publication now keeps original identity values for validation and uses + separate string fields only in reports. Date/set refusals serialize cleanly + through both CLIs; historical skips also retain serializable identity fields. + The regression and history controls exit 0: 14 passed, 12 warnings. + +- New publication vintage segments and derived build-ID segments now pass the + same canonical segment validator, preventing API-provided slashes or '..' + from moving an otherwise recognized route. Fetch applies the vintage check + before publisher I/O; historical raw objects retain their existing routes. + All 11 failing-first namespace cases pass (99 deselected, 12 warnings). + +### Final verification in progress + +- Final focused integration exits 0: 434 passed, 40 warnings in 11.11s. +- Final `uv run ruff check .` exits 0 (`All checks passed!`). Final + `uv run ruff format --check` on the 8 changed Python files exits 0 + (`8 files already formatted`). Both use the permitted UV cache. +- `git diff --check fa98993a..HEAD` and the tracked `db/data` no-change check + both exit 0. Worktree clean before this journal update. +- Restart the full suite at code commit `19d037f` with UV offline, offline + OpenTimestamps client selection, and live Supabase credentials removed from + the child environment. This keeps the required full run free of network. + +### Final state, done, and next (peer round 4) + +**State:** complete. All nine findings and the peer-review follow-ups are fixed +in small detached-HEAD commits. Final code commit: `19d037f`; the completed +full run included the subsequent journal commit `9fa6a1d`. + +**Done:** + +- Full `uv run pytest -q -p no:cacheprovider` returned direct exit 0: + **1,238 passed, 7 skipped, 42 warnings in 1,468.40s (24m 28s)**. + The child environment used the permitted UV cache, `UV_OFFLINE=1`, an offline + OpenTimestamps client command, and no live Supabase URL/service/secret keys. + Exact invocation and output are in `/tmp/chronicle-226-round4/out.md` and + `/tmp/chronicle-226-round4/full-pytest.log`. +- Final whole-repository Ruff lint exited 0. Formatting checks on every changed + Python file exited 0: 8 files already formatted. Final focused integration + also exited 0: 434 passed, 40 warnings. +- No tracked `db/data/**` changes, GitHub/network access, pushes, branches, + stashes, or edits to PR #227's worktree. Shared helper names remain intact. +- The external report contains each finding's red command and observed failure, + fix, regression names, and commit SHA, plus the final verification and + deliberate boundaries. It records the corrected CLI-test invocation and the + interrupted first full run without counting either as passing verification. + +**Next:** none in this fix lane. No push or branch operation is authorized. diff --git a/README.md b/README.md index f75fe697..3574847a 100644 --- a/README.md +++ b/README.md @@ -65,10 +65,12 @@ contract that aligns it to another period (see | Microcosm Target Contracts | Selection, measurement bindings, and active subset | Period alignment, support-aware activation, solver inputs, diagnostics | The storage split is documented in -[`docs/storage-architecture.md`](docs/storage-architecture.md): `ledger-raw` -stores immutable source bytes, `ledger-derived` stores reproducible build +[`docs/storage-architecture.md`](docs/storage-architecture.md): a raw R2 archive +stores immutable source bytes, a derived R2 archive stores reproducible build artifacts, and Supabase/Postgres hosts the queryable relational Chronicle registry -mirrored from accepted builds. +mirrored from accepted builds. The bucket names are configuration +(`$CHRONICLE_R2_RAW_BUCKET` and `$CHRONICLE_R2_DERIVED_BUCKET`), still defaulting +to the ledger-era `ledger-raw` and `ledger-derived`. ## Repository Model @@ -247,7 +249,7 @@ This writes: source_regions.jsonl facts.jsonl consumer_facts.jsonl - ledger.db + chronicle.db reports/ source_rows.json source_cells.json @@ -349,11 +351,14 @@ needed, even when your Cloudflare user belongs to several accounts: # One-time per machine (opens a browser consent page): bunx wrangler login -# One-time per account (already done for the PolicyEngine account): -uv run chronicle bootstrap-r2 --raw-bucket ledger-raw --derived-bucket ledger-derived +# One-time per account (already done for the PolicyEngine account). The bucket +# flags default to $CHRONICLE_R2_RAW_BUCKET / $CHRONICLE_R2_DERIVED_BUCKET: +uv run chronicle bootstrap-r2 # Fetch/register a source artifact, write db/data/.../manifest.yaml, and upload -# the exact bytes to R2 when Wrangler is authenticated: +# the exact bytes to R2 when Wrangler is authenticated. Pass --manifest when the +# package directory keeps more than one manifest (ira_contributions keeps a +# traditional and a Roth one): uv run chronicle fetch-artifact \ --url https://www.irs.gov/pub/irs-soi/23in12ms.xls \ --source-id irs_soi \ @@ -364,11 +369,20 @@ uv run chronicle fetch-artifact \ --table "Publication 1304 Table 1.2" \ --upload-r2 +# Re-fetching is safe: identical bytes keep the recorded storage.r2 block, and +# bytes that disagree with what the entry identifies -- its declared sha256, or +# its recorded content-addressed key once published -- are refused. When a +# publisher has re-published +# under the same URL and vintage, register the revision explicitly — the new +# bytes get their own content-addressed key and the superseded object is kept +# in storage.previous_r2: +uv run chronicle fetch-artifact ... --record-revision + # Audit local manifests and checksums: uv run chronicle inventory-artifacts --root db/data -# Upload all existing manifest-declared local artifacts to ledger-raw and write -# storage.r2 metadata back into the manifests: +# Upload all existing manifest-declared local artifacts to the raw archive and +# write storage.r2 metadata back into the manifests: uv run chronicle publish-raw --root db/data ``` @@ -406,10 +420,10 @@ To prepare the deterministic SQLite artifact for a hosted Supabase/Postgres mirror, export each relational table to JSONL plus a manifest: ```bash -uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace +uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace ``` -To publish the deterministic build outputs to the `ledger-derived` R2 bucket: +To publish the deterministic build outputs to the derived R2 archive: ```bash uv run chronicle publish-derived \ @@ -420,13 +434,14 @@ uv run chronicle publish-derived \ --build-artifacts-out /tmp/chronicle-build-artifacts.jsonl ``` -The Supabase schema for this mirror lives at -`supabase/migrations/20260504_chronicle_bronze.sql`. Raw government spreadsheets are -mirrored as artifact metadata plus one row per parsed cell, not one tidy table -per sheet. Chronicle does not host raw survey microdata tables. +Before loading, create and apply a Supabase/Postgres migration that creates the +mirror tables in the schema selected for the load, then expose that schema +through the Supabase Data API. Raw government spreadsheets are mirrored as +artifact metadata plus one row per parsed cell, not one tidy table per sheet. +Chronicle does not host raw survey microdata tables. -After the migration is applied and the `chronicle` schema is exposed through the -Supabase Data API, accepted mirror exports can be upserted with: +After that deployment migration is applied, accepted mirror exports can be +upserted with: ```bash uv run chronicle load-supabase-mirror \ @@ -434,9 +449,21 @@ uv run chronicle load-supabase-mirror \ --build-artifacts /tmp/chronicle-build-artifacts.jsonl ``` +With no schema environment override and no `--schema`, this command +writes to `ledger`. To load a migrated `chronicle` schema instead, set +`CHRONICLE_SCHEMA=chronicle` or pass `--schema chronicle`. + Use `--dry-run` first to validate JSONL row counts and file coverage without writing to Supabase. +Chronicle settings are read chronicle-first: `CHRONICLE_X` wins, and the +ledger-era `POLICYENGINE_LEDGER_X` and `LEDGER_X` spellings still work behind a +one-time deprecation warning naming the variable to move to. +[`docs/storage-architecture.md`](docs/storage-architecture.md#environment-variable-rename-window) +lists every variable in that window, and +[Bucket Cutover](docs/storage-architecture.md#bucket-cutover) covers the R2 +bucket rename. + Chronicle facts keep source concepts and canonical concepts separately. For example, the SOI Table 1.1 adjusted gross income column is preserved as `irs_soi.adjusted_gross_income`, while the canonical concept is diff --git a/chronicle/__init__.py b/chronicle/__init__.py index 04d50943..e3ddfb06 100644 --- a/chronicle/__init__.py +++ b/chronicle/__init__.py @@ -13,6 +13,7 @@ "consumer_contract", "core", "database", + "env", "facts", "harness", "jurisdictions", diff --git a/chronicle/artifacts.py b/chronicle/artifacts.py index fa56d620..876d9730 100644 --- a/chronicle/artifacts.py +++ b/chronicle/artifacts.py @@ -9,23 +9,368 @@ from datetime import UTC, datetime from pathlib import Path import posixpath +import re import shlex import sqlite3 +import stat import subprocess -from typing import Any +from typing import Any, Mapping from urllib.parse import unquote, urlparse import httpx import yaml +from chronicle.database import ( + CHRONICLE_DB_FILENAME, + CHRONICLE_DB_FILENAMES, + LEGACY_CHRONICLE_DB_FILENAME, +) +from chronicle.env import env_value from chronicle.epoch import EMIT_EPOCH, Epoch, canonicalize_key, hash_domain - - +from chronicle.registration import ( + filename_key, + is_bare_filename, + is_manifest_filename, + load_manifest_document, + matching_directory_entry, + package_manifest_paths, + validate_package_directory, +) + + +R2_RAW_BUCKET_ENV = "CHRONICLE_R2_RAW_BUCKET" +R2_DERIVED_BUCKET_ENV = "CHRONICLE_R2_DERIVED_BUCKET" + +# The bucket defaults stay at their ledger-era names. Archived witness records +# pin raw R2 URLs by hash, so ledger-raw and ledger-derived are preserved +# read-only forever and no recorded manifest URI is ever rewritten. The env +# vars exist so the cutover in docs/storage-architecture.md can be rehearsed, +# and so flipping to chronicle-raw/chronicle-derived is a default change rather +# than a code change (PolicyEngine/chronicle#143, mechanism 3). DEFAULT_R2_RAW_BUCKET = "ledger-raw" DEFAULT_R2_DERIVED_BUCKET = "ledger-derived" DEFAULT_R2_PREFIX = "raw" DEFAULT_R2_DERIVED_PREFIX = "derived" +# Most packages keep one manifest.yaml. Publisher directories that feed several +# source packages keep one manifest each -- db/data/irs_soi/ira_contributions +# holds manifest_traditional_source_package.yaml beside the Roth one -- so the +# name is an input, not a constant, wherever a caller addresses a package. +DEFAULT_MANIFEST_FILENAME = "manifest.yaml" + + +def bare_filename(value: Any, *, what: str = "filename") -> str: + """Return ``value`` as a bare filename, refusing any other spelling. + + ``./table.csv``, ``nested/table.csv`` and an absolute path all resolve + outside the one-name manifest contract once joined under a package, so + fetch validates the spelling before reading publisher bytes. + """ + if not is_bare_filename(value): + raise ArtifactFilenameError( + f"{what} must be a bare filename inside the package directory, not " + f"{value!r}; it may not carry a directory, '.', '..', a trailing " + "slash, surrounding whitespace, or an absolute path." + ) + return str(value) + + +def _root_manifest_paths(root: Path, manifest_filename: str) -> list[Path]: + """Return the manifests a root sweep addresses. + + The default is package discovery, not one literal filename: both YAML + extensions and every ``manifest_`` sibling participate. A caller + that supplies another filename keeps the historical exact-name override. + """ + selected_name = _manifest_path(Path(), manifest_filename).name + if selected_name != DEFAULT_MANIFEST_FILENAME: + candidates = [path for path in root.rglob("*") if path.name == selected_name] + else: + candidates = [ + path for path in root.rglob("*") if is_manifest_filename(path.name) + ] + for path in candidates: + _require_regular_manifest_file(path) + return sorted(candidates) + + +def _require_regular_manifest_file(path: Path) -> None: + """Refuse a manifest-named entry that is not a regular, non-symlink file. + + ``is_file`` follows symlinks, so a dangling symlink, a symlink to a + directory, or any other non-regular entry would silently vanish from a + sweep and from sibling-registry checks; a registry entry that cannot be + read as a manifest is a defect to surface, never to skip. + """ + if path.is_symlink() or not path.is_file(): + raise MalformedManifestError( + f"{path} carries a manifest name but is not a regular file; " + "Chronicle will not sweep past it or register beside it." + ) + + +def _manifest_path(output: Path, manifest_filename: str) -> Path: + """Return the named manifest inside ``output``. + + The name is a filename, not a path: it selects among the manifests a + package directory keeps, and must not reach outside it. + """ + name = str(manifest_filename) + if not is_bare_filename(name) or any(character in name for character in "*?[]"): + raise ManifestNameError( + "Manifest must name a file inside the package directory, not " + f"{manifest_filename!r}." + ) + if not is_manifest_filename(name): + raise ManifestNameError( + f"Manifest must be named {DEFAULT_MANIFEST_FILENAME} or " + f"manifest_.yaml, not {manifest_filename!r}: the sweeps " + "address a package's manifests by those names, and a manifest " + "under any other name is invisible to them." + ) + return output / name + + +def _refuse_a_stray_default_manifest(output: Path, manifest_path: Path) -> None: + """Refuse to create any new manifest beside a package's registry. + + Every supported spelling participates: a missing ``manifest.yaml`` beside + ``manifest.yml`` or ``Manifest.yaml`` is just as ambiguous as one beside a + named manifest, and a mistyped named selector must not create a parallel + registry. Operators may create an intentional empty sibling explicitly, + then select that existing file. + """ + paths = package_manifest_paths(output) + if ( + any(path.name == manifest_path.name for path in paths) + or manifest_path.is_symlink() + ): + return + siblings = [path.name for path in paths] + if not siblings: + return + raise AmbiguousManifestError( + f"{output} already keeps {', '.join(siblings)}; refusing to create " + f"{manifest_path.name} beside that registry. Pass --manifest to name " + "an existing manifest, or create an intentional empty sibling " + "explicitly before fetching into it." + ) + + +def _package_manifests( + output: Path, + manifest_path: Path, + existing_manifest: dict[str, Any], +) -> dict[str, dict[str, Any]]: + """Return every manifest the package directory keeps, by path. + + The byte boundary is the file in the directory, not whichever manifest a + fetch selected. A malformed sibling is therefore a pre-I/O refusal: until + Chronicle can read every owner, it cannot safely overwrite shared bytes. + """ + try: + paths = package_manifest_paths(output) + except ValueError as error: + # Explicit sweep selectors still inspect every sibling registry. Keep + # discovery refusals in the shared exception family both CLIs report. + raise MalformedManifestError(str(error)) from error + by_name: dict[str, Path] = {} + for path in paths: + key = filename_key(path.name) + previous = by_name.get(key) + if previous is not None and previous != path: + raise AmbiguousManifestError( + f"{previous} and {path} have the same normalized manifest name " + f"{key!r}. Physically distinct manifest aliases can hide one " + "another's registrations; keep exactly one spelling." + ) + by_name[key] = path + selected_alias = by_name.get(filename_key(manifest_path.name)) + if selected_alias is not None and selected_alias != manifest_path: + raise AmbiguousManifestError( + f"{manifest_path} and existing {selected_alias} have the same " + "normalized manifest name. Selecting one spelling would hide the " + "other's registrations; address the existing manifest or remove " + "the duplicate." + ) + + manifests: dict[str, dict[str, Any]] = {str(manifest_path): existing_manifest} + for path in paths: + if path == manifest_path: + continue + sibling = _read_manifest(path) + _manifest_files(sibling, path) + manifests[str(path)] = sibling + return manifests + + +def _manifest_files(payload: dict[str, Any], manifest_path: Path) -> dict[str, Any]: + """Return a manifest's ``files`` block, refusing one that is not a mapping. + + ``inventory-artifacts`` and ``publish-raw`` report the same document as + ``files must be a mapping``; a fetch must refuse it before reading the + publisher, or the write fails only after the local artifact has been + overwritten and any upload has run. + """ + files = payload.get("files") + if files is None: + # A bare ``files:`` line parses as None: no entries, like an absent + # block. The writer normalizes it to a mapping before recording into it. + return {} + if not isinstance(files, dict): + raise MalformedManifestError( + f"{manifest_path} files must be a mapping; it parses as a " + f"{type(files).__name__}. Chronicle will not overwrite a manifest " + "it cannot read." + ) + return files + + +def _assert_manifest_identifies( + existing_manifest: dict[str, Any], + manifest_path: Path, + *, + source_id: str, + package_id: str, +) -> None: + """Refuse to fetch into a manifest that identifies another package. + + The R2 key and registration identity are built from the fetch arguments. + Recording them in a manifest that declares different identifiers would + leave one entry making two incompatible provenance claims. + """ + for field, value in (("source_id", source_id), ("package_id", package_id)): + if field not in existing_manifest: + continue + declared = _require_identity_segment( + existing_manifest[field], what=f"{manifest_path} {field}" + ) + if declared != value: + raise SourceArtifactManifestError( + f"{manifest_path} declares {field}={declared!r}; refusing to " + f"fetch {field}={value!r} into it. Fetch into the package the " + "manifest identifies, or into that package's own directory." + ) + + +def default_r2_raw_bucket() -> str: + """Resolve the raw bucket: ``$CHRONICLE_R2_RAW_BUCKET`` or the default.""" + return env_value(R2_RAW_BUCKET_ENV, default=DEFAULT_R2_RAW_BUCKET) + + +def default_r2_derived_bucket() -> str: + """Resolve the derived bucket: ``$CHRONICLE_R2_DERIVED_BUCKET`` or default.""" + return env_value(R2_DERIVED_BUCKET_ENV, default=DEFAULT_R2_DERIVED_BUCKET) + + +def default_r2_derived_prefix() -> str: + """Resolve the derived route shared by publication and fact refusals.""" + return env_value("CHRONICLE_R2_DERIVED_PREFIX", default=DEFAULT_R2_DERIVED_PREFIX) + + +def is_derived_r2_route(bucket: str, key: str) -> bool: + """Whether an R2 bucket/key pair addresses derived build output. + + Resolve publication configuration at validation time: an operator may use + a bucket or prefix with no ``derived`` marker in its spelling. Archived + rename-window routes remain derived after the active destination changes. + """ + derived_buckets = { + "ledger-derived", + "chronicle-derived", + DEFAULT_R2_DERIVED_BUCKET, + default_r2_derived_bucket(), + } + derived_prefixes = { + "derived", + resolve_r2_prefix( + prefix=None, + default_prefix=DEFAULT_R2_DERIVED_PREFIX, + ), + resolve_r2_prefix( + prefix=None, + default_prefix=default_r2_derived_prefix(), + ), + } + # Configured routes extend the boundary; they never narrow it. A bucket + # ending in ``-derived`` (any case) or a ``derived/`` key was derived + # before routes became configurable, and archived facts still cite such + # routes, so the legacy spelling rule stays alongside the configured set. + return ( + bucket in derived_buckets + or bucket.casefold().endswith("-derived") + or any( + key == prefix or key.startswith(f"{prefix}/") for prefix in derived_prefixes + ) + ) + + +class SourceArtifactManifestError(RuntimeError): + """A manifest refuses the write a fetch is about to make. + + The checks that raise these run before the publisher is read, so an + ordinary refusal costs nothing and leaves the package exactly as it was. + They are repeated immediately before the manifest is rewritten, so no + caller can reach a false-provenance write by another route. + """ + + +class SourceArtifactRevisionError(SourceArtifactManifestError): + """Fetched bytes are not the bytes the manifest entry identifies. + + A manifest entry identifies specific bytes: by its declared ``sha256``, and + -- once published -- by a content-addressed R2 key that repeats them. When + a publisher re-publishes under the same URL and vintage, rewriting that + entry would attach its provenance, and any recorded URI, to bytes it never + described. Chronicle refuses instead: same vintage plus new bytes is a new + release revision (docs/adr-chronicle-fact-identity-v2.md), registered with + ``fetch-artifact --record-revision``. + """ + + +class ManifestNameError(SourceArtifactManifestError, ValueError): + """A manifest name is not a bare filename inside the package directory. + + Also a :class:`ValueError` for callers that validated the name that way + before the CLI learned to report it as an ordinary manifest refusal. + """ + + +class ArtifactFilenameError(SourceArtifactManifestError, ValueError): + """An artifact filename is not a bare, non-manifest package filename.""" + + +class IdentitySegmentError(SourceArtifactManifestError, ValueError): + """A registration identity (source_id / package_id) is not one canonical + R2 key segment.""" + + +class AmbiguousManifestError(SourceArtifactManifestError): + """The default manifest name would create a manifest beside the ones a + package already keeps (PolicyEngine/chronicle#225).""" + + +class MalformedManifestError(SourceArtifactManifestError): + """A manifest document, or a block inside one, is not a mapping. + + Reading such a file as an absent manifest would let a fetch replace it with + a single entry, dropping whatever the unreadable document recorded. + """ + + +class RecordedR2LocatorError(SourceArtifactManifestError): + """A recorded ``storage.r2`` block does not locate exactly one object. + + The provider and URI must explicitly identify R2. ``provider``, ``bucket``, + ``key`` and ``uri`` all describe the same object, so any additional fields + have to agree, and the key has to carry the ``{sha256}/{filename}`` tail + that says which bytes it holds. A block whose fields contradict each other + has no single answer to "which bytes does this entry claim R2 holds", and + preserving or publishing under it would ship whichever field the reader + happened to consult. + """ + + # New UK and New Zealand uploads are namespaced by country. US objects predate # the country segment and deliberately keep their legacy ``raw/{source_id}`` # and ``derived/{source_id}`` shapes. Publisher directories are the stable @@ -75,6 +420,70 @@ def to_dict(self) -> dict[str, str]: } +# A raw key ends in {sha256}/{filename} (see build_r2_key), so the segment +# before the filename is what says which bytes the object holds. +_SHA256_KEY_SEGMENT = re.compile(r"[0-9a-f]{64}") + + +@dataclass(frozen=True) +class RecordedR2Object: + """The R2 object a manifest entry's ``storage.r2`` block claims exists. + + Built only by :func:`_validated_recorded_r2`, so every instance names one + object whose locator fields agree with each other. + """ + + provider: str + bucket: str + key: str + sha256: str + filename: str + + @property + def uri(self) -> str: + """Return the storage URI the recorded fields spell out.""" + return f"{self.provider}://{self.bucket}/{self.key}" + + +@dataclass(frozen=True) +class RecordedIdentity: + """The bytes a manifest entry says its vintage currently holds. + + From the recorded object's content-addressed key once the entry has been + published, and from the entry's own declared ``sha256``/``filename`` before + that. ``r2`` is None in the second case: protection does not wait for an + upload to have happened. + """ + + sha256: str + filename: str + size_bytes: int | None + declared_sha256: str | None + r2: RecordedR2Object | None + + def holds(self, *, sha256: str, filename: str) -> bool: + """Whether this identity is exactly the given bytes under that name. + + The filename participates only when the entry records one: a published + key always carries it, an entry that declares bytes and no name does + not, and inventing a mismatch there would refuse a re-fetch of the very + bytes the entry describes. + """ + if self.sha256 != sha256: + return False + return not self.filename or self.filename == Path(filename).name + + +@dataclass(frozen=True) +class _ManifestFileOwner: + """One manifest entry that names a package-local artifact.""" + + manifest_path: Path + vintage: Any + spec: dict[str, Any] + identity: RecordedIdentity | None + + @dataclass(frozen=True) class ArtifactCommandResult: """Result from a storage command.""" @@ -221,16 +630,24 @@ class RawArtifactPublishEntry: r2_location: ArtifactStorageLocation | None upload: ArtifactCommandResult | None errors: tuple[str, ...] = () + skipped: str | None = None + + @property + def uploaded(self) -> bool: + """Whether this run uploaded the artifact.""" + return self.upload is not None and self.upload.ok @property def valid(self) -> bool: - """Whether this raw artifact uploaded and was registered.""" - return not self.errors and self.upload is not None and self.upload.ok + """Whether this raw artifact is published: uploaded now, or already + held by the recorded object in a preserved bucket (``skipped``).""" + return not self.errors and (self.skipped is not None or self.uploaded) def to_dict(self) -> dict[str, Any]: """Return a JSON-serializable entry.""" return { "valid": self.valid, + "skipped": self.skipped, "manifest_path": self.manifest_path, "source_id": self.source_id, "package_id": self.package_id, @@ -265,7 +682,10 @@ def counts(self) -> dict[str, int]: return { "manifest_count": len(manifest_paths), "artifact_count": len(self.entries), - "uploaded_count": sum(1 for entry in self.entries if entry.valid), + "uploaded_count": sum(1 for entry in self.entries if entry.uploaded), + "skipped_count": sum( + 1 for entry in self.entries if entry.skipped is not None + ), "failed_count": sum(1 for entry in self.entries if not entry.valid), "r2_link_count": sum( 1 for entry in self.entries if entry.r2_location is not None @@ -391,33 +811,166 @@ def fetch_source_artifact( source_page: str | None = None, table: str | None = None, filename: str | None = None, + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, upload_r2: bool = False, - r2_bucket: str = DEFAULT_R2_RAW_BUCKET, + record_revision: bool = False, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", ) -> ArtifactFetchReport: - """Fetch/register a source artifact and optionally upload it to R2.""" + """Fetch/register a source artifact and optionally upload it to R2. + + ``manifest_filename`` names the manifest inside ``output_dir`` the entry + belongs to. Packages that split one publisher directory across several + source packages keep one manifest each, so a fetch that always wrote + ``manifest.yaml`` would write a fresh manifest beside the real ones and + never see the entry it is revising. + + ``record_revision`` opts into registering a publisher revision: the fetched + bytes get their own content-addressed key under the configured bucket and + the superseded object moves to ``storage.previous_r2``. Without it, bytes + that disagree with the entry's recorded identity raise + :class:`SourceArtifactRevisionError` before anything is overwritten. + """ + r2_bucket = r2_bucket or default_r2_raw_bucket() output = Path(output_dir) + manifest_path = _manifest_path(output, manifest_filename) + what = ( + "--filename" if filename is not None else "The filename inferred from the URL" + ) + artifact_filename = bare_filename( + filename if filename is not None else _infer_artifact_filename(source_url), + what=what, + ) + if is_manifest_filename(artifact_filename): + raise ArtifactFilenameError( + f"{what} {artifact_filename!r} is a manifest name. An artifact may " + "not be named like a manifest, which it would overwrite; pass " + "--filename with the publisher's name for the bytes." + ) + # These fields become object-key path segments even when this fetch does + # not upload. Validate them before reading the publisher so a malformed + # registration identity cannot overwrite package-local bytes and fail only + # when the prospective R2 key is constructed below. + _require_identity_segment(source_id, what="source_id") + _require_identity_segment(package_id, what="package_id") + _require_identity_segment(str(year), what="year") resolved_r2_prefix = resolve_r2_prefix( prefix=r2_prefix, default_prefix=DEFAULT_R2_PREFIX, source_id=source_id, package_path=output, ) + # Read and validate the entry being written before anything is fetched: a + # manifest Chronicle cannot read, or a recorded block that names two + # different objects, is a refusal that need not touch the publisher. + try: + _refuse_a_stray_default_manifest(output, manifest_path) + except SourceArtifactManifestError: + raise + except ValueError as error: + # package_manifest_paths refuses non-regular manifest-named entries + # with a plain ValueError; surface it as the manifest error the CLI + # reports rather than a traceback. + raise MalformedManifestError(str(error)) from error + existing_manifest = _read_manifest(manifest_path) + _manifest_files(existing_manifest, manifest_path) + _assert_manifest_identifies( + existing_manifest, + manifest_path, + source_id=source_id, + package_id=package_id, + ) + vintage_key, _existing_value, selected_spec, _index = _select_vintage_entry( + existing_manifest, + manifest_path=manifest_path, + year=year, + ) + recorded_identity = _recorded_identity( + selected_spec, + manifest_path=manifest_path, + year=vintage_key, + ) + manifests = _package_manifests(output, manifest_path, existing_manifest) + owners = _manifest_file_owners( + manifests, + filename=artifact_filename, + initializing=(manifest_path, vintage_key), + ) + _assert_shared_owner_identities_agree(owners, filename=artifact_filename) + + try: + existing_target = matching_directory_entry(output, artifact_filename) + except ValueError as error: + raise ArtifactFilenameError(str(error)) from error + if existing_target is not None: + if existing_target.is_symlink(): + raise ArtifactFilenameError( + f"{existing_target} is a symbolic link. Chronicle will not " + "fetch through a package-local link or overwrite its target." + ) + if existing_target.name != artifact_filename: + raise ArtifactFilenameError( + f"{existing_target} has the same normalized filename as " + f"{artifact_filename!r}. Chronicle will not create a " + "physically distinct alias; pass --filename " + f"{existing_target.name!r}." + ) + if not existing_target.is_file(): + raise ArtifactFilenameError( + f"{existing_target} exists but is not a regular file. " + "Chronicle will not overwrite it with publisher bytes." + ) + fetched_at = datetime.now(UTC).replace(microsecond=0).isoformat() - content, inferred_filename = _read_artifact(source_url) - artifact_filename = filename or inferred_filename - if not artifact_filename: - raise ValueError("Could not infer artifact filename; pass --filename.") + content, _inferred_filename = _read_artifact(source_url) + + sha256 = hashlib.sha256(content).hexdigest() + size_bytes = len(content) + + # Guard before the cached artifact is touched. A rejected fetch must leave + # the recorded bytes and their manifest entry exactly as they were. + _assert_recorded_identity_holds_these_bytes( + recorded_identity, + manifest_path=manifest_path, + year=vintage_key, + filename=artifact_filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + record_revision=record_revision, + ) + if not record_revision: + _assert_siblings_record_these_bytes( + manifests, + manifest_path=manifest_path, + vintage=vintage_key, + filename=artifact_filename, + sha256=sha256, + ) + for owner in owners: + # The selected owner is (manifest, vintage key): a YAML anchor can make + # two vintages share one dict object, and object identity would skip + # both. + if owner.manifest_path == manifest_path and str(owner.vintage) == str( + vintage_key + ): + continue + _assert_recorded_identity_holds_these_bytes( + owner.identity, + manifest_path=owner.manifest_path, + year=owner.vintage, + filename=artifact_filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + record_revision=record_revision, + ) output.mkdir(parents=True, exist_ok=True) local_path = output / artifact_filename local_path.write_bytes(content) - sha256 = hashlib.sha256(content).hexdigest() - size_bytes = len(content) - manifest_path = output / "manifest.yaml" - r2_location = ArtifactStorageLocation( provider="r2", bucket=r2_bucket, @@ -456,6 +1009,7 @@ def fetch_source_artifact( size_bytes=size_bytes, fetched_at=fetched_at, r2_location=(r2_location if upload_r2 and r2_upload and r2_upload.ok else None), + record_revision=record_revision, ) return ArtifactFetchReport( @@ -475,6 +1029,26 @@ def fetch_source_artifact( ) +def _derived_artifact_paths(input_path: Path) -> list[Path]: + """Preflight every build-tree entry before reading or publishing any file.""" + if not stat.S_ISDIR(input_path.lstat().st_mode): + raise ValueError(f"derived_root_not_regular_directory:{input_path}") + artifacts: list[Path] = [] + + def visit(directory: Path) -> None: + for path in sorted(directory.iterdir()): + mode = path.lstat().st_mode + if stat.S_ISDIR(mode): + visit(path) + elif stat.S_ISREG(mode): + artifacts.append(path) + else: + raise ValueError(f"derived_entry_not_regular_file:{path}") + + visit(input_path) + return sorted(artifacts) + + def publish_derived_artifacts( input_dir: str | Path, *, @@ -482,13 +1056,55 @@ def publish_derived_artifacts( package_id: str, year: int, build_id: str | None = None, - r2_bucket: str = DEFAULT_R2_DERIVED_BUCKET, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", build_artifacts_output: str | Path | None = None, ) -> DerivedArtifactPublishReport: """Upload a deterministic build output directory to the derived R2 bucket.""" + r2_bucket = r2_bucket or default_r2_derived_bucket() input_path = Path(input_dir) + try: + _require_identity_segment(source_id, what="source_id") + _require_identity_segment(package_id, what="package_id") + _require_identity_segment(str(year), what="year") + except SourceArtifactManifestError as error: + return DerivedArtifactPublishReport( + input_dir=str(input_path), + source_id=source_id, + package_id=package_id, + year=year, + build_id=build_id or "", + entries=(), + build_artifacts_path=str(build_artifacts_output) + if build_artifacts_output + else None, + errors=(f"r2_identity_invalid:{error}",), + ) + try: + resolved_r2_prefix = resolve_r2_prefix( + prefix=r2_prefix, + default_prefix=default_r2_derived_prefix(), + source_id=source_id, + ) + if not is_derived_r2_route(r2_bucket, resolved_r2_prefix): + raise ValueError( + "Set CHRONICLE_R2_DERIVED_BUCKET or CHRONICLE_R2_DERIVED_PREFIX " + "to identify a custom derived route before publishing to it." + ) + except ValueError as error: + return DerivedArtifactPublishReport( + input_dir=str(input_path), + source_id=source_id, + package_id=package_id, + year=year, + build_id=build_id or "", + entries=(), + build_artifacts_path=str(build_artifacts_output) + if build_artifacts_output + else None, + errors=(f"derived_route_invalid:{error}",), + ) if not input_path.exists(): return DerivedArtifactPublishReport( input_dir=str(input_path), @@ -516,6 +1132,22 @@ def publish_derived_artifacts( errors=(f"input_dir_is_not_directory:{input_path}",), ) + try: + artifact_paths = _derived_artifact_paths(input_path) + except (OSError, ValueError) as error: + return DerivedArtifactPublishReport( + input_dir=str(input_path), + source_id=source_id, + package_id=package_id, + year=year, + build_id=build_id or "", + entries=(), + build_artifacts_path=str(build_artifacts_output) + if build_artifacts_output + else None, + errors=(str(error),), + ) + resolved_build_id = build_id or infer_build_id(input_path) if not resolved_build_id: return DerivedArtifactPublishReport( @@ -536,6 +1168,7 @@ def publish_derived_artifacts( # input failure like the ones above, reported rather than raised. try: canonicalize_key("build", resolved_build_id) + _require_identity_segment(resolved_build_id, what="build_id") except ValueError: return DerivedArtifactPublishReport( input_dir=str(input_path), @@ -550,14 +1183,8 @@ def publish_derived_artifacts( errors=("malformed_build_id",), ) - resolved_r2_prefix = resolve_r2_prefix( - prefix=r2_prefix, - default_prefix=DEFAULT_R2_DERIVED_PREFIX, - source_id=source_id, - ) entries: list[DerivedArtifactUploadEntry] = [] errors: list[str] = [] - artifact_paths = sorted(path for path in input_path.rglob("*") if path.is_file()) for artifact_path in artifact_paths: relative_path = artifact_path.relative_to(input_path).as_posix() if relative_path == "build_artifacts.jsonl": @@ -614,15 +1241,17 @@ def publish_derived_artifacts( def publish_source_artifacts( root: str | Path, *, - manifest_filename: str = "manifest.yaml", + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, source_id: str | None = None, package_id: str | None = None, - r2_bucket: str = DEFAULT_R2_RAW_BUCKET, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", ) -> RawArtifactPublishReport: """Upload manifest-declared raw source artifacts and record R2 locations.""" + r2_bucket = r2_bucket or default_r2_raw_bucket() root_path = Path(root) + _manifest_path(Path(), manifest_filename) if not root_path.exists(): return RawArtifactPublishReport( root=str(root_path), @@ -632,37 +1261,102 @@ def publish_source_artifacts( entries: list[RawArtifactPublishEntry] = [] errors: list[str] = [] - for manifest_path in sorted(root_path.rglob(manifest_filename)): + prepared: list[tuple[Path, dict[str, Any], dict[str, Any], str, str]] = [] + preflight_failures: list[RawArtifactPublishEntry] = [] + # ``--source-id`` / ``--package-id`` complete or confirm the identity of + # every manifest this sweep selects (one with an explicit selector, all of + # a directory's manifests by default). Decide eligibility from the whole + # selection up front, so a selected sibling met through another selected + # manifest's package preflight is not mistaken for an unselected one. + selected_manifest_paths = list(_root_manifest_paths(root_path, manifest_filename)) + selected_manifests = set(selected_manifest_paths) + for manifest_path in selected_manifest_paths: try: - manifest = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - except (OSError, yaml.YAMLError) as exc: + manifest = _read_manifest(manifest_path) + files = _manifest_files(manifest, manifest_path) + package_manifests = _package_manifests( + manifest_path.parent, manifest_path, manifest + ) + _assert_package_file_owner_identities_agree(package_manifests) + except (OSError, SourceArtifactManifestError) as exc: errors.append(f"Could not read {manifest_path}: {exc}") continue - manifest_source_id = source_id or manifest.get("source_id") - manifest_package_id = package_id or manifest.get("package_id") - files = manifest.get("files") or {} - if not manifest_source_id: - errors.append(f"Manifest missing source_id: {manifest_path}") - continue - if not manifest_package_id: - errors.append(f"Manifest missing package_id: {manifest_path}") - continue - if not isinstance(files, dict): - errors.append(f"Manifest files must be a mapping: {manifest_path}") - continue + manifest_source_id = ( + source_id if source_id is not None else manifest.get("source_id") + ) + manifest_package_id = ( + package_id if package_id is not None else manifest.get("package_id") + ) - try: - resolved_r2_prefix = resolve_r2_prefix( - prefix=r2_prefix, - default_prefix=DEFAULT_R2_PREFIX, - source_id=str(manifest_source_id), - package_path=manifest_path, + for package_manifest_name, package_manifest in package_manifests.items(): + package_manifest_path = Path(package_manifest_name) + # A selected sibling takes the overrides like the selected manifest + # itself; an unselected sibling is only preflighted so the package + # boundary holds and keeps its own identifiers, which may + # legitimately differ. + if package_manifest_path == manifest_path: + package_source_id = manifest_source_id + package_id_value = manifest_package_id + elif package_manifest_path in selected_manifests: + package_source_id = ( + source_id + if source_id is not None + else package_manifest.get("source_id") + ) + package_id_value = ( + package_id + if package_id is not None + else package_manifest.get("package_id") + ) + else: + package_source_id = package_manifest.get("source_id") + package_id_value = package_manifest.get("package_id") + package_files = _manifest_files(package_manifest, package_manifest_path) + for year, spec in package_files.items(): + entry, _updated_spec = _publish_raw_manifest_entry( + package_manifest_path, + package_source_id, + package_id_value, + year, + spec, + r2_bucket=r2_bucket, + r2_prefix=r2_prefix, + wrangler_command=wrangler_command, + preflight_only=True, + manifest_identity=package_manifest, + ) + if entry.errors: + preflight_failures.append(entry) + prepared.append( + ( + manifest_path, + manifest, + files, + manifest_source_id, + manifest_package_id, ) - except ValueError as exc: - errors.append(f"Could not resolve R2 prefix for {manifest_path}: {exc}") - continue + ) + + # A root sweep is one requested publish operation. Validate every selected + # package before the first uploader call or manifest rewrite, otherwise a + # malformed later package can make the command fail after earlier packages + # have already changed external and local state. + if errors or preflight_failures: + entries.extend(preflight_failures) + return RawArtifactPublishReport( + root=str(root_path), + entries=tuple(entries), + errors=tuple(errors), + ) + for ( + manifest_path, + manifest, + files, + manifest_source_id, + manifest_package_id, + ) in prepared: updated = False for year, spec in files.items(): entry, updated_spec = _publish_raw_manifest_entry( @@ -672,16 +1366,19 @@ def publish_source_artifacts( year, spec, r2_bucket=r2_bucket, - r2_prefix=resolved_r2_prefix, + r2_prefix=r2_prefix, wrangler_command=wrangler_command, + manifest_identity=manifest, ) entries.append(entry) if updated_spec is not None and isinstance(spec, dict): spec.update(updated_spec) updated = True if updated: - manifest.setdefault("source_id", manifest_source_id) - manifest.setdefault("package_id", manifest_package_id) + if manifest_source_id: + manifest.setdefault("source_id", manifest_source_id) + if manifest_package_id: + manifest.setdefault("package_id", manifest_package_id) manifest_path.write_text( yaml.safe_dump(manifest, sort_keys=False), encoding="utf-8", @@ -738,10 +1435,11 @@ def write_build_artifacts_jsonl( def inventory_source_artifacts( root: str | Path, *, - manifest_filename: str = "manifest.yaml", + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, ) -> ArtifactInventoryReport: """Inventory manifest-declared source artifacts under a root directory.""" root_path = Path(root) + _manifest_path(Path(), manifest_filename) errors: list[str] = [] entries: list[ArtifactInventoryEntry] = [] if not root_path.exists(): @@ -758,22 +1456,23 @@ def inventory_source_artifacts( errors=(f"Root does not exist: {root_path}",), ) - manifests = sorted(root_path.rglob(manifest_filename)) - for manifest_path in manifests: + manifest_paths = _root_manifest_paths(root_path, manifest_filename) + for manifest_path in manifest_paths: try: - manifest = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - files = manifest.get("files") or {} - except (OSError, yaml.YAMLError) as exc: + manifest = _read_manifest(manifest_path) + files = _manifest_files(manifest, manifest_path) + package_manifests = _package_manifests( + manifest_path.parent, manifest_path, manifest + ) + _assert_package_file_owner_identities_agree(package_manifests) + except (OSError, SourceArtifactManifestError) as exc: errors.append(f"Could not read {manifest_path}: {exc}") continue - if not isinstance(files, dict): - errors.append(f"Manifest files must be a mapping: {manifest_path}") - continue for year, spec in files.items(): entries.append(_inventory_entry(manifest_path, year, spec)) counts = { - "manifest_count": len(manifests), + "manifest_count": len(manifest_paths), "artifact_count": len(entries), "missing_count": sum(1 for entry in entries if not entry.exists), "checksum_mismatch_count": sum( @@ -791,12 +1490,15 @@ def inventory_source_artifacts( def bootstrap_r2_buckets( *, - raw_bucket: str = DEFAULT_R2_RAW_BUCKET, - derived_bucket: str = DEFAULT_R2_DERIVED_BUCKET, + raw_bucket: str | None = None, + derived_bucket: str | None = None, wrangler_command: str = "npx wrangler", ) -> R2BootstrapReport: """Create the R2 buckets Chronicle expects, if Wrangler is authenticated.""" - buckets = (raw_bucket, derived_bucket) + buckets = ( + raw_bucket or default_r2_raw_bucket(), + derived_bucket or default_r2_derived_bucket(), + ) commands: list[ArtifactCommandResult] = [] errors: list[str] = [] @@ -959,7 +1661,7 @@ def build_derived_r2_key( """Build the canonical R2 key for a derived build artifact.""" resolved_prefix = resolve_r2_prefix( prefix=prefix, - default_prefix=DEFAULT_R2_DERIVED_PREFIX, + default_prefix=default_r2_derived_prefix(), source_id=source_id, ) return posixpath.join( @@ -1010,9 +1712,9 @@ def infer_build_id(input_dir: str | Path) -> str | None: if build_id: return str(build_id) - db_path = input_path / "ledger.db" + db_path = input_path / CHRONICLE_DB_FILENAME if not db_path.exists(): - db_path = input_path / "ledger.db" + db_path = input_path / LEGACY_CHRONICLE_DB_FILENAME if db_path.exists(): with sqlite3.connect(db_path) as connection: row = connection.execute( @@ -1043,32 +1745,768 @@ def _filename_from_url(source_url: str) -> str: return Path(unquote(parsed.path)).name -def _upsert_manifest( - manifest_path: Path, - *, - source_id: str, - package_id: str, - dataset: str, - source_page: str, - table: str, - year: int, - filename: str, +def _infer_artifact_filename(source_url: str) -> str: + """Return the filename :func:`_read_artifact` would report, without I/O. + + The name is a pure function of the URL: the last path segment for http(s) + and ``file://`` URLs, the basename for a bare path. Resolving it before + the read lets every filename guard run before the publisher is touched. + """ + parsed = urlparse(source_url) + if parsed.scheme in ("http", "https"): + return _filename_from_url(source_url) + if parsed.scheme == "file": + return Path(unquote(parsed.path)).name + if not parsed.scheme: + return Path(source_url).name + raise ValueError(f"Unsupported source URL scheme: {parsed.scheme}") + + +def _read_manifest(manifest_path: Path) -> dict[str, Any]: + """Return a manifest's parsed payload, refusing a document it cannot read. + + An absent or empty manifest reads as an empty mapping: ``fetch-artifact`` + writes the first entry into a package that has none. A document that parses + as anything else -- a list, a scalar, a truncated or half-merged file -- is + not an absent manifest, and treating it as one would let the fetch replace + it with a single entry and drop everything it recorded. + """ + if manifest_path.is_symlink(): + raise MalformedManifestError( + f"{manifest_path} is a symlink; manifest reads and writes require " + "a regular file at its lexical package path." + ) + if not manifest_path.exists(): + return {} + try: + payload = load_manifest_document(manifest_path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise MalformedManifestError( + f"{manifest_path} is not valid YAML: {exc}" + ) from exc + if payload is None: + return {} + if not isinstance(payload, dict): + raise MalformedManifestError( + f"{manifest_path} must be a YAML mapping; it parses as a " + f"{type(payload).__name__}. Chronicle will not overwrite a manifest " + "it cannot read." + ) + return payload + + +def _select_vintage_entry( + payload: dict[str, Any], + *, + manifest_path: Path, + year: Any, +) -> tuple[Any, Any, dict[str, Any], int | None]: + """Locate the entry a fetch revises: ``(key, files[key], entry, index)``. + + Integer and quoted-integer keys are two spellings of one vintage. Preserve + the spelling already present, refuse a manifest that contains both, and + reject any present non-mapping entry before publisher I/O. The final tuple + slot matches the stacked #227 selector; table manifests never use a list + index. + """ + files = payload.get("files") if isinstance(payload, dict) else None + if files is None: + return year, None, {}, None + if not isinstance(files, dict): + raise MalformedManifestError( + f"{manifest_path} files must be a mapping; it is a " + f"{type(files).__name__}. Chronicle will not write into a manifest " + "it cannot read." + ) + + forms: tuple[Any, ...] + if isinstance(year, bool): + forms = (year,) + elif isinstance(year, int): + forms = (year, str(year)) + else: + text = str(year) + if text.isdecimal() and (text == "0" or not text.startswith("0")): + forms = (year, int(text)) + else: + forms = (year,) + present = [form for form in forms if form in files] + if len(present) > 1: + raise MalformedManifestError( + f"{manifest_path}: Vintage {year!r} is recorded under both keys " + f"{present!r}; one vintage has one key. Merge the entries by hand " + "first. Chronicle will not choose which entry is the record." + ) + if not present: + return year, None, {}, None + key = present[0] + existing = files[key] + if not isinstance(existing, dict): + raise MalformedManifestError( + f"{manifest_path} entry {key!r} must be a mapping; it is a " + f"{type(existing).__name__}." + ) + return key, existing, existing, None + + +def _recorded_storage(spec: Any) -> dict[str, Any]: + """Return a manifest file spec's recorded ``storage`` block, if any.""" + if not isinstance(spec, dict): + return {} + storage = spec.get("storage") + return storage if isinstance(storage, dict) else {} + + +def _recorded_r2(spec: Any) -> dict[str, Any]: + """Return the recorded ``storage.r2`` block verbatim, if any. + + Raw access, for callers that carry the block forward as history. Callers + that reason about which object it names go through + :func:`_validated_recorded_r2` instead. + """ + recorded = _recorded_storage(spec).get("r2") + return recorded if isinstance(recorded, dict) else {} + + +def _split_r2_uri(uri: str) -> tuple[str, str, str] | None: + """Split ``provider://bucket/key`` into its three parts, or None.""" + provider, separator, remainder = uri.partition("://") + if not separator or not provider: + return None + bucket, separator, key = remainder.partition("/") + if not separator or not bucket or not key: + return None + return (provider, bucket, key) + + +def _validated_recorded_storage( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> dict[str, Any]: + """Return the entry's ``storage`` mapping, refusing a malformed one.""" + if not isinstance(spec, dict) or "storage" not in spec: + return {} + storage = spec["storage"] + if not isinstance(storage, dict): + raise MalformedManifestError( + f"{manifest_path} entry {year!r} storage must be a mapping; it is a " + f"{type(storage).__name__}." + ) + if "previous_r2" in storage and not isinstance(storage["previous_r2"], list): + previous = storage["previous_r2"] + raise MalformedManifestError( + f"{manifest_path} entry {year!r} storage.previous_r2 must be a " + f"list; it is a {type(previous).__name__}. Chronicle will not " + "discard malformed archived provenance." + ) + return storage + + +def _validated_recorded_r2( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> RecordedR2Object | None: + """Return the object a recorded ``storage.r2`` block names, or None. + + The block must explicitly record provider ``r2`` and an ``r2://`` URI. + Every additional locator field it supplies is cross-checked against that + URI: ``key`` against its path, ``bucket`` against its authority, and the + resulting key against the canonical content-addressed shape + :func:`build_r2_key` writes. Reading one field and trusting the rest is + what lets a block that says two different things survive a preserve or a + publish. + """ + storage = _validated_recorded_storage(spec, manifest_path=manifest_path, year=year) + if "r2" not in storage: + return None + block = storage["r2"] + where = f"{manifest_path} entry {year!r} storage.r2" + if not isinstance(block, dict): + raise MalformedManifestError( + f"{where} must be a mapping; it is a {type(block).__name__}." + ) + + supplied: dict[str, str] = {} + for field in ("provider", "bucket", "key", "uri"): + value = block.get(field) + if value is None: + continue + if not isinstance(value, str) or not value.strip(): + raise RecordedR2LocatorError( + f"{where}: {field} must be a non-empty string, not {value!r}." + ) + supplied[field] = value + + provider = supplied.get("provider") + bucket = supplied.get("bucket") + key = supplied.get("key") + uri = supplied.get("uri") + missing_required = [ + field for field in ("provider", "uri") if not supplied.get(field) + ] + if missing_required: + raise RecordedR2LocatorError( + f"{where}: records no {', '.join(missing_required)}. A block under " + "storage.r2 must explicitly record provider='r2' and an r2:// URI." + ) + if provider != "r2": + raise RecordedR2LocatorError( + f"{where}: provider={provider!r} does not identify R2. A block " + "under storage.r2 must use provider='r2' and an r2:// URI, not " + f"{provider}://." + ) + if uri is not None: + parts = _split_r2_uri(uri) + if parts is None: + raise RecordedR2LocatorError( + f"{where}: uri {uri!r} is not provider://bucket/key." + ) + for field, value, from_uri in zip( + ("provider", "bucket", "key"), (provider, bucket, key), parts + ): + if value is not None and value != from_uri: + raise RecordedR2LocatorError( + f"{where}: {field}={value!r} contradicts uri {uri!r}, which " + f"names {from_uri!r}. The block records two different " + "objects, so Chronicle cannot say which bytes it claims." + ) + provider, bucket, key = ( + provider or parts[0], + bucket or parts[1], + key or parts[2], + ) + + missing = [ + field + for field, value in ( + ("provider", provider), + ("bucket", bucket), + ("key", key), + ) + if not value + ] + if missing: + raise RecordedR2LocatorError( + f"{where}: records no {', '.join(missing)}. A recorded block has to " + "locate its object: provider, bucket and key, or a uri that " + "supplies them." + ) + segments = key.split("/") + if ( + len(segments) < 2 + or not all(segments) + or not _SHA256_KEY_SEGMENT.fullmatch(segments[-2]) + ): + raise RecordedR2LocatorError( + f"{where}: key {key!r} is not content-addressed. A raw key ends in " + "{sha256}/{filename}, which is what says the object holds the " + "entry's bytes; Chronicle will not guess for a key that does not." + ) + return RecordedR2Object( + provider=provider, + bucket=bucket, + key=key, + sha256=segments[-2], + filename=segments[-1], + ) + + +def _recorded_identity( + spec: Any, + *, + manifest_path: Path, + year: Any, +) -> RecordedIdentity | None: + """Return what a manifest entry says its vintage holds, if anything. + + A published entry is identified by its recorded object's content-addressed + key. An entry that has not been published yet -- registered without an + upload, or left behind by a failed one -- is identified by its own declared + ``sha256`` and ``filename``. Both are recorded identities, and a fetch of + different bytes over either one is a publisher revision. + """ + recorded_r2 = _validated_recorded_r2(spec, manifest_path=manifest_path, year=year) + declared_sha256 = spec.get("sha256") if isinstance(spec, dict) else None + declared_sha256 = declared_sha256 if isinstance(declared_sha256, str) else None + declared_filename = spec.get("filename") if isinstance(spec, dict) else None + declared_filename = ( + declared_filename if isinstance(declared_filename, str) else None + ) + size_bytes = spec.get("size_bytes") if isinstance(spec, dict) else None + size_bytes = size_bytes if isinstance(size_bytes, int) else None + if recorded_r2 is not None: + return RecordedIdentity( + sha256=recorded_r2.sha256, + filename=recorded_r2.filename, + # Only report a size the recorded key agrees with: an entry can + # arrive here already describing the new bytes. + size_bytes=size_bytes if declared_sha256 == recorded_r2.sha256 else None, + declared_sha256=declared_sha256, + r2=recorded_r2, + ) + if not declared_sha256: + return None + return RecordedIdentity( + sha256=declared_sha256, + filename=Path(declared_filename).name if declared_filename else "", + size_bytes=size_bytes, + declared_sha256=declared_sha256, + r2=None, + ) + + +def _manifest_file_owners( + manifests: Mapping[str, dict[str, Any]], + *, + filename: str, + initializing: tuple[Path, Any] | None = None, +) -> list[_ManifestFileOwner]: + """Return every entry in a package directory that names ``filename``. + + ``initializing`` names the ``(manifest_path, vintage)`` entry the calling + command is about to identify -- a predeclared entry (``filename`` and + ``source_url`` only) on its first fetch. That entry has no identity yet + and is not an owner; every other entry naming the file must already + record one, because Chronicle cannot tell whether overwriting the shared + bytes would change what an unidentified sibling means. + """ + wanted = filename_key(filename) + owners: list[_ManifestFileOwner] = [] + for name, payload in manifests.items(): + manifest_path = Path(name) + for vintage, spec in _manifest_files(payload, manifest_path).items(): + if not isinstance(spec, dict): + raise MalformedManifestError( + f"{manifest_path} entry {vintage!r} must be a mapping; it " + f"is a {type(spec).__name__}. Chronicle cannot decide " + "whether it owns a shared package-local file." + ) + recorded_name = spec.get("filename") + if recorded_name is None: + continue + if not is_bare_filename(recorded_name): + raise MalformedManifestError( + f"{manifest_path} entry {vintage!r} filename must be a " + f"bare package-local name, not {recorded_name!r}." + ) + if filename_key(recorded_name) != wanted: + continue + identity = _recorded_identity( + spec, + manifest_path=manifest_path, + year=vintage, + ) + if identity is None: + if ( + initializing is not None + and manifest_path == initializing[0] + and str(vintage) == str(initializing[1]) + ): + # The entry this command identifies: not an owner yet. + continue + raise MalformedManifestError( + f"{manifest_path} entry {vintage!r} names " + f"{recorded_name!r} but records no sha256 identity. " + "Chronicle cannot safely overwrite an unidentifiable " + "shared file." + ) + owners.append( + _ManifestFileOwner( + manifest_path=manifest_path, + vintage=vintage, + spec=spec, + identity=identity, + ) + ) + return owners + + +def _assert_shared_owner_identities_agree( + owners: list[_ManifestFileOwner], + *, + filename: str, +) -> None: + """Refuse an already-contradictory set of owners before publisher I/O.""" + if not owners: + return + first = owners[0] + first_identity = first.identity + assert first_identity is not None + for owner in owners[1:]: + identity = owner.identity + assert identity is not None + if identity.sha256 == first_identity.sha256 and filename_key( + identity.filename + ) == filename_key(first_identity.filename): + continue + raise SourceArtifactManifestError( + f"{first.manifest_path} entry {first.vintage!r} and " + f"{owner.manifest_path} entry {owner.vintage!r} both name " + f"{filename!r} but identify different bytes. One package-local " + "file must have one recorded identity; reconcile the manifests " + "before fetching it again." + ) + + +def _effective_recorded_digest( + manifest_name: str, vintage: Any, entry: Mapping[str, Any] +) -> str | None: + """Return the digest an entry's recorded R2 key encodes, if it has one. + + Two manifests may identify one package-local file through identical + content-addressed R2 locators without declaring ``sha256``; refetching + those bytes records ``sha256`` on the selected manifest only. The + directory-level collision check must compare the identities the entries + *effectively* record -- the same identity :func:`_recorded_identity` + resolves -- so a sibling that only carries the locator does not read as + an empty digest. A malformed locator is the per-entry preflight's error, + not a collision: fall back to the declared field for it. + """ + try: + recorded = _validated_recorded_r2( + entry, manifest_path=Path(manifest_name), year=vintage + ) + except SourceArtifactManifestError: + return None + return None if recorded is None else recorded.sha256 + + +def _assert_package_file_owner_identities_agree( + manifests: Mapping[str, dict[str, Any]], +) -> None: + """Refuse contradictory identities for any package-local filename. + + Publish and inventory sweep a manifest at a time, but the physical byte is + shared by every manifest in its directory. Validate every identified owner + as one package boundary before a selected manifest can upload anything. + Entry-shape and local-file errors remain the per-entry preflight's job. + """ + collision_codes = validate_package_directory( + manifests, entry_digest=_effective_recorded_digest + ) + if collision_codes: + raise SourceArtifactManifestError( + "Package manifests identify different bytes for one package-local " + f"filename: {', '.join(collision_codes)}. Reconcile the manifests " + "before publishing or inventorying that directory." + ) + + owners_by_filename: dict[str, list[_ManifestFileOwner]] = {} + display_names: dict[str, str] = {} + unidentified: dict[str, list[tuple[Path, Any, str]]] = {} + for name, payload in manifests.items(): + manifest_path = Path(name) + for vintage, spec in _manifest_files(payload, manifest_path).items(): + if not isinstance(spec, dict): + continue + recorded_name = spec.get("filename") + if not is_bare_filename(recorded_name): + continue + try: + identity = _recorded_identity( + spec, + manifest_path=manifest_path, + year=vintage, + ) + except SourceArtifactManifestError: + # The complete per-entry preflight reports the precise locator + # or history error without letting another entry upload first. + continue + key = filename_key(recorded_name) + display_names.setdefault(key, str(recorded_name)) + if identity is None: + unidentified.setdefault(key, []).append( + (manifest_path, vintage, str(recorded_name)) + ) + continue + owners_by_filename.setdefault(key, []).append( + _ManifestFileOwner( + manifest_path=manifest_path, + vintage=vintage, + spec=spec, + identity=identity, + ) + ) + for key, owners in owners_by_filename.items(): + _assert_shared_owner_identities_agree( + owners, + filename=display_names[key], + ) + # An entry that records no identity yet is not a collision (nothing to + # contradict), but the command that identifies it will hash the shared + # bytes. Those bytes must already be what the identified owners record, + # otherwise identifying it would split one package-local file into two + # identities. Check before any upload or manifest rewrite. + for key, pending in unidentified.items(): + owners = owners_by_filename.get(key) + if not owners: + continue + expected = owners[0].identity + assert expected is not None + for manifest_path, vintage, recorded_name in pending: + try: + local = matching_directory_entry(manifest_path.parent, recorded_name) + except ValueError: + # Conflicting spellings are the per-entry preflight's refusal. + continue + if local is None or local.is_symlink() or not local.is_file(): + continue + actual = hashlib.sha256(local.read_bytes()).hexdigest() + if actual == expected.sha256: + continue + raise SourceArtifactManifestError( + f"{manifest_path} entry {vintage!r} names {recorded_name!r} " + f"without a recorded identity, and the package-local bytes " + f"(sha256={actual}) are not what {owners[0].manifest_path} entry " + f"{owners[0].vintage!r} records for it (sha256=" + f"{expected.sha256}). Identifying this entry would give one " + "package-local file two identities; reconcile the manifests " + "or the file before publishing or inventorying the directory." + ) + + +def _assert_siblings_record_these_bytes( + manifests: Mapping[str, dict[str, Any]], + *, + manifest_path: Path, + vintage: Any, + filename: str, + sha256: str, +) -> None: + """Refuse a default fetch that would stale another manifest's owner.""" + for owner in _manifest_file_owners( + manifests, filename=filename, initializing=(manifest_path, vintage) + ): + if owner.manifest_path == manifest_path: + continue + identity = owner.identity + assert identity is not None + if identity.sha256 == sha256: + continue + raise SourceArtifactRevisionError( + f"{owner.manifest_path} entry {owner.vintage!r} records " + f"{filename!r} as sha256={identity.sha256}; this fetch would write " + f"sha256={sha256} to the same package-local file. Re-run with " + "--record-revision to update every owner together." + ) + + +def _revision_error_message( + *, + manifest_path: Path, + year: Any, + filename: str, + identity: RecordedIdentity, + sha256: str, + size_bytes: int, + r2_bucket: str, +) -> str: + """Explain a refused fetch: recorded identity, fetched identity, next step.""" + records = ( + f"already records the R2 object {identity.r2.uri}, which holds" + if identity.r2 is not None + else "already records" + ) + message = ( + f"{manifest_path} entry {year!r} {records} " + f"sha256={identity.sha256} " + f"filename={identity.filename or 'unknown'} " + f"size_bytes=" + f"{identity.size_bytes if identity.size_bytes is not None else 'unknown'}. " + f"The fetched bytes are sha256={sha256} filename={Path(filename).name} " + f"size_bytes={size_bytes}. Chronicle will not rewrite a vintage that " + "identifies specific bytes to describe bytes it never identified." + ) + if identity.r2 is not None and identity.declared_sha256 not in ( + None, + identity.sha256, + ): + message += ( + f" (The entry also declares sha256={identity.declared_sha256}, which " + "its own R2 key contradicts: an earlier fetch rewrote the hash " + "without moving the object.)" + ) + return message + ( + " The same vintage with new bytes is a new release revision " + "(docs/adr-chronicle-fact-identity-v2.md). Re-run with " + "--record-revision to store the fetched bytes under their own " + f"content-addressed key in {r2_bucket} and keep the superseded object " + "in storage.previous_r2." + ) + + +def _assert_recorded_identity_holds_these_bytes( + identity: RecordedIdentity | None, + *, + manifest_path: Path, + year: Any, + filename: str, + sha256: str, + size_bytes: int, + r2_bucket: str, + record_revision: bool, +) -> None: + """Refuse a publisher revision that has not been opted into.""" + if identity is None or identity.holds(sha256=sha256, filename=filename): + return + if identity.sha256 == sha256: + # The recorded object holds exactly these bytes under another name. A + # rename is not a publisher revision, so --record-revision does not + # apply, and silently adopting the new name would leave the entry's + # filename disagreeing with the key its own storage block records. + raise SourceArtifactRevisionError( + f"{manifest_path} entry {year!r} already records these exact bytes " + f"(sha256={sha256}) as filename={identity.filename}; this fetch " + f"names them {Path(filename).name}. A rename is not a release " + "revision, so --record-revision does not apply. Re-run with " + f"--filename {identity.filename} to keep the recorded identity." + ) + if record_revision: + return + raise SourceArtifactRevisionError( + _revision_error_message( + manifest_path=manifest_path, + year=year, + filename=filename, + identity=identity, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + ) + ) + + +def _superseding_storage( + recorded_spec: dict[str, Any], + *, + recorded_r2: RecordedR2Object | None, + new_r2: dict[str, Any] | None, + superseded_at: str, +) -> dict[str, Any]: + """Return a storage block in which the recorded object becomes history. + + ``storage.r2`` only ever names the object that holds the entry's current + bytes. The superseded block is appended, oldest first, to + ``storage.previous_r2`` so the earlier bytes stay addressable by the URI + archived witness records already pin. An entry that was never published has + no object to supersede, and gets no ``previous_r2`` key. + """ + storage = dict(_recorded_storage(recorded_spec)) + previous = storage.get("previous_r2") + entries = list(previous) if isinstance(previous, list) else [] + if recorded_r2 is not None: + entry = dict(_recorded_r2(recorded_spec)) + entry["sha256"] = recorded_r2.sha256 + if recorded_spec.get("sha256") == recorded_r2.sha256: + # Only carry metadata the superseded key agrees with: a manifest + # can arrive here already describing the new bytes. + for field in ("size_bytes", "fetched_at", "source_url"): + value = recorded_spec.get(field) + if value is not None: + entry[field] = value + entry["superseded_at"] = superseded_at + entries.append(entry) + if entries: + storage["previous_r2"] = entries + if new_r2 is None: + storage.pop("r2", None) + else: + storage["r2"] = new_r2 + return storage + + +#: Entry fields a fetch owns. Every other field already recorded on the entry +#: is carried forward during a refetch or explicit publisher revision. +_FETCH_OWNED_FIELDS: frozenset[str] = frozenset( + { + "filename", + "source_url", + "sha256", + "size_bytes", + "fetched_at", + "storage", + } +) + + +def _storage_for_fetched_identity( + recorded_spec: dict[str, Any], + *, + identity: RecordedIdentity | None, + filename: str, + sha256: str, + new_r2: dict[str, Any] | None, + fetched_at: str, +) -> dict[str, Any]: + """Return one owner's storage after a refetch or explicit revision.""" + recorded_storage = _recorded_storage(recorded_spec) + holds = identity is not None and identity.holds( + sha256=sha256, + filename=filename, + ) + if holds and identity.r2 is not None: + # A same-byte copy does not replace the object's recorded history. + return {**recorded_storage, "r2": _recorded_r2(recorded_spec)} + if identity is not None and not holds: + return _superseding_storage( + recorded_spec, + recorded_r2=identity.r2, + new_r2=new_r2, + superseded_at=fetched_at, + ) + if new_r2 is not None: + return {**recorded_storage, "r2": new_r2} + return dict(recorded_storage) + + +def _upsert_manifest( + manifest_path: Path, + *, + source_id: str, + package_id: str, + dataset: str, + source_page: str, + table: str, + year: int, + filename: str, source_url: str, sha256: str, size_bytes: int, fetched_at: str, r2_location: ArtifactStorageLocation | None, + record_revision: bool = False, ) -> None: - if manifest_path.exists(): - payload = yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} - else: - payload = {} + payload = _read_manifest(manifest_path) + _manifest_files(payload, manifest_path) + _assert_manifest_identifies( + payload, + manifest_path, + source_id=source_id, + package_id=package_id, + ) + manifests = _package_manifests(manifest_path.parent, manifest_path, payload) + owners = _manifest_file_owners( + manifests, filename=filename, initializing=(manifest_path, year) + ) + _assert_shared_owner_identities_agree(owners, filename=filename) payload.setdefault("source_id", source_id) payload.setdefault("package_id", package_id) payload.setdefault("dataset", dataset) payload.setdefault("source_page", source_page) payload.setdefault("table", table) - payload.setdefault("files", {}) + if payload.get("files") is None: + # setdefault keeps an explicit null (a bare ``files:`` line); the + # entry below needs a mapping to record into. + payload["files"] = {} + key, _existing_value, recorded_spec, _index = _select_vintage_entry( + payload, + manifest_path=manifest_path, + year=year, + ) file_entry: dict[str, Any] = { "filename": filename, "source_url": source_url, @@ -1076,13 +2514,98 @@ def _upsert_manifest( "size_bytes": size_bytes, "fetched_at": fetched_at, } - if r2_location is not None: - file_entry["storage"] = {"r2": r2_location.to_dict()} - payload["files"][year] = file_entry - manifest_path.write_text( - yaml.safe_dump(payload, sort_keys=False), - encoding="utf-8", + for field, value in recorded_spec.items(): + if field not in _FETCH_OWNED_FIELDS and field not in file_entry: + file_entry[field] = value + identity = _recorded_identity(recorded_spec, manifest_path=manifest_path, year=key) + new_r2 = r2_location.to_dict() if r2_location is not None else None + r2_bucket = (new_r2 or {}).get("bucket") or default_r2_raw_bucket() + _assert_recorded_identity_holds_these_bytes( + identity, + manifest_path=manifest_path, + year=key, + filename=filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + record_revision=record_revision, ) + if not record_revision: + _assert_siblings_record_these_bytes( + manifests, + manifest_path=manifest_path, + vintage=key, + filename=filename, + sha256=sha256, + ) + for owner in owners: + if owner.manifest_path == manifest_path and owner.spec is recorded_spec: + continue + _assert_recorded_identity_holds_these_bytes( + owner.identity, + manifest_path=owner.manifest_path, + year=owner.vintage, + filename=filename, + sha256=sha256, + size_bytes=size_bytes, + r2_bucket=r2_bucket, + record_revision=record_revision, + ) + + storage = _storage_for_fetched_identity( + recorded_spec, + identity=identity, + filename=filename, + sha256=sha256, + new_r2=new_r2, + fetched_at=fetched_at, + ) + # An entry that has no storage to record carries no empty block: a + # revision over a never-published entry supersedes nothing. + if storage: + file_entry["storage"] = storage + payload["files"][key] = file_entry + + revision = any( + owner.identity is not None + and not owner.identity.holds(sha256=sha256, filename=filename) + for owner in owners + ) or (identity is not None and not identity.holds(sha256=sha256, filename=filename)) + changed_paths = {manifest_path} + if record_revision and revision: + for owner in owners: + if owner.manifest_path == manifest_path and str(owner.vintage) == str(key): + continue + revised_entry = dict(owner.spec) + revised_entry.update( + { + "filename": filename, + "sha256": sha256, + "size_bytes": size_bytes, + "fetched_at": fetched_at, + } + ) + owner_storage = _storage_for_fetched_identity( + owner.spec, + identity=owner.identity, + filename=filename, + sha256=sha256, + new_r2=new_r2, + fetched_at=fetched_at, + ) + if owner_storage: + revised_entry["storage"] = owner_storage + else: + revised_entry.pop("storage", None) + manifests[str(owner.manifest_path)]["files"][owner.vintage] = revised_entry + changed_paths.add(owner.manifest_path) + + rendered = { + path: yaml.safe_dump(manifests[str(path)], sort_keys=False) + for path in changed_paths + } + for path, text in sorted(rendered.items(), key=lambda item: str(item[0])): + path.write_text(text, encoding="utf-8") def _upload_r2_object( @@ -1115,21 +2638,121 @@ def _publish_raw_manifest_entry( spec: Any, *, r2_bucket: str, - r2_prefix: str, + r2_prefix: str | None, wrangler_command: str, + preflight_only: bool = False, + manifest_identity: dict[str, Any] | None = None, ) -> tuple[RawArtifactPublishEntry, dict[str, Any] | None]: errors: list[str] = [] + # Validate original values below; reports must still serialize refusals of + # YAML dates, sets, and other non-string declarations through the CLI. + reported_source_id = str(source_id) if source_id is not None else "" + reported_package_id = str(package_id) if package_id is not None else "" if not isinstance(spec, dict): spec = {} errors.append("malformed_file_spec") filename = str(spec.get("filename") or "") - artifact_path = manifest_path.parent / filename + artifact_path = manifest_path.parent + sha256_actual = None + size_bytes = None + + def refuse(reason: str | None = None) -> tuple[RawArtifactPublishEntry, None]: + """Report the entry unpublished, with nothing uploaded or rewritten.""" + if reason is not None: + errors.append(reason) + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=reported_source_id, + package_id=reported_package_id, + year=str(year), + filename=filename, + local_path=str(artifact_path), + sha256=sha256_actual, + size_bytes=size_bytes, + r2_location=None, + upload=None, + errors=tuple(errors), + ), + None, + ) + + if filename and not is_bare_filename(filename): + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=reported_source_id, + package_id=reported_package_id, + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + sha256=None, + size_bytes=None, + r2_location=None, + upload=None, + errors=(f"non_canonical_filename:{filename}",), + ), + None, + ) + if is_manifest_filename(filename): + return ( + RawArtifactPublishEntry( + manifest_path=str(manifest_path), + source_id=reported_source_id, + package_id=reported_package_id, + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + sha256=None, + size_bytes=None, + r2_location=None, + upload=None, + errors=(f"manifest_named_filename:{filename}",), + ), + None, + ) + try: + recorded_r2 = _validated_recorded_r2( + spec, manifest_path=manifest_path, year=year + ) + except SourceArtifactManifestError as error: + # A block that does not name one object cannot be treated as history, + # and publishing under it would ship whichever field was read. + return refuse(f"recorded_r2_locator_invalid:{error}") + if recorded_r2 is None: + try: + _require_identity_segment(source_id, what="source_id") + _require_identity_segment(package_id, what="package_id") + _require_identity_segment(str(year), what="year") + _assert_manifest_identifies( + manifest_identity or {}, + manifest_path, + source_id=source_id, + package_id=package_id, + ) + except SourceArtifactManifestError as error: + return refuse(f"r2_identity_invalid:{error}") + + try: + artifact_path = ( + matching_directory_entry(manifest_path.parent, filename) + or manifest_path.parent / filename + ) + if filename and artifact_path.name != filename: + errors.append(f"artifact_spelling_mismatch:{filename}:{artifact_path.name}") + except ValueError: + errors.append(f"duplicate_artifact_spellings:{filename}") + artifact_path = manifest_path.parent / filename sha256_expected = spec.get("sha256") sha256_actual = None size_bytes = None if not filename: errors.append("missing_filename") - elif not artifact_path.exists(): + elif errors: + pass + elif artifact_path.is_symlink(): + errors.append(f"artifact_path_is_symlink:{filename}") + elif not artifact_path.is_file(): errors.append("missing_file") else: content = artifact_path.read_bytes() @@ -1139,23 +2762,75 @@ def _publish_raw_manifest_entry( errors.append("checksum_mismatch") if errors: + return refuse() + + if recorded_r2 is not None and (recorded_r2.sha256, recorded_r2.filename) != ( + sha256_actual or "", + Path(filename).name, + ): + # The recorded object is addressed by different bytes, so it is not + # this file's history. Uploading anyway would either publish under a + # key that misdescribes its content or restate a URI that belongs to + # the superseded bytes. Registering a publisher revision is + # `fetch-artifact --record-revision`, not a publish-time rewrite. + return refuse( + "recorded_r2_identity_mismatch:" + f"recorded_sha256={recorded_r2.sha256}:" + f"recorded_filename={recorded_r2.filename}:" + f"local_sha256={sha256_actual}:" + f"local_filename={Path(filename).name}" + ) + + if recorded_r2 is not None: + # A recorded content-addressed object whose tail identifies the bytes + # in hand is published history. Its source/package/year route may + # predate today's country prefix or intentionally represent the + # publisher's explicit route (for example Statbel's 2023 snapshots and + # USDA's cross-manifest archive). Reconstructing a current route and + # requiring equality would rewrite that history during a bucket + # cutover. Only the checksum/filename tail decides byte identity. + skipped = "recorded_r2_already_published" + if recorded_r2.bucket != r2_bucket: + skipped = ( + "recorded_r2_bucket_is_preserved_history:" + f"recorded={recorded_r2.bucket}:requested={r2_bucket}" + ) return ( RawArtifactPublishEntry( manifest_path=str(manifest_path), - source_id=source_id, - package_id=package_id, + source_id=reported_source_id, + package_id=reported_package_id, year=str(year), filename=filename, local_path=str(artifact_path), sha256=sha256_actual, size_bytes=size_bytes, - r2_location=None, + r2_location=ArtifactStorageLocation( + provider="r2", + bucket=recorded_r2.bucket, + key=recorded_r2.key, + ), upload=None, - errors=tuple(errors), + errors=(), + skipped=skipped, ), None, ) + if not source_id: + return refuse("missing_source_id") + if not package_id: + return refuse("missing_package_id") + try: + resolved_r2_prefix = resolve_r2_prefix( + prefix=r2_prefix, + default_prefix=DEFAULT_R2_PREFIX, + source_id=source_id, + package_path=manifest_path, + ) + except ValueError as error: + return refuse(f"r2_prefix_invalid:{error}") + location = ArtifactStorageLocation( provider="r2", bucket=r2_bucket, @@ -1165,31 +2840,24 @@ def _publish_raw_manifest_entry( year=year, sha256=sha256_actual or "", filename=filename, - prefix=r2_prefix, + prefix=resolved_r2_prefix, package_path=manifest_path, ), ) - storage = spec.get("storage") if isinstance(spec.get("storage"), dict) else {} - recorded_r2 = storage.get("r2") if isinstance(storage.get("r2"), dict) else {} - recorded_key = recorded_r2.get("key") - if recorded_key and recorded_key != location.key: - errors.append( - "recorded_r2_key_disagrees_with_country_prefix:" - f"recorded={recorded_key}:expected={location.key}" - ) + if preflight_only: return ( RawArtifactPublishEntry( manifest_path=str(manifest_path), - source_id=source_id, - package_id=package_id, + source_id=reported_source_id, + package_id=reported_package_id, year=str(year), filename=filename, local_path=str(artifact_path), sha256=sha256_actual, size_bytes=size_bytes, - r2_location=None, + r2_location=location, upload=None, - errors=tuple(errors), + errors=(), ), None, ) @@ -1217,8 +2885,8 @@ def _publish_raw_manifest_entry( return ( RawArtifactPublishEntry( manifest_path=str(manifest_path), - source_id=source_id, - package_id=package_id, + source_id=reported_source_id, + package_id=reported_package_id, year=str(year), filename=filename, local_path=str(artifact_path), @@ -1242,13 +2910,82 @@ def _inventory_entry( spec = {} errors.append("malformed_file_spec") filename = str(spec.get("filename") or "") - artifact_path = manifest_path.parent / filename - exists = bool(filename) and artifact_path.exists() + r2 = None + recorded_r2 = None + try: + recorded_r2 = _validated_recorded_r2( + spec, manifest_path=manifest_path, year=year + ) + except SourceArtifactManifestError as error: + errors.append(f"recorded_r2_locator_invalid:{error}") + if recorded_r2 is not None and ( + recorded_r2.filename != filename + or (spec.get("sha256") is not None and recorded_r2.sha256 != spec["sha256"]) + ): + errors.append("recorded_r2_identity_mismatch") + if errors: + return ArtifactInventoryEntry( + manifest_path=str(manifest_path), + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + exists=False, + sha256_expected=spec.get("sha256"), + sha256_actual=None, + size_bytes=None, + source_url=spec.get("source_url"), + r2=None, + errors=tuple(errors), + ) + if filename and not is_bare_filename(filename): + return ArtifactInventoryEntry( + manifest_path=str(manifest_path), + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + exists=False, + sha256_expected=spec.get("sha256"), + sha256_actual=None, + size_bytes=None, + source_url=spec.get("source_url"), + r2=r2, + errors=(f"non_canonical_filename:{filename}",), + ) + if is_manifest_filename(filename): + return ArtifactInventoryEntry( + manifest_path=str(manifest_path), + year=str(year), + filename=filename, + local_path=str(manifest_path.parent), + exists=False, + sha256_expected=spec.get("sha256"), + sha256_actual=None, + size_bytes=None, + source_url=spec.get("source_url"), + r2=r2, + errors=(f"manifest_named_filename:{filename}",), + ) + try: + artifact_path = ( + matching_directory_entry(manifest_path.parent, filename) + or manifest_path.parent / filename + ) + if filename and artifact_path.name != filename: + errors.append(f"artifact_spelling_mismatch:{filename}:{artifact_path.name}") + except ValueError: + errors.append(f"duplicate_artifact_spellings:{filename}") + artifact_path = manifest_path.parent / filename + symlink = bool(filename) and artifact_path.is_symlink() + exists = bool(filename) and not errors and not symlink and artifact_path.is_file() sha256_expected = spec.get("sha256") sha256_actual = None size_bytes = None if not filename: errors.append("missing_filename") + elif errors: + pass + elif symlink: + errors.append(f"artifact_path_is_symlink:{filename}") elif not exists: errors.append("missing_file") else: @@ -1257,8 +2994,16 @@ def _inventory_entry( size_bytes = len(content) if sha256_expected and sha256_actual != sha256_expected: errors.append("checksum_mismatch") - storage = spec.get("storage") if isinstance(spec, dict) else None - r2 = storage.get("r2") if isinstance(storage, dict) else None + if recorded_r2 is not None: + if sha256_actual is not None and sha256_actual != recorded_r2.sha256: + errors.append("recorded_r2_identity_mismatch") + if not errors: + r2 = { + "provider": recorded_r2.provider, + "bucket": recorded_r2.bucket, + "key": recorded_r2.key, + "uri": recorded_r2.uri, + } return ArtifactInventoryEntry( manifest_path=str(manifest_path), year=str(year), @@ -1275,7 +3020,7 @@ def _inventory_entry( def _derived_artifact_kind(artifact_name: str) -> str: - if artifact_name in {"ledger.db", "ledger.db"}: + if artifact_name in CHRONICLE_DB_FILENAMES: return "sqlite_database" if artifact_name.endswith(".jsonl"): return "jsonl" @@ -1308,6 +3053,33 @@ def _clean_key_part(value: str) -> str: return cleaned.replace(" ", "_") +def _require_identity_segment(value: Any, *, what: str) -> str: + """Require a registration identity to be one canonical key segment. + + ``_clean_key_part`` normalizes what it is given (strips, folds spaces to + underscores) because it also renders legacy recorded values; a NEW + registration identity must already be canonical, or two spellings such as + ``foo bar`` and ``foo_bar`` would collide in one R2 namespace and a + separator would shift the key's path shape. + """ + if ( + not isinstance(value, str) + or not value + or value in (".", "..") + or value != value.strip() + or any(character.isspace() for character in value) + or "/" in value + or "\\" in value + or _clean_key_part(value) != value + ): + raise IdentitySegmentError( + f"{what} must be one canonical R2 key segment (no whitespace, " + f"slashes, or '..'), not {value!r}; R2 key parts cannot be empty " + "or rewritten." + ) + return value + + def _clean_relative_key_parts(value: str) -> tuple[str, ...]: path = Path(value) if path.is_absolute(): diff --git a/chronicle/consumer_contract.py b/chronicle/consumer_contract.py index a1c78325..a98256d8 100644 --- a/chronicle/consumer_contract.py +++ b/chronicle/consumer_contract.py @@ -15,6 +15,7 @@ from pathlib import Path from typing import Any +from chronicle import artifacts from chronicle.core import ( ALLOWED_PROVENANCE_CLASSES, DEFAULT_ASSERTION, @@ -468,6 +469,33 @@ def validate_consumer_fact_contract( ) +def _r2_uri_parts(uri: str) -> tuple[str, str]: + """Split an ``r2://bucket/key`` URI into its bucket and key.""" + if uri[: len("r2://")].lower() != "r2://": + return "", "" + bucket, _, key = uri[len("r2://") :].partition("/") + return bucket, key + + +# The marker a downstream target row carries in its source_record_id. It moves +# with the rename window: producers write `ledger_derived` today and +# `chronicle_derived` once they migrate (PolicyEngine/chronicle#143, mechanism +# 3), so the boundary has to reject both spellings identically or the guard +# stops firing the moment a producer renames. +DERIVED_SOURCE_RECORD_SUFFIXES = frozenset({"ledger_derived", "chronicle_derived"}) + + +def _is_derived_source_record_id(source_record_id: str) -> bool: + """Whether a source_record_id marks a downstream derived target row.""" + _, separator, suffix = source_record_id.rpartition(".") + return bool(separator) and suffix in DERIVED_SOURCE_RECORD_SUFFIXES + + +def _points_at_derived(bucket: str, key: str) -> bool: + """Use the same derived routes publication is permitted to address.""" + return artifacts.is_derived_r2_route(bucket, key) + + def _derived_source_provenance_issue(fact: AggregateFact) -> str | None: """Return a boundary error if a fact is a downstream target derivation.""" source = fact.source @@ -489,26 +517,29 @@ def _derived_source_provenance_issue(fact: AggregateFact) -> str | None: "itself. Target construction, aging, and reconciliation belong in " "Microcosm." ) - if source_file.startswith("ledger-derived:"): + if source_file[: len("r2://")].lower() == "r2://": + source_file_bucket, source_file_key = _r2_uri_parts(source_file) + else: + source_file_bucket, bucket_separator, source_file_key = source_file.partition( + ":" + ) + if not bucket_separator: + source_file_bucket = source_file_key = "" + if _points_at_derived(source_file_bucket, source_file_key): return ( "Chronicle consumer facts must cite raw publisher artifacts. Derived " "target-construction artifacts belong in Microcosm." ) if ( - raw_r2_bucket.endswith("-derived") - or raw_r2_key.startswith("derived/") - or raw_r2_uri.startswith( - ( - "r2://ledger-derived/", - "r2://ledger-raw/derived/", - ) - ) + _points_at_derived(raw_r2_bucket, raw_r2_key) + or _points_at_derived(*_r2_uri_parts(raw_r2_uri)) + or _points_at_derived(*_r2_uri_parts(source.url or "")) ): return ( "Chronicle consumer facts must point at raw source artifacts, not " "derived build artifacts." ) - if source_record_id.endswith(".ledger_derived"): + if _is_derived_source_record_id(source_record_id): return ( "Chronicle source_record_id must identify a publisher-backed row, not " "a downstream derived target row." diff --git a/chronicle/database.py b/chronicle/database.py index 3f18ef78..5297db16 100644 --- a/chronicle/database.py +++ b/chronicle/database.py @@ -42,6 +42,15 @@ LEDGER_DB_SCHEMA_VERSION = schema_id("relational", Epoch.LEDGER) +# New suite outputs write chronicle.db. Existing builds wrote ledger.db and are +# still read and published unchanged, so the legacy name stays accepted for +# inference and artifact classification (PolicyEngine/chronicle#143, +# mechanism 3). The relational schema id above is a frozen machine surface that +# migrates with the epoch lane, not with this rename. +CHRONICLE_DB_FILENAME = "chronicle.db" +LEGACY_CHRONICLE_DB_FILENAME = "ledger.db" +CHRONICLE_DB_FILENAMES = (CHRONICLE_DB_FILENAME, LEGACY_CHRONICLE_DB_FILENAME) + @dataclass(frozen=True) class ChronicleDbBuildReport: diff --git a/chronicle/env.py b/chronicle/env.py new file mode 100644 index 00000000..0d4bbbad --- /dev/null +++ b/chronicle/env.py @@ -0,0 +1,161 @@ +"""Environment configuration for the Chronicle rename window. + +Chronicle's operational stores migrate by dual-run (PolicyEngine/chronicle#143, +mechanism 3): every configuration variable gets a ``CHRONICLE_``-prefixed name +that is read first, while the ledger-era ``LEDGER_`` and +``POLICYENGINE_LEDGER_`` names keep working behind a deprecation warning. That +window lets downstream publish flows migrate on their own schedule instead of +breaking the moment Chronicle ships a rename. + +Names that carry none of those three prefixes are read literally: this helper +renames the ledger-era surface, not every PolicyEngine variable. +""" + +from __future__ import annotations + +import os +from typing import TypeVar +import warnings + +__all__ = [ + "CHRONICLE_ENV_PREFIX", + "CHRONICLE_SCHEMA_ENV", + "ChronicleEnvDeprecationWarning", + "DEFAULT_CHRONICLE_SCHEMA", + "LEGACY_ENV_PREFIXES", + "default_chronicle_schema", + "env_flag", + "env_names", + "env_value", + "reset_env_deprecation_state", +] + +CHRONICLE_ENV_PREFIX = "CHRONICLE_" + +# Ordered most specific first so prefix stripping is unambiguous. +LEGACY_ENV_PREFIXES = ("POLICYENGINE_LEDGER_", "LEDGER_") + +TRUTHY_ENV_VALUES = frozenset({"1", "true", "yes", "on"}) + +CHRONICLE_SCHEMA_ENV = "CHRONICLE_SCHEMA" + +# The hosted Postgres schema is still named "ledger". Renaming the schema value +# is a later slice of PolicyEngine/chronicle#143, coordinated with the CI +# writers that already target it; only the variable that overrides the name has +# moved to the chronicle prefix. +DEFAULT_CHRONICLE_SCHEMA = "ledger" + + +class ChronicleEnvDeprecationWarning(FutureWarning): + """A ledger-era environment variable supplied a Chronicle setting. + + Subclasses :class:`FutureWarning` rather than :class:`DeprecationWarning` + so the notice reaches operators running the CLI, who are the people who + have to move the variable. ``DeprecationWarning`` is silenced by default + outside ``__main__``. + """ + + +_Default = TypeVar("_Default") + +_WARNED_LEGACY_NAMES: set[str] = set() + + +def _env_suffix(name: str) -> str | None: + """Return the rename-window suffix of ``name``, or None if it has none.""" + for prefix in (CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES): + if name.startswith(prefix) and len(name) > len(prefix): + return name[len(prefix) :] + return None + + +def env_names(name: str) -> tuple[str, ...]: + """Return the lookup order for ``name``. + + The chronicle-preferred name comes first, then the ledger-era names that + remain accepted during the migration window. A name outside the rename + window is returned unchanged, as its own single-element lookup order. + """ + suffix = _env_suffix(name) + if suffix is None: + return (name,) + return ( + f"{CHRONICLE_ENV_PREFIX}{suffix}", + *(f"{prefix}{suffix}" for prefix in LEGACY_ENV_PREFIXES), + ) + + +def _warn_legacy(found: str, preferred: str) -> None: + """Warn once per process that a ledger-era variable supplied a value. + + ``stacklevel=4`` walks out through :func:`_first_set` and its public + wrapper so the notice points at the code that asked for the setting. + """ + if found in _WARNED_LEGACY_NAMES: + return + _WARNED_LEGACY_NAMES.add(found) + warnings.warn( + f"{found} is a ledger-era Chronicle environment variable; " + f"set {preferred} instead. The old name is still honored during the " + "Chronicle rename window and will be removed once consumers migrate.", + ChronicleEnvDeprecationWarning, + stacklevel=4, + ) + + +def reset_env_deprecation_state() -> None: + """Forget which legacy names have already warned. Test-support hook.""" + _WARNED_LEGACY_NAMES.clear() + + +def _first_set(names: tuple[str, ...]) -> str | None: + """Return the first set value across ``names``, warning on a legacy hit. + + Both public readers call this at the same stack depth so the deprecation + warning is always attributed to their caller, not to this module. + """ + for name in names: + candidates = env_names(name) + preferred = candidates[0] + for candidate in candidates: + value = os.environ.get(candidate) + if value: + if candidate != preferred: + _warn_legacy(candidate, preferred) + return value + return None + + +def env_value(*names: str, default: _Default = None) -> str | _Default: + """Read the first set value across ``names``, chronicle-preferred first. + + Each name is expanded through :func:`env_names`, so a caller can pass the + chronicle name and still pick up a value set under a ledger-era name. + Empty values are treated as unset, matching the helpers this replaces. + """ + value = _first_set(names) + return default if value is None else value + + +def default_chronicle_schema() -> str: + """Resolve the Chronicle schema: ``$CHRONICLE_SCHEMA``, else the default. + + Every reader of the setting goes through this function so the lookup ladder + and the default have one home. It resolves at call time rather than at + import: a module-level constant binds whatever the shell held when the + module was first imported, which for a library means an arbitrary moment + the caller cannot control, and for the test suite means collection. + """ + return env_value(CHRONICLE_SCHEMA_ENV, default=DEFAULT_CHRONICLE_SCHEMA) + + +def env_flag(*names: str) -> bool: + """Return whether the first set value across ``names`` reads as true. + + The chronicle-preferred name wins even when it reads false, so an operator + who has migrated can turn a flag off without unsetting the legacy name. + """ + value = _first_set(names) + if value is None: + return False + return value.strip().lower() in TRUTHY_ENV_VALUES diff --git a/chronicle/harness.py b/chronicle/harness.py index 298a1380..a100bea4 100644 --- a/chronicle/harness.py +++ b/chronicle/harness.py @@ -5,14 +5,19 @@ import argparse import json import shlex +import sys from pathlib import Path from chronicle.artifacts import ( + DEFAULT_MANIFEST_FILENAME, + DEFAULT_R2_DERIVED_BUCKET, + DEFAULT_R2_RAW_BUCKET, ArtifactFetchReport, ArtifactInventoryReport, DerivedArtifactPublishReport, R2BootstrapReport, RawArtifactPublishReport, + SourceArtifactManifestError, bootstrap_r2_buckets, fetch_source_artifact, inventory_source_artifacts, @@ -33,6 +38,7 @@ ) from chronicle.core import AggregateFact, ValidationReport, validate_facts from chronicle.database import ChronicleDbBuildReport, build_chronicle_db +from chronicle.env import DEFAULT_CHRONICLE_SCHEMA from chronicle.mirror import ( ChronicleMirrorExportReport, SupabaseMirrorLoadReport, @@ -334,12 +340,20 @@ def fetch_artifact_file( source_page: str | None = None, table: str | None = None, filename: str | None = None, + manifest_filename: str = DEFAULT_MANIFEST_FILENAME, upload_r2: bool = False, - r2_bucket: str = "ledger-raw", + record_revision: bool = False, + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", ) -> ArtifactFetchReport: - """Fetch/register a raw source artifact and optionally upload it to R2.""" + """Fetch/register a raw source artifact and optionally upload it to R2. + + ``manifest_filename`` selects which of the package directory's manifests + the entry belongs to. Raises :class:`SourceArtifactRevisionError` when the + fetched bytes are not the bytes that manifest's entry identifies, unless + ``record_revision`` opts into registering the publisher revision. + """ return fetch_source_artifact( source_url, source_id=source_id, @@ -350,7 +364,9 @@ def fetch_artifact_file( source_page=source_page, table=table, filename=filename, + manifest_filename=manifest_filename, upload_r2=upload_r2, + record_revision=record_revision, r2_bucket=r2_bucket, r2_prefix=r2_prefix, wrangler_command=wrangler_command, @@ -372,7 +388,7 @@ def publish_raw_artifact_files( manifest_filename: str = "manifest.yaml", source_id: str | None = None, package_id: str | None = None, - r2_bucket: str = "ledger-raw", + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", ) -> RawArtifactPublishReport: @@ -390,8 +406,8 @@ def publish_raw_artifact_files( def bootstrap_r2_storage( *, - raw_bucket: str = "ledger-raw", - derived_bucket: str = "ledger-derived", + raw_bucket: str | None = None, + derived_bucket: str | None = None, wrangler_command: str = "npx wrangler", ) -> R2BootstrapReport: """Create Chronicle R2 buckets when Wrangler is authenticated.""" @@ -409,7 +425,7 @@ def publish_derived_artifact_files( package_id: str, year: int, build_id: str | None = None, - r2_bucket: str = "ledger-derived", + r2_bucket: str | None = None, r2_prefix: str | None = None, wrangler_command: str = "npx wrangler", build_artifacts_output: str | Path | None = None, @@ -441,12 +457,16 @@ def export_chronicle_db_table_files( def load_supabase_mirror_files( input_dir: str | Path, *, - schema: str = "ledger", + schema: str | None = None, batch_size: int = 500, dry_run: bool = False, build_artifacts_path: str | Path | None = None, ) -> SupabaseMirrorLoadReport: - """Load exported Chronicle JSONL mirror files into Supabase/Postgres.""" + """Load exported Chronicle JSONL mirror files into Supabase/Postgres. + + ``schema`` of None resolves to ``$CHRONICLE_SCHEMA``, else the default + schema, so the hosted mirror writer answers to the renamed variable. + """ table_paths = ( {"build_artifacts": Path(build_artifacts_path)} if build_artifacts_path is not None @@ -836,7 +856,7 @@ def main(argv: list[str] | None = None) -> int: artifact_parser = subparsers.add_parser( "fetch-artifact", - help="Fetch/register a raw source artifact and update manifest.yaml", + help="Fetch/register a raw source artifact and update its manifest", ) artifact_parser.add_argument( "--url", @@ -857,13 +877,23 @@ def main(argv: list[str] | None = None) -> int: "--year", type=int, required=True, - help="Artifact vintage year to record in manifest.yaml", + help="Artifact vintage year to record in the manifest", ) artifact_parser.add_argument( "--out-dir", type=Path, required=True, - help="Directory where the raw artifact and manifest.yaml should live", + help="Directory where the raw artifact and its manifest should live", + ) + artifact_parser.add_argument( + "--manifest", + default=DEFAULT_MANIFEST_FILENAME, + help=( + "Manifest filename inside --out-dir. A publisher directory that " + "feeds several source packages keeps one manifest each, and the " + "entry being revised lives in exactly one of them. Defaults to " + f"{DEFAULT_MANIFEST_FILENAME}." + ), ) artifact_parser.add_argument( "--dataset", @@ -886,10 +916,25 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="Upload the artifact to R2 after local checksum capture.", ) + artifact_parser.add_argument( + "--record-revision", + action="store_true", + help=( + "Register a publisher revision: the fetched bytes get their own " + "content-addressed key under the configured bucket and the " + "superseded object moves to storage.previous_r2. Without this " + "flag, bytes that disagree with what the entry identifies -- its " + "declared sha256, or its recorded content-addressed key once " + "published -- are refused." + ), + ) artifact_parser.add_argument( "--r2-bucket", - default="ledger-raw", - help="R2 bucket for raw artifacts when --upload-r2 is set.", + default=None, + help=( + "R2 bucket for raw artifacts when --upload-r2 is set. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) artifact_parser.add_argument( "--r2-prefix", @@ -923,7 +968,7 @@ def main(argv: list[str] | None = None) -> int: raw_publish_parser = subparsers.add_parser( "publish-raw", - help="Upload manifest-declared raw source artifacts to ledger-raw R2", + help="Upload manifest-declared raw source artifacts to the raw R2 bucket", ) raw_publish_parser.add_argument( "--root", @@ -946,8 +991,11 @@ def main(argv: list[str] | None = None) -> int: ) raw_publish_parser.add_argument( "--r2-bucket", - default="ledger-raw", - help="R2 bucket for immutable raw artifacts.", + default=None, + help=( + "R2 bucket for immutable raw artifacts. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) raw_publish_parser.add_argument( "--r2-prefix", @@ -969,13 +1017,19 @@ def main(argv: list[str] | None = None) -> int: ) r2_parser.add_argument( "--raw-bucket", - default="ledger-raw", - help="R2 bucket name for immutable raw source artifacts.", + default=None, + help=( + "R2 bucket name for immutable raw source artifacts. Defaults to " + f"$CHRONICLE_R2_RAW_BUCKET, else {DEFAULT_R2_RAW_BUCKET}." + ), ) r2_parser.add_argument( "--derived-bucket", - default="ledger-derived", - help="R2 bucket name for derived Chronicle build artifacts.", + default=None, + help=( + "R2 bucket name for derived Chronicle build artifacts. Defaults to " + f"$CHRONICLE_R2_DERIVED_BUCKET, else {DEFAULT_R2_DERIVED_BUCKET}." + ), ) r2_parser.add_argument( "--wrangler-command", @@ -985,7 +1039,7 @@ def main(argv: list[str] | None = None) -> int: derived_publish_parser = subparsers.add_parser( "publish-derived", - help="Upload deterministic Chronicle build outputs to ledger-derived R2", + help="Upload deterministic Chronicle build outputs to the derived R2 bucket", ) derived_publish_parser.add_argument( "--dir", @@ -1014,13 +1068,16 @@ def main(argv: list[str] | None = None) -> int: help=( "Build ID under an accepted epoch prefix, ledger.build.v1: or " "chronicle.build.v2:; any other form is refused. Defaults to " - "the ID inferred from reports or ledger.db." + "the ID inferred from reports, chronicle.db, or a legacy ledger.db." ), ) derived_publish_parser.add_argument( "--r2-bucket", - default="ledger-derived", - help="R2 bucket for derived build artifacts.", + default=None, + help=( + "R2 bucket for derived build artifacts. Defaults to " + f"$CHRONICLE_R2_DERIVED_BUCKET, else {DEFAULT_R2_DERIVED_BUCKET}." + ), ) derived_publish_parser.add_argument( "--r2-prefix", @@ -1075,8 +1132,11 @@ def main(argv: list[str] | None = None) -> int: ) mirror_load_parser.add_argument( "--schema", - default="ledger", - help="Supabase/Postgres schema to load into.", + default=None, + help=( + "Supabase/Postgres schema to load into. Defaults to " + f"$CHRONICLE_SCHEMA, else {DEFAULT_CHRONICLE_SCHEMA}." + ), ) mirror_load_parser.add_argument( "--batch-size", @@ -1305,40 +1365,54 @@ def main(argv: list[str] | None = None) -> int: print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if args.command == "fetch-artifact": - report = fetch_artifact_file( - args.url, - source_id=args.source_id, - package_id=args.package_id, - year=args.year, - output_dir=args.out_dir, - dataset=args.dataset, - source_page=args.source_page, - table=args.table, - filename=args.filename, - upload_r2=args.upload_r2, - r2_bucket=args.r2_bucket, - r2_prefix=args.r2_prefix, - wrangler_command=args.wrangler_command, - ) + try: + report = fetch_artifact_file( + args.url, + source_id=args.source_id, + package_id=args.package_id, + year=args.year, + output_dir=args.out_dir, + dataset=args.dataset, + source_page=args.source_page, + table=args.table, + filename=args.filename, + manifest_filename=args.manifest, + upload_r2=args.upload_r2, + record_revision=args.record_revision, + r2_bucket=args.r2_bucket, + r2_prefix=args.r2_prefix, + wrangler_command=args.wrangler_command, + ) + except SourceArtifactManifestError as error: + print(f"error: {error}", file=sys.stderr) + return 1 print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 if args.command == "inventory-artifacts": - report = inventory_artifact_files( - args.root, - manifest_filename=args.manifest, - ) + try: + report = inventory_artifact_files( + args.root, + manifest_filename=args.manifest, + ) + except SourceArtifactManifestError as error: + print(f"error: {error}", file=sys.stderr) + return 1 print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 if args.command == "publish-raw": - report = publish_raw_artifact_files( - args.root, - manifest_filename=args.manifest, - source_id=args.source_id, - package_id=args.package_id, - r2_bucket=args.r2_bucket, - r2_prefix=args.r2_prefix, - wrangler_command=args.wrangler_command, - ) + try: + report = publish_raw_artifact_files( + args.root, + manifest_filename=args.manifest, + source_id=args.source_id, + package_id=args.package_id, + r2_bucket=args.r2_bucket, + r2_prefix=args.r2_prefix, + wrangler_command=args.wrangler_command, + ) + except SourceArtifactManifestError as error: + print(f"error: {error}", file=sys.stderr) + return 1 print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) return 0 if report.valid else 1 if args.command == "bootstrap-r2": diff --git a/chronicle/mirror.py b/chronicle/mirror.py index 7a3ccaf3..f9c95dc2 100644 --- a/chronicle/mirror.py +++ b/chronicle/mirror.py @@ -11,6 +11,8 @@ from pathlib import Path from typing import Any +from chronicle.env import default_chronicle_schema + LEDGER_MIRROR_TABLES = ( "ledger_builds", "build_artifacts", @@ -200,15 +202,22 @@ def export_chronicle_db_tables( def load_supabase_mirror( input_dir: str | Path, *, - schema: str = "ledger", + schema: str | None = None, batch_size: int = 500, dry_run: bool = False, table_paths: dict[str, str | Path] | None = None, client: Any | None = None, ) -> SupabaseMirrorLoadReport: - """Load exported Chronicle JSONL mirror files into Supabase/Postgres.""" + """Load exported Chronicle JSONL mirror files into Supabase/Postgres. + + ``schema`` defaults to :func:`chronicle.env.default_chronicle_schema`, so + the writer that owns the hosted mirror answers to ``CHRONICLE_SCHEMA`` (and + the ledger-era names behind it) exactly like every other reader of the + setting. The resolved name is reported back in the load report. + """ if batch_size < 1: raise ValueError("batch_size must be at least 1.") + schema = schema or default_chronicle_schema() input_path = Path(input_dir) tables: list[SupabaseTableLoad] = [] errors: list[str] = [] diff --git a/chronicle/registration.py b/chronicle/registration.py new file mode 100644 index 00000000..f9f5393e --- /dev/null +++ b/chronicle/registration.py @@ -0,0 +1,334 @@ +"""Shared source-artifact registration primitives. + +This module holds manifest parsing and filename identity rules used at every +artifact boundary. PR #227 extends the same surface with access-specific +registration; keeping the common functions here lets that stacked work rebase +without inventing parallel helpers. +""" + +from __future__ import annotations + +from pathlib import Path +import re +from typing import Any, Callable, Mapping +import unicodedata + +import yaml + + +class ArtifactFilenameError(ValueError): + """Raised when a filename is not a bare name inside a package directory.""" + + +def is_bare_filename(value: Any) -> bool: + """Whether ``value`` names a file inside a directory, with no path.""" + if value is None: + return False + text = str(value).strip() + if not text or text != str(value) or text in (".", ".."): + return False + if "/" in text or "\\" in text or "\x00" in text: + return False + return Path(text).name == text + + +def bare_filename(value: Any, *, what: str = "filename") -> str: + """Return ``value`` as a bare filename, refusing any other spelling. + + ``./adult.tab``, ``sub/../adult.tab``, ``adult.tab/`` and an absolute path + all resolve to the same file as ``adult.tab`` once joined under the package + directory, so the manifest and every guard use one spelling. + """ + if not is_bare_filename(value): + raise ArtifactFilenameError( + f"{what} must be a bare filename inside the package directory, not " + f"{value!r}; it may not carry a directory, '.', '..', a trailing " + "slash, surrounding whitespace, or an absolute path." + ) + return str(value) + + +def filename_key(value: Any) -> str: + """Return the case-folded, Unicode-normalized comparison key for a name.""" + return unicodedata.normalize("NFC", Path(str(value)).name).casefold() + + +_MANIFEST_FILENAME_RE = re.compile( + r"^manifest(?:_[^/\\]+)?\.ya?ml$", + re.IGNORECASE, +) + + +def is_manifest_filename(value: Any) -> bool: + """Whether ``value`` is a package-manifest filename.""" + return is_bare_filename(value) and bool(_MANIFEST_FILENAME_RE.fullmatch(str(value))) + + +def package_manifest_paths(package_dir: Path) -> list[Path]: + """Return every manifest file a package directory keeps, sorted by name.""" + directory = Path(package_dir) + if not directory.is_dir(): + return [] + manifests = [] + for path in sorted(directory.iterdir()): + if not is_manifest_filename(path.name): + continue + if path.is_symlink() or not path.is_file(): + raise ValueError( + f"{path} carries a manifest name but is not a regular file; " + "Chronicle will not register beside it or sweep past it." + ) + manifests.append(path) + return manifests + + +def validate_package_directory( + manifests: Mapping[str, Mapping[str, Any] | None], + *, + entry_digest: Callable[[str, Any, Mapping[str, Any]], str | None] | None = None, +) -> tuple[str, ...]: + """Return filename-identity collisions across a package's manifests. + + Two manifests may name one physical file only when every identity they + record for it agrees. A differing digest means the same package-local + bytes have two incompatible identities, so no command may act through + either record. An entry that records no identity yet -- no ``sha256`` and + no locator, as a manifest looks before its first fetch or publication -- + cannot contradict an identified owner and is not a collision: the command + that eventually identifies it hashes the shared bytes, so a selected or + partially failed publication never strands its siblings behind a false + collision. + + ``entry_digest(manifest_name, vintage, entry)`` resolves an entry's + *effective* recorded digest -- for example the one its content-addressed + R2 key encodes when the entry declares no ``sha256`` -- and returns + ``None`` to fall back to the declared ``sha256`` field. Without it only the + declared field is compared. + """ + by_name: dict[str, list[tuple[str, str]]] = {} + for name, manifest in manifests.items(): + files = manifest.get("files") if isinstance(manifest, Mapping) else None + if not isinstance(files, Mapping): + continue + for vintage, entry in files.items(): + if not isinstance(entry, Mapping): + continue + filename = entry.get("filename") + if filename is None: + continue + digest = entry_digest(name, vintage, entry) if entry_digest else None + if digest is None: + digest = entry.get("sha256") + digest = digest.strip() if isinstance(digest, str) else "" + if not digest: + # No recorded identity yet: nothing to contradict. + continue + by_name.setdefault(filename_key(filename), []).append((name, digest)) + + errors: list[str] = [] + for key, records in by_name.items(): + if len({name for name, _digest in records}) < 2: + continue + if len({digest for _name, digest in records}) > 1: + errors.append(f"filename_collision_across_manifests:{key}") + return tuple(dict.fromkeys(errors)) + + +def matching_directory_entry(directory: Any, filename: Any) -> Any | None: + """Return the actual directory entry matching a bare filename's safe key. + + Scanning real entries makes the identity rule the same on case-sensitive + and case-folding filesystems, including Unicode-normalized aliases. + """ + if not is_bare_filename(filename) or not directory.is_dir(): + return None + wanted = filename_key(filename) + matches = [ + path + for path in sorted(directory.iterdir(), key=lambda item: item.name) + if filename_key(path.name) == wanted + ] + if len(matches) > 1: + names = ", ".join(repr(path.name) for path in matches) + raise ValueError( + f"{filename!r} matches more than one physical entry ({names}); " + "the package holds conflicting spellings of one artifact identity " + "and must be repaired by hand." + ) + return matches[0] if matches else None + + +class StrictManifestLoader(yaml.SafeLoader): + """A YAML loader that refuses a mapping with duplicate keys. + + PyYAML keeps the last of two equal keys, so ``files:`` recorded twice, or + a vintage recorded as ``2023`` and again as ``2_023`` (the same integer), + would read as one entry and the shadowed entry would be dropped by the + next write. A manifest is the record the byte boundary is decided from, + so a document the loader cannot represent faithfully is malformed. + + ``<<`` merges are honoured with YAML precedence (an explicit key overrides + a merged one) but never by mutating a node: PyYAML's ``flatten_mapping`` + rewrites the node in place, and because construction is lazy a later + merge of an anchored entry would turn that entry's inherited keys into + apparent explicit duplicates. Merge sources are expanded into a fresh + pair list instead, and every mapping reached through a merge gets the + same duplicate check as a mapping the document spells out directly. + """ + + _MERGE_TAG = "tag:yaml.org,2002:merge" + + def _explicit_pairs(self, node: Any, deep: bool) -> tuple[list[Any], list[Any]]: + """Split a mapping node into merge sources and its explicit pairs. + + Refuses duplicate explicit keys on the node's own pair list, before any + merge is consulted. + """ + merge_sources: list[Any] = [] + explicit_pairs: list[Any] = [] + seen: set[Any] = set() + for key_node, value_node in node.value: + if key_node.tag == self._MERGE_TAG: + merge_sources.append(value_node) + continue + key = self.construct_object(key_node, deep=deep) + try: + hash(key) + except TypeError as exc: + raise yaml.constructor.ConstructorError( + "while constructing a mapping", + node.start_mark, + f"found unhashable key ({exc})", + key_node.start_mark, + ) from exc + if key in seen: + raise yaml.constructor.ConstructorError( + "while constructing a mapping", + node.start_mark, + f"found duplicate key {key!r}", + key_node.start_mark, + ) + seen.add(key) + explicit_pairs.append((key_node, value_node)) + return merge_sources, explicit_pairs + + def _merged_pairs( + self, source: Any, deep: bool, active: tuple[int, ...] + ) -> list[Any]: + """Return the pairs a ``<<`` source contributes, validated, unmutated. + + ``active`` holds the mappings whose merges are being expanded on the + way to ``source``. A source that is already active merges itself, + directly or through nested merges, which has no expansion: refuse it + as a ``ConstructorError`` (a ``yaml.YAMLError`` every manifest reader + handles) rather than recursing until the interpreter gives up. + """ + if isinstance(source, yaml.MappingNode): + if id(source) in active: + raise yaml.constructor.ConstructorError( + "while constructing a mapping", + source.start_mark, + "found a recursive merge: a mapping merges itself", + source.start_mark, + ) + nested_sources, explicit = self._explicit_pairs(source, deep) + nested_active = (*active, id(source)) + pairs: list[Any] = [] + for nested in nested_sources: + pairs.extend(self._merged_pairs(nested, deep, nested_active)) + pairs.extend(explicit) + return pairs + if isinstance(source, yaml.SequenceNode): + for subnode in source.value: + if not isinstance(subnode, yaml.MappingNode): + raise yaml.constructor.ConstructorError( + "while constructing a mapping", + source.start_mark, + f"expected a mapping for merging, but found {subnode.id}", + subnode.start_mark, + ) + # YAML merge-key precedence: earlier mappings in a ``<<`` sequence + # win over later ones. Pairs are assigned last-wins, so contribute + # the later mappings first and the first mapping last. + pairs = [] + for subnode in reversed(source.value): + pairs.extend(self._merged_pairs(subnode, deep, active)) + return pairs + raise yaml.constructor.ConstructorError( + "while constructing a mapping", + source.start_mark, + f"expected a mapping or list of mappings for merging, but found {source.id}", + source.start_mark, + ) + + def construct_mapping(self, node: Any, deep: bool = False) -> dict[Any, Any]: + if not isinstance(node, yaml.MappingNode): + raise yaml.constructor.ConstructorError( + None, + None, + f"expected a mapping node, but found {node.id}", + node.start_mark, + ) + merge_sources, explicit_pairs = self._explicit_pairs(node, deep) + merged_pairs: list[Any] = [] + for source in merge_sources: + merged_pairs.extend(self._merged_pairs(source, deep, (id(node),))) + mapping: dict[Any, Any] = {} + # Merged pairs first, explicit pairs last: YAML precedence, last wins. + for key_node, value_node in [*merged_pairs, *explicit_pairs]: + key = self.construct_object(key_node, deep=deep) + mapping[key] = self.construct_object(value_node, deep=deep) + return mapping + + +def validate_manifest_vintages(payload: Any) -> None: + """Refuse different keys that identify one logical ``files`` vintage. + + YAML distinguishes integer ``2024`` from quoted ``"2024"``, but manifest + consumers select or report them as the same vintage. Validate the entire + manifest, including vintages other than the one a caller requested, before + any consumer can read artifact bytes or construct publication routes. + + Leave non-mapping documents and ``files`` blocks to the consumers' existing + shape checks. Labels retain their spelling, including leading zeroes. + """ + files = payload.get("files") if isinstance(payload, Mapping) else None + if not isinstance(files, Mapping): + return + seen: dict[str, Any] = {} + for vintage in files: + identity = str(vintage) + if identity in seen: + raise yaml.YAMLError( + f"Vintage {identity!r} is recorded under both keys " + f"{seen[identity]!r} and {vintage!r}; one vintage has one key. " + "Merge the entries by hand first. Chronicle will not choose " + "which entry is the record." + ) + seen[identity] = vintage + + +def load_manifest_document(text: str) -> Any: + """Parse a manifest document, refusing duplicate keys and vintages. + + Raises :class:`yaml.YAMLError` for keys YAML would silently collapse or + for distinct YAML keys that manifest consumers treat as one vintage. + """ + payload = yaml.load(text, Loader=StrictManifestLoader) # noqa: S506 + validate_manifest_vintages(payload) + return payload + + +__all__ = [ + "ArtifactFilenameError", + "StrictManifestLoader", + "bare_filename", + "filename_key", + "is_bare_filename", + "is_manifest_filename", + "load_manifest_document", + "matching_directory_entry", + "package_manifest_paths", + "validate_manifest_vintages", + "validate_package_directory", +] diff --git a/chronicle/source_package.py b/chronicle/source_package.py index 5f6a5709..0738b08b 100644 --- a/chronicle/source_package.py +++ b/chronicle/source_package.py @@ -3,7 +3,6 @@ from __future__ import annotations import hashlib -import os from dataclasses import dataclass, replace from importlib.resources import files from io import BytesIO @@ -16,6 +15,7 @@ import httpx import yaml +from chronicle.artifacts import SourceArtifactManifestError, _validated_recorded_r2 from chronicle.core import ( ALLOWED_AGGREGATIONS, ALLOWED_ASSERTIONS, @@ -32,7 +32,14 @@ AggregateFact, build_label, ) +from chronicle.env import env_flag, env_value from chronicle.epoch import SCHEMA_IDS, schema_id +from chronicle.registration import ( + is_bare_filename, + is_manifest_filename, + load_manifest_document, + matching_directory_entry, +) from chronicle.sources.cells import ( SourceArtifactMetadata, SourceCell, @@ -392,8 +399,8 @@ "usda_snap/fy2025_monthly_state_caseloads" ), } -SOURCE_ARTIFACT_CACHE_ENV = "LEDGER_SOURCE_ARTIFACT_CACHE_DIR" -SOURCE_ARTIFACT_FETCH_ENV = "LEDGER_SOURCE_ARTIFACT_FETCH" +SOURCE_ARTIFACT_CACHE_ENV = "CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR" +SOURCE_ARTIFACT_FETCH_ENV = "CHRONICLE_SOURCE_ARTIFACT_FETCH" DEFAULT_SOURCE_ARTIFACT_CACHE_DIR = ( Path.home() / ".cache" / "policyengine-chronicle" / "source-artifacts" ) @@ -868,32 +875,177 @@ def _source_artifact_metadata( raw_r2_uri=raw_r2.get("uri"), ) + def _resource_root(self) -> Any: + """Resolve the resource directory, refusing an escape from the package. + + ``resource_directory`` is joined under ``files(resource_package)``; an + absolute value would discard that root entirely, a ``..`` or ``.`` + component would step outside it, and a symlinked ancestor would follow + the link out of the package tree. Every byte and manifest read goes + through here, so the containment check runs before any I/O. + """ + raw = self.resource_directory + parts = str(raw).split("/") + if ( + not isinstance(raw, str) + or not raw + or raw.startswith("/") + or "\\" in raw + or any( + not part or part in (".", "..") or part != part.strip() + for part in parts + ) + ): + raise ValueError( + f"resource_directory must be a relative path of plain segments " + f"inside the resource package, not {raw!r}." + ) + root = files(self.resource_package) + directory = root.joinpath(raw) + if isinstance(root, Path): + current = root + for part in parts: + current = current / part + if current.is_symlink(): + raise ValueError( + f"resource_directory component {current} is a symbolic " + "link. Chronicle will not read source-package data " + "through it." + ) + resolved_root = root.resolve() + if not Path(directory).resolve().is_relative_to(resolved_root): + raise ValueError( + f"resource_directory {raw!r} escapes the resource package " + f"root {resolved_root}." + ) + return directory + + def _resource_entry( + self, + value: Any, + *, + what: str, + require_manifest_name: bool = False, + forbid_manifest_name: bool = False, + ) -> Any: + """Resolve one safe file entry under the package resource directory.""" + if not is_bare_filename(value): + raise ValueError( + f"{what} must be a bare filename inside " + f"{self.resource_directory}, not {value!r}." + ) + name = str(value) + if require_manifest_name and not is_manifest_filename(name): + raise ValueError( + f"{what} must be named manifest.yaml or " + f"manifest_.yaml, not {name!r}." + ) + if forbid_manifest_name and is_manifest_filename(name): + raise ValueError( + f"{what} {name!r} is a manifest name and cannot be read as " + "source artifact bytes." + ) + + directory = self._resource_root() + existing = matching_directory_entry(directory, name) + if existing is None: + return directory.joinpath(name) + is_symlink = getattr(existing, "is_symlink", None) + if callable(is_symlink) and is_symlink(): + raise ValueError( + f"{what} {existing} is a symbolic link. Chronicle will not " + "read source-package data through it." + ) + if existing.name != name: + raise ValueError( + f"{what} {existing} has the same normalized filename as " + f"{name!r}. Keep exactly one spelling in the package." + ) + if not existing.is_file(): + raise ValueError( + f"{what} {existing} is not a regular file. Chronicle will " + "not open non-regular source-package resources." + ) + return existing + + def manifest_resource(self) -> Any: + """Return the validated manifest file this package spec points at.""" + return self._resource_entry( + self.manifest, + what="Source artifact manifest", + require_manifest_name=True, + ) + + def manifest_payload(self) -> dict[str, Any]: + """Load the artifact manifest strictly as a YAML mapping.""" + with self.manifest_resource().open("r", encoding="utf-8") as file: + text = file.read() + try: + payload = load_manifest_document(text) + except yaml.YAMLError as exc: + raise ValueError( + f"{self.resource_directory}/{self.manifest} is not valid YAML: {exc}" + ) from exc + if payload is None: + return {} + if not isinstance(payload, dict): + raise ValueError( + f"{self.resource_directory}/{self.manifest} must be a YAML " + f"mapping; it parses as a {type(payload).__name__}." + ) + return payload + def _artifact_content( self, year: int, ) -> tuple[bytes, str, str, dict[str, str]]: - manifest_path = files(self.resource_package).joinpath( - self.resource_directory, - self.manifest, - ) - with manifest_path.open("r", encoding="utf-8") as file: - manifest = yaml.safe_load(file) + manifest = self.manifest_payload() spec = _year_mapping(manifest["files"], self.artifact_year or year) - artifact_path = files(self.resource_package).joinpath( - self.resource_directory, - spec["filename"], + filename = spec.get("filename") + artifact_path = self._resource_entry( + filename, + what="Source artifact filename", + forbid_manifest_name=True, ) - content = _read_source_artifact_content(artifact_path, spec) + try: + recorded_r2 = _validated_recorded_r2( + spec, + manifest_path=Path(self.resource_directory) / self.manifest, + year=self.artifact_year or year, + ) + except SourceArtifactManifestError as exc: + raise ValueError(str(exc)) from exc expected_sha = spec.get("sha256") + raw_r2 = {} + if recorded_r2 is not None: + if recorded_r2.filename != filename or ( + expected_sha is not None and expected_sha != recorded_r2.sha256 + ): + raise ValueError( + f"Source artifact {filename!r} disagrees with its recorded " + f"R2 identity: storage.r2 names {recorded_r2.filename!r} " + f"with sha256={recorded_r2.sha256}, while the manifest " + f"declares sha256={expected_sha!r}." + ) + expected_sha = recorded_r2.sha256 + # The immutable object also supplies the checksum for a manifest + # without a separate sha256 field. Pass it into the fetch/cache + # reader so wrong publisher bytes are refused before cache writes. + spec = {**spec, "sha256": expected_sha} + raw_r2 = { + "provider": recorded_r2.provider, + "bucket": recorded_r2.bucket, + "key": recorded_r2.key, + "uri": recorded_r2.uri, + } + content = _read_source_artifact_content(artifact_path, spec) if expected_sha: _validate_source_artifact_sha( content, expected_sha=str(expected_sha), - filename=str(spec["filename"]), + filename=str(filename), ) - storage = spec.get("storage") if isinstance(spec, dict) else None - raw_r2 = storage.get("r2") if isinstance(storage, dict) else {} - return content, spec["filename"], spec["source_url"], raw_r2 or {} + return content, str(filename), spec["source_url"], raw_r2 def _sheet_name(self, filename: str, *, year: int) -> str: if self.sheet_name: @@ -2257,7 +2409,7 @@ def _read_source_artifact_content( if cache_path.exists(): return cache_path.read_bytes() - if not _truthy_env(SOURCE_ARTIFACT_FETCH_ENV): + if not env_flag(SOURCE_ARTIFACT_FETCH_ENV): raise FileNotFoundError( f"Source artifact {spec['filename']} is not packaged and was not " f"found in {cache_path}. Set {SOURCE_ARTIFACT_FETCH_ENV}=1 to fetch " @@ -2279,7 +2431,7 @@ def _read_source_artifact_content( def _source_artifact_cache_path(spec: dict[str, Any]) -> Path: cache_root = Path( - _env_value( + env_value( SOURCE_ARTIFACT_CACHE_ENV, default=DEFAULT_SOURCE_ARTIFACT_CACHE_DIR, ) @@ -2316,21 +2468,6 @@ def _validate_source_artifact_sha( ) -def _env_value(*names: str, default: str | Path) -> str | Path: - for name in names: - value = os.environ.get(name) - if value: - return value - return default - - -def _truthy_env(*names: str) -> bool: - return any( - os.environ.get(name, "").strip().lower() in {"1", "true", "yes", "on"} - for name in names - ) - - def _single_archive_member(archive: ZipFile, *, suffixes: tuple[str, ...]) -> str: members = [ name diff --git a/chronicle/suite.py b/chronicle/suite.py index 157b7420..0705c862 100644 --- a/chronicle/suite.py +++ b/chronicle/suite.py @@ -22,7 +22,11 @@ build_fact_key, validate_facts, ) -from chronicle.database import ChronicleDbBuildReport, build_chronicle_db +from chronicle.database import ( + CHRONICLE_DB_FILENAME, + ChronicleDbBuildReport, + build_chronicle_db, +) from chronicle.epoch import canonicalize_key from chronicle.sources.cells import ( SourceCell, @@ -372,7 +376,7 @@ def build_source_suite( concept_report.to_dict(), ) - db_path = output_path / "ledger.db" + db_path = output_path / CHRONICLE_DB_FILENAME db_report = build_chronicle_db( facts, db_path, @@ -1675,7 +1679,7 @@ def _write_package_sidecars(output_path: Path, *, source: str, year: int) -> Non output_path / "source_regions.jsonl", output_path / "facts.jsonl", output_path / "consumer_facts.jsonl", - output_path / "ledger.db", + output_path / CHRONICLE_DB_FILENAME, output_path / "reports" / "source_rows.json", output_path / "reports" / "source_cells.json", output_path / "reports" / "source_regions.json", diff --git a/db/cli.py b/db/cli.py index 52257240..66538149 100644 --- a/db/cli.py +++ b/db/cli.py @@ -276,14 +276,15 @@ def cmd_query(args): def _pe_source_root_env_default(jurisdiction: str) -> str | None: + from chronicle.env import env_value + from .pe_source_inventory import ( PE_UK_DATA_ROOT_ENV, PE_US_DATA_ROOT_ENV, - _env_value, ) env_var = PE_US_DATA_ROOT_ENV if jurisdiction == "us" else PE_UK_DATA_ROOT_ENV - return _env_value(env_var) + return env_value(env_var) def main(): diff --git a/db/pe_source_inventory.py b/db/pe_source_inventory.py index a52fff23..f64ed870 100644 --- a/db/pe_source_inventory.py +++ b/db/pe_source_inventory.py @@ -2,14 +2,15 @@ from __future__ import annotations -import os from pathlib import Path +from chronicle.env import env_value + from .schema import Jurisdiction from .source_files import SourceArtifactSpec, make_slug, make_url_slug -PE_US_DATA_ROOT_ENV = "LEDGER_PE_US_DATA_ROOT" -PE_UK_DATA_ROOT_ENV = "LEDGER_PE_UK_DATA_ROOT" +PE_US_DATA_ROOT_ENV = "CHRONICLE_PE_US_DATA_ROOT" +PE_UK_DATA_ROOT_ENV = "CHRONICLE_PE_UK_DATA_ROOT" SOURCE_SUFFIXES = { ".csv", @@ -272,21 +273,13 @@ ] -def _env_value(*names: str) -> str | None: - for name in names: - value = os.environ.get(name) - if value: - return value - return None - - def _resolve_required_root( root: Path | None, *, flag: str, env_var: str, ) -> Path: - value = root if root is not None else _env_value(env_var) + value = root if root is not None else env_value(env_var) if value is None: raise ValueError(f"{flag} or {env_var} is required.") path = Path(value).expanduser() @@ -612,8 +605,8 @@ def pe_source_specs( ) -> list[SourceArtifactSpec]: """Return source files used by the PE-US and PE-UK calibration pipelines.""" specs: list[SourceArtifactSpec] = [] - us_configured = pe_us_root is not None or _env_value(PE_US_DATA_ROOT_ENV) - uk_configured = pe_uk_root is not None or _env_value(PE_UK_DATA_ROOT_ENV) + us_configured = pe_us_root is not None or env_value(PE_US_DATA_ROOT_ENV) + uk_configured = pe_uk_root is not None or env_value(PE_UK_DATA_ROOT_ENV) if include_us and (us_configured or not include_uk or not uk_configured): specs.extend(pe_us_source_specs(pe_us_root)) if include_uk and (uk_configured or not include_us or not us_configured): diff --git a/db/supabase_client.py b/db/supabase_client.py index e14bb5bb..a8c5dbeb 100644 --- a/db/supabase_client.py +++ b/db/supabase_client.py @@ -4,11 +4,15 @@ Provides connection to PolicyEngine Supabase database for: - Source metadata and dataset registries - Target inputs + +``LEDGER_SCHEMA`` and ``TARGETS_SCHEMA`` remain as deprecated compatibility +snapshots of the environment at import time. Runtime code should call +``chronicle_schema()`` and ``targets_schema()`` so environment overrides are +resolved at the time of use. """ from __future__ import annotations -import os from dataclasses import dataclass from functools import lru_cache from typing import Any, Dict, List, Optional @@ -16,18 +20,40 @@ from supabase import create_client, Client +from chronicle.env import default_chronicle_schema, env_value + +TARGETS_SCHEMA_ENV = "POLICYENGINE_TARGETS_SCHEMA" +DEFAULT_TARGETS_SCHEMA = "targets" + + +def chronicle_schema() -> str: + """Resolve the hosted Chronicle schema for a query. + + Read at call time, not bound at import: an import-time constant fixes the + schema at whatever the environment held when this module was first + imported, which the caller does not control (in the test suite that moment + is collection, before any fixture has isolated the environment). The + hosted schema is still named "ledger" -- only the variable that overrides + it has moved to the chronicle prefix, and renaming the schema value is a + later slice of PolicyEngine/chronicle#143. + """ + return default_chronicle_schema() + -def _env(*names: str) -> str | None: - """Read PolicyEngine-owned storage config.""" - for name in names: - value = os.environ.get(name) - if value: - return value - return None +def targets_schema() -> str: + """Resolve the hosted targets schema. Read at call time, as above. + ``POLICYENGINE_TARGETS_SCHEMA`` names a surface outside the ledger rename + window, so it is read literally. + """ + return env_value(TARGETS_SCHEMA_ENV, default=DEFAULT_TARGETS_SCHEMA) -LEDGER_SCHEMA = _env("POLICYENGINE_LEDGER_SCHEMA") or "ledger" -TARGETS_SCHEMA = _env("POLICYENGINE_TARGETS_SCHEMA") or "targets" + +# Deprecated import compatibility. Preserve the historical import-time +# environment snapshot for downstream code that still imports these names; +# Chronicle's own query paths use the lazy resolvers above. +LEDGER_SCHEMA = chronicle_schema() +TARGETS_SCHEMA = targets_schema() @dataclass @@ -49,14 +75,14 @@ def from_env(cls) -> "SupabaseConfig": Raises: ValueError: If required environment variables are missing """ - url = _env("POLICYENGINE_SUPABASE_URL") + url = env_value("POLICYENGINE_SUPABASE_URL") if not url: raise ValueError( "POLICYENGINE_SUPABASE_URL not set. " "Set this to your Supabase project URL." ) - secret_key = _env( + secret_key = env_value( "POLICYENGINE_SUPABASE_SERVICE_KEY", "POLICYENGINE_SUPABASE_SECRET_KEY", ) @@ -114,7 +140,7 @@ def query_sources( List of source records """ client = get_supabase_client() - query = _table(client, LEDGER_SCHEMA, "sources").select("*") + query = _table(client, chronicle_schema(), "sources").select("*") if jurisdiction: query = query.eq("jurisdiction", jurisdiction) @@ -143,7 +169,9 @@ def query_strata( List of strata records with nested constraints """ client = get_supabase_client() - query = _table(client, TARGETS_SCHEMA, "strata").select("*, stratum_constraints(*)") + query = _table(client, targets_schema(), "strata").select( + "*, stratum_constraints(*)" + ) if jurisdiction: query = query.eq("jurisdiction", jurisdiction) @@ -172,7 +200,7 @@ def query_targets( """ client = get_supabase_client() # Nested join: strata with their stratum_constraints - query = _table(client, TARGETS_SCHEMA, "targets").select( + query = _table(client, targets_schema(), "targets").select( "*, strata(*, stratum_constraints(*)), sources(*)" ) @@ -219,7 +247,7 @@ def insert_targets_batch( for i in range(0, len(targets), chunk_size): chunk = targets[i : i + chunk_size] - _table(client, TARGETS_SCHEMA, "targets").insert(chunk).execute() + _table(client, targets_schema(), "targets").insert(chunk).execute() total += len(chunk) return total diff --git a/docs/agent-source-package-harness.md b/docs/agent-source-package-harness.md index 5f300356..a8da1801 100644 --- a/docs/agent-source-package-harness.md +++ b/docs/agent-source-package-harness.md @@ -14,21 +14,37 @@ lineage, provenance, constraints, and a passing `build-suite` report. The first gate for a new package is source-artifact acquisition. Agents should register raw source files with `uv run chronicle fetch-artifact` before authoring selectors. This writes the local artifact, captures checksum and retrieval -metadata in `manifest.yaml`, and can upload the exact bytes to the private -`ledger-raw` R2 bucket when Wrangler is authenticated. Agents can audit the local +metadata in `manifest.yaml`, and can upload the exact bytes to the private raw +R2 bucket (`ledger-raw` today; overridable with `CHRONICLE_R2_RAW_BUCKET`) when +Wrangler is authenticated. A publisher directory that feeds several source +packages keeps one manifest each, so pass `--manifest ` to address +the right one. Agents can audit the local artifact registry with `uv run chronicle inventory-artifacts --root db/data`. For already-downloaded manifest artifacts, agents should run `uv run chronicle publish-raw --root db/data` to upload checksum-verified bytes to R2 and write `storage.r2` metadata back into each manifest entry. +Both commands treat a manifest entry as a claim about specific bytes: by its +declared `sha256` from the moment it is registered, and by the content-addressed +key of its recorded `storage.r2` block once it is published. Re-fetching or +publishing bytes the entry does not identify is refused; when a publisher has re-published +under the same URL and vintage, register the revision with +`uv run chronicle fetch-artifact ... --record-revision`, which stores the new +bytes under their own key and keeps the superseded object in +`storage.previous_r2`. See +[Publisher Revisions](storage-architecture.md#publisher-revisions). + Builds do not require production raw bytes to be committed to Git. Source packages first read packaged fixture bytes, then -`LEDGER_SOURCE_ARTIFACT_CACHE_DIR` (defaulting to +`CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` (defaulting to `~/.cache/policyengine-chronicle/source-artifacts`). If a manifest artifact is -missing locally, set `LEDGER_SOURCE_ARTIFACT_FETCH=1` to fetch it from the +missing locally, set `CHRONICLE_SOURCE_ARTIFACT_FETCH=1` to fetch it from the manifest `source_url`, verify the declared SHA-256, and write it to that cache. -The old `CHRONICLE_`-prefixed environment variables remain accepted only as -migration fallbacks. +The ledger-era spellings `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` and +`LEDGER_SOURCE_ARTIFACT_FETCH` are still honored during the rename window and +emit a one-time deprecation warning naming the `CHRONICLE_` variable to set +instead; see "Environment Variable Rename Window" in +[`docs/storage-architecture.md`](storage-architecture.md#environment-variable-rename-window). For broad PE source migration, generate the agent queue from the manifest before assigning work: @@ -645,16 +661,19 @@ uv run chronicle build-suite packages/irs_soi/table_1_1 \ --require-axiom-validation ``` -The SQLite `ledger.db` is the source of hosted mirrors. To prepare tables for +The SQLite `chronicle.db` is the source of hosted mirrors. To prepare tables for Supabase/Postgres bulk loading, export the DB artifact rather than inserting cells through the Supabase client: ```bash -uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace +uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace ``` -Accepted build-suite outputs can be published to the private `ledger-derived` R2 -bucket after validation: +Builds produced before the rename wrote `ledger.db`. That name is still read and +published unchanged, so point `--db` at whichever file the build emitted. + +Accepted build-suite outputs can be published to the private derived R2 bucket +after validation: ```bash uv run chronicle publish-derived \ @@ -665,11 +684,11 @@ uv run chronicle publish-derived \ --build-artifacts-out /tmp/chronicle-build-artifacts.jsonl ``` -The SQL schema is checked in at -`supabase/migrations/20260504_chronicle_bronze.sql`. Spreadsheet publications are -stored as immutable artifact metadata and one parsed-cell row per workbook cell. -Agents should not try to normalize irregular government worksheets into tidy -sheet tables before selector specs interpret them. +Before loading, create and apply a Supabase/Postgres migration that creates the +mirror tables in the selected schema. Spreadsheet publications are stored as +immutable artifact metadata and one parsed-cell row per workbook cell. Agents +should not try to normalize irregular government worksheets into tidy sheet +tables before selector specs interpret them. After the DB export and derived publish, agents can validate and load the hosted mirror: @@ -685,8 +704,11 @@ uv run chronicle load-supabase-mirror \ ``` The live load requires `POLICYENGINE_SUPABASE_URL` and -`POLICYENGINE_SUPABASE_SERVICE_KEY`, the Chronicle mirror migration applied, and the -`chronicle` schema exposed by the Supabase Data API. +`POLICYENGINE_SUPABASE_SERVICE_KEY`, the deployment migration applied, and the +selected schema exposed by the Supabase Data API. With no schema environment +override and no `--schema`, the selected schema is `ledger`; set +`CHRONICLE_SCHEMA=chronicle` or pass `--schema chronicle` to load a migrated +`chronicle` schema. ## Declarative Authoring Contract diff --git a/docs/storage-architecture.md b/docs/storage-architecture.md index cf898933..91cacbb1 100644 --- a/docs/storage-architecture.md +++ b/docs/storage-architecture.md @@ -9,17 +9,24 @@ class of Chronicle data belongs. Chronicle uses three storage layers with different jobs. -`ledger-raw` is the immutable source-byte archive. It stores exact publisher +The raw archive is the immutable source-byte store. It holds exact publisher artifacts as fetched: workbooks, CSVs, PDFs, ZIPs, HTML snapshots, and similar government-statistics release files. Raw objects are content-addressed by checksum and should never be overwritten in place. -`ledger-derived` is the reproducible artifact archive. It stores build outputs +The derived archive is the reproducible artifact store. It holds build outputs that Chronicle can regenerate from raw bytes, package specs, parser code, and build configuration. Examples include parsed-cell or parsed-row Parquet/JSONL files, -source record outputs, `ledger.db`, mirror JSONL exports, QA reports, Data +source record outputs, `chronicle.db`, mirror JSONL exports, QA reports, Data Package metadata, and RO-Crate metadata. +Both bucket names are configuration, not constants. The raw archive is +`$CHRONICLE_R2_RAW_BUCKET` and the derived archive is +`$CHRONICLE_R2_DERIVED_BUCKET`; the shipped defaults are still the ledger-era +`ledger-raw` and `ledger-derived`. [Bucket Cutover](#bucket-cutover) records how +those defaults move to `chronicle-raw` and `chronicle-derived` and why the +ledger-era buckets are preserved read-only rather than retired. + Supabase/Postgres is the queryable relational registry for accepted Chronicle builds. It stores rows that applications, agents, and downstream systems need to search and join: source artifacts, source rows/cells, source records, @@ -32,8 +39,8 @@ Hosted tables mirror accepted build outputs and provide a shared query surface. ## Ownership Matrix -| Data class | Git/local package | `ledger-raw` R2 | `ledger-derived` R2 | SQLite `ledger.db` | Supabase/Postgres | -|------------|-------------------|---------------|-------------------|------------------|-------------------| +| Data class | Git/local package | Raw R2 | Derived R2 | SQLite `chronicle.db` | Supabase/Postgres | +|------------|-------------------|--------|------------|-----------------------|-------------------| | Source package specs | Authoritative YAML and parser code | No | Optional packaged snapshot | No | Metadata only | | Raw publisher files | Tiny fixtures only | Authoritative bytes | No | Metadata only | Metadata plus R2 pointer | | Source manifests | Authoritative checked metadata | No | Optional snapshot | Metadata loaded into tables | Queryable artifact registry | @@ -62,8 +69,10 @@ raw/nz/ird/ird-working-for-families-statistics-sept-2025/2024/{sha256}/working-f The implemented country segments are `nz` and `uk`. US objects deliberately retain the legacy shape `raw/{source_id}/...`; migrating those keys requires a separate consumer audit. The fetch and raw-publish commands infer the country -from the package publisher directory. Raw publication refuses to replace a -manifest-recorded key that disagrees with the inferred country path. +from the package publisher directory for new objects. A manifest-recorded raw +object is preserved as history when its content-addressed checksum and filename +tail identify the local bytes, including legacy routes that predate the country +prefix and publisher-explicit routes such as Statbel's 2023 snapshots. New UK and New Zealand derived build artifacts use the same country segment and build-scoped keys so different builds can coexist and be audited: @@ -76,14 +85,122 @@ Examples: ```text derived/uk/ons/ons-mye-2024-uk/2024/{build_id}/source_cells.jsonl -derived/nz/ird/ird-working-for-families-statistics-sept-2025/2024/{build_id}/ledger.db +derived/nz/ird/ird-working-for-families-statistics-sept-2025/2024/{build_id}/chronicle.db ``` Legacy US derived keys likewise remain `derived/{source_id}/...`. +The derived prefix defaults to `derived` and can be configured with +`CHRONICLE_R2_DERIVED_PREFIX`, using the same legacy environment fallback as +the bucket. Publisher and consumer validation must share this route configuration: +facts citing a configured derived bucket or prefix are refused. The archived +`ledger-derived`, `chronicle-derived`, and `derived/` routes remain derived. +An explicit `publish-derived --r2-bucket ... --r2-prefix ...` combination must +use a recognized derived bucket or prefix; configure a custom route through +the environment before publishing it. This keeps custom build locations +identifiable at the publisher-fact boundary. + Derived artifacts are reproducible and may be replaced by a new build, but a specific `{build_id}` path should be immutable once published. +## Publisher Revisions + +A raw key embeds the sha256 of the bytes it holds, so a manifest's recorded +`storage.r2` block is a claim about specific bytes, not a pointer to a file +name. Publishers do not always honor that: on 2026-09-02 the IRS re-published +`22in05ira.xlsx` and `22in06ira.xlsx` under their existing URLs +(PolicyEngine/chronicle#225). + +`fetch-artifact` therefore compares the entry's recorded identity with the +bytes it just fetched, before it writes anything. That identity is the recorded +key's `{sha256}/{filename}` tail once the entry has been published, and the +entry's own declared `sha256` before then — an entry that was registered +without an upload, or whose upload failed, still identifies its bytes, and gets +the same protection: + +- **Identical** — the recorded block is preserved exactly, whichever bucket is + configured now. Re-fetching after the bucket rename copies bytes; it does not + restate where they were first published. +- **Different** — the fetch is refused. Nothing is overwritten: not the cached + artifact, not the manifest entry, and no object is uploaded. The error names + the recorded and the fetched `sha256`/`size_bytes`. Per + `docs/adr-chronicle-fact-identity-v2.md`, the same vintage with new bytes is a + new release revision, so registering it is a decision an operator makes, not a + silent rewrite. + +`--record-revision` makes that decision explicit. The fetched bytes get their +own content-addressed key under the configured bucket — never the recorded key — +and the superseded block moves to `storage.previous_r2`: + +```yaml +files: + 2022: + filename: 22in05ira.xlsx + sha256: + size_bytes: + fetched_at: "2026-09-02T17:04:11+00:00" + storage: + r2: + provider: r2 + bucket: chronicle-raw + key: raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + uri: r2://chronicle-raw/raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + previous_r2: + - provider: r2 + bucket: ledger-raw + key: raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + uri: r2://ledger-raw/raw/irs_soi/soi-table-5/2022//22in05ira.xlsx + sha256: + size_bytes: + fetched_at: "2026-06-11T14:22:05+00:00" + source_url: https://www.irs.gov/pub/irs-soi/22in05ira.xlsx + superseded_at: "2026-09-02T17:04:11+00:00" +``` + +`storage.r2` only ever names the object that holds the entry's current bytes, +and `previous_r2` lists superseded objects oldest first, so the bytes an +archived witness record pinned stay addressable at the URI it pinned. Every +reader — `inventory-artifacts`, `publish-raw`, source-package artifact loading, +and the suite's raw-R2-link acceptance check — reads `storage.r2` alone, so a +revised entry reads exactly like an unrevised one; `publish-raw` preserves the +rest of the `storage` block when it writes back. + +`publish-raw` applies the same identity check before treating a recorded block +as history. A local file the recorded object does not hold is reported as +`recorded_r2_identity_mismatch` and nothing is uploaded: registering a revision +is a fetch-time decision, not a publish-time rewrite. + +### Which manifest + +Most packages keep one `manifest.yaml`. A publisher directory that feeds +several source packages keeps one manifest each — +`db/data/irs_soi/ira_contributions/` holds +`manifest_traditional_source_package.yaml` beside +`manifest_roth_source_package.yaml`. `fetch-artifact --manifest ` +selects the entry whose publisher metadata the fetch updates; defaulting to +`manifest.yaml` there would write a third manifest neither package reads. A +physical artifact can also be owned by several entries in that directory (the +tracked USDA SNAP archive spans two manifests, and SSA extracts have semantic +aliases within one). Chronicle compares every such owner before overwriting the +file. A changed archive is refused by default; `--record-revision` updates every +owner to the new checksum and preserves each owner's own R2 block in +`storage.previous_r2`. The manifest selector must be a filename inside +`--out-dir`, not a path. + +### What a recorded block has to say + +A `storage.r2` block must explicitly say `provider: r2` and carry an `r2://` +URI. Its `provider`, `bucket`, `key` and `uri` all describe one object, so every +additional field that is present is cross-checked against the URI: the key +against its path, the bucket against its authority, the provider against its +scheme, and the resulting key against the content-addressed +`{sha256}/{filename}` shape. A block whose fields disagree does not answer +"which bytes does this entry claim R2 holds", so it is an error rather than +something to preserve or publish under. Likewise a manifest that parses as +anything other than a mapping is refused rather than treated as absent — +reading it as absent would let the next fetch replace the file with a single +entry. + ## Relational Registry Contract The hosted `chronicle` schema should be the lookup surface for Chronicle, not the place @@ -104,9 +221,10 @@ The registry should expose: authority, legal vintage, and evidence; - build metadata, validation status, and derived artifact R2 bucket/key/URI. -The current Supabase migration mirrors the core relational tables and includes -R2 location fields for raw source artifacts and derived build artifacts, so the -registry can serve as the shared index over both R2 buckets. +A deployment migration for the selected Supabase schema must mirror the core +relational tables and include R2 location fields for raw source artifacts and +derived build artifacts, so the registry can serve as the shared index over +both R2 buckets. ## Build And Publish Flow @@ -114,20 +232,24 @@ The intended flow is: 1. Register raw source artifacts with `uv run chronicle fetch-artifact`, which writes local bytes, records checksums in `manifest.yaml`, and can upload the - exact bytes to `ledger-raw`. Existing manifest-declared artifacts can be + exact bytes to the raw archive. Re-fetching an entry whose bytes the + publisher has changed is refused unless the revision is registered with + `--record-revision`; see [Publisher Revisions](#publisher-revisions). + Existing manifest-declared artifacts can be checksum-validated, uploaded, and linked with `uv run chronicle publish-raw`. Production package specs may omit raw bytes from Git as long as the manifest keeps `source_url` and SHA-256 metadata; builds can fill - `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` by setting - `LEDGER_SOURCE_ARTIFACT_FETCH=1`. The old `CHRONICLE_`-prefixed environment - variables remain accepted only as migration fallbacks. + `CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` by setting + `CHRONICLE_SOURCE_ARTIFACT_FETCH=1`. Ledger-era spellings of both still work; + see [Environment Variable Rename Window](#environment-variable-rename-window). 2. Validate and build a source package with `uv run chronicle validate-package` and `uv run chronicle build-suite`. 3. Produce local deterministic outputs: parsed rows/cells, source records, - aggregate facts, `ledger.db`, QA reports, Data Package metadata, and RO-Crate - metadata. + aggregate facts, `chronicle.db`, QA reports, Data Package metadata, and + RO-Crate metadata. Builds before this rename wrote `ledger.db`; every reader + still accepts that name. 4. Export relational mirror files with `uv run chronicle export-db-tables`. -5. Publish derived build outputs to `ledger-derived`: +5. Publish derived build outputs to the derived archive: ```bash uv run chronicle publish-derived \ @@ -146,9 +268,181 @@ The intended flow is: --build-artifacts /tmp/chronicle-build-artifacts.jsonl ``` -The Supabase project must have the checked migration applied and the `chronicle` -schema exposed in PostgREST/Data API settings before the REST loader can write -to it. Use `--dry-run` to verify local JSONL files without writing. +The Supabase project must have a deployment migration for the selected schema +applied and that schema exposed in PostgREST/Data API settings before the REST +loader can write to it. The load defaults to `ledger`; set +`CHRONICLE_SCHEMA=chronicle` or pass `--schema chronicle` to target a migrated +`chronicle` schema. Use `--dry-run` to verify local JSONL files without writing. + +## Environment Variable Rename Window + +Every Chronicle setting is read chronicle-first by one shared helper, +`chronicle/env.py`. For a setting `X`, the lookup order is: + +1. `CHRONICLE_X` +2. `POLICYENGINE_LEDGER_X` +3. `LEDGER_X` + +The first name that holds a non-empty value wins. When that name is a ledger-era +one, the process emits a single `ChronicleEnvDeprecationWarning` naming the +`CHRONICLE_`-prefixed variable to set instead. The warning fires once per legacy +name per process, and it subclasses `FutureWarning` rather than +`DeprecationWarning` so it actually reaches operators running the CLI. + +Two consequences are worth stating outright, because both are the reverse of +what a naive fallback would do: + +- The `CHRONICLE_` name wins even when its value reads false. An operator who + has migrated can set `CHRONICLE_SOURCE_ARTIFACT_FETCH=0` and have the flag + turn off, without first hunting down a stale `LEDGER_SOURCE_ARTIFACT_FETCH=1` + somewhere in their profile. +- An empty value counts as unset, so exporting an empty `CHRONICLE_` name does + not mask a set legacy name. + +| Chronicle name | Ledger-era names still accepted | Meaning | +|----------------|--------------------------------|---------| +| `CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR` | `LEDGER_SOURCE_ARTIFACT_CACHE_DIR` | Where fetched raw bytes are cached; defaults to `~/.cache/policyengine-chronicle/source-artifacts` | +| `CHRONICLE_SOURCE_ARTIFACT_FETCH` | `LEDGER_SOURCE_ARTIFACT_FETCH` | Fetch a missing manifest artifact from its `source_url` during a build | +| `CHRONICLE_PE_US_DATA_ROOT` | `LEDGER_PE_US_DATA_ROOT` | Local checkout root for PE US source inventory | +| `CHRONICLE_PE_UK_DATA_ROOT` | `LEDGER_PE_UK_DATA_ROOT` | Local checkout root for PE UK source inventory | +| `CHRONICLE_SCHEMA` | `POLICYENGINE_LEDGER_SCHEMA`, `LEDGER_SCHEMA` | Postgres schema the Supabase client reads and `load-supabase-mirror` writes; defaults to `ledger` | +| `CHRONICLE_R2_RAW_BUCKET` | `LEDGER_R2_RAW_BUCKET` | Raw R2 archive bucket; defaults to `ledger-raw` | +| `CHRONICLE_R2_DERIVED_BUCKET` | `LEDGER_R2_DERIVED_BUCKET` | Derived R2 archive bucket; defaults to `ledger-derived` | + +The two R2 rows are new in this window rather than renamed: those buckets were +hardcoded before, so the ledger-era spellings are accepted for consistency, not +because anything ever set them. + +Variables carrying none of the three prefixes are read literally. This helper +renames the ledger-era surface, not every PolicyEngine variable, so +`POLICYENGINE_SUPABASE_URL`, `POLICYENGINE_SUPABASE_SERVICE_KEY` and +`POLICYENGINE_TARGETS_SCHEMA` keep their names and gain no aliases. + +The hosted schema *value* is a separate migration. `CHRONICLE_SCHEMA` renames +the variable that overrides the schema; the schema still defaults to `ledger`, +and the mirror table names are unchanged. Those move in a later slice +coordinated with the CI writers. + +Every reader of the setting resolves it the same way, at call time. That +includes the writer: `load-supabase-mirror` takes its `--schema` default from +`CHRONICLE_SCHEMA`, so setting the variable to rehearse a cutover moves the +mirror load with the client rather than leaving it pointed at `ledger`. An +explicit `--schema` still wins. + +## Bucket Cutover + +Chronicle's operational stores migrate by dual-run +(PolicyEngine/chronicle#143, mechanism 3): stand up the chronicle-named home, +backfill it, repoint writers, retire the old home. The R2 buckets take one +exception to the last step. Archived witness records pin raw R2 URLs by hash, so +`ledger-raw` and `ledger-derived` are preserved read-only forever rather than +deleted, and manifests keep the `storage.r2` URIs they already recorded as +historical truth. A backfill copies bytes into the new bucket; it never rewrites +where those bytes were first published. `publish-raw` and `fetch-artifact` +enforce that: a recorded block that addresses the bytes in hand is preserved +whichever bucket is configured, and `publish-raw` reports such an entry as +`skipped` (already published under the recorded bucket) rather than restating +it under a different one, so a sweep over a fully published tree stays green +after the flip. Bytes that the recorded object does not hold are not that +object's history at all; see [Publisher Revisions](#publisher-revisions). + +The cutover therefore has one irreversible-looking step that is in fact additive +(creating and filling the new buckets), one cheap reversible step (flipping the +defaults, which is a one-line change in `chronicle/artifacts.py`), and no +deletion step at all. + +### 1. Create the new buckets + +Bucket creation needs a Cloudflare login carrying R2 permissions, so it is an +operator step rather than something CI can do. `wrangler.toml` already pins the +PolicyEngine account (`account_id = "20d90f557651969925eece96e58e24dc"`), so no +`CLOUDFLARE_ACCOUNT_ID` is needed even for a user who belongs to several +accounts: + +```bash +bunx wrangler login +uv run chronicle bootstrap-r2 --raw-bucket chronicle-raw --derived-bucket chronicle-derived +``` + +`bootstrap-r2` verifies authentication with `wrangler whoami` before creating +anything, and creating a bucket that already exists is not an error. + +### 2. Enumerate what has to be copied + +Tracked manifests are the authoritative registry of raw objects. Every one of +them points at `ledger-raw` today: + +```bash +git ls-files '*manifest*.yaml' '*manifest*.yml' \ + | xargs grep -ho 'r2://ledger-raw/[^"'"'"' ]*' | sort -u > /tmp/chronicle-raw-objects.txt +wc -l < /tmp/chronicle-raw-objects.txt +``` + +That is 186 distinct objects at `ff3efd3`, spread over 154 manifest files. Recount +rather than trusting the number: source packages land continuously, and each new +package adds objects. + +The derived bucket needs no backfill. Derived artifacts are reproducible by +definition and are already keyed by `{build_id}`, so a rebuild republishes them +into whichever bucket is configured. + +### 3. Backfill-copy the raw objects + +Keys are content-addressed and identical across buckets, so the copy is a +straight get/put per object: + +```bash +mkdir -p /tmp/chronicle-r2-backfill +while read -r uri; do + key=${uri#r2://ledger-raw/} + dest=/tmp/chronicle-r2-backfill/$key + mkdir -p "$(dirname "$dest")" + bunx wrangler r2 object get "ledger-raw/$key" --file "$dest" --remote + bunx wrangler r2 object put "chronicle-raw/$key" --file "$dest" --remote +done < /tmp/chronicle-raw-objects.txt +``` + +### 4. Verify the copy against the keys themselves + +Every raw key ends `.../{sha256}/{filename}`, so the key is its own checksum +witness and verification needs no manifest lookup: + +```bash +while read -r uri; do + key=${uri#r2://ledger-raw/} + expected=$(printf '%s\n' "$key" | awk -F/ '{print $(NF-1)}') + actual=$(shasum -a 256 "/tmp/chronicle-r2-backfill/$key" | cut -d' ' -f1) + [ "$expected" = "$actual" ] || echo "MISMATCH $key" +done < /tmp/chronicle-raw-objects.txt +``` + +Silence means every downloaded object hashes to the checksum its key claims. +That covers the read from `ledger-raw`; to cover the write to `chronicle-raw`, +re-download each key from the new bucket into a second directory and rerun the +same loop against it. + +### 5. Flip the defaults, in a follow-up PR + +Once the new buckets are filled and verified, change `DEFAULT_R2_RAW_BUCKET` and +`DEFAULT_R2_DERIVED_BUCKET` in `chronicle/artifacts.py` to `chronicle-raw` and +`chronicle-derived`. Until then, operators can opt in per-shell: + +```bash +export CHRONICLE_R2_RAW_BUCKET=chronicle-raw +export CHRONICLE_R2_DERIVED_BUCKET=chronicle-derived +``` + +New raw publications land in the new bucket from that point. Manifests written +before the flip keep pointing at `ledger-raw`, which is why the old bucket stays +readable. A `publish-raw --root db/data` sweep after the flip reports every +already-published entry as `skipped` with its recorded `ledger-raw` location +(`skipped_count` in the report) and exits 0; only bytes that no recorded object +holds are uploaded, into `chronicle-raw`. + +### 6. Set the ledger-era buckets read-only + +`ledger-raw` and `ledger-derived` keep serving archived witness records after the +flip. They should accept no further writes and should never be deleted. ## Non-Goals diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 00000000..3d315441 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,50 @@ +"""Shared fixtures for the Chronicle test suite. + +Chronicle is mid-rename (PolicyEngine/chronicle#143, mechanism 3), so its +settings answer to three prefixes at once: ``CHRONICLE_``, and the ledger-era +``POLICYENGINE_LEDGER_`` and ``LEDGER_``. Any of them can be set in an +operator's shell, and many tests assert the defaults those variables override. +Isolation therefore belongs to the whole suite, not to one module. +""" + +from __future__ import annotations + +import os + +import pytest + +from chronicle.env import ( + CHRONICLE_ENV_PREFIX, + LEGACY_ENV_PREFIXES, + reset_env_deprecation_state, +) + +RENAME_WINDOW_PREFIXES = (CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES) + + +def pytest_configure(config): + """Strip the rename window before collection imports a single module. + + The autouse fixture below runs per test, which is too late for anything a + module does while being imported. Collection happens after this hook, so + clearing here means no module can read an operator's ``LEDGER_SCHEMA`` + (warning as it goes, or freezing it into a constant) before a fixture has + had the chance to isolate it. Modules should resolve settings at call time + rather than at import; this hook makes that property testable instead of + depending on which shell ran pytest. + """ + for name in list(os.environ): + if name.startswith(RENAME_WINDOW_PREFIXES): + del os.environ[name] + reset_env_deprecation_state() + + +@pytest.fixture(autouse=True) +def isolated_rename_window_env(monkeypatch): + """Run every test with no rename-window variable inherited from the shell.""" + for name in list(os.environ): + if name.startswith(RENAME_WINDOW_PREFIXES): + monkeypatch.delenv(name, raising=False) + reset_env_deprecation_state() + yield + reset_env_deprecation_state() diff --git a/tests/test_chronicle_artifact_peer4.py b/tests/test_chronicle_artifact_peer4.py new file mode 100644 index 00000000..f63f5935 --- /dev/null +++ b/tests/test_chronicle_artifact_peer4.py @@ -0,0 +1,525 @@ +"""Failing-first regressions for PR #226 peer round 4.""" + +import hashlib +import json +import os +from pathlib import Path + +import pytest +import yaml + +from chronicle.artifacts import ( + ArtifactCommandResult, + SourceArtifactManifestError, + fetch_source_artifact, + inventory_source_artifacts, + publish_derived_artifacts, + publish_source_artifacts, +) +from chronicle.cli import main as cli_main +from chronicle.harness import main as harness_main + + +def _package(tmp_path, *, filename="table.csv", manifest_name="manifest.yaml"): + package = tmp_path / "package" + package.mkdir() + content = b"publisher,value\nexample,123\n" + (package / filename).write_bytes(content) + manifest_path = package / manifest_name + manifest = { + "source_id": "publisher", + "package_id": "package", + "files": { + 2024: { + "filename": filename, + "source_url": "https://example.test/table.csv", + "sha256": hashlib.sha256(content).hexdigest(), + "size_bytes": len(content), + } + }, + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + return package, manifest_path, manifest + + +def _no_upload(monkeypatch): + def unexpected_upload(*args, **kwargs): + pytest.fail("uploader reached before refusal") + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", unexpected_upload) + + +@pytest.mark.parametrize("shape", ["file-link", "directory-link", "dangling", "fifo"]) +def test_derived_preflights_complete_tree_before_reads_or_uploads( + tmp_path, monkeypatch, shape +): + suite = tmp_path / "suite" + suite.mkdir() + (suite / "a_good.jsonl").write_bytes(b"{}\n") + outside = tmp_path / "outside" + outside.mkdir() + (outside / "secret.csv").write_bytes(b"outside bytes") + unsafe = suite / "z_unsafe" + if shape == "file-link": + unsafe.symlink_to(outside / "secret.csv") + elif shape == "directory-link": + unsafe.symlink_to(outside, target_is_directory=True) + elif shape == "dangling": + unsafe.symlink_to(outside / "missing") + else: + os.mkfifo(unsafe) + output = tmp_path / "registry.jsonl" + output.write_text("sentinel\n") + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("build read reached before tree refusal") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + monkeypatch.setattr("chronicle.artifacts.infer_build_id", unexpected_read) + report = publish_derived_artifacts( + suite, + source_id="publisher", + package_id="package", + year=2024, + build_artifacts_output=output, + ) + assert not report.valid + assert report.entries == () + assert any("regular" in error or "symlink" in error for error in report.errors) + assert output.read_text() == "sentinel\n" + + +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +@pytest.mark.parametrize("bad_id", ["foo bar", "a/b", "..", 123, None, ""]) +@pytest.mark.parametrize("operation", ["raw", "derived"]) +def test_new_publication_refuses_noncanonical_identity_before_reads( + tmp_path, monkeypatch, field, bad_id, operation +): + package, manifest_path, manifest = _package(tmp_path) + kwargs = {"source_id": "publisher", "package_id": "package"} + kwargs[field] = bad_id + if operation == "raw": + manifest[field] = bad_id + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("artifact read reached with noncanonical publication identity") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + monkeypatch.setattr("chronicle.artifacts.infer_build_id", unexpected_read) + if operation == "raw": + report = publish_source_artifacts(package) + else: + report = publish_derived_artifacts(package, year=2024, **kwargs) + assert not report.valid + assert any( + "identity" in error or "source_id" in error or "package_id" in error + for error in ( + *report.errors, + *(e for entry in report.entries for e in entry.errors), + ) + ) + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +@pytest.mark.parametrize("declaration", [" padded ", 123, None, " ", ""]) +def test_fetch_refuses_noncanonical_present_manifest_identity_before_io( + tmp_path, monkeypatch, field, declaration +): + package, manifest_path, manifest = _package(tmp_path) + args = {"source_id": "publisher", "package_id": "package"} + args[field] = "padded" if declaration == " padded " else "123" + manifest[field] = declaration + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("publisher read reached with noncanonical manifest declaration") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + with pytest.raises(SourceArtifactManifestError, match=field): + fetch_source_artifact( + "https://example.test/table.csv", + year=2024, + output_dir=package, + filename="table.csv", + **args, + ) + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("operation", ["publish", "inventory"]) +@pytest.mark.parametrize( + ("physical", "declared"), + [("table.csv", "TABLE.csv"), ("café.csv", "cafe\u0301.csv")], +) +def test_sweeps_refuse_single_normalized_artifact_alias_before_read( + tmp_path, monkeypatch, operation, physical, declared +): + package, manifest_path, manifest = _package(tmp_path, filename=physical) + # On filesystems that normalize Unicode on creation, choose the other + # logical spelling from the actual directory entry returned by iterdir. + actual = next( + path.name for path in package.iterdir() if path.name != manifest_path.name + ) + if actual == declared: + declared = physical + manifest["files"][2024]["filename"] = declared + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("artifact read reached through a normalized alias") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + function = ( + publish_source_artifacts + if operation == "publish" + else inventory_source_artifacts + ) + report = function(package) + assert not report.valid + assert any( + "spelling" in error for entry in report.entries for error in entry.errors + ) + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("operation", ["publish-raw", "inventory-artifacts"]) +@pytest.mark.parametrize("shape", ["directory", "dangling", "fifo"]) +@pytest.mark.parametrize("entrypoint", ["function", "harness", "cli"]) +def test_explicit_sweep_reports_nonregular_manifest_sibling( + tmp_path, monkeypatch, capsys, operation, shape, entrypoint +): + package, manifest_path, _manifest = _package( + tmp_path, manifest_name="manifest_selected.yaml" + ) + sibling = package / "manifest_sibling.yaml" + if shape == "directory": + sibling.mkdir() + elif shape == "dangling": + sibling.symlink_to(package / "missing") + else: + os.mkfifo(sibling) + before = manifest_path.read_text() + _no_upload(monkeypatch) + if entrypoint == "function": + function = ( + publish_source_artifacts + if operation == "publish-raw" + else inventory_source_artifacts + ) + report = function(package, manifest_filename=manifest_path.name) + assert not report.valid + assert "regular file" in " ".join(report.errors) + else: + args = [operation, "--root", str(package), "--manifest", manifest_path.name] + if entrypoint == "harness": + assert harness_main(args) == 1 + else: + monkeypatch.setattr("sys.argv", ["chronicle", *args]) + with pytest.raises(SystemExit) as exit_info: + cli_main() + assert exit_info.value.code == 1 + assert "regular file" in json.dumps(json.loads(capsys.readouterr().out)) + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("location", ["root", "nested", "excluded-registry"]) +def test_derived_preflight_rejects_symlinks_at_every_tree_boundary( + tmp_path, monkeypatch, location +): + suite = tmp_path / "suite" + outside = tmp_path / "outside" + outside.mkdir() + (outside / "secret.csv").write_bytes(b"outside") + if location == "root": + suite.symlink_to(outside, target_is_directory=True) + else: + suite.mkdir() + target = suite / "build_artifacts.jsonl" + if location == "nested": + (suite / "reports").mkdir() + target = suite / "reports" / "database.json" + target.symlink_to(outside / "secret.csv") + _no_upload(monkeypatch) + report = publish_derived_artifacts( + suite, + source_id="publisher", + package_id="package", + year=2024, + build_id="ledger.build.v1:peer4", + ) + assert not report.valid + assert report.entries == () + + +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +@pytest.mark.parametrize("bad_id", ["foo bar", "a/b", "..", 123, ""]) +def test_raw_publication_refuses_noncanonical_explicit_identity_before_read( + tmp_path, monkeypatch, field, bad_id +): + package, manifest_path, _manifest = _package(tmp_path) + before = manifest_path.read_text() + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("artifact read reached with invalid explicit identity") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + report = publish_source_artifacts(package, **{field: bad_id}) + assert not report.valid + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize( + "defect", + [ + "contradictory", + "incomplete", + "non-r2", + "empty", + "sha256", + "filename", + "local-digest", + ], +) +def test_inventory_refuses_invalid_recorded_r2_and_does_not_count_link( + tmp_path, monkeypatch, defect +): + package, manifest_path, manifest = _package(tmp_path) + spec = manifest["files"][2024] + key = f"raw/publisher/package/2024/{spec['sha256']}/table.csv" + block = { + "provider": "r2", + "bucket": "archive", + "key": key, + "uri": f"r2://archive/{key}", + } + if defect == "contradictory": + block["bucket"] = "different" + elif defect == "incomplete": + del block["provider"] + elif defect == "non-r2": + block["provider"] = "s3" + block["uri"] = f"s3://archive/{key}" + elif defect == "empty": + block = {} + elif defect == "sha256": + spec["sha256"] = "0" * 64 + elif defect == "filename": + block["key"] = key.replace("table.csv", "other.csv") + block["uri"] = f"r2://archive/{block['key']}" + else: + del spec["sha256"] + (package / "table.csv").write_bytes(b"different local bytes") + spec["storage"] = {"r2": block} + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + if defect != "local-digest": + + def unexpected_read(*args, **kwargs): + pytest.fail("inventory read bytes before refusing invalid R2 identity") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + report = inventory_source_artifacts(package) + assert not report.valid + assert report.counts["r2_link_count"] == 0 + assert report.entries[0].r2 is None + assert any("recorded_r2" in error for error in report.entries[0].errors) + assert manifest_path.read_text() == before + + +def test_inventory_accepts_consistent_uri_only_r2_locator(tmp_path): + package, manifest_path, manifest = _package(tmp_path) + spec = manifest["files"][2024] + spec["storage"] = { + "r2": { + "provider": "r2", + "uri": f"r2://archive/raw/publisher/package/2024/{spec['sha256']}/table.csv", + } + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + report = inventory_source_artifacts(package) + assert report.valid + assert report.counts["r2_link_count"] == 1 + + +def test_derived_publication_refuses_unrecognized_explicit_route_before_read( + tmp_path, monkeypatch +): + suite = tmp_path / "suite" + suite.mkdir() + (suite / "facts.jsonl").write_bytes(b"{}\n") + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("build read reached for an unrecognizable derived route") + + monkeypatch.setattr("chronicle.artifacts.infer_build_id", unexpected_read) + report = publish_derived_artifacts( + suite, + source_id="publisher", + package_id="package", + year=2024, + r2_bucket="chronicle-builds", + r2_prefix="builds", + ) + assert not report.valid + assert "derived_route" in " ".join(report.errors) + + +@pytest.mark.parametrize( + "env_prefix", ["CHRONICLE_", "POLICYENGINE_LEDGER_", "LEDGER_"] +) +def test_derived_publication_propagates_configured_prefix( + tmp_path, monkeypatch, env_prefix +): + from chronicle.consumer_contract import _points_at_derived + + suite = tmp_path / "suite" + suite.mkdir() + (suite / "facts.jsonl").write_bytes(b"{}\n") + monkeypatch.setenv(f"{env_prefix}R2_DERIVED_BUCKET", "chronicle-builds") + monkeypatch.setenv(f"{env_prefix}R2_DERIVED_PREFIX", "builds") + uploaded = [] + + def upload(location, *_args, **_kwargs): + uploaded.append(location) + return ArtifactCommandResult( + command=("test",), returncode=0, stdout="", stderr="" + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", upload) + report = publish_derived_artifacts( + suite, + source_id="publisher", + package_id="package", + year=2024, + build_id="ledger.build.v1:peer4", + ) + assert report.valid + assert len(uploaded) == 1 + assert uploaded[0].key.startswith("builds/") + assert _points_at_derived(uploaded[0].bucket, uploaded[0].key) + + +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +@pytest.mark.parametrize("declaration", [None, 123, "historical name"]) +def test_raw_publication_preserves_history_without_new_identity_requirements( + tmp_path, monkeypatch, field, declaration +): + package, manifest_path, manifest = _package(tmp_path) + spec = manifest["files"][2024] + key = f"historical/route/{spec['sha256']}/table.csv" + spec["storage"] = {"r2": {"provider": "r2", "uri": f"r2://archive/{key}"}} + manifest[field] = declaration + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + _no_upload(monkeypatch) + report = publish_source_artifacts(package) + assert report.valid + assert report.entries[0].skipped + assert report.entries[0].r2_location.uri == f"r2://archive/{key}" + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +@pytest.mark.parametrize("yaml_identity", ["2024-01-01", "!!set {legacy: null}"]) +@pytest.mark.parametrize("entrypoint", ["harness", "cli"]) +def test_raw_cli_serializes_noncanonical_yaml_identity_refusals( + tmp_path, monkeypatch, capsys, field, yaml_identity, entrypoint +): + package, manifest_path, manifest = _package(tmp_path) + manifest[field] = yaml.safe_load(yaml_identity) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_text() + _no_upload(monkeypatch) + args = ["publish-raw", "--root", str(package)] + if entrypoint == "harness": + assert harness_main(args) == 1 + else: + monkeypatch.setattr("sys.argv", ["chronicle", *args]) + with pytest.raises(SystemExit) as exit_info: + cli_main() + assert exit_info.value.code == 1 + payload = json.loads(capsys.readouterr().out) + assert not payload["valid"] + assert isinstance(payload["entries"][0][field], str) + assert "r2_identity_invalid" in " ".join(payload["entries"][0]["errors"]) + assert manifest_path.read_text() == before + + +@pytest.mark.parametrize("bad_year", ["/2024", "..", "2024/elsewhere"]) +@pytest.mark.parametrize("operation", ["raw", "derived"]) +def test_publication_refuses_vintage_namespace_escape_before_reads( + tmp_path, monkeypatch, bad_year, operation +): + package, manifest_path, manifest = _package(tmp_path) + manifest["files"][bad_year] = manifest["files"].pop(2024) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + _no_upload(monkeypatch) + + def unexpected_read(*args, **kwargs): + pytest.fail("publication read reached with a vintage namespace escape") + + monkeypatch.setattr(Path, "read_bytes", unexpected_read) + if operation == "raw": + report = publish_source_artifacts(package) + else: + report = publish_derived_artifacts( + package, + source_id="publisher", + package_id="package", + year=bad_year, + build_id="ledger.build.v1:peer4", + r2_bucket="custom-store", + ) + assert not report.valid + + +@pytest.mark.parametrize( + "bad_build_id", ["ledger.build.v1:bad/id", "ledger.build.v1:bad id"] +) +def test_derived_publication_refuses_noncanonical_build_segment( + tmp_path, monkeypatch, bad_build_id +): + suite = tmp_path / "suite" + suite.mkdir() + (suite / "facts.jsonl").write_bytes(b"{}\n") + _no_upload(monkeypatch) + report = publish_derived_artifacts( + suite, + source_id="publisher", + package_id="package", + year=2024, + build_id=bad_build_id, + ) + assert not report.valid + assert report.errors == ("malformed_build_id",) + + +@pytest.mark.parametrize("bad_year", ["/2024", "..", "2024/elsewhere"]) +def test_fetch_refuses_vintage_namespace_escape_before_publisher_io( + tmp_path, monkeypatch, bad_year +): + package, _manifest_path, _manifest = _package(tmp_path) + + def unexpected_read(*args, **kwargs): + pytest.fail("fetch reached publisher with a vintage namespace escape") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + with pytest.raises(SourceArtifactManifestError, match="year"): + fetch_source_artifact( + "https://example.test/table.csv", + source_id="publisher", + package_id="package", + year=bad_year, + output_dir=package, + ) diff --git a/tests/test_chronicle_artifacts.py b/tests/test_chronicle_artifacts.py index 7104ec03..c72d737c 100644 --- a/tests/test_chronicle_artifacts.py +++ b/tests/test_chronicle_artifacts.py @@ -4,12 +4,23 @@ import hashlib import json +from pathlib import Path +import shutil +import sqlite3 import pytest import yaml from chronicle.cli import main as cli_main from chronicle.artifacts import ( + AmbiguousManifestError, + ArtifactCommandResult, + ArtifactFilenameError, + MalformedManifestError, + ManifestNameError, + RecordedR2LocatorError, + SourceArtifactManifestError, + SourceArtifactRevisionError, build_artifact_key, build_artifact_rows, build_derived_r2_key, @@ -242,6 +253,7 @@ def test_publish_source_artifacts_uploads_manifest_entries(tmp_path): "failed_count": 0, "manifest_count": 1, "r2_link_count": 1, + "skipped_count": 0, "uploaded_count": 1, } assert storage["bucket"] == "ledger-raw" @@ -320,7 +332,7 @@ def test_publish_source_artifacts_uses_country_for_each_manifest(tmp_path): assert "ledger-raw/raw/irs_soi/soi-table/2023/" in commands -def test_publish_source_artifacts_refuses_stale_country_key(tmp_path): +def test_publish_source_artifacts_preserves_a_legacy_countryless_key(tmp_path): output_dir = tmp_path / "data" / "ird" / "wff" source = tmp_path / "wff.xlsx" source.write_bytes(b"official WFF workbook") @@ -343,6 +355,9 @@ def test_publish_source_artifacts_refuses_stale_country_key(tmp_path): ), } } + artifact["storage"]["r2"]["uri"] = ( + f"r2://ledger-raw/{artifact['storage']['r2']['key']}" + ) manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) log = tmp_path / "wrangler.log" wrangler = tmp_path / "wrangler" @@ -351,13 +366,11 @@ def test_publish_source_artifacts_refuses_stale_country_key(tmp_path): report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) - assert not report.valid + assert report.valid assert report.entries[0].upload is None - assert ( - report.entries[0] - .errors[0] - .startswith("recorded_r2_key_disagrees_with_country_prefix:") - ) + assert report.entries[0].errors == () + assert report.entries[0].skipped == "recorded_r2_already_published" + assert report.entries[0].r2_location.key == artifact["storage"]["r2"]["key"] assert not log.exists() @@ -658,3 +671,3277 @@ def test_top_level_cli_dispatches_publish_derived(tmp_path, capsys, monkeypatch) assert exc.value.code == 0 assert payload["valid"] + + +def _sqlite_build(path, build_id): + """Write a minimal build database carrying one ledger_builds row.""" + with sqlite3.connect(path) as connection: + connection.execute("CREATE TABLE ledger_builds (build_id TEXT PRIMARY KEY)") + connection.execute("INSERT INTO ledger_builds VALUES (?)", (build_id,)) + + +@pytest.mark.parametrize("db_name", ["chronicle.db", "ledger.db"]) +def test_infer_build_id_reads_new_and_legacy_database_names(tmp_path, db_name): + suite = tmp_path / "suite" + suite.mkdir() + _sqlite_build(suite / db_name, "ledger.build.v1:from-db") + + assert infer_build_id(suite) == "ledger.build.v1:from-db" + + +def test_infer_build_id_prefers_the_chronicle_database(tmp_path): + suite = tmp_path / "suite" + suite.mkdir() + _sqlite_build(suite / "chronicle.db", "ledger.build.v1:chronicle") + _sqlite_build(suite / "ledger.db", "ledger.build.v1:legacy") + + assert infer_build_id(suite) == "ledger.build.v1:chronicle" + + +@pytest.mark.parametrize("db_name", ["chronicle.db", "ledger.db"]) +def test_publish_derived_classifies_both_database_names(tmp_path, db_name): + suite = tmp_path / "suite" + reports = suite / "reports" + reports.mkdir(parents=True) + build_id = "ledger.build.v1:kind" + (reports / "database.json").write_text(json.dumps({"build_id": build_id})) + (suite / db_name).write_bytes(b"db") + wrangler = tmp_path / "wrangler" + wrangler.write_text("#!/bin/sh\necho ok\n") + wrangler.chmod(0o755) + + report = publish_derived_artifacts( + suite, + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + wrangler_command=str(wrangler), + ) + rows = {row["artifact_name"]: row for row in build_artifact_rows(report)} + + assert rows[db_name]["artifact_kind"] == "sqlite_database" + + +def test_publish_derived_uses_the_configured_bucket(tmp_path, monkeypatch): + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + suite = tmp_path / "suite" + reports = suite / "reports" + reports.mkdir(parents=True) + (reports / "database.json").write_text( + json.dumps({"build_id": "ledger.build.v1:bucket"}) + ) + (suite / "facts.jsonl").write_text("{}\n") + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + + report = publish_derived_artifacts( + suite, + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + wrangler_command=str(wrangler), + ) + + assert report.valid + assert report.entries[0].r2_location.bucket == "chronicle-derived" + assert "chronicle-derived/derived/irs_soi/" in log.read_text() + + +def test_publish_raw_skips_an_object_already_held_by_a_preserved_bucket( + tmp_path, monkeypatch +): + """A recorded storage.r2 bucket is preserved history, not a publish target. + + Archived witness records pin raw R2 URLs by hash, so backfilling the same + bytes into a renamed bucket must not rewrite the manifest. The entry is + already published, so the sweep reports it skipped and stays green: after + the bucket-default flip every entry published before it takes this path. + """ + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-1-1" + source = tmp_path / "soi.xlsx" + source.write_bytes(b"official SOI workbook") + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + ) + manifest_path = output_dir / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + artifact = manifest["files"][2023] + recorded_key = ( + f"raw/irs_soi/soi-table-1-1/2023/{artifact['sha256']}/{artifact['filename']}" + ) + artifact["storage"] = { + "r2": { + "provider": "r2", + "bucket": "ledger-raw", + "key": recorded_key, + "uri": f"r2://ledger-raw/{recorded_key}", + } + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + + before = manifest_path.read_bytes() + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + entry = report.entries[0] + + assert report.valid + assert entry.errors == () + assert entry.upload is None + assert entry.skipped == ( + "recorded_r2_bucket_is_preserved_history:" + "recorded=ledger-raw:requested=chronicle-raw" + ) + assert entry.r2_location is not None + assert entry.r2_location.bucket == "ledger-raw" + assert entry.r2_location.key == recorded_key + assert entry.to_dict()["skipped"] == entry.skipped + assert report.counts["skipped_count"] == 1 + assert report.counts["uploaded_count"] == 0 + assert report.counts["failed_count"] == 0 + assert not log.exists() + assert manifest_path.read_bytes() == before + + +def test_documented_bucket_cutover_sweep_accepts_the_tracked_registry( + tmp_path, monkeypatch, capsys +): + """The documented bucket flip is green for every recorded historical key.""" + tracked_data = Path(__file__).resolve().parents[1] / "db" / "data" + copied_data = tmp_path / "data" + shutil.copytree(tracked_data, copied_data) + manifest_bytes = { + path.relative_to(copied_data): path.read_bytes() + for path in copied_data.rglob("*") + if path.is_file() and path.name.lower().startswith("manifest") + } + uploads = [] + + def non_writing_uploader(location, local_path, *, wrangler_command): + uploads.append((location, local_path, wrangler_command)) + return ArtifactCommandResult( + command=("non-writing-uploader",), + returncode=0, + stdout="", + stderr="", + ) + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", non_writing_uploader) + + exit_code = harness_main( + [ + "publish-raw", + "--root", + str(copied_data), + "--wrangler-command", + "non-writing-uploader", + ] + ) + report = json.loads(capsys.readouterr().out) + counts = report["counts"] + + # The tracked registry grows as packages land, so the sweep is pinned by + # its invariants rather than by today's exact counts: every artifact is a + # preserved-bucket skip with an R2 link, nothing uploads or fails, no + # manifest-level error, exit 0. The floors keep the test meaningful. + observed = (exit_code, report["valid"], len(report["errors"])) + assert observed == (0, True, 0), json.dumps( + {"observed": observed, "counts": counts}, sort_keys=True + ) + assert counts["uploaded_count"] == 0, counts + assert counts["failed_count"] == 0, counts + assert ( + counts["skipped_count"] == counts["artifact_count"] == counts["r2_link_count"] + ), counts + assert counts["artifact_count"] >= 194, counts + assert counts["manifest_count"] >= 161, counts + assert all(entry["skipped"] or entry["upload"] for entry in report["entries"]) + assert uploads == [] + assert { + path.relative_to(copied_data): path.read_bytes() + for path in copied_data.rglob("*") + if path.is_file() and path.name.lower().startswith("manifest") + } == manifest_bytes + + +def test_fetch_artifact_keeps_an_already_recorded_bucket(tmp_path, monkeypatch): + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-1-1" + source = tmp_path / "soi.xlsx" + source.write_bytes(b"official SOI workbook") + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + upload_r2=True, + wrangler_command=str(wrangler), + ) + manifest_path = output_dir / "manifest.yaml" + first = yaml.safe_load(manifest_path.read_text()) + assert first["files"][2023]["storage"]["r2"]["bucket"] == "ledger-raw" + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + report = fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-1-1", + year=2023, + output_dir=output_dir, + upload_r2=True, + wrangler_command=str(wrangler), + ) + second = yaml.safe_load(manifest_path.read_text()) + + # The backfill copy really is uploaded to the new bucket, but the manifest + # keeps recording where the bytes were first published. + assert report.r2_location.bucket == "chronicle-raw" + assert "chronicle-raw" in log.read_text() + assert ( + second["files"][2023]["storage"]["r2"] == first["files"][2023]["storage"]["r2"] + ) + + +# --------------------------------------------------------------------------- +# Publisher revisions +# +# A raw R2 key is content-addressed, so a recorded storage.r2 block is a claim +# about specific bytes. On 2026-09-02 the IRS re-published 22in05ira.xlsx and +# 22in06ira.xlsx under their existing URLs (PolicyEngine/chronicle#225): a +# repeated fetch must never pair those new bytes with the old object's URI. +# --------------------------------------------------------------------------- + +REPUBLISHED_URL = "https://www.irs.gov/pub/irs-soi/22in05ira.xlsx" +REPUBLISHED_FILENAME = "22in05ira.xlsx" +FIRST_PUBLICATION = b"IRA table 5, first publication" +SECOND_PUBLICATION = b"IRA table 5, silently re-published with revised rows" + + +def _wrangler_stub(tmp_path, log): + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + return wrangler + + +def _serve(monkeypatch, content): + """Serve ``content`` from the publisher URL, without touching the network.""" + + def _fake_read_artifact(source_url): + assert source_url == REPUBLISHED_URL + return content, REPUBLISHED_FILENAME + + monkeypatch.setattr("chronicle.artifacts._read_artifact", _fake_read_artifact) + + +def _fetch_republished(output_dir, wrangler, *, upload_r2=True, **kwargs): + return fetch_source_artifact( + REPUBLISHED_URL, + source_id="irs_soi", + package_id="soi-table-5", + year=2022, + output_dir=output_dir, + upload_r2=upload_r2, + wrangler_command=str(wrangler), + **kwargs, + ) + + +def test_repeated_fetch_of_identical_bytes_preserves_the_recorded_block( + tmp_path, monkeypatch +): + """Same bytes: the recorded block survives whatever bucket is configured.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + first = yaml.safe_load(manifest_path.read_text())["files"][2022] + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + report = _fetch_republished(output_dir, wrangler) + second = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + # The backfill copy really goes to the renamed bucket, but the manifest + # keeps recording where these bytes were first published. + assert report.r2_location.bucket == "chronicle-raw" + assert "chronicle-raw" in log.read_text() + assert second["storage"] == first["storage"] + # Field order too, so the block is byte-for-byte identical once dumped. + assert list(second["storage"]["r2"].items()) == list(first["storage"]["r2"].items()) + assert second["storage"]["r2"]["bucket"] == "ledger-raw" + assert "previous_r2" not in second["storage"] + assert second["sha256"] == first["sha256"] + + +@pytest.mark.parametrize( + ("upload_r2", "configured_bucket"), + [ + pytest.param(True, None, id="reuploaded"), + # The two routes that reached a manifest in the wild: a fetch that only + # registers the bytes, and a fetch once the bucket default has moved. + # Both preserved the recorded block while rewriting sha256/size_bytes. + pytest.param(False, None, id="registered-without-upload"), + pytest.param(True, "chronicle-raw", id="after-the-bucket-rename"), + ], +) +def test_repeated_fetch_of_different_bytes_is_refused( + tmp_path, monkeypatch, upload_r2, configured_bucket +): + """A publisher revision must not inherit the recorded object's provenance.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + artifact_path = output_dir / REPUBLISHED_FILENAME + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + manifest_before = manifest_path.read_bytes() + uploads_before = log.read_text() + + if configured_bucket: + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", configured_bucket) + _serve(monkeypatch, SECOND_PUBLICATION) + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_republished(output_dir, wrangler, upload_r2=upload_r2) + + message = str(raised.value) + assert first_report.sha256 in message + assert hashlib.sha256(SECOND_PUBLICATION).hexdigest() in message + assert f"size_bytes={len(FIRST_PUBLICATION)}" in message + assert f"size_bytes={len(SECOND_PUBLICATION)}" in message + assert "release revision" in message + assert "--record-revision" in message + # Nothing was overwritten, copied or uploaded on the way to the refusal. + assert manifest_path.read_bytes() == manifest_before + assert artifact_path.read_bytes() == FIRST_PUBLICATION + assert log.read_text() == uploads_before + + +def test_record_revision_writes_a_new_key_and_keeps_the_previous_object( + tmp_path, monkeypatch +): + """The opt-in records the new bytes' own key under the configured bucket.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + superseded = yaml.safe_load(manifest_path.read_text())["files"][2022] + + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + _serve(monkeypatch, SECOND_PUBLICATION) + report = _fetch_republished(output_dir, wrangler, record_revision=True) + revised = yaml.safe_load(manifest_path.read_text())["files"][2022] + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + assert report.valid + # storage.r2 names the object that holds the entry's current bytes... + assert revised["sha256"] == revised_sha256 + assert revised["size_bytes"] == len(SECOND_PUBLICATION) + assert revised["storage"]["r2"]["bucket"] == "chronicle-raw" + assert revised["storage"]["r2"]["key"] == ( + f"raw/irs_soi/soi-table-5/2022/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert revised["storage"]["r2"]["uri"] == ( + f"r2://chronicle-raw/{revised['storage']['r2']['key']}" + ) + # ...and never the superseded key, which stays addressable as history. + previous = revised["storage"]["previous_r2"] + assert [entry["uri"] for entry in previous] == [superseded["storage"]["r2"]["uri"]] + assert previous[0]["bucket"] == "ledger-raw" + assert previous[0]["sha256"] == first_report.sha256 + assert previous[0]["size_bytes"] == len(FIRST_PUBLICATION) + assert previous[0]["fetched_at"] == superseded["fetched_at"] + assert previous[0]["superseded_at"] == revised["fetched_at"] + assert (output_dir / REPUBLISHED_FILENAME).read_bytes() == SECOND_PUBLICATION + assert f"chronicle-raw/{revised['storage']['r2']['key']}" in log.read_text() + + +def test_a_revised_manifest_still_reads_as_one_r2_linked_artifact( + tmp_path, monkeypatch +): + """storage.previous_r2 is a sibling key, so every storage.r2 reader is intact.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, record_revision=True) + + inventory = inventory_source_artifacts(output_dir) + + assert inventory.valid + assert inventory.counts["r2_link_count"] == 1 + assert inventory.counts["checksum_mismatch_count"] == 0 + assert inventory.entries[0].r2["bucket"] == "ledger-raw" + assert inventory.entries[0].sha256_actual == ( + hashlib.sha256(SECOND_PUBLICATION).hexdigest() + ) + + +def test_publish_raw_refuses_a_file_the_recorded_object_does_not_hold( + tmp_path, monkeypatch +): + """Recorded sha256 != local sha256 is a revision, not a backfill.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + + # Reproduce the state a pre-fix fetch left behind: new bytes on disk, the + # entry's own hash rewritten, the recorded key still addressing the old + # bytes. + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + (output_dir / REPUBLISHED_FILENAME).write_bytes(SECOND_PUBLICATION) + manifest = yaml.safe_load(manifest_path.read_text()) + manifest["files"][2022]["sha256"] = revised_sha256 + manifest["files"][2022]["size_bytes"] = len(SECOND_PUBLICATION) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + manifest_before = manifest_path.read_bytes() + uploads_before = log.read_text() + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + + assert not report.valid + assert report.entries[0].upload is None + assert report.entries[0].r2_location is None + assert report.entries[0].errors == ( + "recorded_r2_identity_mismatch:" + f"recorded_sha256={first_report.sha256}:" + f"recorded_filename={REPUBLISHED_FILENAME}:" + f"local_sha256={revised_sha256}:" + f"local_filename={REPUBLISHED_FILENAME}", + ) + assert log.read_text() == uploads_before + assert manifest_path.read_bytes() == manifest_before + + +def test_publish_raw_uploads_a_registered_revision_and_keeps_its_history( + tmp_path, monkeypatch +): + """Once the revision is registered, publishing it is ordinary work.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, record_revision=True) + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + published = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + assert published["storage"]["r2"]["key"].endswith( + f"/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert [entry["sha256"] for entry in published["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + +def test_fetch_artifact_cli_refuses_a_revision_then_records_it_on_request( + tmp_path, monkeypatch, capsys +): + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + argv = [ + "fetch-artifact", + "--url", + REPUBLISHED_URL, + "--source-id", + "irs_soi", + "--package-id", + "soi-table-5", + "--year", + "2022", + "--out-dir", + str(output_dir), + "--upload-r2", + "--wrangler-command", + str(wrangler), + ] + _serve(monkeypatch, FIRST_PUBLICATION) + assert harness_main(argv) == 0 + capsys.readouterr() + + _serve(monkeypatch, SECOND_PUBLICATION) + refused = harness_main(argv) + refusal = capsys.readouterr() + + assert refused == 1 + assert "--record-revision" in refusal.err + assert refusal.out == "" + + assert harness_main([*argv, "--record-revision"]) == 0 + recorded = json.loads(capsys.readouterr().out) + + assert recorded["sha256"] == hashlib.sha256(SECOND_PUBLICATION).hexdigest() + assert recorded["r2_location"]["key"].endswith( + f"/{recorded['sha256']}/{REPUBLISHED_FILENAME}" + ) + + +def test_record_revision_without_an_upload_records_no_current_object( + tmp_path, monkeypatch +): + """An offline revision keeps history without claiming the new bytes exist. + + Registering a revision without ``--upload-r2`` leaves the entry with no + ``storage.r2`` at all rather than a pointer to bytes R2 does not hold. The + superseded object stays addressable, and a later publish-raw completes the + registration. + """ + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + first_report = _fetch_republished(output_dir, wrangler) + + _serve(monkeypatch, SECOND_PUBLICATION) + _fetch_republished(output_dir, wrangler, upload_r2=False, record_revision=True) + registered = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert "r2" not in registered["storage"] + assert [entry["sha256"] for entry in registered["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + report = publish_source_artifacts(output_dir, wrangler_command=str(wrangler)) + published = yaml.safe_load(manifest_path.read_text())["files"][2022] + revised_sha256 = hashlib.sha256(SECOND_PUBLICATION).hexdigest() + + assert report.valid + assert published["storage"]["r2"]["key"].endswith( + f"/{revised_sha256}/{REPUBLISHED_FILENAME}" + ) + assert [entry["sha256"] for entry in published["storage"]["previous_r2"]] == [ + first_report.sha256 + ] + + +def _shared_archive_entry(content, *, package_id, year, filename="shared.zip"): + sha256 = hashlib.sha256(content).hexdigest() + key = f"raw/usda_snap/{package_id}/{year}/{sha256}/{filename}" + return { + "filename": filename, + "source_url": "https://example.test/shared.zip", + "sha256": sha256, + "size_bytes": len(content), + "fetched_at": "2026-05-11T11:57:29+00:00", + "storage": { + "r2": { + "provider": "r2", + "bucket": "ledger-raw", + "key": key, + "uri": f"r2://ledger-raw/{key}", + } + }, + } + + +def test_shared_archive_revision_is_refused_through_an_unregistered_owner(tmp_path): + """A selected empty vintage cannot bypass another manifest's identity.""" + package = tmp_path / "db" / "data" / "usda_snap" / "fy69_to_current" + package.mkdir(parents=True) + original = b"USDA archive, first publication" + revised = b"USDA archive, revised publication" + filename = "snap-zip-fy69tocurrent-6.zip" + (package / filename).write_bytes(original) + primary_path = package / "manifest.yaml" + primary_path.write_text( + yaml.safe_dump( + { + "source_id": "usda_snap", + "package_id": "usda-snap-fy69-to-current", + "files": {}, + }, + sort_keys=False, + ) + ) + sibling_path = package / "manifest_fy2025_monthly_source_package.yaml" + sibling_path.write_text( + yaml.safe_dump( + { + "source_id": "usda_snap", + "package_id": "usda-snap-fy2025-monthly-state-caseloads", + "files": { + 2025: _shared_archive_entry( + original, + package_id="usda-snap-fy69-to-current", + year=2024, + filename=filename, + ) + }, + }, + sort_keys=False, + ) + ) + publisher = _publish(tmp_path, filename, revised) + before = {path: path.read_bytes() for path in (primary_path, sibling_path)} + + with pytest.raises(SourceArtifactRevisionError): + fetch_source_artifact( + str(publisher), + source_id="usda_snap", + package_id="usda-snap-fy69-to-current", + year=2024, + output_dir=package, + ) + + assert (package / filename).read_bytes() == original + assert {path: path.read_bytes() for path in before} == before + + +def test_record_revision_updates_every_owner_of_usda_shared_archive(tmp_path): + """The tracked USDA two-manifest shape has one physical archive.""" + package = tmp_path / "db" / "data" / "usda_snap" / "fy69_to_current" + package.mkdir(parents=True) + original = b"USDA archive, first publication" + revised = b"USDA archive, revised publication" + revised_sha256 = hashlib.sha256(revised).hexdigest() + filename = "snap-zip-fy69tocurrent-6.zip" + (package / filename).write_bytes(original) + manifests = ( + ( + package / "manifest.yaml", + "usda-snap-fy69-to-current", + 2024, + "usda-snap-fy69-to-current", + 2024, + ), + ( + package / "manifest_fy2025_monthly_source_package.yaml", + "usda-snap-fy2025-monthly-state-caseloads", + 2025, + "usda-snap-fy69-to-current", + 2024, + ), + ) + previous_uris = {} + for path, package_id, vintage, route_package, route_year in manifests: + entry = _shared_archive_entry( + original, + package_id=route_package, + year=route_year, + filename=filename, + ) + entry["source_table"] = f"owner {vintage}" + previous_uris[path] = entry["storage"]["r2"]["uri"] + path.write_text( + yaml.safe_dump( + { + "source_id": "usda_snap", + "package_id": package_id, + "files": {vintage: entry}, + }, + sort_keys=False, + ) + ) + publisher = _publish(tmp_path, filename, revised) + + fetch_source_artifact( + str(publisher), + source_id="usda_snap", + package_id="usda-snap-fy69-to-current", + year=2024, + output_dir=package, + record_revision=True, + ) + + assert (package / filename).read_bytes() == revised + for path, _package_id, vintage, _route_package, _route_year in manifests: + entry = yaml.safe_load(path.read_text())["files"][vintage] + assert entry["sha256"] == revised_sha256 + assert entry["size_bytes"] == len(revised) + assert entry["source_table"] == f"owner {vintage}" + assert "r2" not in entry["storage"] + assert [item["uri"] for item in entry["storage"]["previous_r2"]] == [ + previous_uris[path] + ] + + +def test_record_revision_updates_every_same_manifest_owner(tmp_path): + """SSA-style semantic aliases of one file share one byte identity.""" + package = tmp_path / "db" / "data" / "ssa" / "supplement" + package.mkdir(parents=True) + original = b"SSA extracted table, first publication" + revised = b"SSA extracted table, revised publication" + revised_sha256 = hashlib.sha256(revised).hexdigest() + filename = "ssa_oasdi_ssi_2024.csv" + (package / filename).write_bytes(original) + manifest_path = package / "manifest.yaml" + entries = { + 2024: _shared_archive_entry( + original, + package_id="ssa-annual-statistical-supplement-2025", + year=2024, + filename=filename, + ), + "extracted_targets": _shared_archive_entry( + original, + package_id="ssa-annual-statistical-supplement-2025", + year="extracted_targets", + filename=filename, + ), + } + for entry in entries.values(): + entry["source_url"] = "https://example.test/ssa.csv" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "ssa", + "package_id": "ssa-annual-statistical-supplement-2025", + "files": entries, + }, + sort_keys=False, + ) + ) + publisher = _publish(tmp_path, filename, revised) + + fetch_source_artifact( + str(publisher), + source_id="ssa", + package_id="ssa-annual-statistical-supplement-2025", + year=2024, + output_dir=package, + record_revision=True, + ) + + updated = yaml.safe_load(manifest_path.read_text())["files"] + assert {entry["sha256"] for entry in updated.values()} == {revised_sha256} + assert { + item["sha256"] + for entry in updated.values() + for item in entry["storage"]["previous_r2"] + } == {hashlib.sha256(original).hexdigest()} + + +def test_a_recorded_block_that_only_carries_a_uri_is_still_recognized( + tmp_path, monkeypatch +): + """Identity reads the URI when a hand-written block records no key.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + manifest_path = output_dir / "manifest.yaml" + _serve(monkeypatch, FIRST_PUBLICATION) + _fetch_republished(output_dir, wrangler) + manifest = yaml.safe_load(manifest_path.read_text()) + recorded = manifest["files"][2022]["storage"]["r2"] + manifest["files"][2022]["storage"]["r2"] = { + "provider": recorded["provider"], + "bucket": recorded["bucket"], + "uri": recorded["uri"], + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + uri_only = manifest["files"][2022]["storage"]["r2"] + + report = _fetch_republished(output_dir, wrangler) + preserved = yaml.safe_load(manifest_path.read_text())["files"][2022] + + assert report.valid + assert preserved["storage"]["r2"] == uri_only + + _serve(monkeypatch, SECOND_PUBLICATION) + with pytest.raises(SourceArtifactRevisionError): + _fetch_republished(output_dir, wrangler) + + +# --------------------------------------------------------------------------- +# Manifest addressing, entry identity, and recorded locators +# +# Everything below concerns the state a fetch reads before it writes: which +# manifest it reads, what that manifest's entry says its vintage holds, and +# whether the recorded R2 block names one object or two. +# --------------------------------------------------------------------------- + +TRADITIONAL_MANIFEST = "manifest_traditional_source_package.yaml" +ROTH_MANIFEST = "manifest_roth_source_package.yaml" + + +def _publish(tmp_path, name, content): + """Write bytes a fetch can read as a local publisher path.""" + path = tmp_path / "publisher" / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return path + + +def _fetch_local(output_dir, source_path, *, package_id="soi-table-5", **kwargs): + return fetch_source_artifact( + str(source_path), + source_id="irs_soi", + package_id=package_id, + year=2022, + output_dir=output_dir, + **kwargs, + ) + + +def _entry(manifest_path): + return yaml.safe_load(manifest_path.read_text())["files"][2022] + + +def test_fetch_artifact_writes_the_manifest_it_was_given(tmp_path): + """One publisher directory, two source packages, two manifests. + + db/data/irs_soi/ira_contributions keeps the traditional and Roth IRA + packages side by side. A fetch that always wrote manifest.yaml would write + a third manifest neither package reads. + """ + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + roth = _publish(tmp_path, "22in06ira.xlsx", b"roth IRA table") + package.mkdir(parents=True) + for name, package_id in ( + (TRADITIONAL_MANIFEST, "soi-ira-traditional-contributions-2022"), + (ROTH_MANIFEST, "soi-ira-roth-contributions-2022"), + ): + (package / name).write_text( + yaml.safe_dump( + {"source_id": "irs_soi", "package_id": package_id, "files": {}}, + sort_keys=False, + ) + ) + + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + _fetch_local( + package, + roth, + package_id="soi-ira-roth-contributions-2022", + manifest_filename=ROTH_MANIFEST, + ) + + assert sorted(path.name for path in package.glob("manifest*.yaml")) == [ + ROTH_MANIFEST, + TRADITIONAL_MANIFEST, + ] + assert not (package / "manifest.yaml").exists() + assert _entry(package / TRADITIONAL_MANIFEST)["filename"] == "22in05ira.xlsx" + assert _entry(package / ROTH_MANIFEST)["filename"] == "22in06ira.xlsx" + assert _entry(package / TRADITIONAL_MANIFEST)["sha256"] == ( + hashlib.sha256(b"traditional IRA table").hexdigest() + ) + + +def test_a_revision_is_refused_in_the_manifest_that_records_it(tmp_path): + """The IRA revision workflow the docs cite, on a two-manifest package.""" + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + recorded = (package / TRADITIONAL_MANIFEST).read_bytes() + + # The IRS re-publishes under the same URL and vintage. + traditional.write_bytes(b"traditional IRA table, revised rows") + + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + + assert TRADITIONAL_MANIFEST in str(raised.value) + assert (package / TRADITIONAL_MANIFEST).read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == b"traditional IRA table" + + # Without the flag the same fetch would address a manifest.yaml that no + # package reads and that protects nothing: the #225 path. It is refused, + # naming the manifests the directory keeps, and nothing is written. + with pytest.raises(AmbiguousManifestError) as stray: + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + ) + + assert TRADITIONAL_MANIFEST in str(stray.value) + assert "--manifest" in str(stray.value) + assert not (package / "manifest.yaml").exists() + assert (package / TRADITIONAL_MANIFEST).read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == b"traditional IRA table" + + +def test_fetch_artifact_cli_refuses_a_stray_default_manifest(tmp_path, capsys): + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + _fetch_local( + package, + traditional, + package_id="soi-ira-traditional-contributions-2022", + manifest_filename=TRADITIONAL_MANIFEST, + ) + argv = [ + "fetch-artifact", + "--url", + str(traditional), + "--source-id", + "irs_soi", + "--package-id", + "soi-ira-traditional-contributions-2022", + "--year", + "2022", + "--out-dir", + str(package), + ] + + assert harness_main(argv) == 1 + + err = capsys.readouterr().err + assert err.startswith("error: ") + assert TRADITIONAL_MANIFEST in err + assert not (package / "manifest.yaml").exists() + + +def test_fetch_refuses_a_stray_default_beside_a_case_variant_named_manifest( + tmp_path, monkeypatch +): + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + package.mkdir(parents=True) + named_manifest = package / "MANIFEST_TRADITIONAL.YML" + named_manifest.write_text("source_id: irs_soi\nfiles: {}\n") + source = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + + def unexpected_read(_source_url): + raise AssertionError("a case-variant named manifest did not block I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(AmbiguousManifestError, match="MANIFEST_TRADITIONAL.YML"): + _fetch_local(package, source) + + assert not (package / "manifest.yaml").exists() + + +def test_a_same_bytes_rename_is_refused_by_name_not_as_a_revision(tmp_path): + """Identical bytes under another filename are neither a revision nor a + re-fetch: the entry's filename must keep agreeing with its recorded key.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5") + _fetch_local(package, source, filename="table-5.xlsx", upload_r2=False) + recorded = (package / "manifest.yaml").read_bytes() + + for record_revision in (False, True): + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local( + package, source, upload_r2=False, record_revision=record_revision + ) + message = str(raised.value) + assert "rename is not a release revision" in message + assert "filename=table-5.xlsx" in message + assert "names them 22in05ira.xlsx" in message + assert "--filename table-5.xlsx" in message + + assert (package / "manifest.yaml").read_bytes() == recorded + assert not (package / "22in05ira.xlsx").exists() + assert (package / "table-5.xlsx").read_bytes() == b"IRA table 5" + + +@pytest.mark.parametrize( + "manifest_filename", + ["../manifest.yaml", "nested/manifest.yaml", "", " ", ".", ".."], +) +def test_a_manifest_name_must_stay_inside_the_package(tmp_path, manifest_filename): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "table.xlsx", b"table") + + with pytest.raises(ValueError, match="inside the package directory"): + _fetch_local(package, source, manifest_filename=manifest_filename) + + assert not package.exists() + + +@pytest.mark.parametrize( + ("source_url", "filename", "message"), + [ + pytest.param("publisher.csv", "manifest.yaml", "manifest name", id="default"), + pytest.param( + "publisher.csv", "MANIFEST_NAMED.YML", "manifest name", id="named" + ), + pytest.param( + "publisher.csv", "nested/publisher.csv", "bare filename", id="nested" + ), + pytest.param( + "https://publisher.test/manifest.yaml", + None, + "manifest name", + id="inferred", + ), + ], +) +def test_artifact_filename_is_refused_before_publisher_io( + tmp_path, monkeypatch, source_url, filename, message +): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text("source_id: irs_soi\npackage_id: soi-table\nfiles: {}\n") + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("an invalid artifact filename reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(SourceArtifactManifestError, match=message): + fetch_source_artifact( + source_url, + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + filename=filename, + ) + + assert manifest_path.read_bytes() == before + + +@pytest.mark.parametrize( + ("source_id", "package_id"), + [ + pytest.param("/", "package", id="source-id"), + pytest.param("publisher", "/", id="package-id"), + ], +) +def test_fetch_refuses_invalid_r2_identity_before_publisher_io( + tmp_path, monkeypatch, source_id, package_id +): + package = tmp_path / "db" / "data" / "publisher" / "package" + package.mkdir(parents=True) + artifact_path = package / "table.csv" + artifact_path.write_bytes(b"registered publisher bytes") + + def unexpected_read(_source_url): + raise AssertionError("an invalid R2 identity reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(ValueError, match="canonical R2 key segment"): + fetch_source_artifact( + "https://publisher.test/table.csv", + source_id=source_id, + package_id=package_id, + year=2024, + output_dir=package, + ) + + assert artifact_path.read_bytes() == b"registered publisher bytes" + assert not (package / "manifest.yaml").exists() + + +def test_manifest_name_must_be_discoverable_before_publisher_io(tmp_path, monkeypatch): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table" + source = _publish(tmp_path, "table.csv", b"publisher table") + + def unexpected_read(_source_url): + raise AssertionError("an undiscoverable manifest name reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(ManifestNameError, match="invisible"): + _fetch_local(package, source, manifest_filename="custom.yaml") + + assert not package.exists() + + +def test_fetch_artifact_cli_reports_a_manifest_name_outside_the_package( + tmp_path, capsys +): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "table.xlsx", b"table") + argv = [ + "fetch-artifact", + "--url", + str(source), + "--source-id", + "irs_soi", + "--package-id", + "soi-table-5", + "--year", + "2022", + "--out-dir", + str(package), + "--manifest", + "../manifest.yaml", + ] + + assert harness_main(argv) == 1 + + err = capsys.readouterr().err + assert err.startswith("error: ") + assert "inside the package directory" in err + assert not package.exists() + assert not (tmp_path / "db" / "data" / "irs_soi" / "manifest.yaml").exists() + + +def test_fetch_artifact_cli_targets_the_named_manifest(tmp_path, capsys): + package = tmp_path / "db" / "data" / "irs_soi" / "ira_contributions" + traditional = _publish(tmp_path, "22in05ira.xlsx", b"traditional IRA table") + argv = [ + "fetch-artifact", + "--url", + str(traditional), + "--source-id", + "irs_soi", + "--package-id", + "soi-ira-traditional-contributions-2022", + "--year", + "2022", + "--out-dir", + str(package), + "--manifest", + TRADITIONAL_MANIFEST, + ] + + assert harness_main(argv) == 0 + payload = json.loads(capsys.readouterr().out) + + assert payload["manifest_path"].endswith(TRADITIONAL_MANIFEST) + assert not (package / "manifest.yaml").exists() + + traditional.write_bytes(b"traditional IRA table, revised rows") + + assert harness_main(argv) == 1 + assert TRADITIONAL_MANIFEST in capsys.readouterr().err + + +@pytest.mark.parametrize( + ("existing_name", "requested_name"), + [ + pytest.param("manifest.yml", "manifest.yaml", id="yml-default"), + pytest.param("Manifest.yaml", "manifest.yaml", id="case-variant-default"), + pytest.param( + "manifest_monthly_source_package.yaml", + "manifest_monthy_source_package.yaml", + id="mistyped-named-manifest", + ), + ], +) +def test_fetch_refuses_to_create_any_manifest_beside_an_existing_registry( + tmp_path, monkeypatch, existing_name, requested_name +): + package = tmp_path / "db" / "data" / "usda_snap" / "fy69_to_current" + package.mkdir(parents=True) + existing = package / existing_name + existing.write_text( + "source_id: usda_snap\npackage_id: usda-snap-fy69-to-current\nfiles: {}\n" + ) + before = existing.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("ambiguous manifest creation reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(AmbiguousManifestError, match=existing_name): + fetch_source_artifact( + "https://example.test/snap.zip", + source_id="usda_snap", + package_id="usda-snap-fy69-to-current", + year=2024, + output_dir=package, + manifest_filename=requested_name, + ) + + assert existing.read_bytes() == before + assert requested_name not in {path.name for path in package.iterdir()} + + +def test_fetch_refuses_a_symlinked_manifest_before_publisher_io(tmp_path, monkeypatch): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table" + package.mkdir(parents=True) + outside_manifest = tmp_path / "outside-manifest.yaml" + outside_manifest.write_text( + "source_id: irs_soi\npackage_id: soi-table\nfiles: {}\n" + ) + manifest_path = package / "manifest.yaml" + manifest_path.symlink_to(outside_manifest) + before = outside_manifest.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("a symlinked manifest reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(MalformedManifestError, match="symlink"): + fetch_source_artifact( + "https://example.test/table.xlsx", + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + + assert manifest_path.is_symlink() + assert outside_manifest.read_bytes() == before + + +def test_fetch_refuses_physically_distinct_normalized_manifest_aliases( + tmp_path, monkeypatch +): + package = tmp_path / "db" / "data" / "publisher" / "package" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text("source_id: publisher\npackage_id: package\nfiles: {}\n") + case_alias = package / "Manifest.yaml" + monkeypatch.setattr( + "chronicle.artifacts.package_manifest_paths", + lambda _package: [manifest_path, case_alias], + ) + + def unexpected_read(_source_url): + raise AssertionError("normalized manifest aliases reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(AmbiguousManifestError, match="normalized manifest name"): + fetch_source_artifact( + "https://example.test/table.csv", + source_id="publisher", + package_id="package", + year=2024, + output_dir=package, + ) + + +def test_fetch_refuses_a_symlinked_artifact_target_before_publisher_io( + tmp_path, monkeypatch +): + package = tmp_path / "db" / "data" / "publisher" / "package" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text("source_id: publisher\npackage_id: package\nfiles: {}\n") + outside = tmp_path / "outside.csv" + outside.write_bytes(b"outside bytes") + artifact_path = package / "table.csv" + artifact_path.symlink_to(outside) + before = {manifest_path: manifest_path.read_bytes(), outside: outside.read_bytes()} + + def unexpected_read(_source_url): + raise AssertionError("symlinked artifact target reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(ArtifactFilenameError, match="symbolic link"): + fetch_source_artifact( + "https://example.test/table.csv", + source_id="publisher", + package_id="package", + year=2024, + output_dir=package, + ) + + assert artifact_path.is_symlink() + assert {path: path.read_bytes() for path in before} == before + + +@pytest.mark.parametrize( + "manifest_filename", + [ + pytest.param("../manifest.yaml", id="parent"), + pytest.param("manifest_*.yaml", id="star-glob"), + pytest.param("manifest_?.yml", id="question-glob"), + pytest.param("manifest_[ab].yaml", id="character-class-glob"), + ], +) +def test_sweep_manifest_selector_must_be_a_literal_supported_filename( + tmp_path, manifest_filename +): + root = tmp_path / "requested-root" + package = root / "package" + package.mkdir(parents=True) + content = b"publisher table" + (package / "table.csv").write_bytes(content) + (package / "manifest_a.yaml").write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": { + 2024: { + "filename": "table.csv", + "sha256": hashlib.sha256(content).hexdigest(), + } + }, + }, + sort_keys=False, + ) + ) + outside_manifest = root.parent / "manifest.yaml" + outside_manifest.write_text("files: {}\n") + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + before = { + path: path.read_bytes() + for path in (package / "manifest_a.yaml", outside_manifest) + } + + with pytest.raises(ManifestNameError, match="Manifest"): + publish_source_artifacts( + root, + manifest_filename=manifest_filename, + wrangler_command=str(wrangler), + ) + + assert {path: path.read_bytes() for path in before} == before + assert not log.exists() + + +@pytest.mark.parametrize( + "operation", [inventory_source_artifacts, publish_source_artifacts] +) +def test_invalid_sweep_manifest_selector_is_refused_even_when_root_is_missing( + tmp_path, operation +): + with pytest.raises(ManifestNameError): + operation(tmp_path / "missing", manifest_filename="../manifest.yaml") + + +@pytest.mark.parametrize("command", ["inventory-artifacts", "publish-raw"]) +def test_sweep_cli_reports_an_invalid_manifest_selector(command, tmp_path, capsys): + exit_code = harness_main( + [ + command, + "--root", + str(tmp_path), + "--manifest", + "../manifest.yaml", + ] + ) + + captured = capsys.readouterr() + assert exit_code == 1 + assert captured.out == "" + assert captured.err.startswith("error: ") + + +# --------------------------------------------------------------------------- +# Identity without a recorded R2 object +# --------------------------------------------------------------------------- + + +def _failing_wrangler(tmp_path, log): + wrangler = tmp_path / "failing-wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\nexit 1\n") + wrangler.chmod(0o755) + return wrangler + + +def test_a_registered_entry_is_protected_before_it_is_ever_published(tmp_path): + """No storage.r2 yet is not no identity: the entry declares its bytes.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + first = _fetch_local(package, source, upload_r2=False) + recorded = (package / "manifest.yaml").read_bytes() + + assert "storage" not in _entry(package / "manifest.yaml") + + # Same bytes: an ordinary repeated fetch, not a revision. + assert _fetch_local(package, source, upload_r2=False).sha256 == first.sha256 + + source.write_bytes(b"IRA table 5, silently re-published") + with pytest.raises(SourceArtifactRevisionError) as raised: + _fetch_local(package, source, upload_r2=False) + + message = str(raised.value) + assert first.sha256 in message + assert hashlib.sha256(b"IRA table 5, silently re-published").hexdigest() in message + assert "size_bytes=30" in message + assert "--record-revision" in message + assert (package / "manifest.yaml").read_bytes() == recorded + assert (package / "22in05ira.xlsx").read_bytes() == ( + b"IRA table 5, first publication" + ) + + +def test_a_failed_upload_does_not_disable_revision_protection(tmp_path): + """The state #225 hit: bytes registered, upload failed, no storage.r2.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + log = tmp_path / "wrangler.log" + wrangler = _failing_wrangler(tmp_path, log) + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + + report = _fetch_local( + package, source, upload_r2=True, wrangler_command=str(wrangler) + ) + recorded = (package / "manifest.yaml").read_bytes() + + assert report.errors == ("r2_upload_failed",) + assert "storage" not in _entry(package / "manifest.yaml") + + source.write_bytes(b"IRA table 5, silently re-published") + with pytest.raises(SourceArtifactRevisionError): + _fetch_local(package, source, upload_r2=True, wrangler_command=str(wrangler)) + + assert (package / "manifest.yaml").read_bytes() == recorded + + +def test_record_revision_over_an_unpublished_entry_supersedes_nothing(tmp_path): + """There is no object to keep, so the entry gets no previous_r2 key.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5, first publication") + _fetch_local(package, source, upload_r2=False) + + source.write_bytes(b"IRA table 5, silently re-published") + report = _fetch_local(package, source, upload_r2=False, record_revision=True) + revised = _entry(package / "manifest.yaml") + + assert report.valid + assert revised["sha256"] == ( + hashlib.sha256(b"IRA table 5, silently re-published").hexdigest() + ) + assert "storage" not in revised + + +# --------------------------------------------------------------------------- +# Recorded locator cross-checks +# --------------------------------------------------------------------------- + + +def _recorded_package(tmp_path, content=b"IRA table 5, first publication"): + """A package whose entry records a published, content-addressed object.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + wrangler = _wrangler_stub(tmp_path, tmp_path / "wrangler.log") + source = _publish(tmp_path, "22in05ira.xlsx", content) + report = _fetch_local( + package, source, upload_r2=True, wrangler_command=str(wrangler) + ) + return package, source, report + + +def _rewrite_recorded_r2(package, mutate, manifest="manifest.yaml"): + manifest_path = package / manifest + payload = yaml.safe_load(manifest_path.read_text()) + mutate(payload["files"][2022]["storage"]) + manifest_path.write_text(yaml.safe_dump(payload, sort_keys=False)) + return manifest_path + + +def _other_sha256(): + return hashlib.sha256(b"some other object entirely").hexdigest() + + +@pytest.mark.parametrize( + "previous_r2", + [ + pytest.param({}, id="mapping"), + pytest.param("not a list", id="scalar"), + pytest.param(None, id="null"), + ], +) +def test_fetch_refuses_non_list_previous_r2_before_publisher_io( + tmp_path, monkeypatch, previous_r2 +): + """Malformed archived provenance must not be replaced by a new history.""" + package, source, _report = _recorded_package(tmp_path) + manifest_path = _rewrite_recorded_r2( + package, + lambda storage: storage.__setitem__("previous_r2", previous_r2), + ) + artifact_path = package / "22in05ira.xlsx" + before = { + manifest_path: manifest_path.read_bytes(), + artifact_path: artifact_path.read_bytes(), + } + source.write_bytes(b"IRA table 5, revised publication") + + def unexpected_read(_source_url): + raise AssertionError("malformed previous_r2 reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises( + MalformedManifestError, + match=r"storage[.]previous_r2 must be a list", + ): + _fetch_local(package, source, upload_r2=False, record_revision=True) + + assert {path: path.read_bytes() for path in before} == before + + +def _contradict_key(storage): + key = storage["r2"]["key"] + storage["r2"]["key"] = key.replace(key.split("/")[-2], _other_sha256()) + + +def _contradict_bucket(storage): + storage["r2"]["bucket"] = "some-other-bucket" + + +def _contradict_provider(storage): + storage["r2"]["provider"] = "s3" + + +def _mangle_uri(storage): + storage["r2"]["uri"] = "r2:/ledger-raw-missing-a-slash" + + +def _drop_the_locator(storage): + storage["r2"] = {"provider": "r2", "bucket": "ledger-raw"} + + +def _flatten_the_key(storage): + storage["r2"]["key"] = "raw/irs_soi/22in05ira.xlsx" + storage["r2"]["uri"] = f"r2://ledger-raw/{storage['r2']['key']}" + + +@pytest.mark.parametrize( + ("mutate", "expected"), + [ + pytest.param(_contradict_key, "contradicts uri", id="key-vs-uri"), + pytest.param(_contradict_bucket, "contradicts uri", id="bucket-vs-uri"), + pytest.param( + _contradict_provider, "does not identify R2", id="provider-vs-uri" + ), + pytest.param(_mangle_uri, "is not provider://bucket/key", id="uri-shape"), + pytest.param(_drop_the_locator, "records no uri", id="no-locator"), + pytest.param( + _flatten_the_key, "is not content-addressed", id="not-content-addressed" + ), + ], +) +def test_a_recorded_block_that_names_two_objects_is_refused(tmp_path, mutate, expected): + """A contradictory locator is an error, never a silently preserved block. + + The key-vs-uri case is the one that used to pass: identity was read from + the key alone, so a block whose uri named different bytes was carried + forward verbatim, and the manifest kept publishing a URI for an object it + no longer described. + """ + package, source, _ = _recorded_package(tmp_path) + manifest_path = _rewrite_recorded_r2(package, mutate) + recorded = manifest_path.read_bytes() + + # Identical bytes: the fetch would otherwise preserve the recorded block. + with pytest.raises(RecordedR2LocatorError) as raised: + _fetch_local(package, source, upload_r2=False) + + assert expected in str(raised.value) + assert manifest_path.read_bytes() == recorded + + +def test_a_malformed_storage_block_is_not_treated_as_absent(tmp_path): + package, source, _ = _recorded_package(tmp_path) + manifest_path = _rewrite_recorded_r2( + package, lambda storage: storage.update({"r2": ["r2://ledger-raw/raw/key"]}) + ) + recorded = manifest_path.read_bytes() + + with pytest.raises(MalformedManifestError, match="must be a mapping"): + _fetch_local(package, source, upload_r2=False) + + assert manifest_path.read_bytes() == recorded + + +def test_publish_raw_refuses_a_contradictory_recorded_block(tmp_path): + """Nothing is uploaded under a block that does not name one object.""" + package, _, _ = _recorded_package(tmp_path) + log = tmp_path / "publish.log" + wrangler = _wrangler_stub(tmp_path, log) + manifest_path = _rewrite_recorded_r2(package, _contradict_key) + recorded = manifest_path.read_bytes() + + report = publish_source_artifacts(package, wrangler_command=str(wrangler)) + + assert not report.valid + assert report.entries[0].upload is None + assert report.entries[0].errors[0].startswith("recorded_r2_locator_invalid:") + assert "contradicts uri" in report.entries[0].errors[0] + assert not log.exists() + assert manifest_path.read_bytes() == recorded + + +# --------------------------------------------------------------------------- +# Malformed manifests +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "document", + [ + pytest.param("- one entry\n- another\n", id="list"), + pytest.param("a bare scalar\n", id="scalar"), + pytest.param("files: [\n", id="unparseable"), + ], +) +def test_a_malformed_manifest_is_refused_before_anything_is_fetched(tmp_path, document): + """Not an absent manifest: refusing it protects what it still records. + + The publisher path does not exist, so reaching the fetch at all would raise + FileNotFoundError instead. Getting MalformedManifestError is what says the + manifest was read and refused first. + """ + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text(document) + + with pytest.raises(MalformedManifestError): + _fetch_local(package, tmp_path / "publisher" / "never-read.xlsx") + + assert manifest_path.read_text() == document + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize( + "document", + [ + "files:\n- not a mapping\n", + "files: 3\n", + "source_id: irs_soi\nfiles: text\n", + ], +) +def test_a_non_mapping_files_block_is_refused_before_anything_is_fetched( + tmp_path, document +): + """The same document inventory-artifacts and publish-raw report as + 'files must be a mapping'; a fetch must not overwrite the artifact first.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text(document) + + with pytest.raises(MalformedManifestError, match="files must be a mapping"): + _fetch_local(package, tmp_path / "publisher" / "never-read.xlsx") + + assert manifest_path.read_text() == document + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize( + "document", + [ + "", + "\n", + "{}\n", + "# only a comment\n", + "files:\n", + "source_id: irs_soi\nfiles:\n", + ], +) +def test_an_empty_manifest_still_reads_as_absent(tmp_path, document): + """Including a bare ``files:`` line, which parses as an explicit null: the + fetch records into a fresh mapping rather than failing after the write.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text(document) + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5") + + report = _fetch_local(package, source, upload_r2=False) + + assert report.valid + assert _entry(package / "manifest.yaml")["filename"] == "22in05ira.xlsx" + + +def test_a_malformed_manifest_is_reported_by_inventory_and_publish(tmp_path): + """Neither sweep may crash on, or silently skip, a document it cannot read.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text("- not a mapping\n") + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + + assert not inventory.valid + assert inventory.entries == () + assert "must be a YAML mapping" in inventory.errors[0] + assert not published.valid + assert published.entries == () + assert "must be a YAML mapping" in published.errors[0] + + +@pytest.mark.parametrize( + "duplicate_document", + [ + pytest.param( + "source_id: hidden_source\n" + "source_id: irs_soi\n" + "package_id: soi-table-5\n" + "files: {}\n", + id="source-id", + ), + pytest.param( + "source_id: irs_soi\n" + "package_id: hidden-package\n" + "package_id: soi-table-5\n" + "files: {}\n", + id="package-id", + ), + pytest.param( + "source_id: irs_soi\n" + "package_id: soi-table-5\n" + "files:\n" + " 2022:\n" + " filename: hidden.xlsx\n" + f" sha256: {hashlib.sha256(b'hidden bytes').hexdigest()}\n" + "files: {}\n", + id="files", + ), + pytest.param( + "source_id: irs_soi\n" + "package_id: soi-table-5\n" + "files:\n" + " 2022:\n" + " filename: hidden.xlsx\n" + f" sha256: {hashlib.sha256(b'hidden bytes').hexdigest()}\n" + " 2022: {}\n", + id="vintage", + ), + ], +) +def test_fetch_refuses_duplicate_manifest_keys_before_publisher_io( + tmp_path, monkeypatch, duplicate_document +): + """A lossy YAML parse must never decide which identity gets rewritten.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text(duplicate_document) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("duplicate manifest keys reached publisher I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(MalformedManifestError, match="duplicate key"): + fetch_source_artifact( + "https://example.test/table.xlsx", + source_id="irs_soi", + package_id="soi-table-5", + year=2022, + output_dir=package, + ) + + assert manifest_path.read_bytes() == before + + +# --------------------------------------------------------------------------- +# Sol gate round 3: fetch preflight and in-place manifest updates +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ( + "mismatched_field", + "declared_source_id", + "declared_package_id", + "source_id", + "package_id", + ), + [ + pytest.param( + "source_id", + "other_source", + "requested-package", + "requested_source", + "requested-package", + id="source-id", + ), + pytest.param( + "package_id", + "usda_snap", + "usda-snap-fy69-to-current", + "usda_snap", + "usda-snap-fy2025-monthly-state-caseloads", + id="package-id", + ), + ], +) +def test_fetch_refuses_a_selected_manifest_for_another_package_before_io( + tmp_path, + monkeypatch, + mismatched_field, + declared_source_id, + declared_package_id, + source_id, + package_id, +): + package = tmp_path / "db" / "data" / "usda_snap" / "fy69_to_current" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": declared_source_id, + "package_id": declared_package_id, + "files": {}, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("a mismatched manifest must be refused before I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(SourceArtifactManifestError) as raised: + fetch_source_artifact( + "https://example.test/snap-zip-fy69tocurrent-6.zip", + source_id=source_id, + package_id=package_id, + year=2025, + output_dir=package, + ) + + message = str(raised.value) + declared = { + "source_id": declared_source_id, + "package_id": declared_package_id, + }[mismatched_field] + requested = {"source_id": source_id, "package_id": package_id}[mismatched_field] + assert f"{mismatched_field}={declared!r}" in message + assert f"{mismatched_field}={requested!r}" in message + assert manifest_path.read_bytes() == before + assert list(package.iterdir()) == [manifest_path] + + +def test_fetch_uses_a_quoted_year_key_for_revision_protection(tmp_path): + package = tmp_path / "db" / "data" / "irs_soi" / "table" + package.mkdir(parents=True) + artifact_path = package / "table.xlsx" + original = b"original publisher bytes" + revised = b"silently revised publisher bytes" + artifact_path.write_bytes(original) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "irs_soi", + "package_id": "soi-table", + "files": { + "2024": { + "filename": artifact_path.name, + "source_url": "https://example.test/table.xlsx", + "sha256": hashlib.sha256(original).hexdigest(), + "size_bytes": len(original), + } + }, + }, + sort_keys=False, + ) + ) + source = _publish(tmp_path, artifact_path.name, revised) + before = manifest_path.read_bytes() + + with pytest.raises(SourceArtifactRevisionError): + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + + assert manifest_path.read_bytes() == before + assert artifact_path.read_bytes() == original + + +def test_fetch_refuses_both_spellings_of_one_year_before_io(tmp_path, monkeypatch): + package = tmp_path / "db" / "data" / "irs_soi" / "table" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "irs_soi", + "package_id": "soi-table", + "files": { + 2024: {"filename": "numeric.xlsx"}, + "2024": {"filename": "quoted.xlsx"}, + }, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("ambiguous year keys must be refused before I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(MalformedManifestError, match="both keys"): + fetch_source_artifact( + "https://example.test/table.xlsx", + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + + assert manifest_path.read_bytes() == before + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize( + "file_spec", + [ + pytest.param([], id="list"), + pytest.param("not a mapping", id="string"), + pytest.param(0, id="zero"), + pytest.param(False, id="false"), + pytest.param(None, id="null"), + ], +) +def test_fetch_refuses_a_non_mapping_year_entry_before_io( + tmp_path, monkeypatch, file_spec +): + package = tmp_path / "db" / "data" / "irs_soi" / "table" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "irs_soi", + "package_id": "soi-table", + "files": {2024: file_spec}, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("a malformed year entry must be refused before I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(MalformedManifestError, match="entry 2024.*mapping"): + fetch_source_artifact( + "https://example.test/table.xlsx", + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + + assert manifest_path.read_bytes() == before + assert list(package.iterdir()) == [manifest_path] + + +@pytest.mark.parametrize("revision", [False, True], ids=["refetch", "revision"]) +def test_fetch_carries_forward_fields_it_does_not_own(tmp_path, revision): + package = tmp_path / "db" / "data" / "irs_soi" / "table" + source = _publish(tmp_path, "table.xlsx", b"original publisher bytes") + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + manifest_path = package / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + metadata = { + "source_table": "Publisher table 7", + "notes": "Keep this review note.", + "source_urls": ["https://example.test/landing-page"], + "archive_member": "table.csv", + "year": 2024, + } + manifest["files"][2024].update(metadata) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + if revision: + source.write_bytes(b"publisher revision") + + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + record_revision=revision, + ) + + updated = yaml.safe_load(manifest_path.read_text())["files"][2024] + for field, value in metadata.items(): + assert updated.get(field) == value + + +# --------------------------------------------------------------------------- +# Sol gate round 3: whole-tree manifest discovery and files-block shape +# --------------------------------------------------------------------------- + + +def _write_sweep_manifests(root): + manifest_names = ( + "manifest.yaml", + "manifest.yml", + "manifest_named.yaml", + "manifest_named.yml", + "Manifest_Mixed.YAML", + ) + for index, manifest_name in enumerate(manifest_names): + package = root / f"package-{index}" + package.mkdir(parents=True) + content = f"publisher artifact {index}".encode() + filename = f"artifact-{index}.csv" + (package / filename).write_bytes(content) + (package / manifest_name).write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": f"package-{index}", + "files": { + 2024: { + "filename": filename, + "sha256": hashlib.sha256(content).hexdigest(), + "size_bytes": len(content), + } + }, + }, + sort_keys=False, + ) + ) + decoy = root / "decoy" / "manifest-not-a-package.yaml" + decoy.parent.mkdir() + decoy.write_text("this: is not a package manifest\n") + + +def test_inventory_default_sweep_discovers_every_package_manifest(tmp_path): + root = tmp_path / "data" + _write_sweep_manifests(root) + + report = inventory_source_artifacts(root) + + assert report.valid + assert report.counts["manifest_count"] == 5 + assert report.counts["artifact_count"] == 5 + assert {entry.manifest_path.rsplit("/", 1)[-1] for entry in report.entries} == { + "manifest.yaml", + "manifest.yml", + "manifest_named.yaml", + "manifest_named.yml", + "Manifest_Mixed.YAML", + } + + +def test_publish_default_sweep_discovers_every_package_manifest(tmp_path): + root = tmp_path / "data" + _write_sweep_manifests(root) + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + + report = publish_source_artifacts(root, wrangler_command=str(wrangler)) + + assert report.valid + assert report.counts["manifest_count"] == 5 + assert report.counts["artifact_count"] == 5 + assert report.counts["uploaded_count"] == 5 + assert len(log.read_text().splitlines()) == 5 + + +@pytest.mark.parametrize( + "files", + [ + pytest.param([], id="empty-list"), + pytest.param("", id="empty-string"), + pytest.param(0, id="zero"), + pytest.param(False, id="false"), + ], +) +def test_sweeps_reject_falsy_non_mapping_files_blocks(tmp_path, files): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": files, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package, wrangler_command=str(wrangler)) + + assert (inventory.valid, published.valid) == (False, False) + assert "files must be a mapping" in inventory.errors[0] + assert "files must be a mapping" in published.errors[0] + assert inventory.entries == () + assert published.entries == () + assert not log.exists() + assert manifest_path.read_bytes() == before + + +def test_sweeps_treat_a_null_files_block_as_absent(tmp_path): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": None, + }, + sort_keys=False, + ) + ) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + + assert inventory.valid + assert published.valid + + +# --------------------------------------------------------------------------- +# Manifest-declared artifact paths +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("path_kind", ["absolute", "parent"]) +def test_sweeps_refuse_non_bare_artifact_filenames_without_reading_them( + tmp_path, path_kind +): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + outside = tmp_path / "data" / "outside.csv" + outside.write_bytes(b"outside publisher bytes") + filename = str(outside) if path_kind == "absolute" else "../outside.csv" + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": { + 2024: { + "filename": filename, + "sha256": hashlib.sha256(outside.read_bytes()).hexdigest(), + } + }, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package, wrangler_command=str(wrangler)) + expected = f"non_canonical_filename:{filename}" + + assert not inventory.valid + assert inventory.entries[0].errors == (expected,) + assert inventory.entries[0].local_path == str(package) + assert not published.valid + assert published.entries[0].errors == (expected,) + assert published.entries[0].upload is None + assert published.entries[0].local_path == str(package) + assert not log.exists() + assert manifest_path.read_bytes() == before + + +def test_sweeps_refuse_a_symlinked_artifact_without_reading_it(tmp_path): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + outside = tmp_path / "outside.csv" + outside.write_bytes(b"outside publisher bytes") + artifact_path = package / "table.csv" + artifact_path.symlink_to(outside) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": { + 2024: { + "filename": artifact_path.name, + "sha256": hashlib.sha256(outside.read_bytes()).hexdigest(), + } + }, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package, wrangler_command=str(wrangler)) + expected = "artifact_path_is_symlink:table.csv" + + assert not inventory.valid + assert inventory.entries[0].errors == (expected,) + assert not inventory.entries[0].exists + assert not published.valid + assert published.entries[0].errors == (expected,) + assert published.entries[0].upload is None + assert not log.exists() + assert manifest_path.read_bytes() == before + assert artifact_path.is_symlink() + + +@pytest.mark.parametrize( + "bad_kind", + [ + pytest.param("parent", id="parent-path"), + pytest.param("symlink", id="symlink"), + pytest.param("manifest-name", id="manifest-name"), + pytest.param("previous-r2", id="malformed-history"), + ], +) +def test_publish_preflights_every_entry_before_any_upload( + tmp_path, monkeypatch, bad_kind +): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + first = b"first publisher table" + second = b"second publisher table" + (package / "one.csv").write_bytes(first) + outside = tmp_path / "data" / "outside.csv" + outside.write_bytes(second) + second_path = package / "two.csv" + bad_filename = "two.csv" + bad_storage = None + if bad_kind == "parent": + bad_filename = "../outside.csv" + elif bad_kind == "symlink": + second_path.symlink_to(outside) + elif bad_kind == "manifest-name": + bad_filename = "manifest.yaml" + else: + second_path.write_bytes(second) + bad_storage = {"previous_r2": {"not": "a list"}} + bad_entry = { + "filename": bad_filename, + "source_url": "https://example.test/two.csv", + "sha256": hashlib.sha256(second).hexdigest(), + "size_bytes": len(second), + } + if bad_storage is not None: + bad_entry["storage"] = bad_storage + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": { + 2023: { + "filename": "one.csv", + "source_url": "https://example.test/one.csv", + "sha256": hashlib.sha256(first).hexdigest(), + "size_bytes": len(first), + }, + 2024: bad_entry, + }, + }, + sort_keys=False, + ) + ) + before = manifest_path.read_bytes() + uploads = [] + + def non_writing_uploader(location, local_path, *, wrangler_command): + uploads.append((location, local_path, wrangler_command)) + return ArtifactCommandResult( + command=("non-writing-uploader",), + returncode=0, + stdout="", + stderr="", + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", non_writing_uploader) + + report = publish_source_artifacts(package) + + assert not report.valid + assert uploads == [] + assert manifest_path.read_bytes() == before + + +def test_publish_preflights_every_sibling_manifest_before_any_upload( + tmp_path, monkeypatch +): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + content = b"publisher table" + (package / "table.csv").write_bytes(content) + manifests = { + package / "manifest_a.yaml": { + "source_id": "publisher", + "package_id": "package-a", + "files": { + 2024: { + "filename": "table.csv", + "sha256": hashlib.sha256(content).hexdigest(), + } + }, + }, + package / "manifest_b.yaml": { + "source_id": "publisher", + "package_id": "package-b", + "files": { + 2024: { + "filename": "manifest.yaml", + "sha256": hashlib.sha256(b"not a manifest").hexdigest(), + } + }, + }, + } + for path, payload in manifests.items(): + path.write_text(yaml.safe_dump(payload, sort_keys=False)) + before = {path: path.read_bytes() for path in manifests} + uploads = [] + + def non_writing_uploader(location, local_path, *, wrangler_command): + uploads.append((location, local_path, wrangler_command)) + return ArtifactCommandResult( + command=("non-writing-uploader",), + returncode=0, + stdout="", + stderr="", + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", non_writing_uploader) + + report = publish_source_artifacts(package) + + assert not report.valid + assert uploads == [] + assert any( + "manifest_named_filename:manifest.yaml" in entry.errors + for entry in report.entries + ) + assert {path: path.read_bytes() for path in manifests} == before + + +def test_publish_preflights_entire_root_before_any_upload(tmp_path, monkeypatch): + root = tmp_path / "data" + good_package = root / "a_good" + bad_package = root / "z_bad" + good_package.mkdir(parents=True) + bad_package.mkdir(parents=True) + good_content = b"good publisher table" + bad_content = b"bad publisher table" + (good_package / "good.csv").write_bytes(good_content) + (bad_package / "bad.csv").write_bytes(bad_content) + manifests = { + good_package / "manifest.yaml": { + "source_id": "publisher", + "package_id": "good-package", + "files": { + 2024: { + "filename": "good.csv", + "sha256": hashlib.sha256(good_content).hexdigest(), + } + }, + }, + bad_package / "manifest.yaml": { + "source_id": "publisher", + "package_id": "bad-package", + "files": { + 2024: { + "filename": "../bad.csv", + "sha256": hashlib.sha256(bad_content).hexdigest(), + } + }, + }, + } + for path, payload in manifests.items(): + path.write_text(yaml.safe_dump(payload, sort_keys=False)) + before = {path: path.read_bytes() for path in manifests} + uploads = [] + + def non_writing_uploader(location, local_path, *, wrangler_command): + uploads.append((location, local_path, wrangler_command)) + return ArtifactCommandResult( + command=("non-writing-uploader",), + returncode=0, + stdout="", + stderr="", + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", non_writing_uploader) + + report = publish_source_artifacts(root) + + assert not report.valid + assert uploads == [] + assert any( + "non_canonical_filename:../bad.csv" in entry.errors for entry in report.entries + ) + assert {path: path.read_bytes() for path in manifests} == before + + +def test_sweeps_refuse_conflicting_owners_across_package_manifests( + tmp_path, monkeypatch +): + package = tmp_path / "data" / "package" + package.mkdir(parents=True) + content = b"publisher table" + filename = "table.csv" + (package / filename).write_bytes(content) + manifest_paths = ( + package / "manifest_a.yaml", + package / "manifest_b.yaml", + ) + for path, sha256 in zip( + manifest_paths, + (hashlib.sha256(content).hexdigest(), hashlib.sha256(b"other").hexdigest()), + ): + path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": path.stem, + "files": { + 2024: { + "filename": filename, + "sha256": sha256, + "size_bytes": len(content), + } + }, + }, + sort_keys=False, + ) + ) + before = {path: path.read_bytes() for path in manifest_paths} + uploads = [] + + def non_writing_uploader(location, local_path, *, wrangler_command): + uploads.append((location, local_path, wrangler_command)) + return ArtifactCommandResult( + command=("non-writing-uploader",), + returncode=0, + stdout="", + stderr="", + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", non_writing_uploader) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + + assert not inventory.valid + assert not published.valid + assert any("identify different bytes" in error for error in inventory.errors) + assert any("identify different bytes" in error for error in published.errors) + assert uploads == [] + assert {path: path.read_bytes() for path in manifest_paths} == before + + +# --------------------------------------------------------------------------- +# Sol gate round 3: canonical R2 locators before bucket-cutover skips +# --------------------------------------------------------------------------- + + +def test_publish_preserves_an_explicit_historical_route_during_bucket_cutover( + tmp_path, monkeypatch +): + package = tmp_path / "db" / "data" / "irs_soi" / "table" + source = _publish(tmp_path, "table.xlsx", b"publisher table") + fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table", + year=2024, + output_dir=package, + ) + manifest_path = package / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + spec = manifest["files"][2024] + wrong_key = f"raw/irs_soi/other-package/2023/{spec['sha256']}/{spec['filename']}" + spec["storage"] = { + "r2": { + "provider": "r2", + "bucket": "ledger-raw", + "key": wrong_key, + "uri": f"r2://ledger-raw/{wrong_key}", + } + } + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_bytes() + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + log = tmp_path / "wrangler.log" + wrangler = _wrangler_stub(tmp_path, log) + + report = publish_source_artifacts(package, wrangler_command=str(wrangler)) + + assert report.valid + assert report.entries[0].upload is None + assert report.entries[0].errors == () + assert report.entries[0].skipped == ( + "recorded_r2_bucket_is_preserved_history:" + "recorded=ledger-raw:requested=chronicle-raw" + ) + assert report.entries[0].r2_location.key == wrong_key + assert not log.exists() + assert manifest_path.read_bytes() == before + + +def _make_recorded_locator_use_s3(package): + manifest_path = package / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + r2 = manifest["files"][2022]["storage"]["r2"] + r2["provider"] = "s3" + r2["uri"] = f"s3://{r2['bucket']}/{r2['key']}" + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + return manifest_path + + +def test_fetch_refuses_a_self_consistent_non_r2_locator_before_io( + tmp_path, monkeypatch +): + package, source, _report = _recorded_package(tmp_path) + manifest_path = _make_recorded_locator_use_s3(package) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("a non-R2 storage.r2 locator must be refused before I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(RecordedR2LocatorError, match="provider.*r2"): + _fetch_local(package, source, upload_r2=False) + + assert manifest_path.read_bytes() == before + + +@pytest.mark.parametrize("missing_field", ["provider", "uri"]) +def test_fetch_refuses_an_incomplete_r2_locator_before_io( + tmp_path, monkeypatch, missing_field +): + package, source, _report = _recorded_package(tmp_path) + manifest_path = package / "manifest.yaml" + manifest = yaml.safe_load(manifest_path.read_text()) + manifest["files"][2022]["storage"]["r2"].pop(missing_field) + manifest_path.write_text(yaml.safe_dump(manifest, sort_keys=False)) + before = manifest_path.read_bytes() + + def unexpected_read(_source_url): + raise AssertionError("an incomplete storage.r2 locator reached I/O") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + + with pytest.raises(RecordedR2LocatorError, match=missing_field): + _fetch_local(package, source, upload_r2=False) + + assert manifest_path.read_bytes() == before + + +def test_publish_refuses_a_self_consistent_non_r2_locator(tmp_path, monkeypatch): + package, _source, _report = _recorded_package(tmp_path) + manifest_path = _make_recorded_locator_use_s3(package) + before = manifest_path.read_bytes() + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + log = tmp_path / "publish.log" + wrangler = _wrangler_stub(tmp_path, log) + + report = publish_source_artifacts(package, wrangler_command=str(wrangler)) + + assert not report.valid + assert report.entries[0].upload is None + assert report.entries[0].skipped is None + assert report.entries[0].errors[0].startswith("recorded_r2_locator_invalid:") + assert "provider" in report.entries[0].errors[0] + assert not log.exists() + assert manifest_path.read_bytes() == before + + +# --------------------------------------------------------------------------- +# Sol gate round 3: identity segments, alias enumeration, non-regular manifests +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "bad_id", + ["irs soi", "a/b", "..", " irs_soi", "irs_soi ", "a\\b", "a\tb"], +) +@pytest.mark.parametrize("field", ["source_id", "package_id"]) +def test_fetch_refuses_noncanonical_identity_segments_before_io( + tmp_path, monkeypatch, bad_id, field +): + """A registration identity that _clean_key_part would rewrite (or that + embeds separators) must be refused, never normalized into a different + R2 namespace.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "table.xlsx", b"table") + + def unexpected_read(_url): + raise AssertionError("publisher read reached with a bad identity") + + monkeypatch.setattr("chronicle.artifacts._read_artifact", unexpected_read) + kwargs = {"source_id": "irs_soi", "package_id": "soi-table-5"} + kwargs[field] = bad_id + + with pytest.raises(SourceArtifactManifestError, match="segment"): + fetch_source_artifact( + str(source), + year=2022, + output_dir=package, + **kwargs, + ) + + assert not package.exists() + + +def test_matching_directory_entry_refuses_multiple_normalized_aliases(): + """Two physical entries sharing one normalized key are a package defect; + returning the first spelling would silently ignore the other bytes.""" + from types import SimpleNamespace + + from chronicle.registration import matching_directory_entry + + entries = [ + SimpleNamespace(name="TABLE.CSV"), + SimpleNamespace(name="other.csv"), + SimpleNamespace(name="table.csv"), + ] + directory = SimpleNamespace(is_dir=lambda: True, iterdir=lambda: iter(entries)) + + with pytest.raises(ValueError, match="TABLE.CSV.*table.csv|table.csv.*TABLE.CSV"): + matching_directory_entry(directory, "table.csv") + + assert matching_directory_entry(directory, "other.csv").name == "other.csv" + + +def test_publish_and_inventory_report_duplicate_artifact_aliases(tmp_path, monkeypatch): + """A duplicate-alias defect surfaces as an entry error, not a crash and + not a silent first-match read.""" + output_dir = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "22in05ira.xlsx", b"IRA table 5") + _fetch_local(output_dir, source, upload_r2=False) + + def duplicate_alias(_directory, filename): + raise ValueError(f"{filename!r} matches two physical spellings in the package.") + + monkeypatch.setattr("chronicle.artifacts.matching_directory_entry", duplicate_alias) + + inventory = inventory_source_artifacts(output_dir) + published = publish_source_artifacts(output_dir) + + assert not inventory.valid + assert any( + "duplicate_artifact_spellings" in error + for entry in inventory.entries + for error in entry.errors + ) + assert not published.valid + assert any( + "duplicate_artifact_spellings" in error + for entry in published.entries + for error in entry.errors + ) + + +@pytest.mark.parametrize("shape", ["dangling", "directory"]) +def test_sweeps_refuse_non_regular_manifest_entries(tmp_path, shape): + """A manifest-named entry that is not a regular file must fail the sweep + loudly instead of vanishing from it.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + target = package / "manifest.yaml" + if shape == "dangling": + target.symlink_to(package / "nowhere.yaml") + else: + target.mkdir() + + with pytest.raises(SourceArtifactManifestError, match="regular file"): + inventory_source_artifacts(tmp_path / "db" / "data") + with pytest.raises(SourceArtifactManifestError, match="regular file"): + publish_source_artifacts(tmp_path / "db" / "data") + + +def test_fetch_refuses_a_dangling_manifest_symlink_instead_of_creating_one( + tmp_path, +): + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + package.mkdir(parents=True) + (package / "manifest.yaml").symlink_to(package / "nowhere.yaml") + source = _publish(tmp_path, "table.xlsx", b"table") + + with pytest.raises(SourceArtifactManifestError, match="regular file"): + _fetch_local(package, source, upload_r2=False) + + assert (package / "manifest.yaml").is_symlink() + assert not (package / "table.xlsx").exists() + + +def test_record_revision_updates_every_owner_even_when_yaml_aliases_share_one_entry( + tmp_path, +): + """Two vintages sharing one physical file are two owners even when the + manifest spelled them with a YAML anchor and alias (one dict object); + identifying the selected owner by object identity skipped both.""" + package = tmp_path / "db" / "data" / "irs_soi" / "soi-table-5" + source = _publish(tmp_path, "archive.zip", b"first publication") + _fetch_local(package, source, upload_r2=False) + manifest_path = package / "manifest.yaml" + entry = _entry(manifest_path) + lines = [ + "source_id: irs_soi", + "package_id: soi-table-5", + "files:", + " 2022: &shared", + ] + for field, value in entry.items(): + lines.append(f" {field}: {json.dumps(value)}") + lines.append(" 2023: *shared") + manifest_path.write_text("\n".join(lines) + "\n") + old_sha = entry["sha256"] + + source.write_bytes(b"second publication, revised rows") + report = fetch_source_artifact( + str(source), + source_id="irs_soi", + package_id="soi-table-5", + year=2023, + output_dir=package, + upload_r2=False, + record_revision=True, + ) + + assert report.valid + manifest = yaml.safe_load(manifest_path.read_text()) + new_sha = hashlib.sha256(b"second publication, revised rows").hexdigest() + assert manifest["files"][2023]["sha256"] == new_sha + assert manifest["files"][2022]["sha256"] == new_sha, ( + "the aliased owner kept the superseded checksum" + ) + assert old_sha != new_sha + + +def test_identical_byte_refetch_keeps_r2_identified_shared_file_valid(tmp_path): + """Two manifests may identify one package-local file through identical + content-addressed R2 locators without declaring ``sha256``. Refetching + the same bytes records ``sha256`` on the selected manifest only; the + sibling's effective identity (its recorded R2 key) still agrees, so the + package directory must stay valid for every sweep.""" + from chronicle.artifacts import ( + _effective_recorded_digest, + default_r2_raw_bucket, + ) + from chronicle.registration import validate_package_directory + + def collisions(manifests): + # The sweeps resolve each entry's effective identity (its recorded + # content-addressed R2 key when no ``sha256`` is declared) exactly + # like this before comparing owners. + return validate_package_directory( + manifests, entry_digest=_effective_recorded_digest + ) + + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + content = b"shared publisher table" + filename = "shared.csv" + sha256 = hashlib.sha256(content).hexdigest() + (package / filename).write_bytes(content) + source = tmp_path / "publisher-download.csv" + source.write_bytes(content) + bucket = default_r2_raw_bucket() + manifest_paths = (package / "manifest_a.yaml", package / "manifest_b.yaml") + for path in manifest_paths: + key = build_r2_key( + source_id="publisher", + package_id=path.stem, + year=2024, + sha256=sha256, + filename=filename, + ) + path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": path.stem, + "files": { + 2024: { + "filename": filename, + "source_url": str(source), + "storage": { + "r2": { + "provider": "r2", + "bucket": bucket, + "key": key, + "uri": f"r2://{bucket}/{key}", + } + }, + } + }, + }, + sort_keys=False, + ) + ) + manifests = {str(path): yaml.safe_load(path.read_text()) for path in manifest_paths} + assert collisions(manifests) == () + assert inventory_source_artifacts(package).valid + + report = fetch_source_artifact( + str(source), + source_id="publisher", + package_id="manifest_a", + year=2024, + output_dir=package, + filename=filename, + manifest_filename="manifest_a.yaml", + ) + assert report.valid + assert report.sha256 == sha256 + + manifests = {str(path): yaml.safe_load(path.read_text()) for path in manifest_paths} + assert manifests[str(manifest_paths[0])]["files"][2024]["sha256"] == sha256 + assert "sha256" not in manifests[str(manifest_paths[1])]["files"][2024] + assert collisions(manifests) == () + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + for sweep in (inventory, published): + assert not any("filename_collision" in error for error in sweep.errors) + assert not any("identify different bytes" in error for error in sweep.errors) + assert sweep.valid + + +def _write_unidentified_shared_manifests(package, source, *, filename="shared.csv"): + """Two manifests naming one package-local file with no identity yet.""" + manifest_paths = (package / "manifest_a.yaml", package / "manifest_b.yaml") + for path in manifest_paths: + path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": path.stem, + "files": { + 2024: { + "filename": filename, + "source_url": str(source), + } + }, + }, + sort_keys=False, + ) + ) + return manifest_paths + + +def _fake_wrangler(tmp_path): + log = tmp_path / "wrangler.log" + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + return wrangler, log + + +def test_selected_publication_of_a_shared_unidentified_file_keeps_siblings_valid( + tmp_path, +): + """Two manifests may name one package-local file before either records an + identity. Publishing only one of them records its checksum and locator; + the sibling, which still records nothing, must not become a false + collision, and its own later publication must record the same identity.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + content = b"shared publisher table" + (package / "shared.csv").write_bytes(content) + source = tmp_path / "publisher-download.csv" + source.write_bytes(content) + manifest_a, manifest_b = _write_unidentified_shared_manifests(package, source) + wrangler, log = _fake_wrangler(tmp_path) + expected_sha256 = hashlib.sha256(content).hexdigest() + + assert inventory_source_artifacts(package).valid + + selected = publish_source_artifacts( + package, manifest_filename="manifest_a.yaml", wrangler_command=str(wrangler) + ) + assert selected.valid + assert selected.counts["uploaded_count"] == 1 + assert yaml.safe_load(manifest_a.read_text())["files"][2024]["sha256"] == ( + expected_sha256 + ) + assert "sha256" not in yaml.safe_load(manifest_b.read_text())["files"][2024] + + inventory = inventory_source_artifacts(package) + assert inventory.valid, inventory.errors + assert not any("filename_collision" in error for error in inventory.errors) + + sibling = publish_source_artifacts( + package, manifest_filename="manifest_b.yaml", wrangler_command=str(wrangler) + ) + assert sibling.valid, sibling.errors + assert sibling.counts["uploaded_count"] == 1 + recorded_b = yaml.safe_load(manifest_b.read_text())["files"][2024] + recorded_a = yaml.safe_load(manifest_a.read_text())["files"][2024] + assert recorded_b["sha256"] == expected_sha256 + assert recorded_b["storage"]["r2"]["key"].endswith(f"/{expected_sha256}/shared.csv") + assert recorded_a["storage"]["r2"]["key"].endswith(f"/{expected_sha256}/shared.csv") + assert len(log.read_text().splitlines()) == 2 + + everything = publish_source_artifacts(package, wrangler_command=str(wrangler)) + assert everything.valid, everything.errors + assert everything.counts["uploaded_count"] == 0 + assert everything.counts["skipped_count"] == 2 + + +def test_partial_upload_failure_of_a_shared_file_stays_retryable(tmp_path, monkeypatch): + """A full sweep whose second upload fails records the first manifest's + identity only. The retry must not report a collision for the sibling that + still records nothing, and must finish recording the same identity.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + content = b"shared publisher table" + (package / "shared.csv").write_bytes(content) + source = tmp_path / "publisher-download.csv" + source.write_bytes(content) + manifest_a, manifest_b = _write_unidentified_shared_manifests(package, source) + expected_sha256 = hashlib.sha256(content).hexdigest() + uploads = [] + failures = {"remaining": 1} + + def uploader(location, local_path, *, wrangler_command): + uploads.append(location) + if len(uploads) == 2 and failures["remaining"]: + failures["remaining"] -= 1 + return ArtifactCommandResult( + command=("failing-uploader",), returncode=1, stdout="", stderr="boom" + ) + return ArtifactCommandResult( + command=("uploader",), returncode=0, stdout="", stderr="" + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", uploader) + + first = publish_source_artifacts(package) + assert not first.valid + assert first.counts["uploaded_count"] == 1 + assert first.counts["failed_count"] == 1 + recorded = { + path.name: yaml.safe_load(path.read_text())["files"][2024] + for path in (manifest_a, manifest_b) + } + identified = [name for name, entry in recorded.items() if "sha256" in entry] + assert len(identified) == 1 + + inventory = inventory_source_artifacts(package) + assert inventory.valid, inventory.errors + assert not any("filename_collision" in error for error in inventory.errors) + + retry = publish_source_artifacts(package) + assert retry.valid, retry.errors + assert retry.counts["uploaded_count"] == 1 + assert retry.counts["skipped_count"] == 1 + assert retry.counts["failed_count"] == 0 + for path in (manifest_a, manifest_b): + entry = yaml.safe_load(path.read_text())["files"][2024] + assert entry["sha256"] == expected_sha256 + assert entry["storage"]["r2"]["key"].endswith(f"/{expected_sha256}/shared.csv") + assert len(uploads) == 3 + + +def test_sweeps_refuse_identifying_a_shared_file_whose_bytes_changed( + tmp_path, monkeypatch +): + """An unidentified sibling is not a collision, but identifying it would + hash the package-local bytes. When those bytes no longer match what the + identified owner records, every sweep refuses before any upload.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + old_content = b"shared publisher table" + new_content = b"shared publisher table, revised" + (package / "shared.csv").write_bytes(new_content) + source = tmp_path / "publisher-download.csv" + source.write_bytes(new_content) + manifest_a, manifest_b = _write_unidentified_shared_manifests(package, source) + identified = yaml.safe_load(manifest_a.read_text()) + identified["files"][2024]["sha256"] = hashlib.sha256(old_content).hexdigest() + identified["files"][2024]["size_bytes"] = len(old_content) + manifest_a.write_text(yaml.safe_dump(identified, sort_keys=False)) + uploads = [] + + def unexpected_uploader(location, local_path, *, wrangler_command): + uploads.append(location) + return ArtifactCommandResult( + command=("uploader",), returncode=0, stdout="", stderr="" + ) + + monkeypatch.setattr("chronicle.artifacts._upload_r2_object", unexpected_uploader) + + inventory = inventory_source_artifacts(package) + published = publish_source_artifacts(package) + sibling_only = publish_source_artifacts( + package, manifest_filename="manifest_b.yaml" + ) + + for sweep in (inventory, published, sibling_only): + assert not sweep.valid + assert any("two identities" in error for error in sweep.errors), sweep.errors + assert uploads == [] + assert "sha256" not in yaml.safe_load(manifest_b.read_text())["files"][2024] + + +@pytest.mark.parametrize("record_revision", [False, True]) +def test_first_fetch_initializes_a_predeclared_entry_without_identity( + tmp_path, record_revision +): + """A manifest may predeclare an entry with only ``filename`` and + ``source_url``. Its first fetch identifies it: the entry is the one being + initialized, not an unidentifiable owner, so both the fetch preflight + and the manifest writer must accept it (default and --record-revision).""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + source = tmp_path / "publisher-download.csv" + content = b"first publisher bytes" + source.write_bytes(content) + manifest_path = package / "manifest.yaml" + manifest_path.write_text( + yaml.safe_dump( + { + "source_id": "publisher", + "package_id": "package", + "files": {2024: {"filename": "table.csv", "source_url": str(source)}}, + }, + sort_keys=False, + ) + ) + assert not (package / "table.csv").exists() + + report = fetch_source_artifact( + str(source), + source_id="publisher", + package_id="package", + year=2024, + output_dir=package, + filename="table.csv", + record_revision=record_revision, + ) + + assert report.valid, report.errors + expected_sha256 = hashlib.sha256(content).hexdigest() + assert report.sha256 == expected_sha256 + assert (package / "table.csv").read_bytes() == content + entry = yaml.safe_load(manifest_path.read_text())["files"][2024] + assert entry["filename"] == "table.csv" + assert entry["sha256"] == expected_sha256 + assert entry["size_bytes"] == len(content) + assert inventory_source_artifacts(package).valid + + again = fetch_source_artifact( + str(source), + source_id="publisher", + package_id="package", + year=2024, + output_dir=package, + filename="table.csv", + ) + assert again.valid, again.errors + assert again.sha256 == expected_sha256 + + +def test_fetch_still_refuses_an_unidentified_owner_in_another_manifest(tmp_path): + """Only the entry being initialized is exempt: another manifest naming the + same package-local file without an identity keeps refusing the fetch + before any byte is written, because the fetched bytes would silently + define what that sibling means.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + source = tmp_path / "publisher-download.csv" + source.write_bytes(b"first publisher bytes") + manifest_a, manifest_b = _write_unidentified_shared_manifests( + package, source, filename="table.csv" + ) + before = {path: path.read_text() for path in (manifest_a, manifest_b)} + + with pytest.raises(MalformedManifestError, match="records no sha256 identity"): + fetch_source_artifact( + str(source), + source_id="publisher", + package_id="manifest_a", + year=2024, + output_dir=package, + filename="table.csv", + manifest_filename="manifest_a.yaml", + ) + + assert not (package / "table.csv").exists() + assert {path: path.read_text() for path in (manifest_a, manifest_b)} == before + + +@pytest.mark.parametrize("missing", ["source_id", "package_id"]) +def test_selected_publication_overrides_apply_only_to_the_selected_manifest( + tmp_path, missing +): + """``--source-id`` / ``--package-id`` complete the selected manifest's + identity. An unselected sibling that declares a different identifier is + preflighted with its own identifiers, not the override, so the selected + publication succeeds and the sibling is left untouched.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + content_a = b"table a" + content_b = b"table b" + (package / "table_a.csv").write_bytes(content_a) + (package / "table_b.csv").write_bytes(content_b) + identity_a = {"source_id": "publisher_a", "package_id": "package_a"} + identity_b = {"source_id": "publisher_b", "package_id": "package_b"} + override = {missing: identity_a[missing]} + declared_a = {key: value for key, value in identity_a.items() if key != missing} + manifest_a = package / "manifest_a.yaml" + manifest_b = package / "manifest_b.yaml" + manifest_a.write_text( + yaml.safe_dump( + { + **declared_a, + "files": { + 2024: { + "filename": "table_a.csv", + "source_url": "https://publisher.test/a", + "sha256": hashlib.sha256(content_a).hexdigest(), + "size_bytes": len(content_a), + } + }, + }, + sort_keys=False, + ) + ) + manifest_b.write_text( + yaml.safe_dump( + { + **identity_b, + "files": { + 2024: { + "filename": "table_b.csv", + "source_url": "https://publisher.test/b", + "sha256": hashlib.sha256(content_b).hexdigest(), + "size_bytes": len(content_b), + } + }, + }, + sort_keys=False, + ) + ) + before_b = manifest_b.read_text() + wrangler, log = _fake_wrangler(tmp_path) + + report = publish_source_artifacts( + package, + manifest_filename="manifest_a.yaml", + wrangler_command=str(wrangler), + **override, + ) + + assert report.valid, report.errors + assert report.counts["uploaded_count"] == 1 + assert report.counts["failed_count"] == 0 + recorded = yaml.safe_load(manifest_a.read_text()) + assert recorded[missing] == identity_a[missing] + assert recorded["files"][2024]["storage"]["r2"]["key"].startswith( + "raw/publisher_a/package_a/2024/" + ) + assert manifest_b.read_text() == before_b + assert len(log.read_text().splitlines()) == 1 + + +@pytest.mark.parametrize("missing", ["source_id", "package_id"]) +def test_default_sweep_overrides_apply_to_every_selected_sibling(tmp_path, missing): + """A default sweep selects every manifest in the directory. The override + must complete the manifest that lacks the identifier and confirm the + sibling that declares the same value, whichever order they are processed + in -- a selected sibling met through another selected manifest's package + preflight is not an unselected one.""" + package = tmp_path / "data" / "publisher" / "package" + package.mkdir(parents=True) + content_a = b"table a" + content_b = b"table b" + (package / "table_a.csv").write_bytes(content_a) + (package / "table_b.csv").write_bytes(content_b) + identity = {"source_id": "publisher", "package_id": "package"} + override = {missing: identity[missing]} + declared_a = {key: value for key, value in identity.items() if key != missing} + manifest_a = package / "manifest_a.yaml" + manifest_b = package / "manifest_b.yaml" + manifest_a.write_text( + yaml.safe_dump( + { + **declared_a, + "files": { + 2024: { + "filename": "table_a.csv", + "source_url": "https://publisher.test/a", + "sha256": hashlib.sha256(content_a).hexdigest(), + "size_bytes": len(content_a), + } + }, + }, + sort_keys=False, + ) + ) + manifest_b.write_text( + yaml.safe_dump( + { + **identity, + "files": { + 2024: { + "filename": "table_b.csv", + "source_url": "https://publisher.test/b", + "sha256": hashlib.sha256(content_b).hexdigest(), + "size_bytes": len(content_b), + } + }, + }, + sort_keys=False, + ) + ) + wrangler, log = _fake_wrangler(tmp_path) + + report = publish_source_artifacts( + package, wrangler_command=str(wrangler), **override + ) + + assert report.valid, report.errors + assert report.counts["uploaded_count"] == 2 + assert report.counts["failed_count"] == 0 + assert not any("r2_identity_invalid" in error for error in report.errors) + for path in (manifest_a, manifest_b): + recorded = yaml.safe_load(path.read_text()) + assert recorded[missing] == identity[missing] + assert recorded["files"][2024]["storage"]["r2"]["key"].startswith( + "raw/publisher/package/2024/" + ) + assert len(log.read_text().splitlines()) == 2 diff --git a/tests/test_chronicle_consumer_contract.py b/tests/test_chronicle_consumer_contract.py index cda97b66..30dd0056 100644 --- a/tests/test_chronicle_consumer_contract.py +++ b/tests/test_chronicle_consumer_contract.py @@ -11,6 +11,7 @@ import pytest +import chronicle.artifacts as artifacts import chronicle.consumer_contract as consumer_contract from chronicle.consumer_contract import ( CONSUMER_FACT_SCHEMA_VERSION, @@ -886,6 +887,41 @@ def test_export_consumer_facts_cli_rejects_contract_invalid_facts(tmp_path, caps }, "irs_soi.ty2024.table.us.taxable_interest_amount.ledger_derived", ), + # The guard used to match two hardcoded URI prefixes, so a URI naming + # any derived bucket other than `ledger-derived` did not match. Once the + # buckets are renamed (PolicyEngine/chronicle#143, mechanism 3) that is + # every derived URI, so the guard has to match on shape. + ( + { + "source_name": "irs_soi", + "source_file": "publisher.xlsx", + "raw_r2_bucket": None, + "raw_r2_key": None, + "raw_r2_uri": "r2://chronicle-derived/derived/source/fact.json", + }, + "publisher.raw.fact", + ), + ( + { + "source_name": "irs_soi", + "source_file": "chronicle-derived:taxable_interest.json", + "raw_r2_bucket": "ledger-raw", + "raw_r2_uri": "r2://ledger-raw/raw/source/publisher.xlsx", + }, + "publisher.raw.fact", + ), + # The derived-row marker renames with everything else, so the guard has + # to reject the chronicle spelling the same way it rejects the ledger + # one (PolicyEngine/chronicle#143, mechanism 3). + ( + { + "source_name": "irs_soi", + "source_file": "publisher.xlsx", + "raw_r2_bucket": "ledger-raw", + "raw_r2_uri": "r2://ledger-raw/raw/source/publisher.xlsx", + }, + "irs_soi.ty2024.table.us.taxable_interest_amount.chronicle_derived", + ), ], ) def test_consumer_contract_rejects_downstream_derived_target_facts( @@ -905,6 +941,157 @@ def test_consumer_contract_rejects_downstream_derived_target_facts( assert "derived_fact_provenance" in {error.code for error in report.errors} +@pytest.mark.parametrize( + "bucket_env", + [ + "CHRONICLE_R2_DERIVED_BUCKET", + "POLICYENGINE_LEDGER_R2_DERIVED_BUCKET", + "LEDGER_R2_DERIVED_BUCKET", + ], +) +@pytest.mark.parametrize( + ("field", "value"), + [ + ("raw_r2_bucket", "chronicle-builds"), + ("raw_r2_uri", "r2://chronicle-builds/builds/source/fact.json"), + ("source_file", "chronicle-builds:builds/source/fact.json"), + ("source_file", "r2://chronicle-builds/builds/source/fact.json"), + ("url", "r2://chronicle-builds/builds/source/fact.json"), + ], +) +def test_consumer_contract_rejects_configured_derived_bucket( + monkeypatch, tmp_path, bucket_env, field, value +): + monkeypatch.setenv(bucket_env, "chronicle-builds") + fact = _soi_agi_fact() + derived = replace(fact, source=replace(fact.source, **{field: value})) + + report = validate_consumer_fact_contract([derived]) + + assert "derived_fact_provenance" in {error.code for error in report.errors} + output = tmp_path / "new-directory" / "consumer_facts.jsonl" + with pytest.raises(ValueError, match="consumer-contract"): + write_consumer_facts_jsonl([derived], output) + assert not output.parent.exists() + + +@pytest.mark.parametrize( + "prefix_env", + [ + None, + "CHRONICLE_R2_DERIVED_PREFIX", + "POLICYENGINE_LEDGER_R2_DERIVED_PREFIX", + "LEDGER_R2_DERIVED_PREFIX", + ], +) +@pytest.mark.parametrize( + ("field", "value"), + [ + ("raw_r2_key", "builds/source/fact.json"), + ("raw_r2_uri", "r2://publisher-archive/builds/source/fact.json"), + ("source_file", "publisher-archive:builds/source/fact.json"), + ("source_file", "r2://publisher-archive/builds/source/fact.json"), + ("url", "r2://publisher-archive/builds/source/fact.json"), + ], +) +def test_consumer_contract_rejects_configured_derived_prefix( + monkeypatch, prefix_env, field, value +): + if prefix_env is None: + monkeypatch.setattr(artifacts, "DEFAULT_R2_DERIVED_PREFIX", "builds") + else: + monkeypatch.setenv(prefix_env, "builds") + fact = _soi_agi_fact() + derived = replace(fact, source=replace(fact.source, **{field: value})) + + report = validate_consumer_fact_contract([derived]) + + assert "derived_fact_provenance" in {error.code for error in report.errors} + + +def test_consumer_contract_derived_routes_match_complete_names(monkeypatch): + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-builds") + monkeypatch.setattr(artifacts, "DEFAULT_R2_DERIVED_PREFIX", "builds") + fact = _soi_agi_fact() + publisher = replace( + fact, + source=replace( + fact.source, + source_file="chronicle-builds-raw:buildstats/source/publisher.csv", + raw_r2_bucket="chronicle-builds-raw", + raw_r2_key="buildstats/source/publisher.csv", + raw_r2_uri="r2://chronicle-builds-raw/buildstats/source/publisher.csv", + ), + ) + + assert validate_consumer_fact_contract([publisher]).valid + + +@pytest.mark.parametrize("field", ["raw_r2_uri", "source_file", "url"]) +@pytest.mark.parametrize("scheme", ["R2", "r2"]) +def test_consumer_contract_rejects_derived_uri_scheme_case( + monkeypatch, tmp_path, field, scheme +): + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-builds") + monkeypatch.setenv("CHRONICLE_R2_DERIVED_PREFIX", "builds") + fact = _soi_agi_fact() + derived = replace( + fact, + source=replace( + fact.source, + **{field: f"{scheme}://chronicle-builds/builds/source/fact.json"}, + ), + ) + + report = validate_consumer_fact_contract([derived]) + + assert "derived_fact_provenance" in {error.code for error in report.errors} + output = tmp_path / "new-directory" / "consumer_facts.jsonl" + with pytest.raises(ValueError, match="consumer-contract"): + write_consumer_facts_jsonl([derived], output) + assert not output.parent.exists() + + +def test_derived_record_marker_is_rejected_in_either_spelling(): + """Both rename-window spellings produce the identical boundary error.""" + fact = _soi_agi_fact() + base = "irs_soi.ty2024.table.us.taxable_interest_amount" + + reports = { + suffix: validate_consumer_fact_contract( + [replace(fact, source_record_id=f"{base}.{suffix}")] + ) + for suffix in ("ledger_derived", "chronicle_derived") + } + + ledger_errors = [ + (error.code, error.message) for error in reports["ledger_derived"].errors + ] + chronicle_errors = [ + (error.code, error.message) for error in reports["chronicle_derived"].errors + ] + assert ledger_errors == chronicle_errors + assert "derived_fact_provenance" in {code for code, _ in ledger_errors} + + +@pytest.mark.parametrize( + "source_record_id", + [ + # A publisher-backed row that merely contains the marker as a word, or + # carries it without the separating dot, is not a derived target row. + "irs_soi.ty2024.table.us.chronicle_derived_totals", + "irs_soi.ty2024.table.us.ledger_derived_totals", + "chronicle_derived", + ], +) +def test_derived_record_marker_matches_the_whole_final_segment(source_record_id): + fact = replace(_soi_agi_fact(), source_record_id=source_record_id) + + report = validate_consumer_fact_contract([fact]) + + assert report.valid + + def test_export_consumer_facts_cli_preserves_decimal_values(tmp_path, capsys): input_path = tmp_path / "facts.jsonl" output_path = tmp_path / "consumer_facts.jsonl" @@ -940,3 +1127,32 @@ def test_contract_reports_malformed_lineage_keys_instead_of_raising(tmp_path): ValueError, match="Cannot export invalid Chronicle consumer-contract facts" ): write_consumer_facts_jsonl([fact], tmp_path / "facts.jsonl") + + +@pytest.mark.parametrize( + ("bucket", "key"), + [ + ("publisher-derived", "exports/facts.jsonl"), + ("PUBLISHER-Derived", "exports/facts.jsonl"), + ("some-archive", "derived/exports/facts.jsonl"), + ], +) +def test_consumer_contract_keeps_rejecting_legacy_derived_routes(bucket, key): + """Configured routes extend the derived boundary; they never narrow it. + A bucket ending in ``-derived`` or a ``derived/`` key was rejected before + routes became configurable and must still be, under default config.""" + fact = _soi_agi_fact() + derived = replace( + fact, + source=replace( + fact.source, + source_file=f"{bucket}:{key}", + raw_r2_bucket=bucket, + raw_r2_key=key, + raw_r2_uri=f"r2://{bucket}/{key}", + ), + ) + + report = validate_consumer_fact_contract([derived]) + + assert "derived_fact_provenance" in {error.code for error in report.errors} diff --git a/tests/test_chronicle_env.py b/tests/test_chronicle_env.py new file mode 100644 index 00000000..2e1f065e --- /dev/null +++ b/tests/test_chronicle_env.py @@ -0,0 +1,402 @@ +"""Tests for the chronicle-first environment read window. + +Chronicle's operational stores migrate by dual-run (PolicyEngine/chronicle#143, +mechanism 3): ``CHRONICLE_*`` names win, ledger-era names keep working behind a +deprecation warning. Every test here is hermetic — the suite-wide +``isolated_rename_window_env`` fixture in ``tests/conftest.py`` strips every +variable in the rename window from the ambient environment first. +""" + +from __future__ import annotations + +import importlib +import json +import os +from pathlib import Path +import subprocess +import sys + +import pytest + +from chronicle.artifacts import ( + DEFAULT_R2_DERIVED_BUCKET, + DEFAULT_R2_RAW_BUCKET, + default_r2_derived_bucket, + default_r2_raw_bucket, +) +from chronicle.env import ( + CHRONICLE_ENV_PREFIX, + ChronicleEnvDeprecationWarning, + DEFAULT_CHRONICLE_SCHEMA, + LEGACY_ENV_PREFIXES, + default_chronicle_schema, + env_flag, + env_names, + env_value, +) +from chronicle.harness import main as harness_main +from chronicle.source_package import ( + SOURCE_ARTIFACT_CACHE_ENV, + SOURCE_ARTIFACT_FETCH_ENV, +) + + +def _fake_wrangler(tmp_path, log): + wrangler = tmp_path / "wrangler" + wrangler.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$*\" >> {log}\necho ok\n") + wrangler.chmod(0o755) + return wrangler + + +# --------------------------------------------------------------------------- +# Lookup order +# --------------------------------------------------------------------------- + + +def test_every_test_runs_with_the_rename_window_cleared(): + """Isolation is suite-wide (tests/conftest.py), not module-scoped. + + Modules well outside this one assert the defaults these variables override + — the raw and derived bucket names, the Supabase schema — so an operator's + shell must not reach any test. + """ + leaked = sorted( + name + for name in os.environ + if name.startswith((CHRONICLE_ENV_PREFIX, *LEGACY_ENV_PREFIXES)) + ) + + assert leaked == [] + + +def test_env_names_puts_chronicle_first_then_ledger_era_names(): + assert env_names("CHRONICLE_SOURCE_ARTIFACT_FETCH") == ( + "CHRONICLE_SOURCE_ARTIFACT_FETCH", + "POLICYENGINE_LEDGER_SOURCE_ARTIFACT_FETCH", + "LEDGER_SOURCE_ARTIFACT_FETCH", + ) + + +def test_env_names_expands_a_ledger_era_name_to_the_same_ladder(): + assert env_names("LEDGER_PE_US_DATA_ROOT") == env_names("CHRONICLE_PE_US_DATA_ROOT") + + +def test_env_names_leaves_variables_outside_the_rename_window_alone(): + assert env_names("POLICYENGINE_SUPABASE_URL") == ("POLICYENGINE_SUPABASE_URL",) + assert env_names("POLICYENGINE_TARGETS_SCHEMA") == ("POLICYENGINE_TARGETS_SCHEMA",) + + +def test_bare_prefix_is_not_treated_as_a_renamed_variable(): + assert env_names("LEDGER_") == ("LEDGER_",) + + +# --------------------------------------------------------------------------- +# Precedence and the deprecation warning +# --------------------------------------------------------------------------- + + +def test_chronicle_name_wins_over_both_ledger_era_names(monkeypatch, recwarn): + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "/chronicle") + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + monkeypatch.setenv("POLICYENGINE_LEDGER_PE_US_DATA_ROOT", "/policyengine-ledger") + + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/chronicle" + assert not [ + warning + for warning in recwarn.list + if issubclass(warning.category, ChronicleEnvDeprecationWarning) + ] + + +def test_ledger_name_alone_still_works_and_warns(monkeypatch): + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/ledger" + + message = str(warnings_raised[0].message) + assert "LEDGER_PE_US_DATA_ROOT" in message + assert "CHRONICLE_PE_US_DATA_ROOT" in message + + +def test_policyengine_ledger_name_alone_still_works_and_warns(monkeypatch): + monkeypatch.setenv("POLICYENGINE_LEDGER_SCHEMA", "ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_value("CHRONICLE_SCHEMA") == "ledger" + + message = str(warnings_raised[0].message) + assert "POLICYENGINE_LEDGER_SCHEMA" in message + assert "CHRONICLE_SCHEMA" in message + + +def test_deprecation_warning_is_raised_once_per_process(monkeypatch, recwarn): + monkeypatch.setenv("LEDGER_PE_UK_DATA_ROOT", "/ledger") + + for _ in range(3): + assert env_value("CHRONICLE_PE_UK_DATA_ROOT") == "/ledger" + + deprecations = [ + warning + for warning in recwarn.list + if issubclass(warning.category, ChronicleEnvDeprecationWarning) + ] + assert len(deprecations) == 1 + + +def test_deprecation_warning_is_attributed_to_the_calling_module(monkeypatch): + monkeypatch.setenv("LEDGER_SOURCE_ARTIFACT_FETCH", "1") + + with pytest.warns(ChronicleEnvDeprecationWarning) as warnings_raised: + assert env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + # env_flag and env_value must report at the same depth, or operators get a + # notice pointing at Chronicle's own source instead of their call site. + assert Path(warnings_raised[0].filename).name == "test_chronicle_env.py" + + +def test_unset_variables_fall_back_to_the_default(): + assert env_value("CHRONICLE_PE_US_DATA_ROOT") is None + assert env_value("CHRONICLE_PE_US_DATA_ROOT", default="/fallback") == "/fallback" + + +def test_empty_values_count_as_unset(monkeypatch): + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "") + monkeypatch.setenv("LEDGER_PE_US_DATA_ROOT", "/ledger") + + with pytest.warns(ChronicleEnvDeprecationWarning): + assert env_value("CHRONICLE_PE_US_DATA_ROOT") == "/ledger" + + +# --------------------------------------------------------------------------- +# Flags +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " on "]) +def test_env_flag_accepts_truthy_spellings(monkeypatch, value): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", value) + assert env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +@pytest.mark.parametrize("value", ["0", "false", "no", "off", "maybe"]) +def test_env_flag_rejects_other_values(monkeypatch, value): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", value) + assert not env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +def test_env_flag_lets_the_chronicle_name_turn_a_legacy_flag_off(monkeypatch): + monkeypatch.setenv("CHRONICLE_SOURCE_ARTIFACT_FETCH", "0") + monkeypatch.setenv("LEDGER_SOURCE_ARTIFACT_FETCH", "1") + + # An operator who has migrated must be able to turn the flag off without + # first hunting down the stale ledger-era variable. + assert not env_flag(SOURCE_ARTIFACT_FETCH_ENV) + + +# --------------------------------------------------------------------------- +# Real call sites +# --------------------------------------------------------------------------- + + +def test_source_artifact_env_constants_are_chronicle_named(): + assert SOURCE_ARTIFACT_CACHE_ENV == "CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR" + assert SOURCE_ARTIFACT_FETCH_ENV == "CHRONICLE_SOURCE_ARTIFACT_FETCH" + + +@pytest.mark.parametrize( + "name", + ["CHRONICLE_SOURCE_ARTIFACT_CACHE_DIR", "LEDGER_SOURCE_ARTIFACT_CACHE_DIR"], +) +def test_source_artifact_cache_dir_honors_both_names(monkeypatch, tmp_path, name): + from chronicle.source_package import _source_artifact_cache_path + + monkeypatch.setenv(name, str(tmp_path)) + + cache_path = _source_artifact_cache_path( + {"filename": "table.xlsx", "sha256": "abc123"} + ) + + assert cache_path == tmp_path / "abc123" / "table.xlsx" + + +@pytest.mark.parametrize( + "name", ["CHRONICLE_PE_US_DATA_ROOT", "LEDGER_PE_US_DATA_ROOT"] +) +def test_pe_source_root_cli_default_honors_both_names(monkeypatch, name): + from db.cli import _pe_source_root_env_default + + monkeypatch.setenv(name, "/pe-us") + + assert _pe_source_root_env_default("us") == "/pe-us" + + +def test_pe_source_inventory_env_constants_are_chronicle_named(): + from db.pe_source_inventory import PE_UK_DATA_ROOT_ENV, PE_US_DATA_ROOT_ENV + + assert PE_US_DATA_ROOT_ENV == "CHRONICLE_PE_US_DATA_ROOT" + assert PE_UK_DATA_ROOT_ENV == "CHRONICLE_PE_UK_DATA_ROOT" + + +def test_db_cli_parser_builds_with_the_env_backed_defaults(monkeypatch, capsys): + """The db CLI builds its parser before dispatching any subcommand. + + Its --pe-us-root/--pe-uk-root defaults call into the env helper, so an + import error there breaks `chronicle init`, `load` and `stats` alike while + the rest of the test suite stays green. + """ + import db.cli + + monkeypatch.setenv("CHRONICLE_PE_US_DATA_ROOT", "/pe-us") + monkeypatch.setattr("sys.argv", ["chronicle", "--help"]) + + with pytest.raises(SystemExit) as exit_info: + db.cli.main() + + assert exit_info.value.code == 0 + assert "Manage Chronicle target input data" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "name", + ["CHRONICLE_SCHEMA", "POLICYENGINE_LEDGER_SCHEMA", "LEDGER_SCHEMA"], +) +def test_supabase_schema_honors_every_name_in_the_window(monkeypatch, name): + """Set after import and still honored: the schema is read at call time.""" + import db.supabase_client + + monkeypatch.setenv(name, "chronicle_probe") + + assert db.supabase_client.chronicle_schema() == "chronicle_probe" + assert default_chronicle_schema() == "chronicle_probe" + + +def test_supabase_schema_default_is_unchanged(): + import db.supabase_client + + # The hosted schema name itself is out of this slice; only the variable + # that overrides it moved. + assert DEFAULT_CHRONICLE_SCHEMA == "ledger" + assert default_chronicle_schema() == "ledger" + assert db.supabase_client.chronicle_schema() == "ledger" + assert db.supabase_client.targets_schema() == "targets" + + +@pytest.mark.parametrize( + "schema_env", + ["CHRONICLE_SCHEMA", "POLICYENGINE_LEDGER_SCHEMA", "LEDGER_SCHEMA"], +) +def test_supabase_schema_compatibility_aliases_honor_import_time_environment( + schema_env, +): + """Deprecated exports retain the environment snapshot existing importers use.""" + environment = os.environ.copy() + for name in (*env_names("CHRONICLE_SCHEMA"), "POLICYENGINE_TARGETS_SCHEMA"): + environment.pop(name, None) + environment[schema_env] = "chronicle_import_probe" + environment["POLICYENGINE_TARGETS_SCHEMA"] = "targets_import_probe" + script = ( + "import json; " + "from db.supabase_client import LEDGER_SCHEMA, TARGETS_SCHEMA; " + "print(json.dumps([LEDGER_SCHEMA, TARGETS_SCHEMA]))" + ) + + completed = subprocess.run( + [sys.executable, "-c", script], + cwd=Path(__file__).resolve().parents[1], + env=environment, + check=True, + capture_output=True, + text=True, + ) + + assert json.loads(completed.stdout) == [ + "chronicle_import_probe", + "targets_import_probe", + ] + + +def test_supabase_schema_is_not_bound_at_import(monkeypatch): + """Compatibility constants do not freeze the runtime schema resolver. + + The deprecated names expose only stable defaults for existing importers. + Query code calls the functions, which still honor an environment change + made after module import. + """ + import db.supabase_client + + assert db.supabase_client.LEDGER_SCHEMA == "ledger" + assert db.supabase_client.TARGETS_SCHEMA == "targets" + + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + monkeypatch.setenv("POLICYENGINE_TARGETS_SCHEMA", "targets_probe") + unreloaded = importlib.import_module("db.supabase_client") + + assert unreloaded.chronicle_schema() == "chronicle_probe" + assert unreloaded.targets_schema() == "targets_probe" + assert unreloaded.LEDGER_SCHEMA == "ledger" + assert unreloaded.TARGETS_SCHEMA == "targets" + + +# --------------------------------------------------------------------------- +# R2 bucket configuration +# --------------------------------------------------------------------------- + + +def test_r2_bucket_defaults_are_still_the_ledger_era_names(): + assert DEFAULT_R2_RAW_BUCKET == "ledger-raw" + assert DEFAULT_R2_DERIVED_BUCKET == "ledger-derived" + assert default_r2_raw_bucket() == "ledger-raw" + assert default_r2_derived_bucket() == "ledger-derived" + + +def test_r2_buckets_follow_the_chronicle_env_vars(monkeypatch): + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + + assert default_r2_raw_bucket() == "chronicle-raw" + assert default_r2_derived_bucket() == "chronicle-derived" + + +def test_r2_buckets_honor_ledger_era_names_with_a_warning(monkeypatch): + monkeypatch.setenv("LEDGER_R2_RAW_BUCKET", "legacy-raw") + + with pytest.warns(ChronicleEnvDeprecationWarning): + assert default_r2_raw_bucket() == "legacy-raw" + + +def test_bootstrap_r2_cli_creates_the_configured_buckets(monkeypatch, tmp_path): + log = tmp_path / "wrangler.log" + wrangler = _fake_wrangler(tmp_path, log) + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + monkeypatch.setenv("CHRONICLE_R2_DERIVED_BUCKET", "chronicle-derived") + + exit_code = harness_main(["bootstrap-r2", "--wrangler-command", str(wrangler)]) + + commands = log.read_text() + assert exit_code == 0 + assert "r2 bucket create chronicle-raw" in commands + assert "r2 bucket create chronicle-derived" in commands + + +def test_bootstrap_r2_cli_flags_still_override_the_environment(monkeypatch, tmp_path): + log = tmp_path / "wrangler.log" + wrangler = _fake_wrangler(tmp_path, log) + monkeypatch.setenv("CHRONICLE_R2_RAW_BUCKET", "chronicle-raw") + + harness_main( + [ + "bootstrap-r2", + "--raw-bucket", + "explicit-raw", + "--derived-bucket", + "explicit-derived", + "--wrangler-command", + str(wrangler), + ] + ) + + commands = log.read_text() + assert "r2 bucket create explicit-raw" in commands + assert "r2 bucket create explicit-derived" in commands + assert "chronicle-raw" not in commands diff --git a/tests/test_chronicle_manifest_reading.py b/tests/test_chronicle_manifest_reading.py new file mode 100644 index 00000000..d6589c48 --- /dev/null +++ b/tests/test_chronicle_manifest_reading.py @@ -0,0 +1,149 @@ +"""Manifest reading contracts shared by every consumer on this branch.""" + +from __future__ import annotations + +import pytest +import yaml + + +def test_strict_loader_keeps_yaml_merge_overrides_and_refuses_explicit_duplicates(): + """A ``<<: *defaults`` merge followed by an explicit override is valid YAML + (the explicit key wins); only two explicit spellings of one key are a + duplicate the loader must refuse.""" + from chronicle.registration import load_manifest_document + + merged = load_manifest_document( + "defaults: &defaults\n" + " source_url: https://publisher.test/a\n" + " filename: table.csv\n" + "files:\n" + " 2024:\n" + " <<: *defaults\n" + " source_url: https://publisher.test/b\n" + " sha256: " + "ab" * 32 + "\n" + ) + + assert merged["files"][2024]["source_url"] == "https://publisher.test/b" + assert merged["files"][2024]["filename"] == "table.csv" + + with pytest.raises(yaml.YAMLError, match="duplicate key 'source_url'"): + load_manifest_document( + "defaults: &defaults\n" + " filename: table.csv\n" + "files:\n" + " 2024:\n" + " <<: *defaults\n" + " source_url: https://publisher.test/a\n" + " source_url: https://publisher.test/b\n" + ) + + +def test_strict_loader_keeps_merge_sequence_precedence_for_artifact_selection(): + """``<<: [first, second]`` selects ``first``'s values: earlier mappings in a + merge sequence take precedence over later ones (YAML merge-key semantics, + and what ``yaml.safe_load`` does), while an explicit key in the entry still + overrides every merged source. The selected ``filename``/``sha256`` pair is + what fetch, publish, and the source-package reader use to pick bytes, so + reversing the precedence silently selects a different artifact.""" + from chronicle.registration import load_manifest_document + + document = ( + "first: &first\n" + " filename: first.csv\n" + " sha256: " + "aa" * 32 + "\n" + " source_url: https://publisher.test/first\n" + "second: &second\n" + " filename: second.csv\n" + " sha256: " + "bb" * 32 + "\n" + " source_url: https://publisher.test/second\n" + " licence: second-only\n" + "files:\n" + " 2024:\n" + " <<: [*first, *second]\n" + " 2023:\n" + " <<: [*first, *second]\n" + " filename: explicit.csv\n" + " 2022:\n" + " <<: [{filename: inline-first.csv}, {filename: inline-second.csv}]\n" + " 2021:\n" + " <<:\n" + " - <<: [*second, *first]\n" + " source_url: https://publisher.test/nested\n" + " - *first\n" + ) + + strict = load_manifest_document(document) + reference = yaml.safe_load(document) + assert strict["files"] == reference["files"] + + selected = strict["files"][2024] + assert selected["filename"] == "first.csv" + assert selected["sha256"] == "aa" * 32 + assert selected["source_url"] == "https://publisher.test/first" + # Keys only the later source carries are still merged in. + assert selected["licence"] == "second-only" + # An explicit key beats every merged source; the rest still follow first. + assert strict["files"][2023]["filename"] == "explicit.csv" + assert strict["files"][2023]["sha256"] == "aa" * 32 + assert strict["files"][2022]["filename"] == "inline-first.csv" + # Nested: the first sequence entry is itself a merge whose own explicit + # key wins inside it, and whose [second, first] order selects second. + nested = strict["files"][2021] + assert nested["filename"] == "second.csv" + assert nested["source_url"] == "https://publisher.test/nested" + + +def test_strict_loader_refuses_recursive_merges_as_yaml_errors(): + """A mapping that merges itself (directly or through a nested merge) has + no expansion. The loader must refuse it with a ``yaml.YAMLError`` that the + manifest-reading commands already handle, never a ``RecursionError``.""" + from chronicle.registration import load_manifest_document + + for document in ( + # Direct self-merge. + "files: &f\n 2024:\n filename: table.csv\n <<: *f\n", + # Self-merge through a nested merge sequence. + "files: &f\n 2024:\n filename: table.csv\n <<:\n - {filename: other.csv}\n - *f\n", + # Self-merge through a merged mapping's own merge. + "a: &a\n filename: table.csv\n <<: {<<: *a}\n", + ): + with pytest.raises(yaml.YAMLError, match="recursive"): + load_manifest_document(document) + + # A merge that only *repeats* a source is not a cycle. + repeated = load_manifest_document( + "d: &d\n filename: table.csv\nfiles:\n 2024:\n <<: [*d, *d]\n" + ) + assert repeated["files"][2024] == {"filename": "table.csv"} + + +def test_strict_loader_does_not_mutate_anchored_entries_when_merged_later(): + """Constructing a later merge of an anchored entry must not turn that + entry's inherited keys into 'explicit' ones: PyYAML constructs lazily and + flattens merged nodes in place.""" + from chronicle.registration import load_manifest_document + + document = load_manifest_document( + "defaults: &d\n" + " source_url: old\n" + "files:\n" + " 2024: &e\n" + " <<: *d\n" + " source_url: new\n" + "latest:\n" + " <<: *e\n" + ) + + assert document["files"][2024]["source_url"] == "new" + assert document["latest"]["source_url"] == "new" + + +def test_strict_loader_validates_duplicate_keys_inside_inline_merges(): + """A mapping reached through ``<<`` is still a mapping the document + spells out; duplicate keys inside it must be refused, not collapsed.""" + from chronicle.registration import load_manifest_document + + with pytest.raises(yaml.YAMLError, match="duplicate key 'files'"): + load_manifest_document( + "<<: {files: {2023: {filename: old.csv}}, files: {2024: {filename: new.csv}}}\n" + ) diff --git a/tests/test_chronicle_manifest_vintages.py b/tests/test_chronicle_manifest_vintages.py new file mode 100644 index 00000000..11e4764c --- /dev/null +++ b/tests/test_chronicle_manifest_vintages.py @@ -0,0 +1,132 @@ +"""Every artifact boundary refuses duplicate logical manifest vintages.""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +import pytest +import yaml + +from chronicle.artifacts import ( + MalformedManifestError, + fetch_source_artifact, + inventory_source_artifacts, + publish_source_artifacts, +) +from chronicle.source_package import SourceArtifactSpec + + +@pytest.fixture +def duplicate_vintage_package(tmp_path): + package = tmp_path / "data" / "irs_soi" / "table" + package.mkdir(parents=True) + files = {} + for vintage, filename in ( + (2023, "selected.csv"), + (2024, "numeric.csv"), + ("2024", "quoted.csv"), + ): + content = f"publisher bytes for {filename}\n".encode() + (package / filename).write_bytes(content) + files[vintage] = { + "filename": filename, + "source_url": f"https://example.test/{filename}", + "sha256": hashlib.sha256(content).hexdigest(), + "size_bytes": len(content), + } + (package / "manifest.yaml").write_text( + yaml.safe_dump( + { + "source_id": "irs_soi", + "package_id": "soi-table", + "files": files, + }, + sort_keys=False, + ) + ) + return package + + +def _unexpected_artifact_io(*_args, **_kwargs): + raise AssertionError("duplicate logical vintages reached artifact I/O") + + +@pytest.mark.parametrize("year", [2023, 2024]) +def test_fetch_refuses_logical_vintage_duplicates_anywhere_before_io( + duplicate_vintage_package, monkeypatch, year +): + package = duplicate_vintage_package + before = {path.name: path.read_bytes() for path in package.iterdir()} + monkeypatch.setattr("chronicle.artifacts._read_artifact", _unexpected_artifact_io) + monkeypatch.setattr( + "chronicle.artifacts._upload_r2_object", _unexpected_artifact_io + ) + + with pytest.raises(MalformedManifestError, match="both keys"): + fetch_source_artifact( + "https://example.test/selected.csv", + source_id="irs_soi", + package_id="soi-table", + year=year, + output_dir=package, + upload_r2=True, + ) + + assert {path.name: path.read_bytes() for path in package.iterdir()} == before + + +@pytest.mark.parametrize( + "operation", [publish_source_artifacts, inventory_source_artifacts] +) +def test_sweeps_refuse_logical_vintage_duplicates_before_artifact_io( + duplicate_vintage_package, monkeypatch, operation +): + package = duplicate_vintage_package + before = {path.name: path.read_bytes() for path in package.iterdir()} + read_bytes = Path.read_bytes + + def checked_read_bytes(path): + if path.parent == package and path.suffix == ".csv": + _unexpected_artifact_io() + return read_bytes(path) + + with monkeypatch.context() as guarded: + guarded.setattr(Path, "read_bytes", checked_read_bytes) + guarded.setattr( + "chronicle.artifacts._upload_r2_object", _unexpected_artifact_io + ) + report = operation(package) + + assert not report.valid + assert report.entries == () + assert any("both keys" in error for error in report.errors) + assert {path.name: path.read_bytes() for path in package.iterdir()} == before + + +@pytest.mark.parametrize("year", [2023, 2024]) +def test_source_loader_refuses_logical_vintage_duplicates_anywhere_before_io( + duplicate_vintage_package, tmp_path, monkeypatch, year +): + package = duplicate_vintage_package + before = {path.name: path.read_bytes() for path in package.iterdir()} + monkeypatch.setattr("chronicle.source_package.files", lambda _package: tmp_path) + monkeypatch.setattr( + "chronicle.source_package._read_source_artifact_content", + _unexpected_artifact_io, + ) + artifact = SourceArtifactSpec( + source_name="irs_soi", + source_table="Table", + resource_package="test_resources", + resource_directory="data/irs_soi/table", + manifest="manifest.yaml", + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + ) + + with pytest.raises(ValueError, match="both keys"): + artifact._artifact_content(year) + + assert {path.name: path.read_bytes() for path in package.iterdir()} == before diff --git a/tests/test_chronicle_mirror.py b/tests/test_chronicle_mirror.py index 787429c1..b9026cae 100644 --- a/tests/test_chronicle_mirror.py +++ b/tests/test_chronicle_mirror.py @@ -3,7 +3,12 @@ from __future__ import annotations import json +from pathlib import Path +import re +import pytest + +from chronicle.env import ChronicleEnvDeprecationWarning, DEFAULT_CHRONICLE_SCHEMA from chronicle.harness import main as harness_main from chronicle.mirror import ( LEDGER_MIRROR_TABLES, @@ -18,7 +23,7 @@ def test_export_chronicle_db_tables_writes_jsonl_and_manifest(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -53,7 +58,7 @@ def test_export_chronicle_db_tables_writes_jsonl_and_manifest(tmp_path): def test_export_chronicle_db_tables_orders_rows_deterministically(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" first_output_dir = tmp_path / "mirror-first" second_output_dir = tmp_path / "mirror-second" build_chronicle_db( @@ -72,7 +77,7 @@ def test_export_chronicle_db_tables_orders_rows_deterministically(tmp_path): def test_export_db_tables_cli_emits_manifest_summary(tmp_path, capsys): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -99,7 +104,7 @@ def test_export_db_tables_cli_emits_manifest_summary(tmp_path, capsys): def test_load_supabase_mirror_dry_run_counts_exported_rows(tmp_path): - db_path = tmp_path / "ledger.db" + db_path = tmp_path / "chronicle.db" output_dir = tmp_path / "mirror" build_chronicle_db( build_soi_table_1_1_facts(2023), @@ -189,6 +194,155 @@ def test_load_supabase_mirror_cli_dry_run(tmp_path, capsys): assert payload["table_count"] == len(LEDGER_MIRROR_TABLES) +# --------------------------------------------------------------------------- +# Schema configuration +# +# The mirror loader is the primary writer into the hosted schema, so it is the +# call site CHRONICLE_SCHEMA has to reach (PolicyEngine/chronicle#143, +# mechanism 3). It defaulted to the literal "ledger" while only the read-side +# client honored the renamed variable, which would have sent a rehearsal load +# into production the moment an operator set it. +# --------------------------------------------------------------------------- + + +def _empty_mirror(tmp_path): + mirror_dir = tmp_path / "mirror" + mirror_dir.mkdir() + for table in LEDGER_MIRROR_TABLES: + (mirror_dir / f"{table}.jsonl").write_text("") + return mirror_dir + + +def _one_build_artifact(tmp_path): + path = tmp_path / "build_artifacts.jsonl" + path.write_text( + json.dumps( + { + "build_artifact_key": "ledger.build_artifact.v1:test", + "build_id": "ledger.build.v1:test", + "artifact_kind": "json", + "artifact_name": "reports/build_summary.json", + "sha256": "abc", + "size_bytes": 3, + "r2_bucket": "ledger-derived", + "r2_key": "derived/test", + "r2_uri": "r2://ledger-derived/derived/test", + }, + sort_keys=True, + ) + + "\n" + ) + return path + + +def _load_into_fake_client(tmp_path, **kwargs): + client = _FakeSupabaseClient() + report = load_supabase_mirror( + _empty_mirror(tmp_path), + table_paths={"build_artifacts": _one_build_artifact(tmp_path)}, + client=client, + **kwargs, + ) + return report, client + + +def test_load_supabase_mirror_defaults_to_the_ledger_schema(tmp_path): + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "ledger" + assert [upsert[0] for upsert in client.upserts] == ["ledger"] + + +def test_load_supabase_mirror_writes_to_the_chronicle_schema(tmp_path, monkeypatch): + """The renamed variable configures the writer, not just the reader.""" + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "chronicle_probe" + assert [upsert[0] for upsert in client.upserts] == ["chronicle_probe"] + + +@pytest.mark.parametrize("name", ["POLICYENGINE_LEDGER_SCHEMA", "LEDGER_SCHEMA"]) +def test_load_supabase_mirror_honors_a_ledger_era_schema_name( + tmp_path, monkeypatch, name +): + monkeypatch.setenv(name, "legacy_probe") + + with pytest.warns(ChronicleEnvDeprecationWarning): + report, client = _load_into_fake_client(tmp_path) + + assert report.schema == "legacy_probe" + assert [upsert[0] for upsert in client.upserts] == ["legacy_probe"] + + +def test_an_explicit_schema_still_wins_over_the_environment(tmp_path, monkeypatch): + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + + report, client = _load_into_fake_client(tmp_path, schema="explicit_probe") + + assert report.schema == "explicit_probe" + assert [upsert[0] for upsert in client.upserts] == ["explicit_probe"] + + +def test_load_supabase_mirror_cli_writes_to_the_configured_schema( + tmp_path, monkeypatch, capsys +): + """The CLI resolves the same way when no --schema is supplied.""" + client = _FakeSupabaseClient() + monkeypatch.setattr("chronicle.mirror._get_supabase_client", lambda: client) + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + argv = [ + "load-supabase-mirror", + "--dir", + str(_empty_mirror(tmp_path)), + "--build-artifacts", + str(_one_build_artifact(tmp_path)), + ] + + exit_code = harness_main(argv) + payload = json.loads(capsys.readouterr().out) + + assert exit_code == 0 + assert payload["schema"] == "chronicle_probe" + assert [upsert[0] for upsert in client.upserts] == ["chronicle_probe"] + + assert harness_main([*argv, "--schema", "explicit_probe"]) == 0 + assert json.loads(capsys.readouterr().out)["schema"] == "explicit_probe" + assert [upsert[0] for upsert in client.upserts] == [ + "chronicle_probe", + "explicit_probe", + ] + + +def _readme_supabase_cutover_section(): + readme = (Path(__file__).parents[1] / "README.md").read_text() + return readme[ + readme.index("To prepare the deterministic SQLite artifact") : readme.index( + "Chronicle settings are read" + ) + ] + + +def test_readme_supabase_cutover_documents_the_runtime_schema_default(): + section = _readme_supabase_cutover_section() + + assert f"writes to `{DEFAULT_CHRONICLE_SCHEMA}`" in section + assert "With no schema environment override and no `--schema`" in section + assert "`CHRONICLE_SCHEMA=chronicle`" in section + assert "`--schema chronicle`" in section + + +def test_readme_supabase_cutover_only_names_checked_in_migrations(): + section = _readme_supabase_cutover_section() + repository = Path(__file__).parents[1] + migration_paths = re.findall(r"`([^`\n]+[.]sql)`", section) + missing = [path for path in migration_paths if not (repository / path).is_file()] + + assert "create and apply a Supabase/Postgres migration" in section + assert missing == [] + + class _FakeSupabaseClient: def __init__(self): self.upserts = [] diff --git a/tests/test_chronicle_namespace.py b/tests/test_chronicle_namespace.py index 9b0ffd41..a8ce2b63 100644 --- a/tests/test_chronicle_namespace.py +++ b/tests/test_chronicle_namespace.py @@ -8,11 +8,7 @@ query_targets, ) from db.schema import Target as DbTarget -from db.supabase_client import ( - LEDGER_SCHEMA, - TARGETS_SCHEMA, - query_targets as db_query_targets, -) +from db.supabase_client import query_targets as db_query_targets def test_chronicle_targets_reexport_schema_objects(): @@ -29,8 +25,43 @@ def test_chronicle_client_reexports_supabase_client(): def test_chronicle_supabase_schema_boundaries_are_defaulted(): - assert LEDGER_SCHEMA == "ledger" - assert TARGETS_SCHEMA == "targets" + """The schema names resolve per call, so this reads the cleared window. + + ``db.supabase_client`` is imported at collection, before any fixture runs. + Resolving the schema there — as an import-time constant — would bind an + operator's ``CHRONICLE_SCHEMA`` (or a ledger-era name, warning as it went) + into the module for the whole session, and no fixture could take it back. + Reading at call time is what makes this a test of the defaults rather than + of the shell. + """ + from db import supabase_client + + assert supabase_client.chronicle_schema() == "ledger" + assert supabase_client.targets_schema() == "targets" + + +def test_chronicle_supabase_schema_compatibility_aliases_are_defaulted(): + """The shipped module keeps its pre-rename import surface.""" + from chronicle.env import DEFAULT_CHRONICLE_SCHEMA + from db.supabase_client import ( + DEFAULT_TARGETS_SCHEMA, + LEDGER_SCHEMA, + TARGETS_SCHEMA, + ) + + assert LEDGER_SCHEMA == DEFAULT_CHRONICLE_SCHEMA == "ledger" + assert TARGETS_SCHEMA == DEFAULT_TARGETS_SCHEMA == "targets" + + +def test_chronicle_supabase_schema_follows_the_environment(monkeypatch): + """The renamed variable reaches the client after it has been imported.""" + from db import supabase_client + + monkeypatch.setenv("CHRONICLE_SCHEMA", "chronicle_probe") + monkeypatch.setenv("POLICYENGINE_TARGETS_SCHEMA", "targets_probe") + + assert supabase_client.chronicle_schema() == "chronicle_probe" + assert supabase_client.targets_schema() == "targets_probe" def test_chronicle_normalization_exports_helpers(): diff --git a/tests/test_chronicle_source_package.py b/tests/test_chronicle_source_package.py index ae6f0166..aeda3df0 100644 --- a/tests/test_chronicle_source_package.py +++ b/tests/test_chronicle_source_package.py @@ -5,8 +5,9 @@ from copy import deepcopy import hashlib from io import BytesIO +import os from pathlib import Path -from zipfile import ZipFile +from zipfile import Path as ZipPath, ZipFile import openpyxl import pytest @@ -244,9 +245,7 @@ def test_hmrc_cgt_reuses_one_publisher_series_across_definition_years(): package = load_source_package("hmrc-cgt-statistics-2026") package_facts = package.build_facts(2026) facts = [ - fact - for fact in package_facts - if fact.measure.concept == "hmrc.cgt_tax_total" + fact for fact in package_facts if fact.measure.concept == "hmrc.cgt_tax_total" ] assert {fact.entity.name for fact in package_facts} == {"tax_unit"} @@ -1083,6 +1082,338 @@ def test_source_artifact_loader_fetches_missing_artifact_when_enabled( assert _source_artifact_cache_path(spec).read_bytes() == content +@pytest.fixture +def recorded_r2_artifact(tmp_path, monkeypatch): + resource_root = tmp_path / "resources" + resource_dir = resource_root / "data" / "publisher" / "package" + resource_dir.mkdir(parents=True) + content = b"publisher source artifact" + digest = hashlib.sha256(content).hexdigest() + key = f"raw/publisher/package/2024/{digest}/table.csv" + entry = { + "filename": "table.csv", + "source_url": "https://example.test/table.csv", + "sha256": digest, + "storage": { + "r2": { + "provider": "r2", + "bucket": "ledger-raw", + "key": key, + "uri": f"r2://ledger-raw/{key}", + } + }, + } + artifact = SourceArtifactSpec( + source_name="publisher", + source_table="Table", + resource_package="test_resources", + resource_directory="data/publisher/package", + manifest="manifest.yaml", + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + artifact_year=2024, + ) + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: resource_root + ) + return artifact, resource_dir, content, entry + + +@pytest.mark.parametrize( + "invalid_locator", + [ + "contradictory-key", + "contradictory-bucket", + "missing-provider", + "missing-uri", + "non-r2", + "null-block", + "checksum-identity", + "filename-identity", + ], +) +def test_source_artifact_spec_refuses_invalid_recorded_r2_before_read( + recorded_r2_artifact, monkeypatch, invalid_locator +): + artifact, resource_dir, content, entry = recorded_r2_artifact + recorded = entry["storage"]["r2"] + if invalid_locator == "contradictory-key": + recorded["key"] = recorded["key"].replace("table.csv", "other.csv") + elif invalid_locator == "contradictory-bucket": + recorded["bucket"] = "other-raw" + elif invalid_locator == "missing-provider": + del recorded["provider"] + elif invalid_locator == "missing-uri": + del recorded["uri"] + elif invalid_locator == "non-r2": + recorded["provider"] = "s3" + recorded["uri"] = recorded["uri"].replace("r2://", "s3://") + elif invalid_locator == "null-block": + entry["storage"]["r2"] = None + elif invalid_locator == "checksum-identity": + entry["sha256"] = "0" * 64 + else: + recorded["key"] = recorded["key"].replace("table.csv", "other.csv") + recorded["uri"] = recorded["uri"].replace("table.csv", "other.csv") + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump({"files": {2024: entry}}) + ) + (resource_dir / "table.csv").write_bytes(content) + + def unexpected_read(_artifact_path, _spec): + raise AssertionError("invalid R2 provenance reached artifact I/O") + + monkeypatch.setattr( + "chronicle.source_package._read_source_artifact_content", unexpected_read + ) + with pytest.raises(ValueError, match="storage.r2|recorded R2"): + artifact._artifact_content(2024) + + +@pytest.mark.parametrize("location", ["local", "fetch"]) +def test_source_artifact_spec_checks_recorded_r2_digest_without_declared_checksum( + recorded_r2_artifact, tmp_path, monkeypatch, location +): + artifact, resource_dir, _content, entry = recorded_r2_artifact + del entry["sha256"] + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump({"files": {2024: entry}}) + ) + changed_content = b"different publisher bytes" + cache = tmp_path / "cache" + monkeypatch.setenv(SOURCE_ARTIFACT_CACHE_ENV, str(cache)) + if location == "local": + (resource_dir / "table.csv").write_bytes(changed_content) + else: + monkeypatch.setenv(SOURCE_ARTIFACT_FETCH_ENV, "1") + monkeypatch.setattr( + "chronicle.source_package._fetch_source_artifact_content", + lambda _url: changed_content, + ) + + with pytest.raises(ValueError, match="checksum mismatch"): + artifact._artifact_content(2024) + assert not cache.exists() + + +def test_source_artifact_spec_accepts_consistent_recorded_r2(recorded_r2_artifact): + artifact, resource_dir, content, entry = recorded_r2_artifact + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump({"files": {2024: entry}}) + ) + (resource_dir / "table.csv").write_bytes(content) + + assert artifact._artifact_content(2024) == ( + content, + "table.csv", + entry["source_url"], + entry["storage"]["r2"], + ) + + +@pytest.mark.parametrize("entry_kind", ["directory", "fifo"]) +@pytest.mark.parametrize("resource_kind", ["manifest", "artifact"]) +def test_source_artifact_spec_refuses_non_regular_resource_before_open( + recorded_r2_artifact, monkeypatch, entry_kind, resource_kind +): + artifact, resource_dir, _content, entry = recorded_r2_artifact + name = "manifest.yaml" if resource_kind == "manifest" else "table.csv" + resource = resource_dir / name + if entry_kind == "directory": + resource.mkdir() + else: + os.mkfifo(resource) + if resource_kind == "artifact": + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump({"files": {2024: entry}}) + ) + + def unexpected_read(_artifact_path, _spec): + raise AssertionError("non-regular artifact reached artifact I/O") + + monkeypatch.setattr( + "chronicle.source_package._read_source_artifact_content", unexpected_read + ) + with pytest.raises(ValueError, match="not a regular file"): + if resource_kind == "manifest": + # Resolve without opening: a broken guard must fail promptly even + # for a FIFO, whose actual open would block the test process. + artifact.manifest_resource() + else: + artifact._artifact_content(2024) + + +def test_source_artifact_spec_reads_regular_importlib_zip_resources( + recorded_r2_artifact, monkeypatch +): + artifact, _resource_dir, content, entry = recorded_r2_artifact + buffer = BytesIO() + with ZipFile(buffer, "w") as archive: + archive.writestr("data/publisher/package/table.csv", content) + archive.writestr( + "data/publisher/package/manifest.yaml", + yaml.safe_dump({"files": {2024: entry}}), + ) + with ZipFile(buffer) as archive: + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: ZipPath(archive) + ) + assert artifact._artifact_content(2024)[0] == content + + +def test_source_artifact_spec_fetches_absent_regular_resource( + recorded_r2_artifact, tmp_path, monkeypatch +): + artifact, resource_dir, content, entry = recorded_r2_artifact + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump({"files": {2024: entry}}) + ) + monkeypatch.setenv(SOURCE_ARTIFACT_CACHE_ENV, str(tmp_path / "cache")) + monkeypatch.setenv(SOURCE_ARTIFACT_FETCH_ENV, "1") + monkeypatch.setattr( + "chronicle.source_package._fetch_source_artifact_content", lambda _url: content + ) + + assert artifact._artifact_content(2024)[0] == content + assert _source_artifact_cache_path(entry).read_bytes() == content + + +@pytest.mark.parametrize( + "path_kind", + ["absolute", "parent", "symlink", "normalized-alias", "manifest-name"], +) +def test_source_artifact_spec_refuses_unsafe_manifest_filename_before_read( + tmp_path, monkeypatch, path_kind +): + resource_root = tmp_path / "resources" + resource_dir = resource_root / "data" / "publisher" / "package" + resource_dir.mkdir(parents=True) + outside = resource_dir.parent / "outside.csv" + outside.write_bytes(b"outside publisher bytes") + if path_kind == "absolute": + filename = str(outside) + elif path_kind == "parent": + filename = "../outside.csv" + else: + filename = "table.csv" + if path_kind == "symlink": + (resource_dir / filename).symlink_to(outside) + elif path_kind == "normalized-alias": + (resource_dir / "TABLE.csv").write_bytes(outside.read_bytes()) + else: + filename = "manifest_artifact.yaml" + (resource_dir / filename).write_bytes(outside.read_bytes()) + (resource_dir / "manifest.yaml").write_text( + yaml.safe_dump( + { + "files": { + 2024: { + "filename": filename, + "source_url": outside.as_uri(), + "sha256": hashlib.sha256(outside.read_bytes()).hexdigest(), + } + } + }, + sort_keys=False, + ) + ) + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: resource_root + ) + + def unexpected_artifact_read(_artifact_path, _spec): + raise AssertionError("unsafe artifact path reached artifact I/O") + + monkeypatch.setattr( + "chronicle.source_package._read_source_artifact_content", + unexpected_artifact_read, + ) + artifact = SourceArtifactSpec( + source_name="publisher", + source_table="Table", + resource_package="test_resources", + resource_directory="data/publisher/package", + manifest="manifest.yaml", + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + artifact_year=2024, + ) + + if path_kind == "symlink": + message = "symbolic link" + elif path_kind == "normalized-alias": + message = "normalized filename" + elif path_kind == "manifest-name": + message = "manifest name" + else: + message = "bare filename" + with pytest.raises(ValueError, match=message): + artifact._artifact_content(2024) + + +@pytest.mark.parametrize( + "path_kind", ["absolute", "parent", "symlink", "normalized-alias", "unsupported"] +) +def test_source_artifact_spec_refuses_unsafe_manifest_path_before_artifact_read( + tmp_path, monkeypatch, path_kind +): + resource_root = tmp_path / "resources" + resource_dir = resource_root / "data" / "publisher" / "package" + resource_dir.mkdir(parents=True) + payload = { + "files": { + 2024: { + "filename": "table.csv", + "source_url": "https://example.test/table.csv", + } + } + } + outside_manifest = resource_dir.parent / "outside-manifest.yaml" + outside_manifest.write_text(yaml.safe_dump(payload, sort_keys=False)) + if path_kind == "absolute": + manifest_name = str(outside_manifest) + elif path_kind == "parent": + manifest_name = "../outside-manifest.yaml" + elif path_kind == "symlink": + manifest_name = "manifest.yaml" + (resource_dir / manifest_name).symlink_to(outside_manifest) + elif path_kind == "normalized-alias": + manifest_name = "manifest.yaml" + (resource_dir / "Manifest.yaml").write_text( + yaml.safe_dump(payload, sort_keys=False) + ) + else: + manifest_name = "registry.yaml" + (resource_dir / manifest_name).write_text(yaml.safe_dump(payload)) + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: resource_root + ) + + def unexpected_artifact_read(_artifact_path, _spec): + raise AssertionError("unsafe manifest path reached artifact I/O") + + monkeypatch.setattr( + "chronicle.source_package._read_source_artifact_content", + unexpected_artifact_read, + ) + artifact = SourceArtifactSpec( + source_name="publisher", + source_table="Table", + resource_package="test_resources", + resource_directory="data/publisher/package", + manifest=manifest_name, + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + artifact_year=2024, + ) + + with pytest.raises(ValueError): + artifact._artifact_content(2024) + + def test_source_package_path_builds_valid_soi_table_1_4_facts(): package_path = REPO_ROOT / "packages" / "irs_soi" / "table_1_4" package = load_source_package(package_path) @@ -4284,3 +4615,59 @@ def test_build_facts_with_label_year_does_not_crash(): # Record-set periods are literal 2024, so a label build still resolves them. assert facts assert all(fact.period.value == 2024 for fact in facts) + + +@pytest.mark.parametrize( + "bad_directory", + ["/etc", "../outside", "data/../../outside", "data/./publisher"], +) +def test_resource_directory_must_stay_inside_the_resource_package( + tmp_path, monkeypatch, bad_directory +): + """`resource_directory` is joined under the resource package root; an + absolute or parent-traversing value escapes it and must be refused + before any read.""" + resource_root = tmp_path / "pkg" + resource_root.mkdir() + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: resource_root + ) + artifact = SourceArtifactSpec( + source_name="publisher", + source_table="Table", + resource_package="test_resources", + resource_directory=bad_directory, + manifest="manifest.yaml", + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + artifact_year=2024, + ) + + with pytest.raises(ValueError, match="resource_directory"): + artifact._artifact_content(2024) + + +def test_resource_directory_refuses_a_symlinked_ancestor(tmp_path, monkeypatch): + resource_root = tmp_path / "pkg" + real = tmp_path / "elsewhere" / "package" + real.mkdir(parents=True) + (resource_root / "data").mkdir(parents=True) + (resource_root / "data" / "publisher").symlink_to(real.parent) + monkeypatch.setattr( + "chronicle.source_package.files", lambda _package: resource_root + ) + artifact = SourceArtifactSpec( + source_name="publisher", + source_table="Table", + resource_package="test_resources", + resource_directory="data/publisher/package", + manifest="manifest.yaml", + vintage="2024", + extracted_at="2026-09-04", + extraction_method="test", + artifact_year=2024, + ) + + with pytest.raises(ValueError, match="symbolic link|resource_directory"): + artifact._artifact_content(2024) diff --git a/tests/test_chronicle_suite.py b/tests/test_chronicle_suite.py index 6911d6e2..c08f8abe 100644 --- a/tests/test_chronicle_suite.py +++ b/tests/test_chronicle_suite.py @@ -96,7 +96,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): assert (output_dir / "source_regions.jsonl").exists() assert (output_dir / "facts.jsonl").exists() assert (output_dir / "consumer_facts.jsonl").exists() - assert (output_dir / "ledger.db").exists() + assert (output_dir / "chronicle.db").exists() assert (output_dir / "datapackage.json").exists() assert (output_dir / "ro-crate-metadata.json").exists() assert (output_dir / "reports" / "source_regions.json").exists() @@ -124,7 +124,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): "source_regions.jsonl", "facts.jsonl", "consumer_facts.jsonl", - "ledger.db", + "chronicle.db", "reports/build_summary.json", "reports/source_regions.json", "reports/selectors.json", @@ -150,7 +150,7 @@ def test_build_source_suite_writes_artifacts_and_reports(tmp_path): "concept_alignment_validation_skipped" ] - with sqlite3.connect(output_dir / "ledger.db") as connection: + with sqlite3.connect(output_dir / "chronicle.db") as connection: facts_count = connection.execute( "SELECT COUNT(*) FROM aggregate_facts" ).fetchone()[0] @@ -184,7 +184,7 @@ def test_build_source_suite_supports_soi_table_1_4(tmp_path): "concept_alignment_validation_skipped" ) assert (output_dir / "source_regions.jsonl").exists() - assert (output_dir / "ledger.db").exists() + assert (output_dir / "chronicle.db").exists() def test_agent_acceptance_accepts_aggregate_income_range_source_rows(): @@ -739,7 +739,7 @@ def test_build_suite_cli_emits_json_summary(tmp_path, capsys): assert payload["outputs"]["source_regions"] == str( output_dir / "source_regions.jsonl" ) - assert payload["outputs"]["database"] == str(output_dir / "ledger.db") + assert payload["outputs"]["database"] == str(output_dir / "chronicle.db") assert payload["outputs"]["consumer_facts"] == str( output_dir / "consumer_facts.jsonl" )