ADR: raw microdata in Chronicle is identity, not content

.github/chronicle-agents.yml

@@ -43,4 +43,4 @@ required_judges:
   ledger-contract:
     verdict: PASS if schema or identity changes preserve source provenance and do not move Microcosm responsibilities into Chronicle.
   ledger-boundary:
-    verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle.
+    verdict: PASS if reconciliation, aging, imputation, support-aware activation, and solver construction remain outside Chronicle; no microdata records, rows, columns, row values, or cells enter any Chronicle parsed-source surface, registry, derived artifact, or journal; no fact computed from raw microdata by Chronicle or by a PolicyEngine-side consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a Chronicle registration) enters Chronicle, however many intermediate artifacts stand between them, while a value that a third party asserted and published, whether the microdata's own publisher or another, is an ordinary fact with ordinary provenance; no licensed or restricted microdata bytes enter any Chronicle store, and public microdata bytes enter only with artifact-bound redistribution evidence (docs/adr-chronicle-raw-microdata-identity.md).

AGENTS.md

@@ -19,6 +19,14 @@ Do not put Microcosm work in Chronicle:
 - no solver-ready target construction
 - no target profiles or model-measurement bindings
 - no PolicyEngine-computed values stored as facts
+- no microdata records, rows, columns, row values, or cells, no facts
+  computed from raw microdata by Chronicle or by a PolicyEngine-side consumer
+  (Microcosm, PolicyEngine, Thesis), and no licensed or restricted microdata
+  bytes in any Chronicle store (registering a release's identity is allowed
+  once chronicle#221 lands its access-aware refusals, and until then no
+  microdata release may be pointed at any Chronicle command; any value a
+  third party asserted and published remains an ordinary fact; see
+  `docs/adr-chronicle-raw-microdata-identity.md`)
 
 Chronicle records every fact's publisher reference period. Consumers own and
 enforce any declaration that aligns those facts to another period; Chronicle

README.md

@@ -12,7 +12,13 @@ Chronicle may normalize structure: parse files, type values, declare units and
 scales, assign geography and period identifiers, preserve lineage back to
 source artifacts, and publish source-backed facts. Chronicle does not own
 selection or measurement contracts, reconcile inconsistent sources, impute
-missing data, store raw survey microdata, or execute simulator-specific calibration.
+missing data, parse survey or administrative microdata into records, rows,
+columns, row values, or cells, derive facts from raw microdata, hold licensed
+or restricted microdata bytes, or execute simulator-specific calibration. Once
+chronicle#221 lands, Chronicle registers the microdata releases its consumers
+build from (publisher, vintage, checksum, licence) and archives the bytes of
+public-use files that carry artifact-bound redistribution evidence; see
+[`docs/adr-chronicle-raw-microdata-identity.md`](docs/adr-chronicle-raw-microdata-identity.md).
 
 Microcosm consumes Chronicle facts, owns the contracts that select and bind
 them, applies declared period alignment, and runs calibration. Thesis can
@@ -49,13 +55,16 @@ The load-bearing rule:
 > consumer or transform it in ways that change its meaning.
 
 The store is facts-only, and the line is who asserted the value. Everything a
-publisher asserted — including the publisher's own projections — is a fact.
-Everything PolicyEngine computes (aged, uprated, forecast, or reconciled
-levels) is a downstream build artifact and never enters the store; Microcosm
-owns aging as a named, versioned model over Chronicle growth-factor facts. A
-fact's `period` is the period its value refers to. Consumers must enforce any
-contract that aligns it to another period (see
-[`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)).
+publisher asserted — including the publisher's own projections — is a fact,
+except publisher-authored values at microdata grain, which are content and
+never facts; only the values a publisher asserted and published over that
+microdata are facts, whether the microdata's own publisher or another (see
+`docs/adr-chronicle-raw-microdata-identity.md`). Everything PolicyEngine
+computes (aged, uprated, forecast, or reconciled levels) is a downstream build
+artifact and never enters the store; Microcosm owns aging as a named, versioned
+model over Chronicle growth-factor facts. A fact's `period` is the period its
+value refers to. Consumers must enforce any contract that aligns it to another
+period (see [`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)).
 
 | Layer | Owns | Examples |
 |-------|------|----------|

chronicle/__init__.py

@@ -1,9 +1,13 @@
 """Chronicle source-data foundation.
 
-Chronicle owns government-statistics releases: source artifacts, source-backed
-facts, constraints, and provenance. Selection contracts, raw microdata storage,
-source reconciliation, aging, imputation, target activation, and calibration
-belong in downstream systems such as Microcosm.
+Chronicle owns government-statistics releases: source artifacts (and, once
+chronicle#221 lands, registrations of raw microdata releases plus custody of
+public-use bytes with redistribution evidence), source-backed facts,
+constraints, and provenance. Microdata content (records, rows, columns, row
+values, cells, and facts computed from raw microdata by Chronicle or a
+PolicyEngine-side consumer), licensed or restricted microdata bytes, selection
+contracts, source reconciliation, aging, imputation, target activation, and
+calibration belong in downstream systems such as Microcosm.
 """
 
 __all__ = [

chronicle/boundary.py

@@ -0,0 +1,26 @@
+"""The microdata boundary, stated once so docs, the judge contract, and tests agree.
+
+``docs/adr-chronicle-raw-microdata-identity.md`` decides that Chronicle registers
+raw microdata releases by identity and stores none of their content. These are
+the three negative clauses that decision reduces to. ``.github/chronicle-agents.yml``
+carries them verbatim in the ``ledger-boundary`` judge verdict, and
+``tests/test_chronicle_governance.py`` asserts that it does, so the wording cannot
+regress or invert without failing the suite.
+"""
+
+from __future__ import annotations
+
+MICRODATA_BOUNDARY_CLAUSES: tuple[str, ...] = (
+    "no microdata records, rows, columns, row values, or cells enter any "
+    "Chronicle parsed-source surface, registry, derived artifact, or journal",
+    "no fact computed from raw microdata by Chronicle or by a PolicyEngine-side "
+    "consumer (Microcosm, PolicyEngine, Thesis, or any system that builds from a "
+    "Chronicle registration) enters Chronicle, however many intermediate artifacts "
+    "stand between them, while a value that a third party asserted and published, "
+    "whether the microdata's own publisher or another, is an ordinary fact with "
+    "ordinary provenance",
+    "no licensed or restricted microdata bytes enter any Chronicle store, and "
+    "public microdata bytes enter only with artifact-bound redistribution evidence",
+)
+
+__all__ = ["MICRODATA_BOUNDARY_CLAUSES"]

docs/adr-chronicle-facts-only.md

@@ -9,11 +9,15 @@ Chronicle stores source-published values only. The boundary is **who asserted
 the value**, not level versus projection:
 
 1. Anything a publisher asserted is a fact — including the publisher's own
-   projections. "CBO's January 2026 baseline projects individual income tax
-   receipts of $X in 2027" is a source-backed claim with lineage, exactly
-   like an SOI observation. These facts carry
-   `assertion: source_projection`; measured or administered outcomes carry
-   the default `assertion: observation`.
+   projections — with one grain exception added by
+   `docs/adr-chronicle-raw-microdata-identity.md`: a publisher-authored value
+   at microdata grain (a record, row, column, row value, or cell of a microdata
+   release) is content, not a fact; only the values a publisher asserted and
+   published over that microdata are facts. "CBO's January 2026 baseline
+   projects individual income tax receipts of $X in 2027" is a source-backed
+   claim with lineage, exactly like an SOI observation. These facts carry
+   `assertion: source_projection`; measured or administered outcomes carry the
+   default `assertion: observation`.
 2. Anything PolicyEngine computed — an aged, uprated, forecast, or
    reconciled level — is never a Chronicle object. Such values are regenerable
    build artifacts and live in the consumer (Microcosm calibration owns

docs/adr-chronicle-raw-microdata-identity.md

@@ -0,0 +1,160 @@
+# ADR: Raw microdata in Chronicle is identity, not content
+
+Status: accepted 2026-09-02. Enforcement lands in two steps and this ADR's
+guarantees are prospective until they merge: chronicle#221 (PR #227: manifest
+classification, the access-aware refusals in `validate-package`,
+`fetch-artifact`, `publish-raw`, and the source-package byte reader, the
+redistribution allowlist, artifact-bound licence evidence, untracked staging
+for public microdata bytes) and chronicle#238 (asserting principal and
+root-artifact lineage on fact provenance). Until #227 merges, no microdata
+release may be pointed at any Chronicle command: today they materialize,
+upload, and parse every input. Until #238 merges, the derived-fact prohibition
+is a review obligation, not a validator.
+
+## Decision
+
+Chronicle registers every raw microdata release its consumers build from, and
+stores none of their content. "Content" means any parsed representation:
+microdata records, rows, columns, row values, or cells, and any fact computed
+from them by Chronicle or by a PolicyEngine-side consumer (Microcosm,
+PolicyEngine, Thesis, or any system that builds from a Chronicle registration).
+A value that a third party asserted and published, whether the microdata's own
+publisher or another, is an ordinary fact. Custody of a public-use file's bytes
+under artifact-bound redistribution evidence is not content; custody of
+licensed or restricted bytes is never taken at all.
+
+1. **Classification is explicit.** Every manifest created or modified after
+   this ADR declares `kind`, either `publisher_table` or `microdata_release`.
+   `validate-package`, `fetch-artifact`, `publish-raw`, and the source-package
+   byte reader refuse a new or modified manifest without it, at every entry
+   point. The manifests that exist at #227's merge commit are grandfathered as
+   publisher tables by an explicit frozen list checked into the repository; a
+   kindless manifest outside that list is an error, never a publisher table by
+   default. `microdata_release` is what lets every parser refuse the file, so a
+   public microdata release can never be mistaken for a public aggregate
+   workbook.
+2. **Registration.** A microdata release (CPS ASEC, ACS PUMS, SCF, SIPP, FRS,
+   BE-SILC, the IRS PUF, and their successors in every jurisdiction) is a
+   source artifact registered by identity: publisher, source URL or access
+   route, vintage, SHA-256, size, licence, an access class from a closed set
+   (`public`, `licensed`, `restricted`), and the attestation fields in
+   decision 5. One package per publisher release, its files listed under the
+   vintage year; a registration is identified by
+   `{source_id, package_id, year, sha256, filename}`. Keys follow the
+   convention in `docs/storage-architecture.md`:
+   `raw/{country}/{source_id}/{package_id}/{year}/{sha256}/{filename}` for
+   publishers mapped to a country segment (UK and New Zealand today) and the
+   legacy `raw/{source_id}/{package_id}/{year}/{sha256}/{filename}` shape for
+   every unmapped publisher, US sources included. No source package parses
+   the file.
+3. **Bytes only with artifact-bound redistribution evidence.** Being
+   downloadable is not a licence, and an allowlisted licence name is not
+   evidence that this file was issued under it. Chronicle archives a release's
+   bytes only when `access` is `public`, the recorded `licence` is on
+   Chronicle's allowlist of redistributable terms (maintained in code with the
+   evidence for each term: a U.S. Government work under 17 U.S.C. §105, the
+   Open Government Licence v3, CC0, CC BY), and the entry carries
+   `licence_evidence` binding the artifact to the term: issuer, licence
+   identifier and version, the scope statement, a durable evidence URL, and
+   the covered SHA-256. The `ledger-boundary` judge verifies that the evidence
+   names the artifact. Any public-download file without that evidence is
+   classed `licensed`. Licensed or restricted files (FRS under the UKDS
+   agreement, BE-SILC scientific-use files, the IRS PUF) are registered
+   hash-only: no bytes in any Chronicle store, no Chronicle credential grants
+   access to them, and their bytes stay in the licensed environments
+   consumers already operate. Public microdata bytes are staged in an
+   untracked, transient directory outside `db/data/**` during acquisition and
+   uploaded from there; a repository guard refuses tracked microdata bytes.
+   Git custody of a microdata release is always manifest-only.
+4. **No content.** No microdata record, row, column, row value, or cell enters
+   `source_records`, `source_rows`, `source_columns`, `source_row_values`,
+   `source_cells`, the relational registry, the derived-artifact bucket, or the
+   journal, whether the release is public or gated. No fact computed from raw
+   microdata by Chronicle or by a PolicyEngine-side consumer enters Chronicle,
+   however many intermediate artifacts stand between the microdata and the
+   value. The test is who asserted the value: a value that a third party
+   asserted and published, whether the microdata's own publisher (Census over
+   the ASEC) or another publisher (JCT or TPC over the IRS PUF, JRC over
+   EU-SILC), is an ordinary fact with ordinary provenance; a value Chronicle or
+   a PolicyEngine-side consumer computed is not. This is enforceable only once
+   facts carry an asserting principal and root-artifact lineage (`asserted_by`,
+   `root_artifacts`; chronicle#238): a fact rooted in a `microdata_release`
+   registration is then rejected unless `asserted_by` is a third-party
+   publisher, never Chronicle or a PolicyEngine-side consumer. Until then
+   reviewers enforce it by hand.
+5. **What a registration attests, and who.** Each registration records
+   `hash_source` and its attester:
+   - `chronicle_fetch`: Chronicle fetched the bytes and hashed them;
+     `attested_by: chronicle`, `verified_at` is the fetch date. The
+     registration attests the bytes.
+   - `consumer_attested`: a consumer that holds the bytes recomputed the hash
+     and recorded evidence; `attested_by` is that consumer,
+     `attestation_evidence` points at its record, `verified_at` is its
+     comparison date. The registration attests the bytes on that consumer's
+     word.
+   - `consumer_pin`: transcribed from a consumer's reviewed pin without
+     recomputation; `attested_by` is that consumer, `pinned_from` names the
+     repository path and commit, and there is no `verified_at`. The
+     registration attests the pin, not the bytes.
+   The witnessed journal (the release manifests on `codex/thesis-ledger-facts`,
+   each committing to the journal state that first covers a registration's
+   manifest hash) bounds when the registration existed. It bounds when the
+   bytes existed only for `chronicle_fetch` and `consumer_attested` entries.
+6. **Consumers point at the registration.** A Microcosm source-stage manifest
+   that names a microdata artifact carries the Chronicle artifact reference
+   and the same SHA-256, so every root of a build graph resolves to one
+   witnessed registration and a build fails closed when its local bytes
+   differ from the registered ones.
+
+## Why
+
+- **The publisher record is the thing to witness.** Publishers revise and
+  withdraw microdata files: the IRS withdrew the public-use file in 2026, and
+  Census reissues ASEC files under the same vintage label. A registered hash
+  with a witnessed time is the only durable statement that a given release
+  existed with those bytes. This is the same transparency property Chronicle
+  already provides for published tables, applied to the files calibration
+  actually starts from.
+- **Pins today are scattered and unwitnessed.** Microcosm pins raw inputs in
+  per-country manifests, checkpoint metadata, code constants, and command-line
+  arguments, with no shared registry, no licence record, and no timestamp
+  anyone outside the build can check. A build's root inputs deserve the same
+  declared identity as every other node.
+- **Content would break what makes the store useful.** Row-level microdata
+  would grow the relational registry and journal by orders of magnitude,
+  churn on every reissue, and put access-controlled bytes inside the one
+  system whose value is that anyone can verify it. Thesis resolves forecasts
+  against Chronicle observations; microdata are not observations of anything
+  Thesis scores.
+- **This keeps the 2026-06-30 ruling.** PR #68 removed microdata parsers,
+  adapters, and tracked raw storage from the package. Nothing here brings any
+  of that back. The code the enforcement PRs add refuses to read microdata
+  rather than reading it.
+
+## Consequences
+
+- `docs/adr-chronicle-facts-only.md` is amended: "anything a publisher
+  asserted is a fact" now excludes publisher-authored values at microdata
+  grain, which are content; only the values a publisher asserted and
+  published over that microdata are facts, whether the microdata's own
+  publisher or another. README's boundary block says the same.
+- Manifests gain `kind`, `licence`, `access`, `licence_evidence`,
+  `hash_source`, `attested_by`, and per-source `verified_at` /
+  `attestation_evidence` / `pinned_from`; validators refuse the combinations
+  decisions 1, 3, and 5 forbid (#227).
+- Fact provenance gains `asserted_by` and `root_artifacts` (#238).
+- `docs/storage-architecture.md` narrows its non-goal to the three clauses in
+  `chronicle/boundary.py`; its ownership matrix gains a row for microdata
+  releases and states that Git custody is manifest-only with untracked
+  staging. `docs/target-construction-harness-plan.md` is scoped to publisher
+  aggregate artifacts.
+- `chronicle/boundary.py` states the three negative clauses once;
+  `.github/chronicle-agents.yml` carries them verbatim in the
+  `ledger-boundary` judge verdict; `tests/test_chronicle_governance.py`
+  asserts each complete clause, so the wording cannot regress or invert.
+- Microcosm's raw-input entries reference Chronicle registrations
+  (PolicyEngine/microcosm#848).
+- Flip conditions for revisiting content: a consumer other than Microcosm
+  needs row-level evidence from Chronicle, or a publisher grants
+  redistribution of a currently licensed file and Chronicle has an
+  access-controlled tier designed for it. Neither holds today.

docs/architecture.md

@@ -16,7 +16,7 @@ validation, and build-harness layer. Jurisdiction source packages such as
 
 | Layer | Owns | Does not own |
 |-------|------|--------------|
-| Chronicle | Source artifacts, provenance, aggregate facts, constraints | Selection and measurement contracts, raw microdata storage, source reconciliation, aging, imputation, active target selection |
+| Chronicle | Source artifacts (including microdata release registrations), provenance, aggregate facts, constraints | Selection and measurement contracts, microdata records, rows, columns, row values, or cells; facts computed from raw microdata by Chronicle or a PolicyEngine-side consumer; licensed or restricted microdata bytes; source reconciliation, aging, imputation, active target selection |
 | Microcosm Targets | Selection and measurement contracts, reconciliation, aging, imputation, active target sets | Source artifact storage and provenance |
 | Microcosm | Entity model, weights, calibration interfaces, calibrated output | Source ETL and source provenance |
 | Jurisdiction source packages | Source-specific parsers and specs that emit Chronicle records | Forked fact or constraint schemas |