Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
d9c3199
Add one shared chronicle-first env helper
MaxGhenis Sep 2, 2026
e2e0af0
Make R2 buckets configurable and emit chronicle.db
MaxGhenis Sep 2, 2026
de03d37
Add hermetic tests for the rename window
MaxGhenis Sep 2, 2026
08d950b
Document the rename window and the bucket cutover
MaxGhenis Sep 2, 2026
d032ec5
Pin the boundary guard against the bucket rename
MaxGhenis Sep 2, 2026
b8d717d
Isolate the rename window for every test, not one module
MaxGhenis Sep 2, 2026
5cb357a
Refuse to attach a recorded R2 URI to different bytes
MaxGhenis Sep 2, 2026
58706dd
Document publisher revisions alongside the bucket rename
MaxGhenis Sep 2, 2026
0e7e095
Record the gate round-1 fixes in PROGRESS.md
MaxGhenis Sep 2, 2026
83b8ba0
Read the recorded object's identity from its URI too
MaxGhenis Sep 2, 2026
45b7d25
Pin the preserved block's field order too
MaxGhenis Sep 2, 2026
8cf2886
Say what the reproduction actually ran
MaxGhenis Sep 2, 2026
c074341
Record the round-2 gate findings and the corpus scan
MaxGhenis Sep 2, 2026
5e41b75
Reject the chronicle spelling of the derived-row marker
MaxGhenis Sep 2, 2026
fa4a82f
Let CHRONICLE_SCHEMA configure the mirror writer, at call time
MaxGhenis Sep 2, 2026
e68fdd0
Address the manifest a fetch is actually revising, and read it strictly
MaxGhenis Sep 2, 2026
ad828b1
Document identity, manifest selection and locator checks
MaxGhenis Sep 2, 2026
7b68988
Name the strict storage reader for the pair it belongs to
MaxGhenis Sep 2, 2026
272283e
Record what each round-2 fix does and how it was reproduced
MaxGhenis Sep 2, 2026
7f628d1
Say which fields a recorded block actually needs
MaxGhenis Sep 2, 2026
0605c47
Say what --record-revision is an opt-in over
MaxGhenis Sep 2, 2026
3abacef
Say when the manifest guards actually run
MaxGhenis Sep 2, 2026
f6bce08
Drop a blank line ruff format rejects after the rebase onto #228
MaxGhenis Sep 4, 2026
73a1826
Gate round: manifest-name refusals reach the CLI, files blocks are va…
MaxGhenis Sep 4, 2026
d911e1f
Record into a fresh mapping when a manifest's files block is an expli…
MaxGhenis Sep 4, 2026
2e8aae0
Start Sol gate round progress log
MaxGhenis Sep 4, 2026
7c06b1a
Refuse unsafe fetch manifest rewrites
MaxGhenis Sep 4, 2026
19f0cf2
Sweep every package manifest safely
MaxGhenis Sep 4, 2026
4042819
Validate R2 routes before cutover skips
MaxGhenis Sep 4, 2026
c218dcc
Restore Supabase schema compatibility aliases
MaxGhenis Sep 4, 2026
701712c
Make the Supabase cutover instructions truthful
MaxGhenis Sep 4, 2026
2881a8c
Complete strict artifact preflight coverage
MaxGhenis Sep 4, 2026
0c04260
Record Sol gate final verification
MaxGhenis Sep 4, 2026
8688cbd
Start eight-finding Sol gate progress log
MaxGhenis Sep 4, 2026
7cabf11
Reproduce duplicate manifest key loss
MaxGhenis Sep 4, 2026
a430d2d
Read artifact manifests with strict YAML loader
MaxGhenis Sep 4, 2026
a49e1f4
Record strict manifest loader progress
MaxGhenis Sep 4, 2026
82d05b4
Reproduce artifact and manifest path escapes
MaxGhenis Sep 4, 2026
f2432cd
Harden artifact and manifest path boundaries
MaxGhenis Sep 4, 2026
403e500
Record path-boundary fixes
MaxGhenis Sep 4, 2026
b0d29b7
Reproduce stale shared-file revision owners
MaxGhenis Sep 4, 2026
215cc4f
Update every owner of a revised shared artifact
MaxGhenis Sep 4, 2026
20b4028
Record shared revision ownership fix
MaxGhenis Sep 4, 2026
a644be8
Reproduce tracked registry cutover failures
MaxGhenis Sep 4, 2026
eb4d43a
Preserve compatible historical R2 routes
MaxGhenis Sep 4, 2026
ec902b0
Record cutover compatibility fix
MaxGhenis Sep 4, 2026
5f55184
Reproduce malformed revision history loss
MaxGhenis Sep 4, 2026
9e26ca5
Refuse malformed revision history before I/O
MaxGhenis Sep 4, 2026
f5efc50
Record revision history validation
MaxGhenis Sep 4, 2026
27909da
Reproduce frozen Supabase schema aliases
MaxGhenis Sep 4, 2026
cdc7968
Honor schema env in compatibility exports
MaxGhenis Sep 4, 2026
9060f18
Record schema compatibility fix
MaxGhenis Sep 4, 2026
1096ace
Reproduce residual artifact boundary escapes
MaxGhenis Sep 4, 2026
f2fc086
Close residual artifact path boundaries
MaxGhenis Sep 4, 2026
26d22e4
Record residual path boundary hardening
MaxGhenis Sep 4, 2026
619d5cd
Preflight complete packages before raw publish
MaxGhenis Sep 4, 2026
3db6c85
Reproduce source-package artifact path escapes
MaxGhenis Sep 4, 2026
531de1c
Reproduce normalized source artifact alias
MaxGhenis Sep 4, 2026
70b2d69
Reproduce manifest-named source artifact read
MaxGhenis Sep 4, 2026
bd49f52
Reproduce source-package manifest path escapes
MaxGhenis Sep 4, 2026
f431ae7
Harden source-package artifact paths
MaxGhenis Sep 4, 2026
446bd17
Reproduce residual artifact side effects
MaxGhenis Sep 4, 2026
6f74c4f
Preflight artifact operations before side effects
MaxGhenis Sep 4, 2026
c4f59fe
Record artifact preflight verification
MaxGhenis Sep 4, 2026
7e62a1e
Record final eight-finding verification
MaxGhenis Sep 4, 2026
29336c2
Reproduce Sol round-3 findings: identity segments, alias enumeration,…
MaxGhenis Sep 4, 2026
768bed8
Close Sol round 3: canonical identity segments, alias enumeration, re…
MaxGhenis Sep 4, 2026
0981354
docs: start peer round 4 fix journal
MaxGhenis Sep 4, 2026
317f051
test: reproduce publication preflight and identity findings
MaxGhenis Sep 4, 2026
c88b8c7
fix: validate immutable R2 provenance in source artifact loading
MaxGhenis Sep 4, 2026
1727737
fix: require exact canonical manifest identity declarations
MaxGhenis Sep 4, 2026
12ef53d
fix: validate logical vintages in every manifest consumer
MaxGhenis Sep 4, 2026
ce083e7
fix: refuse facts citing configured derived storage routes
MaxGhenis Sep 4, 2026
8bb8219
fix: refuse nonregular source package resources before opening
MaxGhenis Sep 4, 2026
2c683a4
fix: report nonregular manifest siblings through shared errors
MaxGhenis Sep 4, 2026
2f7cb31
test: exercise sibling refusal through the top-level CLI
MaxGhenis Sep 4, 2026
4cecc72
test: cover locator validation and remaining publication routes
MaxGhenis Sep 4, 2026
1c3a223
fix: preflight the complete derived tree before publication
MaxGhenis Sep 4, 2026
b223066
fix: refuse normalized artifact aliases in publication and inventory
MaxGhenis Sep 4, 2026
1570fec
fix: validate new publication identities before artifact reads
MaxGhenis Sep 4, 2026
b7c88a1
fix: validate recorded R2 identity before counting inventory links
MaxGhenis Sep 4, 2026
9a6345f
fix: bind custom derived publication to recognized provenance routes
MaxGhenis Sep 4, 2026
62cc351
fix: recognize derived R2 provenance regardless of URI scheme case
MaxGhenis Sep 4, 2026
b3511ed
test: cover noncanonical report identities and namespace escapes
MaxGhenis Sep 4, 2026
b87231f
fix: serialize raw identity refusals without weakening validation
MaxGhenis Sep 4, 2026
20663fb
fix: keep publication vintage and build segments inside their routes
MaxGhenis Sep 4, 2026
d6d9fad
docs: record final focused checks and full-suite restart
MaxGhenis Sep 4, 2026
a00e274
docs: record completed peer round 4 verification
MaxGhenis Sep 4, 2026
ba8147a
Pin the cutover sweep test by its invariants, not today's registry co…
MaxGhenis Sep 4, 2026
49a3ad7
Reproduce Astra round-1 findings: legacy -derived routes accepted; YA…
MaxGhenis Sep 4, 2026
e72f194
Keep the legacy -derived route rule beside configured routes; identif…
MaxGhenis Sep 4, 2026
ff99d83
Judge duplicate manifest keys before YAML merges expand
MaxGhenis Sep 4, 2026
9379a54
Expand YAML merges without mutating nodes and validate every merged m…
MaxGhenis Sep 5, 2026
4be3854
test: pin YAML merge-sequence precedence for artifact selection
MaxGhenis Sep 5, 2026
c41567e
fix(registration): restore merge-sequence precedence in the strict lo…
MaxGhenis Sep 5, 2026
3cc4738
test: reproduce recursive YAML merges and a false shared-file collision
MaxGhenis Sep 5, 2026
9681c1c
fix: refuse recursive merges; compare effective identities across man…
MaxGhenis Sep 5, 2026
d62fbe9
test: reproduce false collisions from selected and partially failed p…
MaxGhenis Sep 5, 2026
cd14372
fix: unidentified sibling entries are not collisions; guard their byt…
MaxGhenis Sep 5, 2026
9f7cdf2
test: reproduce first fetch of a predeclared entry without identity
MaxGhenis Sep 5, 2026
7d9fd06
fix(artifacts): let a first fetch initialize the predeclared entry it…
MaxGhenis Sep 5, 2026
2938b2a
test: reproduce selected publication overrides leaking onto sibling m…
MaxGhenis Sep 5, 2026
d544ffe
fix(artifacts): apply publish identity overrides to the selected mani…
MaxGhenis Sep 5, 2026
b3251a3
test: reproduce a default sweep withholding overrides from a selected…
MaxGhenis Sep 5, 2026
3eea393
fix(artifacts): decide publish override eligibility from the whole se…
MaxGhenis Sep 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
783 changes: 741 additions & 42 deletions PROGRESS.md

Large diffs are not rendered by default.

61 changes: 44 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,12 @@ contract that aligns it to another period (see
| Microcosm Target Contracts | Selection, measurement bindings, and active subset | Period alignment, support-aware activation, solver inputs, diagnostics |

The storage split is documented in
[`docs/storage-architecture.md`](docs/storage-architecture.md): `ledger-raw`
stores immutable source bytes, `ledger-derived` stores reproducible build
[`docs/storage-architecture.md`](docs/storage-architecture.md): a raw R2 archive
stores immutable source bytes, a derived R2 archive stores reproducible build
artifacts, and Supabase/Postgres hosts the queryable relational Chronicle registry
mirrored from accepted builds.
mirrored from accepted builds. The bucket names are configuration
(`$CHRONICLE_R2_RAW_BUCKET` and `$CHRONICLE_R2_DERIVED_BUCKET`), still defaulting
to the ledger-era `ledger-raw` and `ledger-derived`.

## Repository Model

Expand Down Expand Up @@ -247,7 +249,7 @@ This writes:
source_regions.jsonl
facts.jsonl
consumer_facts.jsonl
ledger.db
chronicle.db
reports/
source_rows.json
source_cells.json
Expand Down Expand Up @@ -349,11 +351,14 @@ needed, even when your Cloudflare user belongs to several accounts:
# One-time per machine (opens a browser consent page):
bunx wrangler login

# One-time per account (already done for the PolicyEngine account):
uv run chronicle bootstrap-r2 --raw-bucket ledger-raw --derived-bucket ledger-derived
# One-time per account (already done for the PolicyEngine account). The bucket
# flags default to $CHRONICLE_R2_RAW_BUCKET / $CHRONICLE_R2_DERIVED_BUCKET:
uv run chronicle bootstrap-r2

# Fetch/register a source artifact, write db/data/.../manifest.yaml, and upload
# the exact bytes to R2 when Wrangler is authenticated:
# the exact bytes to R2 when Wrangler is authenticated. Pass --manifest when the
# package directory keeps more than one manifest (ira_contributions keeps a
# traditional and a Roth one):
uv run chronicle fetch-artifact \
--url https://www.irs.gov/pub/irs-soi/23in12ms.xls \
--source-id irs_soi \
Expand All @@ -364,11 +369,20 @@ uv run chronicle fetch-artifact \
--table "Publication 1304 Table 1.2" \
--upload-r2

# Re-fetching is safe: identical bytes keep the recorded storage.r2 block, and
# bytes that disagree with what the entry identifies -- its declared sha256, or
# its recorded content-addressed key once published -- are refused. When a
# publisher has re-published
# under the same URL and vintage, register the revision explicitly — the new
# bytes get their own content-addressed key and the superseded object is kept
# in storage.previous_r2:
uv run chronicle fetch-artifact ... --record-revision

# Audit local manifests and checksums:
uv run chronicle inventory-artifacts --root db/data

# Upload all existing manifest-declared local artifacts to ledger-raw and write
# storage.r2 metadata back into the manifests:
# Upload all existing manifest-declared local artifacts to the raw archive and
# write storage.r2 metadata back into the manifests:
uv run chronicle publish-raw --root db/data
```

Expand Down Expand Up @@ -406,10 +420,10 @@ To prepare the deterministic SQLite artifact for a hosted Supabase/Postgres
mirror, export each relational table to JSONL plus a manifest:

```bash
uv run chronicle export-db-tables --db /tmp/chronicle-suite/ledger.db --out /tmp/chronicle-mirror --replace
uv run chronicle export-db-tables --db /tmp/chronicle-suite/chronicle.db --out /tmp/chronicle-mirror --replace
```

To publish the deterministic build outputs to the `ledger-derived` R2 bucket:
To publish the deterministic build outputs to the derived R2 archive:

```bash
uv run chronicle publish-derived \
Expand All @@ -420,23 +434,36 @@ uv run chronicle publish-derived \
--build-artifacts-out /tmp/chronicle-build-artifacts.jsonl
```

The Supabase schema for this mirror lives at
`supabase/migrations/20260504_chronicle_bronze.sql`. Raw government spreadsheets are
mirrored as artifact metadata plus one row per parsed cell, not one tidy table
per sheet. Chronicle does not host raw survey microdata tables.
Before loading, create and apply a Supabase/Postgres migration that creates the
mirror tables in the schema selected for the load, then expose that schema
through the Supabase Data API. Raw government spreadsheets are mirrored as
artifact metadata plus one row per parsed cell, not one tidy table per sheet.
Chronicle does not host raw survey microdata tables.

After the migration is applied and the `chronicle` schema is exposed through the
Supabase Data API, accepted mirror exports can be upserted with:
After that deployment migration is applied, accepted mirror exports can be
upserted with:

```bash
uv run chronicle load-supabase-mirror \
--dir /tmp/chronicle-mirror \
--build-artifacts /tmp/chronicle-build-artifacts.jsonl
```

With no schema environment override and no `--schema`, this command
writes to `ledger`. To load a migrated `chronicle` schema instead, set
`CHRONICLE_SCHEMA=chronicle` or pass `--schema chronicle`.

Use `--dry-run` first to validate JSONL row counts and file coverage without
writing to Supabase.

Chronicle settings are read chronicle-first: `CHRONICLE_X` wins, and the
ledger-era `POLICYENGINE_LEDGER_X` and `LEDGER_X` spellings still work behind a
one-time deprecation warning naming the variable to move to.
[`docs/storage-architecture.md`](docs/storage-architecture.md#environment-variable-rename-window)
lists every variable in that window, and
[Bucket Cutover](docs/storage-architecture.md#bucket-cutover) covers the R2
bucket rename.

Chronicle facts keep source concepts and canonical concepts separately. For example,
the SOI Table 1.1 adjusted gross income column is preserved as
`irs_soi.adjusted_gross_income`, while the canonical concept is
Expand Down
1 change: 1 addition & 0 deletions chronicle/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"consumer_contract",
"core",
"database",
"env",
"facts",
"harness",
"jurisdictions",
Expand Down
Loading
Loading