Skip to content

journal: kept in the configured store, resumed at boot (H3) - #702

Merged
MattJackson merged 18 commits into
predevfrom
lane-task-draft-687-journal-in-store
Oct 10, 2026
Merged

MattJackson merged 18 commits into
predevfrom
lane-task-draft-687-journal-in-store

Conversation

@MattJackson

Copy link
Copy Markdown
Collaborator

Supersedes #687. Carries #549 (lane-gb-loader-wal-edge, merged in at bb5bc0f); #549 is still open, so this PR lands it.

Audit root-R1 #3 (H3, ARCHITECT 2026-10-07 ruling (a)-(e)). On a node with no data directory, the money journal, the deployment keyset and the node identity were lost on every restart. The journal was memory-buffered and shipped to the store adapter's shim, which answered Ok to every batch and kept only a count. As a result, a signed back-dated amend_rate_history and a commit_upgrade could answer 200 and then vanish, and audit records, checkpoints and open-hold recovery were all lost.

What this PR does

  • (a) The journal is kept in the configured store. Each record goes through the store's v3 record_put under schema busbar.journal.v1, keyed by node, node_seq and part (big-endian). A record over 512 bytes crosses as parts. No new store slot is added.
  • (b) Boot resumes the chain. root::boot::book reads the chain back with record_scan and seeds a memory journal from it (Journal::memory_resumed, Wal::memory_seeded). Durability::keeps_chain gates everything rebuilt from a chain: the book, hold recovery, the audit cache and the /audit/range window, the checkpoint anchor, the dated rate-card history and the node amendment journal.
  • (c) Write-behind on one bounded lane owned by the host. JournalLane runs one worker thread and writes in chain order. A refused put stays at the head of the lane and is offered again with a growing pause. When the lane is full, the journal's next batch is turned away (Journal::retaining_at_bound), and a new money-bearing unit is answered 503 overloaded before anything is built for it.
  • (d) Durable admin verbs wait for the store. amend_rate_history and commit_upgrade answer 200 only after the store acknowledged their record. They wait at most 5 s, outside the book's lock. Otherwise they answer 503 with the reason.
  • (e) Prose in durability/mod.rs, boot.rs, store_adapter.rs, AmendmentJournal and keyset.rs is corrected.
  • Store adapter. The record-op Shipper shim is removed, along with ShimState::records_shipped, StoreAdapter::shipper/head/speaks_new_ops and STORE_ABI_WITH_NEW_OPS.
  • Keyset and node identity in the store.
    • The keyset is kept under busbar.keyset.v1, and node ids under busbar.node.v1, keyed by host name.
    • /admin/verify walks every chain the store keeps.
    • Racing first boots redeem redeem_plane_token single-use claims: keyset:deployment, host:<name> and node-id:<id>.
    • A minting claim lapses after 120 s, and a losing redemption cannot push the claim out.
    • Known limit: a host whose name changes on every boot mints a new node id each time. Its old chain stays in the store and is walked by verify, but its open holds are not recovered.

New in this round (merge of predev)

  • Conflicts. The three conflicting test files (root/tests/durability.rs, root/tests/plane_node.rs, units_admin/tests/units_admin.rs) keep both sides' cells. The cells predev added build data-dir books or test-only durability::build books. None of them calls root::boot::book / compose_book, so none needs the store's record slots.
  • kernel-audit: the record names its sealing node; resume rebuilds the heads; one canonicaliser; dead surface struck (Q128) #662 (audit record names its sealing node). AuditRecord.node was busbar_kernel::door::node(), a fresh CSPRNG draw per process. That is not the store-assigned id H3 keys the chain by, so one host's stored chain held records naming a different node on every boot. Fix: a book whose chain the store keeps (build_on_store with no data dir) now seals as the store-assigned node. A disk or store-less book still seals as door::node(). The verify walk accepts the store id (it digests node and does not compare it). The door::node doc says so.
  • money: a running driven unit's counts are journaled as unit.accrued checkpoints (K2-H1, R1-M9, R1-L14) #647 (unit.accrued checkpoints). A flushed checkpoint is Durability::checkpoint_accrual, which goes through journal_mark and then the journal's shipper, which is JournalLane. It is rebuilt by the same resumed-chain replay (build_inner, replay_into, then recover). A new cell restarts over the store with an open driven unit holding a flushed checkpoint.
  • Kind isolation. qa/kind-isolation.toml and xtask/src/gates/kind_isolation.rs cite no deleted item.
  • Deleted items. No reference to records_shipped, speaks_new_ops or STORE_ABI_WITH_NEW_OPS remains in crates/, xtask/, qa/ or docs/. BufferShipper and Wal::memory_buffered are not deleted by this branch: they remain the in-process journal for a book with no store behind it, and predev uses them too.

RED arm: each cell and the neutralisation that makes it fail

Cell Neutralisation → failure
root/tests/durability.rs a_restart_over_the_same_store_keeps_the_chain_the_audit_records_and_the_holds build_on_store passes no resume (journal opens as an empty memory_buffered_to) → restarted book has 0 audit records, no recovered hold, settled 0, numbering from 1
durability.rs a_data_unit_rides_a_refusing_store_and_its_record_lands_once_it_recovers the lane worker drops a refused put instead of keeping it at the head → the record never lands after refuse(false); the next boot does not read it back
durability.rs a_full_lane_refuses_new_money_and_drops_no_record retaining_at_bound removed (journal drops its oldest at the bound) → dropped_total() > 0 and the store is missing records
root/tests/plane_node.rs a_full_journal_lane_refuses_a_new_unit_503_with_the_reason the lane-health gate in Node::answer removed → the unit is admitted and its build runs (the build closure panics)
root/tests/kernel.rs a_back_dated_correction_survives_a_restart_over_the_store keeps_chain reduced to "has a data dir" (cards rebuilt only over a disk) → restarted history is the boot card alone; the window is not repriced
units_admin a_correction_on_a_node_with_no_data_dir_is_kept_by_the_store_and_survives_a_restart the resume neutralised as above → the restarted book replays no amendment
units_admin a_correction_the_store_refuses_answers_503_with_the_reason the verb's healthy() / wait_acked wait removed → 200 over a refusing store
units_admin commit_upgrade_answers_only_once_the_store_kept_its_record the same wait removed from commit_upgrade_effect → 200 over the refusing store
src/tests/tests.rs a_store_without_the_record_slots_refuses_the_boot_by_name root::boot::no_record_slots check removed → the boot opens a book over a store that keeps nothing
durability.rs a_lost_keyset_claim_with_nothing_kept_refuses_until_the_claim_lapses (lapsed claim) the claim answered true without redeeming, or the token not naming its period → the losing boot mints / the claim never lapses
durability.rs a_first_boot_that_lost_the_keyset_claim_takes_the_winners_key, a_first_boot_that_lost_the_host_claim_takes_the_winners_node_id, every_minted_node_id_is_held_by_its_claim claim answered true without redeeming → two keys / two ids for one host / the id's claim is unredeemed
durability.rs a_restart_over_the_same_store_verifies_the_prior_boots_records keyset minted per process instead of read from busbar.keyset.v1 → every resumed record is a verify finding
durability.rs two_nodes_on_one_store_never_collide_and_verify_walks_both_chains node id fixed at 0 → the second node's records overwrite the first's
new durability.rs a_store_kept_chain_seals_every_record_as_the_store_assigned_node sealing_as(busbar_kernel::door::node()) unconditionally (the predev #662 behaviour) → record.node() != node; records name the process draw
new durability.rs a_driven_unit_killed_mid_stream_over_the_store_is_recovered_at_its_flushed_checkpoint the resume neutralised as above → recovered_holds == 0, no recovered posting
busbar-kernel-wal memory_resumed / retaining_at_bound cells Wal::memory_seeded not seeding / retaining_at_bound a no-op → replay empty / oldest dropped

The test double is the in-tree StoreCalls over a map (root/tests/store_double.rs), not a new plugin. This PR's CI is the proof of record. No local build was run.

Customer-visible behaviour: the llm and core request paths are byte-identical to 1.5.5 except in one new state. When the store keeps the journal and the lane to it is full, a new unit answers 503 overloaded with the reason; 1.5.5 kept no journal. The 503 from amend_rate_history and commit_upgrade is new with those verbs, which are themselves new in 1.6.0.

…nd the shipping seam move

The shipping seam the store adapter implements and the log ships through moves into
busbar_contract::ship: Shipper<R> and ShipError, over ShippedRecord (a record's identity
alone). The log's Record and its framing stay in busbar-kernel-wal; Record implements
ShippedRecord, and the log's own shippers and boxes name Shipper<Record>.

durable.rs and its tests move unchanged from busbar-kernel-wal (which never used it) into
busbar-plugin-loader, host-only machinery no plugin image links, so its temp-name counter
stays one per process. busbar-kernel re-exports it from the loader, so every
crate::durable / busbar_kernel::durable call site is unchanged, and the root's keyset
names busbar_kernel::durable (the root names the loader only in root/loader.rs). The structure-lint choke-point owner and class test follow the file.

crates/plugin-loader/Cargo.toml now names no busbar-kernel-* crate (Part 2 #33: plugin
infrastructure is one crate atop the one contract crate). The dead
busbar-plugin-loader -> busbar-kernel-wal [[dep]] row is struck, and both kind-isolation
gates gain a RED plant that grows a loader -> busbar-kernel-ledger edge back: the per-push
gate refuses it as unlisted-dep-edge, the ship twin as a tcb ship-edge.
…red store and resumes it at boot

The root's journal now writes each record through the configured store's v3 record_put slot under
the journal schema, keyed (node, node_seq, part) big-endian so record_scan returns chain order, on a
host-owned bounded write-behind lane that never calls the store under the book's lock. With no data
directory the boot reads the chain back (record_scan) and resumes it (Journal::memory_resumed), and
the dated rate-card history, the amendment journal and the older audit window rebuild whenever the
chain has a durable source (keeps_chain), not only on a disk.

A refused record stays at the head of the lane and is re-offered; a full lane turns the journal's
batch away (the log retains it) and the node fails closed: a new money-bearing unit answers 503 with
the reason, and the journal keeps every record at its bound (retaining_at_bound). amend_rate_history
and commit_upgrade answer 200 only once the store acknowledged their record, 503 with the reason
otherwise. A configured store with no record slots refuses the boot by name.

The store adapter's shipping shim (acknowledged every batch, kept a count) and the stale
STORE_ABI_WITH_NEW_OPS constant are removed, with the prose that described them.

ARCHITECT 2026-10-07 H3 ruling (a)-(e); BUSBAR-1.6.0.md THE DESIGN §7, §11.2, §11.11 R4; Appendix B
Q-STORE = (B).
…he configured store

With no data directory the deployment keyset is minted once into the store's record slots
(busbar.keyset.v1), read back on every boot and by every node on that store, and its fingerprint
is sealed in a Bootstrap record on each chain; a chain whose Bootstrap the store cannot yield
refuses the boot. With a data directory the store supplies the key a first boot adopts, and a
minted key is kept there too. A resumed chain now verifies under the key the boot holds.

Each node takes a stable id from the store's node registry (busbar.node.v1, keyed by host name),
minted on the host's first boot and read back on every later one; journal records are keyed by
it instead of 0, so two nodes on one store never overwrite each other. A data directory's chain
keeps the identity it was written under (Journal::in_directory_adopting). There is no node_id
config key: adding one changes the frozen config-schema snapshot.

/admin/verify walks every chain the store keeps (every registered node and this one), verifying
each journal chain and its audit records, signatures included, against the deployment keyset.

ARCHITECT 2026-10-07 H3 ruling, follow-up (1)-(2).
…he store's single-use claim first

The store ABI has no compare-and-set, so two first boots racing on an empty store could each keep
their own keyset (two signing keys for one deployment) or their own node id for one host (two chains
for one host). Its one atomic step is redeem_plane_token, a single-use test-and-set; minting now
redeems a claim on it first, as oauth_as's store does (ARCHITECT 2026-10-07 H3 ruling).

- keep_keyset: the boot the store answers first on `keyset:deployment` mints and keeps; every other
  boot waits for the winner's write and takes it, and refuses if nothing was kept in time.
- node_id: one minter per host (`host:<name>`), and each minted id is claimed for good
  (`node-id:<id>`) before it is kept, so no two hosts hold one id.
- A minting claim lapses after 120 s, so a winner that died before it kept anything strands the
  store for at most that long; a later boot claims again.

Known limit, documented in store_identity: a host whose name changes on every restart mints a new
id each boot, and its predecessor's chain is walked and verified but its open holds are not
recovered.

RED: a_first_boot_that_lost_the_keyset_claim_takes_the_winners_key,
a_lost_keyset_claim_with_nothing_kept_refuses_until_the_claim_lapses,
a_first_boot_that_lost_the_host_claim_takes_the_winners_node_id,
every_minted_node_id_is_held_by_its_claim (each fails with the claim answered without redeeming).
…e recovery-verb cell's memory governance carries the record slots H3's boot book keeps its journal in
… pass, so a boot crossing a second boundary never finds it lapsed early
…n cannot push it out (a store may re-stamp a token's expiry on every redemption)
…' cells; durability gains two restart cells over the store (a flushed driven-unit checkpoint is recovered, records seal as the store-assigned node)
…node id the store assigned the host (the id the chain is keyed by and /admin/verify walks), not the per-process op-id node

Supersedes #687.
@MattJackson
MattJackson enabled auto-merge October 9, 2026 21:00
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

promote into predev: BOARD @4e2c10ed7: 0 failing test row(s), 11 DENY row(s)

DENY rows (11)

gate row detail
construction one-pick-site 3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8
kind-isolation kind-isolation:deps 3 finding(s), 95 shipped edge instance(s) over 31 class(es), 95 declaration(s); 58 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge legacy -> plane busbar-llm -> busbar-plane-llm is a shipped edge w
kind-isolation kind-isolation:test-deps 3 finding(s), 35 test edge instance(s) over 21 class(es), 35 declaration(s); 20 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge
kind-isolation kind-isolation:law0 18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f
kind-isolation-ship kind-isolation:deps 12 finding(s) over 95 shipped edge(s): ship-edge kernel -> hooks busbar-kernel -> busbar-hook-ranking is 'not-allowed': the architecture grants no kernel -> hooks edge, and the ship criterion is the a
kind-isolation-ship kind-isolation:test-deps 10 finding(s) over 35 test edge(s): ship-edge cleanliness -> export busbar-core-admin -> busbar-export-prometheus is 'not-allowed': the architecture grants no cleanliness -> export edge, and the ship
kind-isolation-ship kind-isolation:law0 18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f
kind-isolation-ship kind-isolation:faces 2 finding(s) over 40 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Transport' 1 time(s) in shipped source — the entry face of kind 'transport'. A trait implementation is
kind-isolation-ship kind-isolation:legacy-drain 3 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier)
ship-ready ship-ready:ship-twin 'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is
structure-lint structure-lint:plane-dup:unledgered 22 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs voice:crates/busbar-voice/src/config.rs (the ledger

Judged against base 92c3f63cc: 0 new red, 0 worse, 5 standing (excused).

tests passed: 24063, failed: 0. Run: https://github.com/GetBusbar/busbar/actions/runs/38018847481 . Artifact verdict-4e2c10ed79a22c897f0045dc196703ae68285ed2 (failures.json, junit.xml, raw.log; 90 days).

@MattJackson MattJackson added fixing and removed fixing labels Oct 9, 2026
…of the shipper seam (the adapter is no shipper; the journal lives in the store), so predev's two shipper cells go with it; predev's no-silent-skip adapter constructor is kept
@MattJackson MattJackson added fixing and removed fixing labels Oct 10, 2026
…hat the ratchets measured

- plane_node: drive_borrowed now returns Option<Outcome> on predev, so the full-lane refusal
  returns None (the table did not take the unit), not false (E0308).
- durability: ACK_DEADLINE is re-exported only with root-admin (its one caller is units_admin),
  walk_stored_chains only with root-admin or test; unused imports under no-default-features and
  the single-plane builds.
- abi-location:ledger: the branch drained STORE_ABI_WITH_NEW_OPS from store_adapter.rs; its stale
  row is struck (also the xtask selftest abi-location row).
- root_key_spelling: JournalLane's Debug label was the quoted root key "store" in an unarmed file;
  it reads "kept_in" now.
- kind-isolation:law0 (18 -> 20): the new tests named the lane's store label "memory", a store
  instance name (busbar x instance:store 2 -> 14); the label is "test-store". And moving
  durable.rs and its tests to the loader drained busbar-kernel-wal x instance:secret to 0, so that
  row is struck. The figure is back to the base's 18.

Proved on Latchkey: gate kind-isolation (law0 at 18), gate abi-location (green),
selftest abi-location, test -p busbar-kernel --test root_key_spelling, test -p busbar full,
check -p busbar --no-default-features (no warnings).
@MattJackson
MattJackson added this pull request to the merge queue Oct 10, 2026
Merged via the queue into predev with commit 78ccc1e Oct 10, 2026
8 checks passed
@MattJackson
MattJackson deleted the lane-task-draft-687-journal-in-store branch October 10, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant