Repository navigation
journal: kept in the configured store, resumed at boot (H3) - #702
Merged
Merged
Conversation
…nd the shipping seam move The shipping seam the store adapter implements and the log ships through moves into busbar_contract::ship: Shipper<R> and ShipError, over ShippedRecord (a record's identity alone). The log's Record and its framing stay in busbar-kernel-wal; Record implements ShippedRecord, and the log's own shippers and boxes name Shipper<Record>. durable.rs and its tests move unchanged from busbar-kernel-wal (which never used it) into busbar-plugin-loader, host-only machinery no plugin image links, so its temp-name counter stays one per process. busbar-kernel re-exports it from the loader, so every crate::durable / busbar_kernel::durable call site is unchanged, and the root's keyset names busbar_kernel::durable (the root names the loader only in root/loader.rs). The structure-lint choke-point owner and class test follow the file. crates/plugin-loader/Cargo.toml now names no busbar-kernel-* crate (Part 2 #33: plugin infrastructure is one crate atop the one contract crate). The dead busbar-plugin-loader -> busbar-kernel-wal [[dep]] row is struck, and both kind-isolation gates gain a RED plant that grows a loader -> busbar-kernel-ledger edge back: the per-push gate refuses it as unlisted-dep-edge, the ship twin as a tcb ship-edge.
…edge # Conflicts: # Cargo.lock
…red store and resumes it at boot The root's journal now writes each record through the configured store's v3 record_put slot under the journal schema, keyed (node, node_seq, part) big-endian so record_scan returns chain order, on a host-owned bounded write-behind lane that never calls the store under the book's lock. With no data directory the boot reads the chain back (record_scan) and resumes it (Journal::memory_resumed), and the dated rate-card history, the amendment journal and the older audit window rebuild whenever the chain has a durable source (keeps_chain), not only on a disk. A refused record stays at the head of the lane and is re-offered; a full lane turns the journal's batch away (the log retains it) and the node fails closed: a new money-bearing unit answers 503 with the reason, and the journal keeps every record at its bound (retaining_at_bound). amend_rate_history and commit_upgrade answer 200 only once the store acknowledged their record, 503 with the reason otherwise. A configured store with no record slots refuses the boot by name. The store adapter's shipping shim (acknowledged every batch, kept a count) and the stale STORE_ABI_WITH_NEW_OPS constant are removed, with the prose that described them. ARCHITECT 2026-10-07 H3 ruling (a)-(e); BUSBAR-1.6.0.md THE DESIGN §7, §11.2, §11.11 R4; Appendix B Q-STORE = (B).
…he configured store With no data directory the deployment keyset is minted once into the store's record slots (busbar.keyset.v1), read back on every boot and by every node on that store, and its fingerprint is sealed in a Bootstrap record on each chain; a chain whose Bootstrap the store cannot yield refuses the boot. With a data directory the store supplies the key a first boot adopts, and a minted key is kept there too. A resumed chain now verifies under the key the boot holds. Each node takes a stable id from the store's node registry (busbar.node.v1, keyed by host name), minted on the host's first boot and read back on every later one; journal records are keyed by it instead of 0, so two nodes on one store never overwrite each other. A data directory's chain keeps the identity it was written under (Journal::in_directory_adopting). There is no node_id config key: adding one changes the frozen config-schema snapshot. /admin/verify walks every chain the store keeps (every registered node and this one), verifying each journal chain and its audit records, signatures included, against the deployment keyset. ARCHITECT 2026-10-07 H3 ruling, follow-up (1)-(2).
…he store's single-use claim first The store ABI has no compare-and-set, so two first boots racing on an empty store could each keep their own keyset (two signing keys for one deployment) or their own node id for one host (two chains for one host). Its one atomic step is redeem_plane_token, a single-use test-and-set; minting now redeems a claim on it first, as oauth_as's store does (ARCHITECT 2026-10-07 H3 ruling). - keep_keyset: the boot the store answers first on `keyset:deployment` mints and keeps; every other boot waits for the winner's write and takes it, and refuses if nothing was kept in time. - node_id: one minter per host (`host:<name>`), and each minted id is claimed for good (`node-id:<id>`) before it is kept, so no two hosts hold one id. - A minting claim lapses after 120 s, so a winner that died before it kept anything strands the store for at most that long; a later boot claims again. Known limit, documented in store_identity: a host whose name changes on every restart mints a new id each boot, and its predecessor's chain is walked and verified but its open holds are not recovered. RED: a_first_boot_that_lost_the_keyset_claim_takes_the_winners_key, a_lost_keyset_claim_with_nothing_kept_refuses_until_the_claim_lapses, a_first_boot_that_lost_the_host_claim_takes_the_winners_node_id, every_minted_node_id_is_held_by_its_claim (each fails with the claim answered without redeeming).
…28-h3-journal-store
…e recovery-verb cell's memory governance carries the record slots H3's boot book keeps its journal in
… pass, so a boot crossing a second boundary never finds it lapsed early
…n cannot push it out (a store may re-stamp a token's expiry on every redemption)
…' cells; durability gains two restart cells over the store (a flushed driven-unit checkpoint is recovered, records seal as the store-assigned node)
…node id the store assigned the host (the id the chain is keyed by and /admin/verify walks), not the per-process op-id node Supersedes #687.
MattJackson
enabled auto-merge
October 9, 2026 21:00
promote into
|
| gate | row | detail |
|---|---|---|
| construction | one-pick-site |
3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8 |
| kind-isolation | kind-isolation:deps |
3 finding(s), 95 shipped edge instance(s) over 31 class(es), 95 declaration(s); 58 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge legacy -> plane busbar-llm -> busbar-plane-llm is a shipped edge w |
| kind-isolation | kind-isolation:test-deps |
3 finding(s), 35 test edge instance(s) over 21 class(es), 35 declaration(s); 20 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge |
| kind-isolation | kind-isolation:law0 |
18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f |
| kind-isolation-ship | kind-isolation:deps |
12 finding(s) over 95 shipped edge(s): ship-edge kernel -> hooks busbar-kernel -> busbar-hook-ranking is 'not-allowed': the architecture grants no kernel -> hooks edge, and the ship criterion is the a |
| kind-isolation-ship | kind-isolation:test-deps |
10 finding(s) over 35 test edge(s): ship-edge cleanliness -> export busbar-core-admin -> busbar-export-prometheus is 'not-allowed': the architecture grants no cleanliness -> export edge, and the ship |
| kind-isolation-ship | kind-isolation:law0 |
18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f |
| kind-isolation-ship | kind-isolation:faces |
2 finding(s) over 40 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Transport' 1 time(s) in shipped source — the entry face of kind 'transport'. A trait implementation is |
| kind-isolation-ship | kind-isolation:legacy-drain |
3 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier) |
| ship-ready | ship-ready:ship-twin |
'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is |
| structure-lint | structure-lint:plane-dup:unledgered |
22 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs voice:crates/busbar-voice/src/config.rs (the ledger |
Judged against base 92c3f63cc: 0 new red, 0 worse, 5 standing (excused).
tests passed: 24063, failed: 0. Run: https://github.com/GetBusbar/busbar/actions/runs/38018847481 . Artifact verdict-4e2c10ed79a22c897f0045dc196703ae68285ed2 (failures.json, junit.xml, raw.log; 90 days).
…of the shipper seam (the adapter is no shipper; the journal lives in the store), so predev's two shipper cells go with it; predev's no-silent-skip adapter constructor is kept
…hat the ratchets measured - plane_node: drive_borrowed now returns Option<Outcome> on predev, so the full-lane refusal returns None (the table did not take the unit), not false (E0308). - durability: ACK_DEADLINE is re-exported only with root-admin (its one caller is units_admin), walk_stored_chains only with root-admin or test; unused imports under no-default-features and the single-plane builds. - abi-location:ledger: the branch drained STORE_ABI_WITH_NEW_OPS from store_adapter.rs; its stale row is struck (also the xtask selftest abi-location row). - root_key_spelling: JournalLane's Debug label was the quoted root key "store" in an unarmed file; it reads "kept_in" now. - kind-isolation:law0 (18 -> 20): the new tests named the lane's store label "memory", a store instance name (busbar x instance:store 2 -> 14); the label is "test-store". And moving durable.rs and its tests to the loader drained busbar-kernel-wal x instance:secret to 0, so that row is struck. The figure is back to the base's 18. Proved on Latchkey: gate kind-isolation (law0 at 18), gate abi-location (green), selftest abi-location, test -p busbar-kernel --test root_key_spelling, test -p busbar full, check -p busbar --no-default-features (no warnings).
…-journal-in-store
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #687. Carries #549 (
lane-gb-loader-wal-edge, merged in at bb5bc0f); #549 is still open, so this PR lands it.Audit root-R1 #3 (H3, ARCHITECT 2026-10-07 ruling (a)-(e)). On a node with no data directory, the money journal, the deployment keyset and the node identity were lost on every restart. The journal was memory-buffered and shipped to the store adapter's shim, which answered Ok to every batch and kept only a count. As a result, a signed back-dated
amend_rate_historyand acommit_upgradecould answer 200 and then vanish, and audit records, checkpoints and open-hold recovery were all lost.What this PR does
record_putunder schemabusbar.journal.v1, keyed by node, node_seq and part (big-endian). A record over 512 bytes crosses as parts. No new store slot is added.root::boot::bookreads the chain back withrecord_scanand seeds a memory journal from it (Journal::memory_resumed,Wal::memory_seeded).Durability::keeps_chaingates everything rebuilt from a chain: the book, hold recovery, the audit cache and the/audit/rangewindow, the checkpoint anchor, the dated rate-card history and the node amendment journal.JournalLaneruns one worker thread and writes in chain order. A refused put stays at the head of the lane and is offered again with a growing pause. When the lane is full, the journal's next batch is turned away (Journal::retaining_at_bound), and a new money-bearing unit is answered 503overloadedbefore anything is built for it.amend_rate_historyandcommit_upgradeanswer 200 only after the store acknowledged their record. They wait at most 5 s, outside the book's lock. Otherwise they answer 503 with the reason.durability/mod.rs,boot.rs,store_adapter.rs,AmendmentJournalandkeyset.rsis corrected.Shippershim is removed, along withShimState::records_shipped,StoreAdapter::shipper/head/speaks_new_opsandSTORE_ABI_WITH_NEW_OPS.busbar.keyset.v1, and node ids underbusbar.node.v1, keyed by host name./admin/verifywalks every chain the store keeps.redeem_plane_tokensingle-use claims:keyset:deployment,host:<name>andnode-id:<id>.New in this round (merge of predev)
root/tests/durability.rs,root/tests/plane_node.rs,units_admin/tests/units_admin.rs) keep both sides' cells. The cells predev added build data-dir books or test-onlydurability::buildbooks. None of them callsroot::boot::book/compose_book, so none needs the store's record slots.node).AuditRecord.nodewasbusbar_kernel::door::node(), a fresh CSPRNG draw per process. That is not the store-assigned id H3 keys the chain by, so one host's stored chain held records naming a different node on every boot. Fix: a book whose chain the store keeps (build_on_storewith no data dir) now seals as the store-assigned node. A disk or store-less book still seals asdoor::node(). The verify walk accepts the store id (it digestsnodeand does not compare it). Thedoor::nodedoc says so.unit.accruedcheckpoints). A flushed checkpoint isDurability::checkpoint_accrual, which goes throughjournal_markand then the journal's shipper, which isJournalLane. It is rebuilt by the same resumed-chain replay (build_inner,replay_into, thenrecover). A new cell restarts over the store with an open driven unit holding a flushed checkpoint.qa/kind-isolation.tomlandxtask/src/gates/kind_isolation.rscite no deleted item.records_shipped,speaks_new_opsorSTORE_ABI_WITH_NEW_OPSremains incrates/,xtask/,qa/ordocs/.BufferShipperandWal::memory_bufferedare not deleted by this branch: they remain the in-process journal for a book with no store behind it, and predev uses them too.RED arm: each cell and the neutralisation that makes it fail
root/tests/durability.rsa_restart_over_the_same_store_keeps_the_chain_the_audit_records_and_the_holdsbuild_on_storepasses no resume (journal opens as an emptymemory_buffered_to) → restarted book has 0 audit records, no recovered hold, settled 0, numbering from 1durability.rsa_data_unit_rides_a_refusing_store_and_its_record_lands_once_it_recoversrefuse(false); the next boot does not read it backdurability.rsa_full_lane_refuses_new_money_and_drops_no_recordretaining_at_boundremoved (journal drops its oldest at the bound) →dropped_total() > 0and the store is missing recordsroot/tests/plane_node.rsa_full_journal_lane_refuses_a_new_unit_503_with_the_reasonNode::answerremoved → the unit is admitted and its build runs (the build closure panics)root/tests/kernel.rsa_back_dated_correction_survives_a_restart_over_the_storekeeps_chainreduced to "has a data dir" (cards rebuilt only over a disk) → restarted history is the boot card alone; the window is not repricedunits_admina_correction_on_a_node_with_no_data_dir_is_kept_by_the_store_and_survives_a_restartunits_admina_correction_the_store_refuses_answers_503_with_the_reasonhealthy()/wait_ackedwait removed → 200 over a refusing storeunits_admincommit_upgrade_answers_only_once_the_store_kept_its_recordcommit_upgrade_effect→ 200 over the refusing storesrc/tests/tests.rsa_store_without_the_record_slots_refuses_the_boot_by_nameroot::boot::no_record_slotscheck removed → the boot opens a book over a store that keeps nothingdurability.rsa_lost_keyset_claim_with_nothing_kept_refuses_until_the_claim_lapses(lapsed claim)truewithout redeeming, or the token not naming its period → the losing boot mints / the claim never lapsesdurability.rsa_first_boot_that_lost_the_keyset_claim_takes_the_winners_key,a_first_boot_that_lost_the_host_claim_takes_the_winners_node_id,every_minted_node_id_is_held_by_its_claimtruewithout redeeming → two keys / two ids for one host / the id's claim is unredeemeddurability.rsa_restart_over_the_same_store_verifies_the_prior_boots_recordsbusbar.keyset.v1→ every resumed record is a verify findingdurability.rstwo_nodes_on_one_store_never_collide_and_verify_walks_both_chainsdurability.rsa_store_kept_chain_seals_every_record_as_the_store_assigned_nodesealing_as(busbar_kernel::door::node())unconditionally (the predev #662 behaviour) →record.node() != node; records name the process drawdurability.rsa_driven_unit_killed_mid_stream_over_the_store_is_recovered_at_its_flushed_checkpointrecovered_holds == 0, no recovered postingmemory_resumed/retaining_at_boundcellsWal::memory_seedednot seeding /retaining_at_bounda no-op → replay empty / oldest droppedThe test double is the in-tree
StoreCallsover a map (root/tests/store_double.rs), not a new plugin. This PR's CI is the proof of record. No local build was run.Customer-visible behaviour: the llm and core request paths are byte-identical to 1.5.5 except in one new state. When the store keeps the journal and the lane to it is full, a new unit answers 503
overloadedwith the reason; 1.5.5 kept no journal. The 503 fromamend_rate_historyandcommit_upgradeis new with those verbs, which are themselves new in 1.6.0.