From 347d5c9e7faa3ab05eb629b6c6226ace260d3cc4 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 9 Aug 2026 12:40:49 +0300 Subject: [PATCH 01/26] docs(roadmap): correct two stale markings now that PR #81 is merged MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified every ✅ in PR #81's closing register against the code rather than trusting the marks: `#W15-3`'s `/cost --release`, `#W15-4`'s durability latch, `#W15-5`'s projection check, `#W15-6`'s folds, `#W15-7`'s refinement, `#W15-9`'s fold guard, `#W15-10`'s escaper, `#W15-13`'s opt-in heal, `#W15-14`'s `hostSleep`, `#W15-15`'s degraded flag, `#W15-22`'s `Reflect.apply`, `#W15-23`'s `TxDb`. All present. ADR-0074/0075/0076 are Accepted and both new ones are indexed. Two markings were wrong, both in the same milestone note: - it said ADR-0074 §1 "remains open". It closed on 2026-08-09 with `/cost --release` (`commands/chat.ts` calls `releaseConservativeCommitments`; `repl-info.ts` renders the durability state and points at the flag). - it called that work "Wave 1.5". That framing was retracted — the work was PR #81's own closing list, not a later wave — and this was the last place still carrying it. Also: the register said the fixes were "committed on `development`". They are merged to `main`. `#W15-1` stays OPEN and is NOT marked. Its decision is made (ADR-0076, Accepted) but no implementation has landed, so the count remains 23 of 24. Marking it now is exactly the false-completion error this register was rebuilt to correct. Refs: PR #81, ADR-0074 §1, ADR-0076 Co-Authored-By: Claude --- docs/roadmap/current.md | 2 +- docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index f2d6d8d4..7f1a80d1 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -244,7 +244,7 @@ Ordered by whether the repo currently states something untrue, then by blast rad **Every item names the check that would close it, not just the defect.** > **Status, 2026-08-09 — 23 of 24 closed.** Everything marked ✅ below is fixed, -> break-verified with the mutation confirmed applied, and committed on `development`. §E's six coverage gaps +> break-verified with the mutation confirmed applied, and **merged to `main` via PR #81** (2026-08-09). §E's six coverage gaps > are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort listener is removed, which > is the unkillable run reproduced exactly rather than an assertion standing in for it. > diff --git a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md index e2e801d5..868b48e2 100644 --- a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md +++ b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md @@ -446,7 +446,7 @@ This substream is the development-process substrate itself — `.github/workflow | In-phase | Completed by | Outcome | |----------|--------------|---------| -| ✅ M2.5.5-1 Safety-critical propagation fixes *(met 2026-07-30, PR #81 — its two CRITICALs, #91 and #228, are closed; ADR-0074 §1 remains open and is tracked as Wave 1.5 in [current.md](../current.md), which this milestone does not gate)* | 2.5.5.A + 2.5.5.C | Two of the three CRITICAL findings closed (`#91` the unredacted approval-preview secret leak; `#228` the chat-persister unhandled-rejection crash — the third, `#56`, lands with 2.5.5.I under M2.5.5-2); the budget governor's concurrency gap and the two-process migration race are fixed | +| ✅ M2.5.5-1 Safety-critical propagation fixes *(met 2026-07-30, PR #81 — its two CRITICALs, #91 and #228, are closed. ADR-0074 §1 was open at that date and CLOSED on 2026-08-09 with `/cost --release`; the "Wave 1.5" framing this note used was retracted — that work was PR #81's own closing list, not a later wave. See [current.md](../current.md).)* | 2.5.5.A + 2.5.5.C | Two of the three CRITICAL findings closed (`#91` the unredacted approval-preview secret leak; `#228` the chat-persister unhandled-rejection crash — the third, `#56`, lands with 2.5.5.I under M2.5.5-2); the budget governor's concurrency gap and the two-process migration race are fixed | | M2.5.5-2 Security-surface hardening | 2.5.5.D + 2.5.5.I | MCP tool-poisoning/config-secret-hijack gaps closed; the third CRITICAL (`#56`) closed as terminal-control sanitization reaches `relavium run`, the error boundary, and the approval card; `chat.ts`/`home-controller.ts` decomposed ahead of 2.6.E/2.6.G | | M2.5.5-3 CLI operator-safety net | 2.5.5.E | The outermost crash/signal/error boundary behaves uniformly across every command; `--version`/`--help` startup tax closed | | M2.5.5-4 Money-safety & process substrate | 2.5.5.B + 2.5.5.H | The three LLM adapters share guard/classification parity and `FallbackChain` is the sole retry authority; CI actually executes the compiled binary and the coverage/dependency/logging gates match what they claim to enforce | From 97376059b0ffd774decf0705089653e285722117 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 9 Aug 2026 15:43:48 +0300 Subject: [PATCH 02/26] =?UTF-8?q?docs(roadmap):=20add=20phase=202.6.5=20?= =?UTF-8?q?=E2=80=94=20core=20reliability=20remediation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three independent core reviews of the post-Wave-1 tree converged, without seeing each other, on the SAME seven blocking gaps: no durable effect journal, stdio MCP spawning before consent, no cross-process run ownership, compaction summaries elevated to `system` authority, a stream ending without a terminal counted as success, the engine not enforcing the authored input contract, and the durable event log not being a gap-free ordered prefix. Three separate reviews landing on the same seven points is not a matter of opinion, and it is why this interlude exists. All three also reached the same verdict on scope: the architecture is right and nothing here calls for a rewrite. The gaps are in the last ten percent of reliable execution — durable ownership, effect identity, trust provenance, stream grammar, admission and ordering — and the existing seams can carry the fixes. The phase document is deliberately SELF-CONTAINED: 40 items (`CR-01`…`CR-95`) each carrying its own evidence, failure scenario, fix and acceptance criteria, so the work can be done from that file alone without the source analyses. It inherits Wave 1's discipline explicitly — ADR before code for anything that changes what a contract means, break-verify with the mutation confirmed applied, never ship a hollow test, record rather than assert, fix the canonical doc in the same change. Three items are already half-closed by Wave 1 and finish first. One of them, `CR-03`, is a propagation gap from `#W15-10`'s own fix: three `--json` paths still use bare `JSON.stringify` and never got the safe serializer — verified in the current tree, not inferred. Wave 2 is amended rather than replaced: its MCP cluster stays and consent-before-spawn joins it as a P0 in the same code area. The realized-cost ledger and the transaction-handle cleanup keep their own tracking and are named in the scope note so nobody schedules them twice. Refs: phase 2.6.5, Wave 2 Co-Authored-By: Claude --- docs/roadmap/current.md | 21 + ...hase-2.6.5-core-reliability-remediation.md | 531 ++++++++++++++++++ 2 files changed, 552 insertions(+) create mode 100644 docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 7f1a80d1..7a7ae061 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -425,6 +425,27 @@ the review of `#W15-16`, fixed with it, and now the signal that test drives. `llm-provider-seam.md` still claims model-discovery/media-poll keep a small SDK retry; ADR-0028's amendment note still says ADR-0074 §2–§5 have not landed. Each could send a future reader back toward a #91-class hole. +### Phase 2.6.5 — Core reliability remediation (before Wave 2) + +Three independent core reviews of the post-Wave-1 tree converged — independently — on the **same seven blocking +gaps**: effect journal, stdio MCP consent-before-spawn, run lease, compaction trust elevation, stream grammar, +input admission, event-log ordering. Three separate reviews landing on the same seven points is not opinion. + +The full, self-contained work list is +[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — 40 items +(`CR-01`…`CR-95`) with evidence, fix and acceptance criteria for each, written so the work can be done from that +document alone. + +Two things it changes about the plan below: + +- **Wave 2 is amended, not replaced.** Its MCP timeout/SSRF/ingress cluster stays; `CR-16` + (consent before a stdio MCP spawn) is added to it as a P0 in the same code area. +- **Its seven P0 items are ADR-first**, in dependency order — an effect journal is meaningless while two + processes can own one run, and run ownership rests on the log being an ordered prefix. + +Three items are already half-closed by Wave 1 and finish first (`CR-01`–`CR-03`); `CR-03` is a propagation gap +from `#W15-10`'s own fix — three `--json` paths never got the safe serializer. + ### Wave 2 — Shut the doors The two named live security holes, one unified filesystem jail, the secrets layer under test — then diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md new file mode 100644 index 00000000..d488173a --- /dev/null +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -0,0 +1,531 @@ +# Phase 2.6.5 — Core reliability remediation (interlude) + +- **Status**: planned +- **Opened**: 2026-08-09 +- **Predecessor**: Wave 1 of the 2.5.5 remediation (complete — PR #81) +- **Successor**: Wave 2 of the 2.5.5 remediation (MCP + fs jail), which this phase AMENDS rather than replaces +- **Kind**: interlude, like [phase 2.5.5](phase-2.5.5-hardening-and-remediation.md) — no new product surface, only + the invariants an existing surface already claims + +## Why this phase exists + +Three independent core reviews of the engine were run against the tree as it stood immediately after Wave 1. +They converged — independently, without seeing each other — on the **same seven blocking gaps**. That +convergence is the reason this phase exists: three separate reviews landing on the same seven points is not a +matter of opinion. + +The verdict all three reached is worth stating, because it shapes every item below: + +> The architecture is right. Nothing here calls for a rewrite. The gaps are in the **last ten percent of +> reliable execution** — durable ownership, effect identity, trust provenance, stream grammar, admission and +> ordering — and the existing seams are good enough to carry the fixes. + +The second reason is sharper. Several of our own canonical documents currently promise behaviour the code does +not deliver: *crash-safe resumable workflow*, *duplicate-free side effect*, *bounded stream*, *first completed +branch wins*, *schema-validated input/output*, *full session-to-workflow graduation*. A claim the code does not +keep is worse than a missing feature, because it is the claim a user plans around. + +## What must NOT change + +Recorded so a remediation pass does not quietly undo a decision that is correct: + +- The pure-TypeScript engine with zero platform imports. +- The `LLMProvider` seam and the rule that no vendor SDK type crosses it. +- Deny-by-default tool policy, the approval chain, and the QuickJS sandbox. +- Strict authored YAML with Zod at every boundary. +- Event-sourced checkpoint/resume as the recovery model. +- Keys in the OS keychain only. + +## Working discipline for this phase + +This phase inherits the discipline Wave 1 arrived at the hard way. It is not optional ceremony; every clause +below exists because its absence produced a real defect during Wave 1. + +1. **A decision gets an ADR before it gets code.** Six of the seven P0 items below change what a contract + *means* — what "exactly once" means, what `system` authority means, what "the stream ended" means. Wave 1 + proved that treating a decision as a task item produces false completion claims. Write the ADR, get it + accepted, then implement. +2. **Break-verify every regression test, and verify the mutation actually applied.** Wave 1 produced three + tests that passed with the production code mutated. Each was found only by checking that the mutation had + landed (e.g. `grep -c` the mutated symbol → 0) before trusting a red. +3. **Never ship a hollow test.** If the honest test cannot be built, leave the gap open, name it, and name the + mutation that would close it. A test that implies coverage it does not have is worse than none. +4. **Record rather than assert.** Any claim that cannot be demonstrated goes in as a limitation, not as a + property. +5. **Fix the canonical doc in the same change.** One canonical home per artifact; a fix that leaves the spec + describing the old behaviour has not landed. +6. **`pnpm run ci` must exit 0.** Check the exit code, not the reporter output. + +## How the work is grouped + +`CR-##` ids below are this phase's own; they are stable and self-contained. Severity is the consensus of the +three reviews. + +- **W0 — finish the halves.** Cheap, already half-done. Leaving them half-done is the exact failure this phase + is correcting. +- **W1 — the seven P0 blockers.** ADR-first, in dependency order. +- **W2…W9 — P1 themes.** Grouped so each theme is one reviewable PR, not sixty scattered tickets. + +--- + +# W0 — Finish what is already half-done + +These are partially closed. Each has a verified open half. + +## CR-01 — `session:cancelled` bypasses the session durability latch · High + +**Evidence.** The CLI chat persister wraps its cost and turn writes in the `persistDurably` failure latch, and +since Wave 1 the `session:compacted` and `session:trimmed` arms go through it too. The `session:cancelled` arm +still calls `deps.store.writeTurn(...)` directly (`apps/cli/src/chat/persister.ts`). + +**Why it matters.** The event bus isolates listener errors from the producer, so a failing DB write lets the +cancel path report success. The durability latch never sets, so nothing gates the next provider egress. + +**Fix.** Route the `session:cancelled` write through `persistDurably`, exactly as its two siblings now are. + +**Acceptance.** A failing `writeTurn` on cancel sets `durabilityFailure`, and the next egress is refused. +Break-verify by removing the wrapper. + +## CR-02 — A failed turn flush leaves the turn counter incremented · Medium + +**Evidence.** `packages/core/src/engine/agent-session.ts` increments `#turnCount` before it awaits +`flushBudgetCommitments()`. Wave 1 fixed the transcript half of this (the flush now precedes the assistant +append, so the rollback's one-message `pop()` is correct again). The counter half was not examined. + +**Why it matters.** A turn whose durability flush rejects has still consumed a slot against the hard turn cap. + +**Fix.** Decide deliberately, then make the code say it: either the counter moves after the flush, or it stays +where it is because an engaged provider call counts regardless — which is what the two catch paths already do +for `engaged` turns. Whichever is chosen, state it in a comment and pin it with a test. + +**Acceptance.** A rejecting flush leaves `#turnCount` at the value the chosen rule requires, asserted directly. + +## CR-03 — Three `--json` paths still bypass the safe serializer · High + +**Evidence.** Wave 1 introduced `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), which losslessly +escapes C1 controls and Trojan-Source bidi characters, and routed the workflow renderer and the record writer +through it. Three sibling paths still use bare `JSON.stringify`: +`apps/cli/src/commands/agent-run.ts:161`, `apps/cli/src/commands/chat-export.ts:90`, +`apps/cli/src/commands/chat.ts:2392`. + +**Why it matters.** Same class as the gap Wave 1 closed: `JSON.stringify` escapes `ESC` but leaves `U+009B` +(8-bit CSI) and the bidi family raw, and these streams carry model-, tool- and persisted content. This is a +propagation gap — the fix landed, the siblings did not get it. + +**Fix.** Route all three through `stringifyJsonLine`. The escape is lossless, so no machine contract changes. + +**Acceptance.** One shared adversarial test drives a C1 + bidi payload through all three surfaces and asserts +the raw code points do not survive while `JSON.parse` round-trips to the identical string. + +--- + +# W1 — The seven P0 blockers + +Dependency order matters. `CR-10 → CR-11 → CR-12` is a chain: an effect journal is meaningless while two +processes can own the same run, and run ownership rests on the log being an ordered prefix. `CR-13`, `CR-14` +and `CR-15` are independent and can run in parallel. + +## CR-10 — The durable event log is not a gap-free ordered prefix · Blocker · needs an ADR + +**Evidence.** The engine assigns sequence numbers centrally but starts each event's persistence independently +for concurrency; the code comment states the split explicitly ("persistence concurrent, delivery serialized"). +The SQLite store writes each event in its own transaction, so a higher sequence can commit while a lower one is +still in flight (`packages/core/src/engine/engine.ts`, `packages/db/src/run-history-store.ts`). + +**Failure.** Event `N`'s write is slow; `N+1` commits; the process dies before `N`. The disk holds a pseudo- +prefix with a hole in its sequence, and the causal predecessor the checkpoint fold depends on is missing. + +**Why it is first.** This is the root cause behind the sum-vs-max decision already recorded for conservative +commitments: concurrent events under a `fan_out` have no canonical `seq` order. Every durability property below +assumes the log is an ordered prefix. + +**Fix.** One ordered append tail per run. `N+1` must not reach the store until `N` is durable. Store-side +compare-and-append against the expected last sequence. + +**Acceptance.** A store harness that delays event `N` and commits `N+1` must be rejected, not accepted. A crash +injected between the two must leave a prefix with no hole. Existing gap-free assertions must still pass. + +## CR-11 — No cross-process run ownership or CAS · Blocker · needs an ADR + +**Evidence.** The engine states that its cross-process guarantee rests on store uniqueness, but the only DB +uniqueness is `(run_id, seq)`. `resumeFromCheckpoint` performs an in-memory check, then loads the checkpoint and +builds an independent `RunExecution`. There is no run lease, fencing token, or checkpoint-cursor CAS. + +**Failure.** Two CLI or desktop processes read the same paused run at the same time and become two independent +side-effect producers for one run. + +**Fix.** A run-level lease with a fencing token, or an atomic CAS on `last_seq`: +`acquireRunLease(runId, ownerId, expectedGeneration, ttl)`. The process that loses the CAS becomes a read-only +observer rather than a second owner. + +**Acceptance.** Two concurrent resumes of one paused run: exactly one proceeds, the other degrades to observer +with a typed, actionable error. A lease that expires mid-run is fenced out of further durable writes. + +## CR-12 — No durable effect/idempotency journal on the hot path · Blocker · needs an ADR + +**Evidence.** `NodeExecContext` carries an attempt number but no run/effect idempotency key. +`ToolDispatchContext` carries a node id but no run correlation or semantic key. The registry calls +`def.dispatch` directly with no durable prepare step. Our own architecture documents state that a stable key +prevents duplicates — that key does not exist in the dispatch context. + +**Failure.** `node:started` persists → an `http_request` POST or MCP mutation completes at the target → the +process dies before the tool result or `node:completed` persists → resume re-runs the node → the target sees +the same effect twice. + +**Impact.** Duplicate ticket, deploy, payment, message or commit. The *duplicate-free* product claim is false +until this closes, and audit/compensation cannot be built on top of it. + +**Fix.** Add `EffectAttemptId = runId + nodeId + nodeAttempt + toolCallId + effectiveArgsHash` to +`ToolDispatchContext`. Side-effectful host ports take it as a required argument. A durable state machine in the +run store: `prepared → dispatched → committed | ambiguous → needs_attention`. + +**Acceptance.** Kill the process after the effect completes at the target — resume produces no duplicate. An +ambiguous non-idempotent effect is never auto-retried. The same effect key arriving from two processes commits +once. + +**Relationship to the realized-cost ledger.** The accepted cost-ledger ADR explicitly scopes effect duplication +out and names this as the decision that owns it. That ADR's implementation and this item are independent; this +one is the larger of the two. + +## CR-13 — Compaction summary is elevated to `system` authority · Blocker (community/untrusted content) · needs an ADR + +**Evidence.** The prior conversation is handed to the summarizer as user-role data; the summarizer's model +output is taken as a plain string, becomes the context preamble, and is concatenated directly into the authored +`agent.system_prompt` on the next turn. Resume and model reseat restore the persisted summary into the same +preamble (`packages/core/src/engine/agent-session.ts`). + +**Attack.** A user message or a read document contains a closing tag for the summary fence plus "ignore previous +instructions". The summarizer preserves it as a task. On the next tool-capable turn those bytes sit in the +`system` role — and survive a restart, because the summary is persisted. An XML fence is not a trust boundary. + +**Conflict.** Our binding security standard treats model output and tool results as untrusted and forbids +concatenating untrusted content into `system`. The compaction ADR fixed exactly that concatenation as its +decision, so this needs an append-only superseding ADR, not a patch. + +**Fix.** Carry the summary as untrusted. Dynamic summary bytes must never reach the `system` builder. Supply the +summary as a separate untrusted content part in user-role context. Make `AgentTurnParams.system` accept only a +branded type the authored builder can produce. + +**Acceptance.** An injected summary containing a fence-closing sequence plus instructions cannot alter tool +authorization on the following turn, before or after a restart. A type-level test proves a dynamic string cannot +be passed as `system`. + +## CR-14 — A stream that ends without a terminal `stop` counts as success · Blocker · needs an ADR + +**Evidence.** The fallback chain emits a success attempt when the provider iterator ends cleanly with no usage. +The agent turn starts with a default `stopReason` of `stop` and returns a normal result without ever seeing a +terminal chunk. **An existing test pins this behaviour as success** — so the fix changes that test, deliberately, +rather than deleting it. + +**Failure.** A transport, proxy or provider closes after `text_delta: "partial"` with no terminal chunk. +Relavium treats the partial text as a completed assistant answer and passes it downstream as a successful node +output. + +**Fix.** Make the stream grammar require exactly one terminal. A clean EOF with no terminal is a `transport` or +`protocol` error. If content was already committed, do not fail over or retry — surface it. + +**Acceptance.** A stream ending without a terminal produces a classified error, not a node success. A +content-committed truncation does not trigger failover. The superseded test is rewritten to assert the new rule +and to keep the reasoning for the behaviour it replaces. + +## CR-15 — The engine does not enforce the authored input contract · Blocker · needs an ADR + +**Evidence.** The shared contract states the engine validates inputs before a run starts. `WorkflowEngine.start()` +passes the caller's `inputs` object straight to the run execution, which stores it without cloning, applying +defaults, or validating against the schema. The input node reads values straight out of that map. The CLI's own +validator does unknown/required checks and coarse coercion only, and explicitly leaves deep validation and +default resolution to the engine. + +**Failure.** An input with a declared default that the caller omitted arrives as `undefined`. A wrong type or a +value outside an enum reaches deep execution. CLI, desktop, extension and the future API behave differently — +which breaks the "one engine, every surface" guarantee directly. + +**Fix.** A pure `resolveAndValidateWorkflowInputs` in core, running before id generation and before the first +event. + +**Acceptance.** Defaults are applied by the engine, not the surface. A type/enum violation fails admission with +a typed error before `run:started`. The same workflow and inputs behave identically through the engine API and +through the CLI. + +## CR-16 — Stdio MCP spawns a local process before consent · Blocker · needs an ADR · **amends Wave 2** + +**Evidence.** The MCP connection is opened while the chat session is being built, before a mode or turn starts; +the agent-run path prepares MCP first and applies mode policy afterwards. An agent or workflow declaration +produces the `command`, `args` and `cwd` spawn spec directly, and the SDK adapter spawns it. + +**Failure.** No tool call is required. `ask` mode does not protect the spawn. `shell: false` does not stop the +declaration from naming `sh`, `bash` or `node`. An imported community artifact executes arbitrary local code at +load time. + +**Fix.** Consent before the first spawn, showing executable, args, cwd, artifact source and hash. Unknown +fingerprint fails closed. In non-interactive mode, fail closed unless an explicit `--allow-mcp-stdio ` +is supplied. Replace auto-start with lazy connect on first actual MCP tool need. + +**Acceptance.** Importing and opening an artifact with a stdio MCP server spawns nothing until consent. A +non-interactive run without the digest flag fails closed with an actionable message. A changed fingerprint +re-prompts. + +**Wave 2 note.** Wave 2 already holds the MCP timeout / SSRF / ingress cluster. This item belongs to the same +code area and the same review; it should be folded into that PR sequence rather than opened separately, but it +is a P0 while the rest of that cluster is P1. + +--- + +# W2 — Liveness and deadlines + +## CR-20 — Agent-node `timeout_ms` is completely inert · High +The node schema accepts the field, but the runner passes only the run-level signal to the agent turn; no timer, +controller or deadline consumes `node.timeout_ms`. An authored liveness bound is silently ignored. +**Fix + acceptance.** Honour it as a real deadline; a node exceeding it fails with a classified timeout, proven +with a fake clock. + +## CR-21 — No Relavium-owned deadline for a normal provider attempt · Medium-High +The chain awaits generate/stream with the caller's signal only; unlike list-models and key validation, it sets +no per-attempt timeout. Without a node or run timeout, the vendor SDK default becomes the product's liveness +semantics — which our security standard forbids for outbound requests. +**Fix + acceptance.** An injected controller/timer factory; merge caller-abort with deadline-abort. A timeout is +`kind: 'timeout'`, a user cancel stays `cancelled`. A pre-content timeout may fail over; a content-committed one +must not. Prove timer cleanup with a fake clock. This is a different timer from CR-20 — both are needed. + +## CR-22 — Gate and run deadlines are not preserved across resume · High +The checkpoint's pending gate carries gate/node/budget data but not an absolute deadline, and the whole-run +timeout is re-armed for its full duration on every resume — so a crash extends the cap. +**Fix + acceptance.** Persist absolute deadlines; resume computes the remaining time. A run crashed and resumed +repeatedly still times out at its original absolute deadline. + +## CR-23 — Exactly-one-terminal is a safety property, not a liveness one · High +Cancel only fires the abort signal, and the terminal waits for the running-node count to reach zero. The node +executor seam accepts an arbitrary promise; honouring the signal is not guaranteed at the type level. An +executor that ignores abort and never settles leaves the run without a terminal forever. +**Fix + acceptance.** A bounded grace period after cancel/timeout, then a generation token that fences the run +state from late outcomes, plus executor quarantine. Tests: a never-settling executor and a late-success executor +both produce exactly one terminal in bounded time, and the late output is not applied. + +--- + +# W3 — Resource governance and bounds + +## CR-30 — The event stream is not really bounded · High +`push()` appends without a capacity check; `whenDrained` is advisory and only awaited at node boundaries, while +token deltas are emitted synchronously. A single provider stream can grow memory without a hard bound. The three +properties "sync producer", "no drop" and "hard bound" cannot all hold — pick two, deliberately. +**Fix + acceptance.** Real backpressure or an explicit, documented drop policy for transient events. A test +drives a fast producer against a slow consumer and asserts a hard ceiling. + +## CR-31 — No safe default concurrency and no absolute graph/retry caps · High +An omitted `max_parallel_nodes` means `Infinity`. There is no absolute cap on authored nodes, edges, fan-out +width, fallback-chain length, retry counts, parallel tools or total attempts; the parser's text-size limit does +not stop many small ones. +**Fix + acceptance.** A small capacity-derived default concurrency that an authored value may only narrow, plus +explicit absolute caps enforced at parse/compile admission with typed errors. + +## CR-32 — Workflow output, state and durable event size are unbounded · High +**Fix + acceptance.** Bound them at the durable boundary with a typed rejection, the same way tool output is +already bounded. + +## CR-33 — Finished runs are retained forever in memory · Medium-High +`WorkflowEngine` keeps completed runs indefinitely. +**Fix + acceptance.** A retention policy with an explicit bound; a long-lived process running many workflows +does not grow without limit. + +--- + +# W4 — MCP hostile boundary *(merge into Wave 2)* + +## CR-40 — Cancellation and deadlines never reach the MCP transport · High +The dispatch context can carry a signal, but the manager's `callTool` chain does not forward it, and neither the +connection nor the stdio SDK call accepts an abort or deadline. After a user cancels, the remote or child effect +continues and child processes can survive. + +## CR-41 — The MCP SSRF floor does not cover DNS or redirects · High +The network config checks the authored hostname and scheme, then hands the raw opener to the SDK. The code +itself documents the DNS-to-private and redirect-to-private holes. A public-looking domain can resolve to +loopback, RFC1918 or a metadata IP. +**Fix.** Apply the built-in egress mechanism — resolve-all, range-block, pinned IP — to the MCP HTTP, SSE and +WebSocket transports. + +## CR-42 — MCP discovery and result ingress are unbounded · High +Page count is capped for `tools/list`, but total tool count, byte size, description and schema size are not, and +a result is fully materialized before core bounding applies. A hostile server can exhaust startup memory, inflate +prompt token cost, or bloat a workflow output. + +**Acceptance for W4.** A hostile-server harness: a server that never answers is cancelled within its deadline +and leaves no orphan process; a DNS record pointing at loopback is refused; oversized discovery and results are +rejected with typed errors before they reach the prompt. + +--- + +# W5 — Media correctness + +## CR-50 — `read_media` does not work end to end · High +The builtin returns a base64 media part, the registry places generic output into the tool result, and the LLM +message schema explicitly forbids raw media bytes there. The canonical alternative is a handle-only attachment, +but all three adapters deliberately drop that field, and production chat does not wire the media-read scope. +There is no registry → turn → chain → adapter path. +**Fix + acceptance.** Either complete the handle path through the adapters, or remove the tool from the catalog +until it works — an advertised tool that cannot work is worse than an absent one. An end-to-end test proves the +chosen answer. + +## CR-51 — Model-level tool/attachment capability is not enforced · High +The catalog carries `toolCall` and `attachment` metadata, but runtime gates on the provider-wide flags and the +adapters attach tools unconditionally. A model the catalog marks as tool-incapable can still be sent tools, and +the chain can treat it as capable instead of pre-skipping it — turning a free skip into a paid 400. + +## CR-52 — Gemini reasoning/tool continuation metadata is lost · Medium-High +Part-level thought signatures on function calls are not captured, and replay drops reasoning after a tool +result. Multi-turn thinking-plus-function-calling continuations can 400 or break semantically. + +## CR-53 — Large media travels fully buffered and base64-encoded · High +Audio and video responses are read into a full buffer and converted to base64; raw buffer, typed view, base64 +string and the store's decoded copy can all exist at once. +**Fix + acceptance.** A bounded stream or download lease from the adapter; the host writes straight to the media +store with a `Content-Length` and streamed-byte ceiling. A large-media test asserts peak memory stays bounded. + +## CR-54 — URL media can produce different bytes on resume · High +Remote URL media is resolved again on resume, and the same URL can return different content, a different +redirect or a different DNS answer. The run appears to have the same inputs while the model sees different bytes. +**Fix + acceptance.** Convert to a content-hashed handle at first resolution; resume must not re-fetch. + +## CR-55 — Missing media rates can bypass a strict cost cap · Blocker for the strict-cap claim +Where a media rate is absent the estimator can fall to zero, the generated catalog projection produces no media +rate, and the DB's media-rate columns are not carried into the listing/overlay path. The governor can then treat +a missing rate as *priced at zero* rather than *unpriced*, admitting paid image/audio/video generation under a +strict cap and reporting `cost:updated = 0`. +**Fix + acceptance.** Missing media rate ⇒ unpriced, never zero. Under a strict cap an unpriced media generation +is refused. Carry the media rates through the overlay path. + +--- + +# W6 — Authoring correctness + +## CR-60 — `merge_strategy: first` is not implemented as specified · High +The canonical table says the first resolved branch wins and the others are ignored. The engine waits for every +branch to settle and the handler takes the first surviving output in declaration order. A slow loser adds +latency and cost; a failing loser can fail the run even when the winner is ready. +**Fix + acceptance.** Either implement race semantics or rename the strategy and correct the canonical table — +but the name and the behaviour must agree. Whichever is chosen, a test pins latency/cost behaviour. + +## CR-61 — Schema fields do not deep-validate · Medium +An agent's `output_schema` becomes a response-format hint and is then only `JSON.parse`d; the transform node's +`output_schema` explicitly does not deep-validate; an agent's `input_schema` is not enforced on the runtime +runner path. A prior ADR required deep validation to land with this work, so this is also a decision-governance +drift. +**Fix + acceptance.** Real deep validation with typed failures, or a documented, deliberately narrowed contract +plus a renamed field. Silence is not an option. + +## CR-62 — Expression dependencies are invisible, so silent mis-routing is possible · High +The DAG derives edges from template references only and deliberately does not order JS-expression reads of run +outputs. A condition that runs before its producer compares `undefined` and silently takes the false branch — a +test currently pins that silence. +**Fix + acceptance.** Either extract dependencies from expressions, or fail loudly when an expression reads an +unresolved output. Not a silent false. + +--- + +# W7 — Agent product correctness + +## CR-70 — Cross-turn tool-call memory does not exist · High (product) +Only the final assistant text enters the cross-turn transcript; within-turn tool call/result pairs are dropped. A +coding agent cannot remember a file it read in the previous turn and calls the tool again. + +## CR-71 — Session transcript and export lose tool history · High +The persister stores the same text-only model, so the exporter — written to derive tool names from assistant +tool-call parts — has nothing to read. The canonical session spec promises a full transcript with a tool union. +**Fix + acceptance for CR-70/71.** Persist the tool pairs (bounded), or narrow the spec. A graduated workflow +must represent the flow that actually happened. + +## CR-72 — Authored agent `memory` policy is inert · Medium +`none | window | summary` is accepted by the schema and consumed by nothing. +**Fix + acceptance.** Implement it or reject it at parse time with a typed error. + +## CR-73 — `invoke_agent` has no production delegate · High (2.6 go/no-go) +The builtin exists in the catalog and errors with tool-unavailable when no delegate is wired; nothing wires one +in production, and the availability filter keeps the delegate-backed tool in the catalog. +**Fix + acceptance.** Wire the delegate, or stop advertising the tool to the model when no delegate exists. + +--- + +# W8 — Provider and conformance + +## CR-80 — An invalid custom base URL fails OPEN to the official endpoint · High (security/compliance) +When the custom provider factory rejects a private, malformed or credential-bearing URL, the error is caught and +the default adapter is left standing — and a test pins that fail-open as correct. A user expecting an internal +gateway silently sends prompts and keys to the official API after a config drift. +**Fix + acceptance.** Fail closed with an explicit message. Rewrite the test that pins the old behaviour, +keeping the reasoning it replaces. + +## CR-81 — Conformance cassettes do not verify the request wire contract · High +A recorded response carries status, content type and body only; replay checks that the request body is parseable +JSON and matches nothing else — not method, path, headers, or a canonical body. A dropped `response_format`, a +lost tool result on continuation, or a broken system-instruction mapper leaves the suite green. +**Fix + acceptance.** Match the canonical request. Mutating the request lowering must redden the suite. + +## CR-82 — Missing or partial usage can be read as zero · High +Carried from our own backlog and re-confirmed by the reviews. +**Fix + acceptance.** Missing usage is *unknown*, never zero, on every path that reaches a cost total. + +--- + +# W9 — Harness honesty and isolation + +## CR-90 — Root coverage collects in-repo worktrees · Medium +The root Vitest include patterns have no exclude for repo-local checkout directories, so a stale worktree adds +test files and 0%-covered sources to the run. In one working tree this produced 471 discovered test files and 23 +suite failures; excluding the worktree path produced 237 files and a clean run. +**Fix + acceptance.** Exclude repo-local checkout areas from both `test.exclude` and `coverage.exclude`. A +self-test creates a fixture worktree and proves root discovery does not see it. + +## CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium +It pins that the live stream reported success; it does not prove durable truth after a restart. +**Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same +terminal type and payload. + +## CR-92 — Terminal persistence failure lets live and durable truth diverge · High +The engine can complete the delivery chain for a terminal event even when its persistence failed, and +reconciliation may later produce a different terminal than the original. Media reclaim can run before the +terminal is durable. A caller can receive a success result and outputs while history shows the run failed. +**Fix + acceptance.** Hold the intended terminal payload in a durable outbox and retry it under the same +identity. If durability is uncertain the API must not say `completed` — it returns a distinct typed result. +Terminal asset cleanup and handle resolution happen only after the terminal is durable. The test proves live, +history, resume and reconcile agree. + +## CR-93 — Process-global catalog and parameter-learning state is not tenant-safe · Medium now, High for cloud +Process-global mutable state is fine for a single local user and wrong for the multi-tenant cloud surface. +**Fix + acceptance.** Scope it per run/session/tenant before any multi-tenant surface ships. Record the decision +even if the work is deferred. + +## CR-94 — One budget approval opens every redispatch without a numeric limit · High +An approved vertex skips pre-egress admission on redispatch, and one agent turn can make many tool rounds plus a +final generation. The user approves a projection they saw, but the approval means "this vertex is exempt", not +"up to this amount". +**Fix + acceptance.** Make the approval an immutable numeric lease bound to money/token/attempt scope, model and +expiry, consumed atomically by each egress. An approved node that exceeds its lease pauses again. + +## CR-95 — A mid-tool-loop budget pause replays the whole loop · High (Blocker while CR-12 is open) +A budget pause becomes a paused outcome; on approval the node is reset to pending and dispatched from the start. +The code already acknowledges that this repeats earlier provider and tool calls. +**Fix + acceptance.** Short term: forbid a mid-loop budget pause (fail closed). Long term: checkpoint the +continuation — provider messages, tool call/result pairs, round index — and resume from that point. A mid-loop +pause plus approval must not repeat a mutation. + +--- + +# Exit criteria + +This phase is done when **all** of the following hold: + +1. Every `CR-##` above is either closed with a break-verified test, or explicitly deferred in this document with + a named reason and the mutation that would close it. No silent omissions. +2. Each of the seven W1 items has an accepted ADR, and the ADRs land in dependency order. +3. Every canonical document that promised behaviour listed here says what the code now does — in particular the + claims about crash-safe resume, duplicate-free effects, bounded streams, first-branch-wins, schema-validated + input/output, and session-to-workflow graduation. +4. `pnpm run ci` exits 0, checked by exit code. +5. A closing register in this file states, per item, the code that closes it — verified by reading the code, not + by trusting the mark. Wave 1's completion claim was wrong twice before this discipline was adopted. + +## Scope note + +Two items already in flight are NOT part of this phase and keep their own tracking: the durable per-attempt +realized-cost ledger (accepted ADR, staged implementation) and the transaction-handle cleanup across the +remaining database stores. They are listed here only so nobody schedules them twice. From 7cf7d8ba36a41b8174c1e67b591ac14dac22b118 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 18:15:31 +0300 Subject: [PATCH 03/26] =?UTF-8?q?docs(roadmap):=20fold=20the=20plan=20revi?= =?UTF-8?q?ew=20of=20phase=202.6.5=20=E2=80=94=20boundary,=20exit=20rule,?= =?UTF-8?q?=20effect=20semantics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An adversarial review of the phase document itself. Every mechanical and structural claim was re-verified against the tree before acting; all of them held. Five blocking corrections: - **Phase boundary.** The metadata named Wave 2 as successor while CR-16 and W4 joined Wave 2 and the exit criteria demanded every item close — so the phase could not close before its own successor. Wave 2's MCP queue now EXECUTES as 2.6.5's W4; Wave 2 keeps the fs jail, secrets, config trust and the 2.5.5 certification. The hostile-MCP security sitting moves with the code, so the reviewer is booked once, not twice. - **#W15-1 is a prerequisite, not a parallel track.** Its five staged steps touch the same four files CR-10 and CR-12 restructure. Landing it after the durability spine means writing its barrier against a persistence path about to change. - **Exit criteria permitted a vacuous close.** The old rule let all 43 items be deferred with a reason and the phase declared complete. Fourteen are now non-deferrable — every W0/W1 item plus CR-50/55/73/80/92 and CR-95's short-term fix — on the ground that each has a cheap fail-closed option, so size argues for the cheap option, never deferral. - **CR-12's "exactly once" was technically unachievable** for an opaque non-idempotent effect: the interval target-completed → process-died → no-receipt is irreducibly ambiguous. Restated as three tiers (idempotency-key / queryable / opaque→ambiguous+ needs_attention, never auto-retried). architectural-principles.md §11 asserts the false version today and is named as a required correction in that ADR's own PR; the ADR now amends ADR-0041 and ADR-0037 rather than sitting beside them. - **CR-17 added** — engine.ts's ResumeFromCheckpointInput documents its own gap: inputs, executionMode and planOptions are caller-supplied and never verified, so a resume can silently be a different run even with a perfect log and a single owner. This falsifies crash-safe resume independently of CR-10/CR-11. Ordering and accuracy: - Oracle before spine — CR-90/CR-91 land first (they are the instrument CR-10/11/12/92 are proven with); CR-92 moves into the spine, since an API returning `completed` while history says `failed` is a runtime defect, not a harness concern. - CR-30 was presented as an open choice; ADR-0036 already decided bounded producer-await no-drop. Restated as "implement the accepted decision". - CR-14 and CR-15 acceptance widened to the full grammar / full InputValidationSchema. - CR-61 split: output_schema deep validation (ADR-0038 drift, needs a validator-dependency ADR) vs new CR-63 — agent input_schema is documented as metadata, so its absence is correct and the action is to verify no doc claims otherwise. - CR-52 now cites ADR-0039's deliberate deferral instead of reading as new drift. - CR-02's decision made from the code: the counter stays, because the rule both catch paths already apply is "count only when a provider engaged". - Unimplementable acceptance criteria fixed: #turnCount is a private field (assert through the turn cap), CR-13 must not assert model behaviour (structural + type-level only), CR-90 uses a synthetic fixture rather than a real git worktree. - Decision/ADR/gate register added; nine items marked decision-open do not start until the maintainer settles them. - Mechanics: 12 H1s → 1, 40 → 45 items, stale last-updated, "six of seven" → eight, review provenance table, and `pnpm coverage` added to the gate (it is a separate required check, not inside `pnpm run ci`, and this phase edits core heavily). pnpm run ci EXIT=0. Co-Authored-By: Claude Opus 5 --- docs/roadmap/current.md | 79 +- ...hase-2.6.5-core-reliability-remediation.md | 674 +++++++++++++----- 2 files changed, 552 insertions(+), 201 deletions(-) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 7a7ae061..1d9ca1bd 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -2,7 +2,7 @@ > Status: Living > -> Last updated: 2026-07-29 +> Last updated: 2026-08-10 - **Related**: [README.md](README.md), [phases/phase-2.5-cli-consolidation.md](phases/phase-2.5-cli-consolidation.md), [phases/phase-2.5.5-hardening-and-remediation.md](phases/phase-2.5.5-hardening-and-remediation.md), [phases/phase-2-cli.md](phases/phase-2-cli.md), [deferred-tasks.md](deferred-tasks.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) @@ -89,7 +89,9 @@ any order"* — never as headcount. flowchart TD W0["Wave 0 — One true baseline
baseline ✅ · CI truth · numbers"] W1["Wave 1 — Stop the bleeding ✅
3 CRITICALs · cost cap · ADR-0074"] - W2["Wave 2 — Shut the doors
MCP · fs jail · secrets
certifies 2.5.5 EXIT 1–3"] + LEDGER["#W15-1 — realized-cost ledger
ADR-0076 implementation"] + P265["Phase 2.6.5 — Core reliability
45 CR items · 8 P0 ADRs
absorbs the hostile-MCP class"] + W2["Wave 2 — Shut the doors
fs jail · secrets · config trust
certifies 2.5.5 EXIT 1–3"] W3["Wave 3 — Clear the ground
god-file decomposition · CLI net"] W4a["Wave 4a — The spine
2.6.A/D/H/K + 2 ADRs"] W4b["Wave 4b — Money floor & close-out
2.5.5.B/C/F/H drain · v0.2.0"] @@ -97,7 +99,7 @@ flowchart TD W5b["Wave 5b — Home becomes the product
2.6.G/I/J · then i18n sweep"] W6["Wave 6 — Hands, voice, lineage
2.6.M/B/E + 2.6.N foundation"] W7["Wave 7 — Orchestration & the gate
2.6.O/P · go/no-go → Phase 3"] - W0 --> W1 --> W2 --> W3 --> W4a --> W5a --> W5b --> W6 --> W7 + W0 --> W1 --> LEDGER --> P265 --> W2 --> W3 --> W4a --> W5a --> W5b --> W6 --> W7 W3 --> W4b W4b -.->|v0.2.0 release| W7 W4a -.->|file-disjoint lanes| W4b @@ -286,6 +288,10 @@ Ordered by whether the repo currently states something untrue, then by blast rad > The break-verify that matters most is step 4's ordering: deleting the `await` must redden a test, or the > ledger records the charge without the guarantee that makes it one. > +> **This is now the prerequisite of Phase 2.6.5, not a parallel track.** Its five steps touch the same four +> files `CR-10` (ordered append tail) and `CR-12` (effect journal) restructure, so it lands first — see the +> 2.6.5 section below. +> > **What ADR-0076 explicitly does NOT close**, so it is not read as more than it is: the cost of a TOOL EFFECT. > A resumed run can still re-execute an `http_request` POST, a `run_command` or an MCP mutation — the > duplicate EFFECT is the harm, and no cost bookkeeping prevents it. That needs a durable effect journal with @@ -425,26 +431,47 @@ the review of `#W15-16`, fixed with it, and now the signal that test drives. `llm-provider-seam.md` still claims model-discovery/media-poll keep a small SDK retry; ADR-0028's amendment note still says ADR-0074 §2–§5 have not landed. Each could send a future reader back toward a #91-class hole. -### Phase 2.6.5 — Core reliability remediation (before Wave 2) +### Phase 2.6.5 — Core reliability remediation (between Wave 1 and Wave 2) Three independent core reviews of the post-Wave-1 tree converged — independently — on the **same seven blocking gaps**: effect journal, stdio MCP consent-before-spawn, run lease, compaction trust elevation, stream grammar, input admission, event-log ordering. Three separate reviews landing on the same seven points is not opinion. The full, self-contained work list is -[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — 40 items -(`CR-01`…`CR-95`) with evidence, fix and acceptance criteria for each, written so the work can be done from that -document alone. - -Two things it changes about the plan below: - -- **Wave 2 is amended, not replaced.** Its MCP timeout/SSRF/ingress cluster stays; `CR-16` - (consent before a stdio MCP spawn) is added to it as a P0 in the same code area. -- **Its seven P0 items are ADR-first**, in dependency order — an effect journal is meaningless while two - processes can own one run, and run ownership rests on the log being an ordered prefix. - -Three items are already half-closed by Wave 1 and finish first (`CR-01`–`CR-03`); `CR-03` is a propagation gap -from `#W15-10`'s own fix — three `--json` paths never got the safe serializer. +[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — **45 items** +(`CR-01`…`CR-95`) with evidence, fix, acceptance criteria and a decision/ADR/gate register, written so the work +can be done from that document alone. An adversarial plan review on 2026-08-10 corrected the phase boundary, +the exit rule and the execution order, and added two items (`CR-17` resume identity, `CR-63` `input_schema` +docs-only). + +**This is the corrected execution order, and it is what the graph above shows:** + +1. **`#W15-1` first.** Its five staged steps touch `run-event.ts`, `engine.ts`, `checkpoint.ts` and + `run-history-store.ts` — the same four files `CR-10` and `CR-12` restructure. Landing it after the durability + spine means writing its barrier against a persistence path that is about to change. +2. **Then Phase 2.6.5, closing before Wave 2 opens.** Which is only true if the work Wave 2 would block on + lives there — so **the hostile-MCP threat class moves into 2.6.5**: `CR-16` (consent before a stdio spawn) + plus `CR-40`–`CR-42` (transport cancellation, DNS/redirect SSRF, ingress bounds). Wave 2's queue item 2 is + **executed as 2.6.5's `W4`**; the 2.5.5 finding ids keep their home in the 2.5.5 phase doc and certify from + there. Wave 2 keeps the MCP name-collision/tool-poisoning trust items, the fs jail, the secrets layer, + persistence-security and the certification. +3. **Within 2.6.5, the oracle comes before the spine.** `CR-90`/`CR-91` (crash + durable-truth harness) land + first because they are the instrument `CR-10 → CR-11 → CR-92 → CR-12` is proven with; `CR-92` moved out of + the harness group into the spine, since an API returning `completed` while history says `failed` is a + runtime defect, not a test concern. + +Two rules that changed with the plan review: + +- **Fourteen items are non-deferrable** — every W0/W1 item plus `CR-50`, `CR-55`, `CR-73`, `CR-80`, `CR-92` and + `CR-95`'s short-term fix. Each has a cheap fail-closed option (refuse, remove, narrow the claim), so "too big + to fix now" argues for the cheap option, never for deferral. The previous exit criterion permitted deferring + all 43 items and declaring the phase complete. +- **The gate is `pnpm run ci` AND `pnpm coverage`.** `coverage` is a separate required CI check and is not + inside the `ci` script; this phase edits `packages/core` heavily, and the CI coverage job does not block on + `core`. + +Three items are already half-closed by Wave 1 and finish right after the oracle (`CR-01`–`CR-03`); `CR-03` is a +propagation gap from `#W15-10`'s own fix — three `--json` paths never got the safe serializer. ### Wave 2 — Shut the doors @@ -453,10 +480,14 @@ certify while the reviews are still booked. 1. 2.5.5.D · **project-layer MCP name collision redirecting a global secret** (G19), then **MCP tool-definition poisoning** (#202). -2. **MCP queue, strict** — connect-phase timeouts on every transport (#35, G32, #205) → transport/discovery/result - size bounds against a hostile server (G33, #201, #209, #288) → structured `serverId`/`reason` discriminants - (#203, #204) → the two fail-loud violations (#206, #207) → per-file transport-adapter tests (#297) → - `CLIENT_INFO.version` (#208). +2. **MCP queue, strict — EXECUTED AS PHASE 2.6.5 `W4`, not here.** Connect-phase timeouts on every transport + (#35, G32, #205) → transport/discovery/result size bounds against a hostile server (G33, #201, #209, #288) → + structured `serverId`/`reason` discriminants (#203, #204) → the two fail-loud violations (#206, #207) → + per-file transport-adapter tests (#297) → `CLIENT_INFO.version` (#208). These finding ids keep their home in + [phases/phase-2.5.5-hardening-and-remediation.md](phases/phase-2.5.5-hardening-and-remediation.md) and + certify from there in step 6 below; the *work* happens in 2.6.5 alongside `CR-16` (consent before a stdio + spawn) and `CR-41` (the DNS/redirect SSRF hole), because they are the same code and the same security + sitting. Splitting them across two phases would book the hostile-MCP reviewer twice. 3. **The fs jail, ONE reviewed change set** — 2.5.5.D · sensitive-credential floor for `.kube`/`.azure`/gcloud (#36, #39) → extract `deepestExistingReal` to `packages/shared` (#235) → **2.6.M's `project`-tier `extraRoots`** and **2.6.N's fs-floor home-anchoring**, both *pulled forward*. `~/.relavium/tmp/` (G21) @@ -470,9 +501,9 @@ certify while the reviews are still booked. context sites (G22). 2.5.5.D · `default_headers` (G20, after D13). 6. **Certify 2.5.5 exit criteria 1, 2 and 3** + the sub-stream D acceptance sign-off. **Closes M2.5.5-2 (D half).** -> Book **five security-review sittings by threat class** (hostile-MCP · fs/path-jail · secrets-at-rest · -> secrets-input · config-trust), not ~30 per-PR passes. Reviewer availability, not code, is this wave's -> critical path. +> Book **four security-review sittings by threat class** (fs/path-jail · secrets-at-rest · secrets-input · +> config-trust), not ~30 per-PR passes. The **hostile-MCP** sitting moved to Phase 2.6.5 with the queue-2 work; +> it covers `CR-16` and `CR-40`–`CR-42` there. Reviewer availability, not code, is this wave's critical path. ### Wave 3 — Clear the ground diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index d488173a..b8c767a6 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -1,11 +1,14 @@ # Phase 2.6.5 — Core reliability remediation (interlude) - **Status**: planned -- **Opened**: 2026-08-09 -- **Predecessor**: Wave 1 of the 2.5.5 remediation (complete — PR #81) -- **Successor**: Wave 2 of the 2.5.5 remediation (MCP + fs jail), which this phase AMENDS rather than replaces -- **Kind**: interlude, like [phase 2.5.5](phase-2.5.5-hardening-and-remediation.md) — no new product surface, only - the invariants an existing surface already claims +- **Opened**: 2026-08-09 · **Plan corrected**: 2026-08-10 +- **Predecessor**: Wave 1 of the 2.5.5 remediation (complete — PR #81), then the `#W15-1` realized-cost + ledger implementation (an accepted ADR with a staged implementation — see [Prerequisite](#prerequisite)) +- **Successor**: Wave 2 of the 2.5.5 remediation, **reduced** — this phase absorbs Wave 2's hostile-MCP + threat class (see [Phase boundary](#phase-boundary)); Wave 2 keeps the filesystem jail, the secrets + layer, persistence-security and the 2.5.5 certification +- **Kind**: interlude, like [phase 2.5.5](phase-2.5.5-hardening-and-remediation.md) — no new product surface, + only the invariants an existing surface already claims ## Why this phase exists @@ -25,6 +28,21 @@ not deliver: *crash-safe resumable workflow*, *duplicate-free side effect*, *bou branch wins*, *schema-validated input/output*, *full session-to-workflow graduation*. A claim the code does not keep is worse than a missing feature, because it is the claim a user plans around. +### Provenance of the findings + +Recorded so the severities below can be audited rather than trusted: + +| Source | Ran against | What it was | +|--------|-------------|-------------| +| Review A | post-Wave-1 tree, 2026-08-08/09 | Full-engine reliability read, findings with file/line evidence | +| Review B | post-Wave-1 tree, 2026-08-09 | Independent full-engine read, no sight of A | +| Review C | post-Wave-1 tree, 2026-08-09 | Independent read scoped to durability, MCP and cost | +| Plan review | this document + tree, 2026-08-10 | Adversarial read of the plan itself; produced `CR-17`, `CR-63`, the phase-boundary correction and the non-deferrable exit rule | + +"Consensus" severity below means at least two of A/B/C rated it independently. Every evidence line was +re-verified against the working tree before it was written here; where a review's claim did not survive that +check it was dropped rather than restated. + ## What must NOT change Recorded so a remediation pass does not quietly undo a decision that is correct: @@ -36,15 +54,50 @@ Recorded so a remediation pass does not quietly undo a decision that is correct: - Event-sourced checkpoint/resume as the recovery model. - Keys in the OS keychain only. +## Phase boundary + +This phase runs **between Wave 1 and Wave 2**, and it closes before Wave 2 opens. That is only true if the +work Wave 2 would otherwise have blocked on lives here, so: + +- **The hostile-MCP threat class moves into this phase.** `CR-16` (consent before a stdio spawn) plus `W4` + (`CR-40`–`CR-42`) are the same code area and the same security-review sitting as Wave 2's MCP queue. Wave 2's + queue item 2 is **executed as this phase's `W4`**; its 2.5.5 finding ids keep their home in + [phase-2.5.5](phase-2.5.5-hardening-and-remediation.md) and are certified from there. +- **Wave 2 keeps** the project-layer MCP name collision and tool-definition poisoning (a trust/config concern, + not the transport boundary), the unified filesystem jail, the secrets layer, persistence-security, and the + certification of 2.5.5 exit criteria 1–3. +- **`current.md` is canonical for execution order** and carries the corrected graph. This section states the + scope line; the ordering statement lives there. + +The alternative — leaving `CR-16`/`W4` in Wave 2 and declaring this phase closed *during* Wave 2 — was +rejected: a phase whose exit criteria cannot be evaluated until a later phase is half-done is not a gate. + +## Prerequisite + +**`#W15-1` — the durable per-attempt realized-cost ledger — lands before `W1` starts.** It is not part of this +phase, but it is not independent of it either. + +[ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md) is Accepted and its implementation +is staged in five steps in [current.md](../current.md). Those five steps touch +`packages/shared/src/run-event.ts`, `packages/core/src/engine/engine.ts`, +`packages/core/src/engine/checkpoint.ts` and `packages/db/src/run-history-store.ts` — the same four files +`CR-10` and `CR-12` restructure. Landing it after the durability spine means writing its barrier against a +persistence path that is about to change, then rebasing it onto the moved floor. Landing it first means its +awaited emit inherits `CR-10`'s ordered tail for free. + +There is a second reason, and it is the one this project keeps re-learning: an accepted ADR with no +implementation is a decision that reads as shipped. Wave 1's completion claim was wrong twice for exactly that +shape. + ## Working discipline for this phase This phase inherits the discipline Wave 1 arrived at the hard way. It is not optional ceremony; every clause below exists because its absence produced a real defect during Wave 1. -1. **A decision gets an ADR before it gets code.** Six of the seven P0 items below change what a contract - *means* — what "exactly once" means, what `system` authority means, what "the stream ended" means. Wave 1 - proved that treating a decision as a task item produces false completion claims. Write the ADR, get it - accepted, then implement. +1. **A decision gets an ADR before it gets code.** All eight `W1` items change what a contract *means* — what + "exactly once" means, what `system` authority means, what "the stream ended" means, what a resume is allowed + to assume. Wave 1 proved that treating a decision as a task item produces false completion claims. Write the + ADR, get it accepted, then implement. 2. **Break-verify every regression test, and verify the mutation actually applied.** Wave 1 produced three tests that passed with the production code mutated. Each was found only by checking that the mutation had landed (e.g. `grep -c` the mutated symbol → 0) before trusting a red. @@ -53,26 +106,121 @@ below exists because its absence produced a real defect during Wave 1. 4. **Record rather than assert.** Any claim that cannot be demonstrated goes in as a limitation, not as a property. 5. **Fix the canonical doc in the same change.** One canonical home per artifact; a fix that leaves the spec - describing the old behaviour has not landed. -6. **`pnpm run ci` must exit 0.** Check the exit code, not the reporter output. + describing the old behaviour has not landed. Where a *standard* states a guarantee the code does not keep, + the correction lands with the ADR, not with the last line of implementation. +6. **Never assert on model behaviour.** A prompt-security test proves a *structural* property — which role a + byte can reach, which type the builder accepts — never "the model did not comply". A test that depends on a + model's judgement is a hollow test under clause 3. +7. **Security review is booked by threat class, not per PR.** Four sittings cover this phase: + **prompt/trust provenance** (`CR-13`), **hostile MCP** (`CR-16`, `CR-40`–`CR-42`), **media bytes** + (`CR-50`, `CR-53`, `CR-54`), **provider/config trust** (`CR-80`). Each sitting produces a recorded + [security-review](../../standards/security-review.md) checklist entry with a direct adversarial test — not a + reviewer's assurance. +8. **The gate is `pnpm run ci` AND `pnpm coverage`, both exit 0, checked by exit code.** `pnpm run ci` does not + include coverage; `coverage` is a separate required check. CI's coverage job enforces the floor for + `@relavium/llm` and `@relavium/mcp` only — `packages/core` is measured but not blocking, a dated carve-out + whose promotion trigger is Wave 3. This phase edits `core` heavily, so the local `pnpm coverage` (which + enforces all three) is the honest gate here. ## How the work is grouped -`CR-##` ids below are this phase's own; they are stable and self-contained. Severity is the consensus of the -three reviews. +`CR-##` ids below are this phase's own; they are **stable** and never renumbered. The thematic groups `W0`…`W9` +are the reference layout. The **execution order is separate from the grouping** and is stated once, below. + +Severity is the consensus of the three reviews, except where an item is marked *(plan review)*. + +### Execution order + +The groups are not the schedule. Test honesty comes first because it is the oracle everything after it is +proven with, and the durability spine is a chain. + +```mermaid +flowchart TD + PRE["Prerequisite
#W15-1 realized-cost ledger"] + ORACLE["Oracle first
CR-90 · CR-91"] + HALVES["W0 — finish the halves
CR-01 · CR-02 · CR-03"] + SPINE["Durability spine
CR-10 → CR-11 → CR-92 → CR-12"] + IND["Independent P0 lines
CR-13 · CR-14+CR-21 · CR-15+CR-17"] + MCP["Hostile MCP
CR-16 · CR-40 · CR-41 · CR-42"] + CLAIM["False-claim blockers
CR-50 · CR-55 · CR-73 · CR-80 · CR-94 · CR-95"] + REST["Remaining P1 themes
W2 · W3 · W5 · W6 · W7 · W8 · W9 residual"] + EXIT["Exit certification"] + PRE --> ORACLE --> HALVES --> SPINE --> CLAIM --> REST --> EXIT + HALVES --> IND --> REST + HALVES --> MCP --> REST +``` + +`CR-92` sits inside the spine, not in `W9`: an API that returns `completed` while durable history says `failed` +is a runtime correctness defect, not a harness concern. `CR-90`/`CR-91` come before the spine because a crash +and durable-truth oracle is what proves `CR-10`/`CR-11`/`CR-12`/`CR-92` landed — building it last would mean +asserting the spine with the harness the reviews already found insufficient. - **W0 — finish the halves.** Cheap, already half-done. Leaving them half-done is the exact failure this phase is correcting. -- **W1 — the seven P0 blockers.** ADR-first, in dependency order. +- **W1 — the P0 blockers.** ADR-first. Seven are the convergent set; `CR-17` was added by the plan review. - **W2…W9 — P1 themes.** Grouped so each theme is one reviewable PR, not sixty scattered tickets. +### Decision, ADR and gate register + +Every row states whether the decision is already made. **An item whose decision column says "open" does not +start until the maintainer settles it** — deciding during implementation is the failure mode this phase exists +to correct. + +| Item | Decision | ADR | Non-deferrable | Security sitting | +|------|----------|-----|----------------|------------------| +| `CR-01` | made (route through the latch) | — | yes | — | +| `CR-02` | made (counter stays; rule = *engaged*) | — | yes | — | +| `CR-03` | made (route through `stringifyJsonLine`) | — | yes | — | +| `CR-10` | made (one ordered append tail per run) | new | yes | — | +| `CR-11` | made (lease + fencing token) | new | yes | — | +| `CR-12` | made (three-tier effect contract) | new, amends [ADR-0041](../../decisions/0041-external-action-governance-seam.md) + [ADR-0037](../../decisions/0037-engine-tool-execution-boundary.md) | yes | — | +| `CR-13` | made (summary is untrusted, never `system`) | new, supersedes [ADR-0062](../../decisions/0062-context-compaction-and-cli-history-commands.md) §1 | yes | prompt/trust | +| `CR-14` | made (exactly one terminal, grammar pinned) | new | yes | — | +| `CR-15` | made (engine-side admission) | new | yes | — | +| `CR-16` | made (consent before spawn, lazy connect) | new | yes | hostile MCP | +| `CR-17` | made (persist and verify resume identity) | with `CR-15`'s | yes | — | +| `CR-20` | made (honour `timeout_ms`) | — | no | — | +| `CR-21` | made (per-attempt deadline) | with `CR-14`'s | no | — | +| `CR-22` | made (absolute deadlines) | — | no | — | +| `CR-23` | **open** — grace period length, quarantine policy | new | no | — | +| `CR-30` | made — implement [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md)'s accepted no-drop producer-await | none (already decided) | no | — | +| `CR-31` | **open** — the cap values | — | no | — | +| `CR-32` | **open** — the bound values | — | no | — | +| `CR-33` | **open** — retention policy shape | — | no | — | +| `CR-40` | made (forward signal + deadline) | with `CR-16`'s | no | hostile MCP | +| `CR-41` | made (apply the built-in egress floor) | with `CR-16`'s | no | hostile MCP | +| `CR-42` | **open** — the ingress bounds | with `CR-16`'s | no | hostile MCP | +| `CR-50` | **open** — complete the handle path, or remove the tool | — | yes | media bytes | +| `CR-51` | made (gate on model-level capability) | — | no | — | +| `CR-52` | made (pull [ADR-0039](../../decisions/0039-same-provider-reasoning-replay.md)'s deferral forward) | ADR-0039 follow-up | no | — | +| `CR-53` | made (bounded stream to the store) | — | no | media bytes | +| `CR-54` | made (content-hashed handle at first resolution) | — | no | media bytes | +| `CR-55` | made (missing rate ⇒ unpriced) | — | yes | — | +| `CR-60` | **open** — race semantics, or rename + correct the table | — | no | — | +| `CR-61` | **open** — needs a JSON-Schema validator dependency | new (dependency ADR) | no | — | +| `CR-62` | **open** — extract dependencies, or fail loudly | — | no | — | +| `CR-63` | made (verify the docs; no runtime change) | — | no | — | +| `CR-70` | **open** — persist bounded tool pairs, or narrow the spec | — | no | — | +| `CR-71` | with `CR-70` | with `CR-70`'s | no | — | +| `CR-72` | **open** — implement, or reject at parse | — | no | — | +| `CR-73` | made (wire it, or stop advertising it) | — | yes | — | +| `CR-80` | made (fail closed) | — | yes | provider/config | +| `CR-81` | made (match the canonical request) | — | no | — | +| `CR-82` | made (missing usage is unknown) | — | no | — | +| `CR-90` | made (exclude repo-local checkouts) | — | no | — | +| `CR-91` | made (build the durable-truth oracle) | — | no | — | +| `CR-92` | made (durable outbox, distinct typed result) | with `CR-10`'s | yes | — | +| `CR-93` | made (scope per run/session/tenant; may defer the work, not the decision) | — | no | — | +| `CR-94` | made (numeric approval lease) | — | no | — | +| `CR-95` | made (short term: fail closed) | — | yes (short term) | — | + --- -# W0 — Finish what is already half-done +## W0 — Finish what is already half-done These are partially closed. Each has a verified open half. -## CR-01 — `session:cancelled` bypasses the session durability latch · High +### CR-01 — `session:cancelled` bypasses the session durability latch · High **Evidence.** The CLI chat persister wraps its cost and turn writes in the `persistDurably` failure latch, and since Wave 1 the `session:compacted` and `session:trimmed` arms go through it too. The `session:cancelled` arm @@ -86,21 +234,27 @@ cancel path report success. The durability latch never sets, so nothing gates th **Acceptance.** A failing `writeTurn` on cancel sets `durabilityFailure`, and the next egress is refused. Break-verify by removing the wrapper. -## CR-02 — A failed turn flush leaves the turn counter incremented · Medium +### CR-02 — A failed turn flush leaves the turn counter incremented · Medium -**Evidence.** `packages/core/src/engine/agent-session.ts` increments `#turnCount` before it awaits -`flushBudgetCommitments()`. Wave 1 fixed the transcript half of this (the flush now precedes the assistant -append, so the rollback's one-message `pop()` is correct again). The counter half was not examined. +**Evidence.** `packages/core/src/engine/agent-session.ts` increments `#turnCount` on the success path *before* +it awaits `flushBudgetCommitments()`. Wave 1 fixed the transcript half of this (the flush now precedes the +assistant append, so the rollback's one-message `pop()` is correct again). The counter half was not examined. -**Why it matters.** A turn whose durability flush rejects has still consumed a slot against the hard turn cap. +**The decision, made.** The counter **stays where it is.** The rule the code already applies on both catch +paths is *count a turn against the hard cap only when a provider actually engaged* — and by the time the +success path reaches the increment, `#runTurn` has resolved, so a provider did engage. A flush rejection after +that point is a durability failure, not evidence the turn never happened. Moving the increment after the flush +would make a billed turn free. -**Fix.** Decide deliberately, then make the code say it: either the counter moves after the flush, or it stays -where it is because an engaged provider call counts regardless — which is what the two catch paths already do -for `engaged` turns. Whichever is chosen, state it in a comment and pin it with a test. +**Fix.** Keep the increment; state the rule in a comment at the success-path site the way the two catch paths +already state it, so the next reader does not re-derive it. Pin it with a test. -**Acceptance.** A rejecting flush leaves `#turnCount` at the value the chosen rule requires, asserted directly. +**Acceptance.** A rejecting `flushBudgetCommitments` on an otherwise-successful turn still consumes a turn +against the cap. `#turnCount` is a JS private field and **must not** be given a production accessor to test — +assert through the observable turn-cap behaviour (`maxTurns: 1`, one flushing-failing turn, the next +`sendMessage` refused by the cap). Break-verify by moving the increment below the flush. -## CR-03 — Three `--json` paths still bypass the safe serializer · High +### CR-03 — Three `--json` paths still bypass the safe serializer · High **Evidence.** Wave 1 introduced `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), which losslessly escapes C1 controls and Trojan-Source bidi characters, and routed the workflow renderer and the record writer @@ -119,13 +273,17 @@ the raw code points do not survive while `JSON.parse` round-trips to the identic --- -# W1 — The seven P0 blockers +## W1 — The P0 blockers + +`CR-10 → CR-11 → CR-92 → CR-12` is a chain: an effect journal is meaningless while two processes can own the +same run, run ownership rests on the log being an ordered prefix, and a terminal that can diverge from durable +truth undermines both. `CR-13`, `CR-14` and `CR-15`+`CR-17` are independent lines that can run in parallel. +`CR-16` opens the hostile-MCP line. -Dependency order matters. `CR-10 → CR-11 → CR-12` is a chain: an effect journal is meaningless while two -processes can own the same run, and run ownership rests on the log being an ordered prefix. `CR-13`, `CR-14` -and `CR-15` are independent and can run in parallel. +Seven of the eight are the set all three reviews converged on. `CR-17` was added by the plan review of this +document and verified against the engine's own interface contract. -## CR-10 — The durable event log is not a gap-free ordered prefix · Blocker · needs an ADR +### CR-10 — The durable event log is not a gap-free ordered prefix · Blocker · needs an ADR **Evidence.** The engine assigns sequence numbers centrally but starts each event's persistence independently for concurrency; the code comment states the split explicitly ("persistence concurrent, delivery serialized"). @@ -144,8 +302,9 @@ compare-and-append against the expected last sequence. **Acceptance.** A store harness that delays event `N` and commits `N+1` must be rejected, not accepted. A crash injected between the two must leave a prefix with no hole. Existing gap-free assertions must still pass. +Break-verify by restoring the concurrent start. -## CR-11 — No cross-process run ownership or CAS · Blocker · needs an ADR +### CR-11 — No cross-process run ownership or fencing · Blocker · needs an ADR **Evidence.** The engine states that its cross-process guarantee rests on store uniqueness, but the only DB uniqueness is `(run_id, seq)`. `resumeFromCheckpoint` performs an in-memory check, then loads the checkpoint and @@ -154,19 +313,23 @@ builds an independent `RunExecution`. There is no run lease, fencing token, or c **Failure.** Two CLI or desktop processes read the same paused run at the same time and become two independent side-effect producers for one run. -**Fix.** A run-level lease with a fencing token, or an atomic CAS on `last_seq`: -`acquireRunLease(runId, ownerId, expectedGeneration, ttl)`. The process that loses the CAS becomes a read-only -observer rather than a second owner. +**The decision, made.** A **run lease with a monotonic fencing token**, checked on every durable write — not a +bare `last_seq` CAS. A CAS on `last_seq` stops two processes writing the same row; it does **not** stop two +processes performing the same external effect before either of them writes anything. The fence is what makes a +stale owner harmless after it loses the lease. + +**Fix.** `acquireRunLease(runId, ownerId, expectedGeneration, ttl)` returning a fencing token; every durable +write carries the token and is rejected if it is stale. The process that loses becomes a read-only observer. **Acceptance.** Two concurrent resumes of one paused run: exactly one proceeds, the other degrades to observer -with a typed, actionable error. A lease that expires mid-run is fenced out of further durable writes. +with a typed, actionable error. A lease that expires mid-run is fenced out of further durable writes — proven +by driving a write with a stale token and asserting the rejection, not by asserting the lease table's contents. -## CR-12 — No durable effect/idempotency journal on the hot path · Blocker · needs an ADR +### CR-12 — No durable effect/idempotency journal on the hot path · Blocker · needs an ADR **Evidence.** `NodeExecContext` carries an attempt number but no run/effect idempotency key. `ToolDispatchContext` carries a node id but no run correlation or semantic key. The registry calls -`def.dispatch` directly with no durable prepare step. Our own architecture documents state that a stable key -prevents duplicates — that key does not exist in the dispatch context. +`def.dispatch` directly with no durable prepare step. **Failure.** `node:started` persists → an `http_request` POST or MCP mutation completes at the target → the process dies before the tool result or `node:completed` persists → resume re-runs the node → the target sees @@ -175,19 +338,43 @@ the same effect twice. **Impact.** Duplicate ticket, deploy, payment, message or commit. The *duplicate-free* product claim is false until this closes, and audit/compensation cannot be built on top of it. +**The contract, stated precisely — three tiers, not one.** A general HTTP POST, MCP mutation or shell command +cannot be made exactly-once by the engine alone. The interval *"the target completed the effect → the process +died → no receipt reached the store"* is irreducibly ambiguous when the target offers no idempotency key and no +way to ask. Promising exactly-once across that gap would be a claim of exactly the kind this phase exists to +remove. The contract is therefore tiered: + +1. **Target accepts an idempotency key** → safe retry under the same key. Effectively exactly-once, and the + only tier that may say so. +2. **Target's outcome is queryable** → reconcile from a receipt lookup before deciding. Exactly-once after + reconciliation. +3. **Opaque, non-idempotent effect** → `dispatched → ambiguous → needs_attention`. **Never auto-retried.** This + is *at-most-once dispatch attempt*, and the docs must say that, not "exactly once". + **Fix.** Add `EffectAttemptId = runId + nodeId + nodeAttempt + toolCallId + effectiveArgsHash` to `ToolDispatchContext`. Side-effectful host ports take it as a required argument. A durable state machine in the -run store: `prepared → dispatched → committed | ambiguous → needs_attention`. +run store: `prepared → dispatched → committed | ambiguous → needs_attention`, with the tier recorded per effect. + +**Canonical docs this must correct, in the ADR's own PR:** -**Acceptance.** Kill the process after the effect completes at the target — resume produces no duplicate. An -ambiguous non-idempotent effect is never auto-retried. The same effect key arriving from two processes commits -once. +- [architectural-principles.md](../../standards/architectural-principles.md) §11 currently states that a stable + key derived from `runId + nodeId + retryCount` means *"a retry never double-applies a side effect"*. That is + false today and remains false for tier 3 after this lands. It must be rewritten to the tiered contract. +- [ADR-0041](../../decisions/0041-external-action-governance-seam.md) places transactional/idempotent wrapping + in the optional enterprise `ActionGuard`. This ADR moves a baseline effect-identity floor below that seam, so + it **amends** ADR-0041 rather than sitting beside it. +- [ADR-0037](../../decisions/0037-engine-tool-execution-boundary.md) owns the `ToolHost` boundary this changes. -**Relationship to the realized-cost ledger.** The accepted cost-ledger ADR explicitly scopes effect duplication -out and names this as the decision that owns it. That ADR's implementation and this item are independent; this -one is the larger of the two. +**Acceptance.** Kill the process after the effect completes at the target — resume produces no duplicate for +tier 1 and tier 2, and produces `needs_attention` with no retry for tier 3. The same effect key arriving from +two processes commits once. A tier-3 effect is never auto-retried, proven by a test that would go red if the +retry path were reachable. -## CR-13 — Compaction summary is elevated to `system` authority · Blocker (community/untrusted content) · needs an ADR +**Relationship to the realized-cost ledger.** [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md) +explicitly scopes effect duplication out and names this as the decision that owns it. That ADR's implementation +is this phase's [prerequisite](#prerequisite); this item is the larger of the two. + +### CR-13 — Compaction summary is elevated to `system` authority · Blocker (untrusted content) · needs an ADR **Evidence.** The prior conversation is handed to the summarizer as user-role data; the summarizer's model output is taken as a plain string, becomes the context preamble, and is concatenated directly into the authored @@ -199,36 +386,60 @@ instructions". The summarizer preserves it as a task. On the next tool-capable t `system` role — and survive a restart, because the summary is persisted. An XML fence is not a trust boundary. **Conflict.** Our binding security standard treats model output and tool results as untrusted and forbids -concatenating untrusted content into `system`. The compaction ADR fixed exactly that concatenation as its -decision, so this needs an append-only superseding ADR, not a patch. - -**Fix.** Carry the summary as untrusted. Dynamic summary bytes must never reach the `system` builder. Supply the -summary as a separate untrusted content part in user-role context. Make `AgentTurnParams.system` accept only a -branded type the authored builder can produce. - -**Acceptance.** An injected summary containing a fence-closing sequence plus instructions cannot alter tool -authorization on the following turn, before or after a restart. A type-level test proves a dynamic string cannot -be passed as `system`. - -## CR-14 — A stream that ends without a terminal `stop` counts as success · Blocker · needs an ADR - -**Evidence.** The fallback chain emits a success attempt when the provider iterator ends cleanly with no usage. -The agent turn starts with a default `stopReason` of `stop` and returns a normal result without ever seeing a -terminal chunk. **An existing test pins this behaviour as success** — so the fix changes that test, deliberately, -rather than deleting it. +concatenating untrusted content into `system`. [ADR-0062](../../decisions/0062-context-compaction-and-cli-history-commands.md) +§1 fixed exactly that concatenation as its decision, so this needs an append-only superseding ADR, not a patch. + +**What the superseding ADR must answer.** ADR-0062 §1 did not choose the preamble by accident — it explicitly +rejected injecting the summary as a transcript message because *"a summary message either forces an +`assistant`-first array Anthropic rejects"*. The new ADR has to answer that, not ignore it. The likely shape is +a separate **untrusted content part inside the first user-role turn**, which is neither a standalone message nor +a `system` concatenation; whatever is chosen, the rejected alternative is engaged on its own terms. + +**Fix.** Carry the summary as untrusted. Dynamic summary bytes must never reach the `system` builder. Make +`AgentTurnParams.system` accept only a branded type the authored builder can produce, so the compiler — not a +convention — enforces it. + +**Acceptance.** Structural and type-level only, per working-discipline clause 6: + +- A type-level test proves a dynamic `string` cannot be passed as `system`; the branded type is constructible + only by the authored-prompt builder. +- Given a summary containing a fence-closing sequence plus instructions, the assembled request has those bytes + in a user-role content part and **zero** occurrences in the `system` field — asserted on the built request, + before any provider call, and again after a restore from persistence and after a model reseat. +- Tool authorization for the following turn is computed from policy alone; a test mutates the summary text + arbitrarily and asserts the resolved tool set is byte-identical. +- **No assertion of the form "the model did not obey the injected instruction."** + +### CR-14 — A stream that ends without a terminal `stop` counts as success · Blocker · needs an ADR + +**Evidence.** The fallback chain emits a success attempt when the provider iterator ends cleanly with no usage +(`packages/llm/src/fallback-chain.ts`). The agent turn starts with a default `stopReason` of `stop` and returns +a normal result without ever seeing a terminal chunk. The canonical `StreamChunkSchema` +(`packages/llm/src/types.ts`) carries both `stop` and `error` as terminal arms. **An existing test pins the +no-terminal case as success** — so the fix changes that test, deliberately, rather than deleting it. **Failure.** A transport, proxy or provider closes after `text_delta: "partial"` with no terminal chunk. Relavium treats the partial text as a completed assistant answer and passes it downstream as a successful node output. -**Fix.** Make the stream grammar require exactly one terminal. A clean EOF with no terminal is a `transport` or -`protocol` error. If content was already committed, do not fail over or retry — surface it. +**Fix — the grammar, stated in full.** The ADR pins all of it, and names which layer enforces it (adapter vs. +`FallbackChain`) so the rule has one owner: -**Acceptance.** A stream ending without a terminal produces a classified error, not a node success. A -content-committed truncation does not trigger failover. The superseded test is rewritten to assert the new rule -and to keep the reasoning for the behaviour it replaces. +1. Exactly one terminal per stream: `stop` **xor** `error`. +2. The terminal is the last chunk. +3. No `*_delta` or content chunk after the terminal. +4. Two `stop`, `stop` + `error`, or two `error` are protocol violations. +5. A clean EOF with no terminal is a `transport`/`protocol` error, never a success. +6. An empty stream that reaches EOF with no chunks at all is an error. +7. A pre-content failure may fail over; a **content-committed** failure must not fail over or retry — it is + surfaced. -## CR-15 — The engine does not enforce the authored input contract · Blocker · needs an ADR +**Acceptance.** One test per numbered rule above, driven through a fake provider. A stream ending without a +terminal produces a classified error, not a node success. A content-committed truncation does not trigger +failover. The superseded test is rewritten to assert the new rule **and keeps a note recording the reasoning for +the behaviour it replaces**, the same way `checkpointer.test.ts` was handled in Wave 1. + +### CR-15 — The engine does not enforce the authored input contract · Blocker · needs an ADR **Evidence.** The shared contract states the engine validates inputs before a run starts. `WorkflowEngine.start()` passes the caller's `inputs` object straight to the run execution, which stores it without cloning, applying @@ -243,11 +454,24 @@ which breaks the "one engine, every surface" guarantee directly. **Fix.** A pure `resolveAndValidateWorkflowInputs` in core, running before id generation and before the first event. -**Acceptance.** Defaults are applied by the engine, not the surface. A type/enum violation fails admission with -a typed error before `run:started`. The same workflow and inputs behave identically through the engine API and -through the CLI. - -## CR-16 — Stdio MCP spawns a local process before consent · Blocker · needs an ADR · **amends Wave 2** +**Acceptance — the whole authored contract, not a sample.** `InputValidationSchema` +(`packages/shared/src/workflow.ts`) carries `format`, `pattern`, `enum`, `min`, `max`, `min_length`, +`max_length`, alongside `required`, `default` and `type`. Each gets a case: + +- A missing `required` input fails admission. +- An unknown input key fails admission. +- Every validation field above rejects a violating value and accepts a conforming one. +- A declared `default` is itself validated — an authored default that violates its own rules fails at parse, + not at run. +- The coercion/strictness split between surface and engine is stated in the ADR and pinned: the CLI's coarse + coercion happens *before* the engine, and the engine is strict about what it receives. +- The engine **clones** the caller's `inputs`; mutating the caller's object after `start()` does not change the + run. +- `start` and `resume` apply the same admission. +- An admission failure produces a typed error and **no `runId`, no `run:started`, no row** — asserted by + checking the store is untouched. + +### CR-16 — Stdio MCP spawns a local process before consent · Blocker · needs an ADR **Evidence.** The MCP connection is opened while the chat session is being built, before a mode or turn starts; the agent-run path prepares MCP first and applies mode policy afterwards. An agent or workflow declaration @@ -261,226 +485,305 @@ load time. fingerprint fails closed. In non-interactive mode, fail closed unless an explicit `--allow-mcp-stdio ` is supplied. Replace auto-start with lazy connect on first actual MCP tool need. -**Acceptance.** Importing and opening an artifact with a stdio MCP server spawns nothing until consent. A -non-interactive run without the digest flag fails closed with an actionable message. A changed fingerprint -re-prompts. +**Acceptance.** Importing and opening an artifact with a stdio MCP server spawns nothing until consent — proven +with a **real spawn counter** injected at the process boundary, asserted at zero, not by inspecting a state +flag. A non-interactive run without the digest flag fails closed with an actionable message. A changed +fingerprint re-prompts. + +### CR-17 — A resume trusts caller-supplied identity it never verifies · Blocker *(plan review)* · ADR with `CR-15` + +**Evidence.** `ResumeFromCheckpointInput` (`packages/core/src/engine/engine.ts`) documents its own gap: the +checkpoint verifies workflow identity, but `inputs`, `executionMode` and `planOptions` are **not** checkpoint- +derived, and the docblock states that passing different ones "would silently diverge the rehydrated execution +from its `run:started` state". The remedy is written there as "a future revision". -**Wave 2 note.** Wave 2 already holds the MCP timeout / SSRF / ingress cluster. This item belongs to the same -code area and the same review; it should be folded into that PR sequence rather than opened separately, but it -is a P0 while the rest of that cluster is P1. +**Failure.** A host resumes with different inputs — a different file path, a different flag, a different +execution mode — and the run continues under a different contract than the one its durable log records. Nothing +errors. This falsifies the *crash-safe resumable workflow* claim independently of `CR-10`/`CR-11`: even with a +perfect log and a single owner, the resumed run can be a different run. + +**Fix.** + +- Persist a canonical, resolved input **snapshot or digest** at admission (after `CR-15` resolves defaults, so + the digest covers what the run actually used). +- Persist `executionMode` and the execution-plan identity; verify or restore them on resume rather than trusting + the caller. +- Verify the workflow by content/plan digest, not only slug/id. +- **Do not persist secret-typed values.** A `secret` input is carried as a key reference plus version, with a + defined re-supply contract on resume; a mismatch is a typed error, never a silent substitution. +- A divergence produces a typed error, never a silent continue. + +**Acceptance.** Resuming with a changed input, a changed `executionMode`, or a changed plan each fails with a +distinct typed error. Resuming with a `secret` input re-supplied at the same version succeeds; at a different +version it fails. The digest of a run started with defaults omitted equals the digest of the same run started +with those defaults written out explicitly. No secret value appears in any persisted row — asserted by scanning +the written rows. --- -# W2 — Liveness and deadlines +## W2 — Liveness and deadlines -## CR-20 — Agent-node `timeout_ms` is completely inert · High +### CR-20 — Agent-node `timeout_ms` is completely inert · High The node schema accepts the field, but the runner passes only the run-level signal to the agent turn; no timer, controller or deadline consumes `node.timeout_ms`. An authored liveness bound is silently ignored. **Fix + acceptance.** Honour it as a real deadline; a node exceeding it fails with a classified timeout, proven with a fake clock. -## CR-21 — No Relavium-owned deadline for a normal provider attempt · Medium-High +### CR-21 — No Relavium-owned deadline for a normal provider attempt · Medium-High The chain awaits generate/stream with the caller's signal only; unlike list-models and key validation, it sets no per-attempt timeout. Without a node or run timeout, the vendor SDK default becomes the product's liveness semantics — which our security standard forbids for outbound requests. **Fix + acceptance.** An injected controller/timer factory; merge caller-abort with deadline-abort. A timeout is `kind: 'timeout'`, a user cancel stays `cancelled`. A pre-content timeout may fail over; a content-committed one -must not. Prove timer cleanup with a fake clock. This is a different timer from CR-20 — both are needed. +must not. Prove timer cleanup with a fake clock. This is a different timer from `CR-20` — both are needed, and +the pre/post-content split must agree with `CR-14`'s rule 7, so it lands on that ADR. -## CR-22 — Gate and run deadlines are not preserved across resume · High +### CR-22 — Gate and run deadlines are not preserved across resume · High The checkpoint's pending gate carries gate/node/budget data but not an absolute deadline, and the whole-run timeout is re-armed for its full duration on every resume — so a crash extends the cap. **Fix + acceptance.** Persist absolute deadlines; resume computes the remaining time. A run crashed and resumed repeatedly still times out at its original absolute deadline. -## CR-23 — Exactly-one-terminal is a safety property, not a liveness one · High +### CR-23 — Exactly-one-terminal is a safety property, not a liveness one · High · **decision open** Cancel only fires the abort signal, and the terminal waits for the running-node count to reach zero. The node executor seam accepts an arbitrary promise; honouring the signal is not guaranteed at the type level. An executor that ignores abort and never settles leaves the run without a terminal forever. +**Open decision.** The grace-period length and whether a quarantined executor is disabled process-wide or per +run. Settle before starting. **Fix + acceptance.** A bounded grace period after cancel/timeout, then a generation token that fences the run state from late outcomes, plus executor quarantine. Tests: a never-settling executor and a late-success executor both produce exactly one terminal in bounded time, and the late output is not applied. --- -# W3 — Resource governance and bounds +## W3 — Resource governance and bounds -## CR-30 — The event stream is not really bounded · High +### CR-30 — The accepted no-drop bounded stream is not implemented · High `push()` appends without a capacity check; `whenDrained` is advisory and only awaited at node boundaries, while -token deltas are emitted synchronously. A single provider stream can grow memory without a hard bound. The three -properties "sync producer", "no drop" and "hard bound" cannot all hold — pick two, deliberately. -**Fix + acceptance.** Real backpressure or an explicit, documented drop policy for transient events. A test -drives a fast producer against a slow consumer and asserts a hard ceiling. - -## CR-31 — No safe default concurrency and no absolute graph/retry caps · High +token deltas are emitted synchronously. A single provider stream can grow memory without a hard bound. + +**This is not an open choice.** [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md) +already decided it: *"buffering is bounded per consumer with a producer-await (no-drop) policy"*, because the +stream must stay gap-free — a drop would force a `sequenceNumber` resync, and reconnect/resync plus event-sourced +resume are both built on that. Implement the accepted decision. A drop policy would need an ADR that supersedes +ADR-0036, and nothing found here argues for one. +**Fix + acceptance.** Real producer-await backpressure with a hard per-consumer ceiling. A test drives a fast +producer against a slow consumer and asserts the buffer never exceeds the ceiling **and** that no sequence +number is skipped. + +### CR-31 — No safe default concurrency and no absolute graph/retry caps · High · **cap values open** An omitted `max_parallel_nodes` means `Infinity`. There is no absolute cap on authored nodes, edges, fan-out width, fallback-chain length, retry counts, parallel tools or total attempts; the parser's text-size limit does not stop many small ones. **Fix + acceptance.** A small capacity-derived default concurrency that an authored value may only narrow, plus -explicit absolute caps enforced at parse/compile admission with typed errors. +explicit absolute caps enforced at parse/compile admission with typed errors. The numbers are a maintainer call. -## CR-32 — Workflow output, state and durable event size are unbounded · High +### CR-32 — Workflow output, state and durable event size are unbounded · High · **bound values open** **Fix + acceptance.** Bound them at the durable boundary with a typed rejection, the same way tool output is -already bounded. +already bounded. The numbers are a maintainer call. -## CR-33 — Finished runs are retained forever in memory · Medium-High +### CR-33 — Finished runs are retained forever in memory · Medium-High · **policy shape open** `WorkflowEngine` keeps completed runs indefinitely. **Fix + acceptance.** A retention policy with an explicit bound; a long-lived process running many workflows -does not grow without limit. +does not grow without limit. Whether retention is count-, age- or memory-based is a maintainer call. --- -# W4 — MCP hostile boundary *(merge into Wave 2)* +## W4 — MCP hostile boundary -## CR-40 — Cancellation and deadlines never reach the MCP transport · High +Executes Wave 2's MCP queue (see [Phase boundary](#phase-boundary)). One security sitting covers this group +plus `CR-16`. + +### CR-40 — Cancellation and deadlines never reach the MCP transport · High The dispatch context can carry a signal, but the manager's `callTool` chain does not forward it, and neither the connection nor the stdio SDK call accepts an abort or deadline. After a user cancels, the remote or child effect continues and child processes can survive. -## CR-41 — The MCP SSRF floor does not cover DNS or redirects · High +### CR-41 — The MCP SSRF floor does not cover DNS or redirects · High The network config checks the authored hostname and scheme, then hands the raw opener to the SDK. The code itself documents the DNS-to-private and redirect-to-private holes. A public-looking domain can resolve to loopback, RFC1918 or a metadata IP. **Fix.** Apply the built-in egress mechanism — resolve-all, range-block, pinned IP — to the MCP HTTP, SSE and WebSocket transports. -## CR-42 — MCP discovery and result ingress are unbounded · High +### CR-42 — MCP discovery and result ingress are unbounded · High · **bound values open** Page count is capped for `tools/list`, but total tool count, byte size, description and schema size are not, and a result is fully materialized before core bounding applies. A hostile server can exhaust startup memory, inflate prompt token cost, or bloat a workflow output. -**Acceptance for W4.** A hostile-server harness: a server that never answers is cancelled within its deadline -and leaves no orphan process; a DNS record pointing at loopback is refused; oversized discovery and results are -rejected with typed errors before they reach the prompt. +**Acceptance for W4.** A hostile-server harness, with adversarial cases rather than a smoke test: a server that +never answers is cancelled within its deadline and leaves **no orphan process** (asserted on the child-process +table, not on the promise); a DNS record resolving to loopback, to an RFC1918 range and to the cloud metadata +address are each refused; a redirect to any of those is refused; oversized discovery and results are rejected +with typed errors **before** they reach the prompt. --- -# W5 — Media correctness +## W5 — Media correctness -## CR-50 — `read_media` does not work end to end · High +### CR-50 — `read_media` does not work end to end · High · **decision open** The builtin returns a base64 media part, the registry places generic output into the tool result, and the LLM message schema explicitly forbids raw media bytes there. The canonical alternative is a handle-only attachment, but all three adapters deliberately drop that field, and production chat does not wire the media-read scope. There is no registry → turn → chain → adapter path. -**Fix + acceptance.** Either complete the handle path through the adapters, or remove the tool from the catalog -until it works — an advertised tool that cannot work is worse than an absent one. An end-to-end test proves the -chosen answer. +**Open decision.** Complete the handle path through the adapters, or remove the tool from the catalog until it +works. Either is acceptable; shipping an advertised tool that cannot work is not. +**Fix + acceptance.** An end-to-end test proves the chosen answer. If removal is chosen, a test asserts the tool +is absent from the catalog the model sees. -## CR-51 — Model-level tool/attachment capability is not enforced · High +### CR-51 — Model-level tool/attachment capability is not enforced · High The catalog carries `toolCall` and `attachment` metadata, but runtime gates on the provider-wide flags and the adapters attach tools unconditionally. A model the catalog marks as tool-incapable can still be sent tools, and the chain can treat it as capable instead of pre-skipping it — turning a free skip into a paid 400. -## CR-52 — Gemini reasoning/tool continuation metadata is lost · Medium-High +### CR-52 — Gemini reasoning/tool continuation metadata is lost · Medium-High Part-level thought signatures on function calls are not captured, and replay drops reasoning after a tool result. Multi-turn thinking-plus-function-calling continuations can 400 or break semantically. +**Governance note.** This is **not** newly discovered drift. [ADR-0039](../../decisions/0039-same-provider-reasoning-replay.md) +deliberately deferred it and recorded it in [deferred-tasks.md](../deferred-tasks.md). This phase pulls that +deferral forward; the item closes by landing the work **and** checking off the deferred-tasks entry, so the two +records do not disagree. -## CR-53 — Large media travels fully buffered and base64-encoded · High +### CR-53 — Large media travels fully buffered and base64-encoded · High Audio and video responses are read into a full buffer and converted to base64; raw buffer, typed view, base64 string and the store's decoded copy can all exist at once. **Fix + acceptance.** A bounded stream or download lease from the adapter; the host writes straight to the media store with a `Content-Length` and streamed-byte ceiling. A large-media test asserts peak memory stays bounded. -## CR-54 — URL media can produce different bytes on resume · High +### CR-54 — URL media can produce different bytes on resume · High Remote URL media is resolved again on resume, and the same URL can return different content, a different redirect or a different DNS answer. The run appears to have the same inputs while the model sees different bytes. -**Fix + acceptance.** Convert to a content-hashed handle at first resolution; resume must not re-fetch. +**Fix + acceptance.** Convert to a content-hashed handle at first resolution; resume must not re-fetch. Pairs +with `CR-17`: the input digest is only meaningful if a URL input's bytes are pinned. -## CR-55 — Missing media rates can bypass a strict cost cap · Blocker for the strict-cap claim +### CR-55 — Missing media rates can bypass a strict cost cap · Blocker for the strict-cap claim Where a media rate is absent the estimator can fall to zero, the generated catalog projection produces no media rate, and the DB's media-rate columns are not carried into the listing/overlay path. The governor can then treat a missing rate as *priced at zero* rather than *unpriced*, admitting paid image/audio/video generation under a strict cap and reporting `cost:updated = 0`. **Fix + acceptance.** Missing media rate ⇒ unpriced, never zero. Under a strict cap an unpriced media generation -is refused. Carry the media rates through the overlay path. +is refused. Carry the media rates through the overlay path. Break-verify by restoring the zero fallback. --- -# W6 — Authoring correctness +## W6 — Authoring correctness -## CR-60 — `merge_strategy: first` is not implemented as specified · High +### CR-60 — `merge_strategy: first` is not implemented as specified · High · **decision open** The canonical table says the first resolved branch wins and the others are ignored. The engine waits for every branch to settle and the handler takes the first surviving output in declaration order. A slow loser adds latency and cost; a failing loser can fail the run even when the winner is ready. -**Fix + acceptance.** Either implement race semantics or rename the strategy and correct the canonical table — -but the name and the behaviour must agree. Whichever is chosen, a test pins latency/cost behaviour. +**Open decision.** Implement race semantics, or rename the strategy and correct the canonical table. The name +and the behaviour must agree; which one moves is a maintainer call. +**Fix + acceptance.** Whichever is chosen, a test pins latency and cost behaviour — a slow loser must not extend +the run's wall clock if race semantics are chosen, and must be documented as doing so if they are not. -## CR-61 — Schema fields do not deep-validate · Medium +### CR-61 — Agent and transform `output_schema` do not deep-validate · Medium · **needs a dependency ADR** An agent's `output_schema` becomes a response-format hint and is then only `JSON.parse`d; the transform node's -`output_schema` explicitly does not deep-validate; an agent's `input_schema` is not enforced on the runtime -runner path. A prior ADR required deep validation to land with this work, so this is also a decision-governance -drift. -**Fix + acceptance.** Real deep validation with typed failures, or a documented, deliberately narrowed contract -plus a renamed field. Silence is not an option. - -## CR-62 — Expression dependencies are invisible, so silent mis-routing is possible · High +`output_schema` explicitly does not deep-validate. + +**The governance drift, stated precisely.** [ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md) +required node-side `output_schema` validation to land **with 1.O in the same PR or behind a hard acceptance +gate**, "never silently deferred". What shipped is +[error-handling.md](../../standards/error-handling.md)'s narrower rule: Phase-1 scope is parse-as-JSON only, and +deep JSON-Schema conformance is a deferred follow-up **because it needs a JSON-Schema validator dependency +behind an ADR** — Zod cannot consume an arbitrary JSON Schema. So the two documents disagree about what was +required, and the narrowing was never recorded as a decision. +**Fix + acceptance.** Either land deep validation (which requires a new-dependency ADR under CLAUDE.md rule 2, +and that ADR is the gate), or record the narrowed contract as a decision that supersedes ADR-0038's clause. +Silence is not an option. A typed `validation` failure on a schema-violating but valid-JSON output is the test +either way — passing if validation lands, documented-as-absent if the narrowing is chosen. + +### CR-62 — Expression dependencies are invisible, so silent mis-routing is possible · High · **decision open** The DAG derives edges from template references only and deliberately does not order JS-expression reads of run outputs. A condition that runs before its producer compares `undefined` and silently takes the false branch — a test currently pins that silence. -**Fix + acceptance.** Either extract dependencies from expressions, or fail loudly when an expression reads an -unresolved output. Not a silent false. +**Open decision.** Extract dependencies from expressions, or fail loudly when an expression reads an unresolved +output. Not a silent false, either way. +**Fix + acceptance.** The pinning test is rewritten to the chosen rule and keeps a note recording the reasoning +for the behaviour it replaces. + +### CR-63 — Agent `input_schema` runtime enforcement: verify the docs, do not implement · Low *(plan review)* +Split out of `CR-61` because it is a different contract with a different answer. +[agent-yaml-spec.md](../../reference/contracts/agent-yaml-spec.md) states that `input_schema` is *"purely +additive metadata — it drives type-safe node chaining and editor (VS Code) completion; it does not change +run-time execution"*. Under that spec, the absence of runtime enforcement is **correct**, not a defect. +**Fix + acceptance.** No runtime change. Grep every canonical document and the product surface for a claim that +an agent's `input_schema` is validated at run time; if one exists, correct it to match the spec. If none exists, +close the item with that finding recorded — do not implement enforcement to satisfy a claim nobody made. --- -# W7 — Agent product correctness +## W7 — Agent product correctness -## CR-70 — Cross-turn tool-call memory does not exist · High (product) +### CR-70 — Cross-turn tool-call memory does not exist · High (product) · **decision open** Only the final assistant text enters the cross-turn transcript; within-turn tool call/result pairs are dropped. A coding agent cannot remember a file it read in the previous turn and calls the tool again. +**Open decision.** Persist the tool pairs (bounded), or narrow the canonical session spec. The bound is part of +the decision. -## CR-71 — Session transcript and export lose tool history · High +### CR-71 — Session transcript and export lose tool history · High The persister stores the same text-only model, so the exporter — written to derive tool names from assistant tool-call parts — has nothing to read. The canonical session spec promises a full transcript with a tool union. -**Fix + acceptance for CR-70/71.** Persist the tool pairs (bounded), or narrow the spec. A graduated workflow -must represent the flow that actually happened. +**Fix + acceptance for CR-70/71.** One decision, one PR. A graduated workflow must represent the flow that +actually happened, or the spec must stop promising it. -## CR-72 — Authored agent `memory` policy is inert · Medium +### CR-72 — Authored agent `memory` policy is inert · Medium · **decision open** `none | window | summary` is accepted by the schema and consumed by nothing. -**Fix + acceptance.** Implement it or reject it at parse time with a typed error. +**Fix + acceptance.** Implement it or reject it at parse time with a typed error. A schema that accepts a field +nothing reads is a claim the code does not keep. -## CR-73 — `invoke_agent` has no production delegate · High (2.6 go/no-go) +### CR-73 — `invoke_agent` has no production delegate · High (2.6 go/no-go) The builtin exists in the catalog and errors with tool-unavailable when no delegate is wired; nothing wires one in production, and the availability filter keeps the delegate-backed tool in the catalog. -**Fix + acceptance.** Wire the delegate, or stop advertising the tool to the model when no delegate exists. +**Fix + acceptance.** Wire the delegate, or stop advertising the tool to the model when no delegate exists. The +cheap option is always available, which is why this is non-deferrable. --- -# W8 — Provider and conformance +## W8 — Provider and conformance -## CR-80 — An invalid custom base URL fails OPEN to the official endpoint · High (security/compliance) +### CR-80 — An invalid custom base URL fails OPEN to the official endpoint · High (security/compliance) When the custom provider factory rejects a private, malformed or credential-bearing URL, the error is caught and the default adapter is left standing — and a test pins that fail-open as correct. A user expecting an internal gateway silently sends prompts and keys to the official API after a config drift. -**Fix + acceptance.** Fail closed with an explicit message. Rewrite the test that pins the old behaviour, -keeping the reasoning it replaces. +**Fix + acceptance.** Fail closed with an explicit message. The error must name the rejected URL's *shape* +without echoing an embedded credential — asserted directly. Rewrite the test that pins the old behaviour, +keeping a note with the reasoning it replaces. -## CR-81 — Conformance cassettes do not verify the request wire contract · High +### CR-81 — Conformance cassettes do not verify the request wire contract · High A recorded response carries status, content type and body only; replay checks that the request body is parseable JSON and matches nothing else — not method, path, headers, or a canonical body. A dropped `response_format`, a lost tool result on continuation, or a broken system-instruction mapper leaves the suite green. -**Fix + acceptance.** Match the canonical request. Mutating the request lowering must redden the suite. +**Fix + acceptance.** Match the canonical request. Mutating the request lowering must redden the suite — proven +by actually applying at least three distinct mutations, not by assertion count. -## CR-82 — Missing or partial usage can be read as zero · High +### CR-82 — Missing or partial usage can be read as zero · High Carried from our own backlog and re-confirmed by the reviews. **Fix + acceptance.** Missing usage is *unknown*, never zero, on every path that reaches a cost total. --- -# W9 — Harness honesty and isolation +## W9 — Harness honesty and isolation -## CR-90 — Root coverage collects in-repo worktrees · Medium +`CR-90` and `CR-91` execute **first** (see [Execution order](#execution-order)); `CR-92` executes inside the +durability spine. They are grouped here by theme only. + +### CR-90 — Root coverage collects in-repo worktrees · Medium The root Vitest include patterns have no exclude for repo-local checkout directories, so a stale worktree adds test files and 0%-covered sources to the run. In one working tree this produced 471 discovered test files and 23 suite failures; excluding the worktree path produced 237 files and a clean run. -**Fix + acceptance.** Exclude repo-local checkout areas from both `test.exclude` and `coverage.exclude`. A -self-test creates a fixture worktree and proves root discovery does not see it. +**Fix + acceptance.** Exclude repo-local checkout areas from both `test.exclude` and `coverage.exclude`. The +self-test uses a **synthetic repo-local checkout fixture** — a directory tree containing a test file and a +source file — not a real `git worktree`; a real worktree makes the test depend on git state and the working +directory, which is exactly the non-determinism this item is about. -## CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium +### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium It pins that the live stream reported success; it does not prove durable truth after a restart. **Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same -terminal type and payload. +terminal type and payload. **This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with**, so +it lands before them: a spine asserted with the harness the reviews already found insufficient is not asserted. -## CR-92 — Terminal persistence failure lets live and durable truth diverge · High +### CR-92 — Terminal persistence failure lets live and durable truth diverge · High · in the durability spine The engine can complete the delivery chain for a terminal event even when its persistence failed, and reconciliation may later produce a different terminal than the original. Media reclaim can run before the terminal is durable. A caller can receive a success result and outputs while history shows the run failed. @@ -489,43 +792,60 @@ identity. If durability is uncertain the API must not say `completed` — it ret Terminal asset cleanup and handle resolution happen only after the terminal is durable. The test proves live, history, resume and reconcile agree. -## CR-93 — Process-global catalog and parameter-learning state is not tenant-safe · Medium now, High for cloud +### CR-93 — Process-global catalog and parameter-learning state is not tenant-safe · Medium now, High for cloud Process-global mutable state is fine for a single local user and wrong for the multi-tenant cloud surface. -**Fix + acceptance.** Scope it per run/session/tenant before any multi-tenant surface ships. Record the decision -even if the work is deferred. +**Fix + acceptance.** Scope it per run/session/tenant before any multi-tenant surface ships. **The decision is +recorded in this phase even if the work is deferred** — a deferral here moves to +[deferred-tasks.md](../deferred-tasks.md) with its trigger (the first multi-tenant surface) named. -## CR-94 — One budget approval opens every redispatch without a numeric limit · High +### CR-94 — One budget approval opens every redispatch without a numeric limit · High An approved vertex skips pre-egress admission on redispatch, and one agent turn can make many tool rounds plus a final generation. The user approves a projection they saw, but the approval means "this vertex is exempt", not "up to this amount". **Fix + acceptance.** Make the approval an immutable numeric lease bound to money/token/attempt scope, model and expiry, consumed atomically by each egress. An approved node that exceeds its lease pauses again. -## CR-95 — A mid-tool-loop budget pause replays the whole loop · High (Blocker while CR-12 is open) +### CR-95 — A mid-tool-loop budget pause replays the whole loop · High (Blocker while CR-12 is open) A budget pause becomes a paused outcome; on approval the node is reset to pending and dispatched from the start. The code already acknowledges that this repeats earlier provider and tool calls. -**Fix + acceptance.** Short term: forbid a mid-loop budget pause (fail closed). Long term: checkpoint the -continuation — provider messages, tool call/result pairs, round index — and resume from that point. A mid-loop -pause plus approval must not repeat a mutation. +**Fix + acceptance.** **Short term (non-deferrable): forbid a mid-loop budget pause — fail closed.** Long term: +checkpoint the continuation — provider messages, tool call/result pairs, round index — and resume from that +point. A mid-loop pause plus approval must not repeat a mutation. The short-term fix closes the claim; the long +term may be deferred with its trigger named. --- -# Exit criteria +## Exit criteria This phase is done when **all** of the following hold: -1. Every `CR-##` above is either closed with a break-verified test, or explicitly deferred in this document with - a named reason and the mutation that would close it. No silent omissions. -2. Each of the seven W1 items has an accepted ADR, and the ADRs land in dependency order. -3. Every canonical document that promised behaviour listed here says what the code now does — in particular the - claims about crash-safe resume, duplicate-free effects, bounded streams, first-branch-wins, schema-validated - input/output, and session-to-workflow graduation. -4. `pnpm run ci` exits 0, checked by exit code. -5. A closing register in this file states, per item, the code that closes it — verified by reading the code, not - by trusting the mark. Wave 1's completion claim was wrong twice before this discipline was adopted. +1. **Every non-deferrable item is closed with a break-verified test.** Non-deferrable means the register above + says so: `CR-01`–`CR-03`, `CR-10`–`CR-17`, `CR-50`, `CR-55`, `CR-73`, `CR-80`, `CR-92`, and `CR-95`'s + short-term fail-closed fix. **None of these may be deferred**, for one reason: each is either a security or + correctness boundary, or a shipped claim the code does not keep — and every one of them has a cheap + fail-closed option (refuse, remove, narrow the claim) available when the full fix is too large. "Too big to + fix now" is an argument for the cheap option, never for deferral. +2. **Every other item is closed, or deferred with a full record.** A deferral is written in + [deferred-tasks.md](../deferred-tasks.md) with its severity, its trigger, and — this is the part that is + usually skipped — **the product claim it narrows**. A deferred item's behaviour must not remain in any + canonical document as a shipped guarantee. +3. **Each `W1` item has an accepted ADR, and the ADRs land in dependency order.** Eight ADRs (or fewer, where + the register above pairs two items onto one), all before their implementation. +4. **Every canonical document that promised behaviour listed here says what the code now does** — in particular + the claims about crash-safe resume, duplicate-free effects, bounded streams, first-branch-wins, + schema-validated input/output, and session-to-workflow graduation. This explicitly includes + [architectural-principles.md](../../standards/architectural-principles.md) §11's idempotency sentence. +5. **Four security-review sittings are recorded**, each with a checklist entry and its adversarial test: + prompt/trust provenance, hostile MCP, media bytes, provider/config trust. +6. **`pnpm run ci` exits 0 and `pnpm coverage` exits 0**, both checked by exit code. `pnpm run ci` alone is not + the merge gate — `coverage` is a separate required check, and this phase edits `packages/core` heavily. +7. **A closing register in this file states, per item, the code that closes it** — verified by reading the code, + not by trusting the mark. Wave 1's completion claim was wrong twice before this discipline was adopted. ## Scope note -Two items already in flight are NOT part of this phase and keep their own tracking: the durable per-attempt -realized-cost ledger (accepted ADR, staged implementation) and the transaction-handle cleanup across the -remaining database stores. They are listed here only so nobody schedules them twice. +One item already in flight is NOT part of this phase and keeps its own tracking: the transaction-handle cleanup +across the remaining database stores. It is listed here only so nobody schedules it twice. + +The realized-cost ledger is **no longer scoped out** — it is this phase's [prerequisite](#prerequisite), for the +file-overlap reason recorded there. From 8d7ffcf8c46d786612ab5626227e46699fceebc0 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 19:11:31 +0300 Subject: [PATCH 04/26] feat(shared,docs): add cost:attempt_settled + ADR-0077 correcting its barrier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 1 of 5, plus the ADR that had to precede it. **ADR-0077 — why an amendment was needed before any code.** ADR-0076 §1 decided the mechanism as "the awaited emit plus an explicit check, NOT a §2-style queue-and-flush", justified by an asymmetry between where the two money events are emitted. That asymmetry does not exist, and the code says so three times: - fallback-chain.ts:148 — `onAttempt?: (record) => void`, synchronous, invoked bare at :902. - agent-turn.ts:783-847 — `settleAtReservedEstimate()` and `cost:updated` are emitted from that ONE callback, a few lines apart. - budget-governor.ts:273 — "The ledger mutation is SYNCHRONOUS ... called from inside the fallback chain's onAttempt callback, WHICH CANNOT AWAIT." So §1 asked for an `await` where none can be placed. node-executor.ts:46-52 had already written the warning for exactly this mistake. ADR-0077 adopts ADR-0074 §2's shape instead: start the write at the settle instant on a chained in-flight promise, join it at THREE barriers — pre-egress, before tool dispatch (the one this adds beyond §2), and the turn/node terminal — each awaiting AND observing, since #emitDurable is total for store faults and resolves. A fold of the ADR review added Decision §4 (one joining entry point owning both money chains, so "await the wrong one" is not expressible) and named the tool-dispatch barrier as the seam CR-12's effect journal will extend. The review also asked whether the session path loses realized cost the same way. It does not, and the ADR now records why with evidence: `recordSessionCost` is `(entry) => void` — a synchronous better-sqlite3 write that `persistDurably` calls inline and re-throws from, so the money is committed before the handler returns. The run path needs a barrier because its store is `persistEvent => Promise` by seam design; the session path is stronger here, not weaker. **Step 1 — the schema.** RUN-only (`runBase`), because the session path already has this ledger. Two fields diverge from their siblings and the divergence is deliberate: `attemptNumber` and `priced` are REQUIRED here where `cost:updated` and `budget:estimate_committed` have them optional. Those two carry historical rows; this type has none, so requiring now is free while requiring LATER would be the one-way door parseStoredRunEvent describes. A ledger row that cannot say which attempt it is, or whether it could be priced, is not a ledger. **Break-verify, with the mutation confirmed applied each time.** Six fixtures, six mutations, all red: drop the refinement; attemptNumber optional; priced optional; attemptNumber nonNegative; costMicrocents as z.number() (reddens both fractional and negative). The first pass was HOLLOW and the mutation check is what caught it. Three reject fixtures omitted `priced` — so they were rejected on the MISSING FIELD and never reached the invariant they were named for; deleting the refinement left the suite green. A fourth then failed on a fractional `cumulativeCostMicrocents` instead of the `costMicrocents` it claimed to test. Every fixture now carries a complete payload with exactly one thing wrong, and says so in a comment. Recorded for steps 3-5: `pnpm turbo run typecheck` passes across all 11 tasks with the new arm unhandled everywhere. No consumer switch has an exhaustiveness guard over RunEvent, so every downstream site must be found by reading, not by the compiler. pnpm run ci EXIT=0. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- ...ses-the-conservative-commitment-barrier.md | 162 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/roadmap/current.md | 13 +- packages/shared/src/constants.ts | 9 +- packages/shared/src/run-event.test.ts | 101 ++++++++++- packages/shared/src/run-event.ts | 112 ++++++++++++ 6 files changed, 392 insertions(+), 6 deletions(-) create mode 100644 docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md diff --git a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md new file mode 100644 index 00000000..683d5091 --- /dev/null +++ b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md @@ -0,0 +1,162 @@ +# ADR-0077: The realized-cost ledger uses ADR-0074 §2's barrier mechanism (amends ADR-0076 §1) + +- **Status**: Accepted +- **Date**: 2026-08-10 +- **Related**: [ADR-0076](0076-durable-per-attempt-realized-cost-ledger.md) §1 (the mechanism this corrects; its decision, event and properties stand), [ADR-0074](0074-durable-conservative-budget-commitments.md) §2 (the mechanism this adopts), [ADR-0038](0038-agentrunner-llm-call-boundary.md) (the one-chain-per-node-execution boundary the barriers sit on), [ADR-0011](0011-internal-llm-abstraction.md) (the `LLMProvider` seam the rejected alternative would have widened), and [sse-event-schema.md](../reference/contracts/sse-event-schema.md) (the canonical event contract). + +## Context + +[ADR-0076](0076-durable-per-attempt-realized-cost-ledger.md) decided that a settled provider attempt's realized +charge becomes a durable run event, `cost:attempt_settled`. That decision is right and is not reopened here. + +Its §1 also decided **how** the write becomes a barrier, and drew a line against its own sibling: + +> **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** §2 needs +> `flushBudgetCommitments` because a conservative commitment is emitted fire-and-forget from inside the +> governor, so something has to join the outstanding writes at the turn boundary. A settled attempt is emitted +> by the engine at the point it settles, on the path that is about to continue, so no queue is needed — and +> adding one would introduce the very concurrency the await removes. + +**The asymmetry that paragraph rests on does not exist**, and the code says so in three places: + +1. The seam's attempt observer is synchronous and returns nothing — + `onAttempt?: (record: AttemptRecord) => void` ([fallback-chain.ts](../../packages/llm/src/fallback-chain.ts)), + invoked as a bare synchronous call inside the chain's attempt loop. +2. Both money events are emitted from **that same callback, a few lines apart** — the conservative + `settleAtReservedEstimate()` and the realized `cost:updated` sit in one `onAttempt` body + ([agent-turn.ts](../../packages/core/src/engine/agent-turn.ts)). There is no "engine path that is about to + continue" for one of them and not the other; there is one synchronous observer for both. +3. The governor's own field documentation already states the constraint in plain words: *"The ledger mutation + is SYNCHRONOUS — `settleAtReservedEstimate` is called from inside the fallback chain's `onAttempt` callback, + **which cannot await**"* ([budget-governor.ts](../../packages/core/src/engine/budget-governor.ts)). + +So §1 asked for an `await` at a point where no `await` can be placed. Worse, the codebase had already written +the warning for exactly this mistake: [node-executor.ts](../../packages/core/src/engine/node-executor.ts)'s +comment on why `budget:estimate_committed` is absent from the streamed in-node event union ends with *"If you +came here to add it, add it to the governor's emit type instead."* + +Two things §1 got **right** are worth keeping explicit, because this ADR narrows one paragraph and not the +section: + +- `#emitDurable` is **total for store faults** — a `persistEvent` rejection is absorbed into the run's failure + state and the promise RESOLVES. Awaiting it alone is therefore never a barrier. §1 named this trap and the + trap is real; it survives unchanged. +- The guarantee §1 wanted — *written before the next thing that can spend or mutate* — is the right guarantee. + Only its shape was wrong. + +## Decision + +**`cost:attempt_settled` uses the same mechanism ADR-0074 §2 built for its estimate twin: the emit is started +synchronously at the settle point onto a chained in-flight promise, and JOINED at every barrier that precedes +spending or mutating. It is not a bare awaited emit.** + +Concretely, three things: + +1. **Start the write at the settle instant.** `onAttempt` cannot await, but it can *begin*. The emit is chained + onto a per-owner in-flight promise exactly as §2's `#commitmentsInFlight` is — which also serializes the + writes, so two settles in the same tick cannot interleave their persists. +2. **Join at three barriers, not two.** §2 joins at the pre-egress check and at the turn/node boundary. + ADR-0076's guarantee names a third thing to precede — the next **tool side effect** — so the ledger's + barrier set is: before the next egress admission, **before tool dispatch**, and at the turn/node terminal. + The tool-dispatch barrier is the one this ADR adds to the §2 shape; without it the ledger would repeat + §2's coverage rather than extend it, and the duplicate-effect window ADR-0076 exists to narrow would stay + open on the path that mutates the world. +3. **Every barrier awaits AND observes.** Because `#emitDurable` is total, a barrier that only awaits proceeds + on a run whose ledger write did not land. Each barrier must therefore also read the failure state and refuse + to continue, and §2's retained-failure pattern (`#commitmentFailure`, re-thrown at the next barrier) is the + shape that carries it — a rejection nobody awaits at the call site would otherwise be unhandled. +4. **One join, not two.** After this ADR there are two chained in-flight promises — the conservative one and + the realized one — and three barriers that must join both. Joining them individually is a correctness bug + waiting for the next barrier someone adds. So the barriers call **one** joining entry point that owns both + chains and both retained failures; there is no supported way to await half the money. This is what turns + the "a future barrier joins only one chain" hazard from a review item into something a caller cannot + express. + +### Why the tool-dispatch barrier matters beyond this ADR + +It is the first place the engine is required to reach a durability checkpoint *before dispatching a side +effect*, and that checkpoint is the seam the durable **effect journal** (`CR-12` in the 2.6.5 phase) needs: +`prepared → dispatched → committed | ambiguous` has to be written at exactly this point in exactly this path. +Landing it here means the effect journal extends an existing barrier rather than threading a new one through +`ToolRegistry.dispatch`. Named so the two are not built twice, and so a reviewer of either can check the other. + +Considered and rejected: + +- **Make the seam's `onAttempt` awaitable** (`void | Promise`, awaited by the chain), so §1's sentence + becomes literally true. Rejected on three counts. It widens `@relavium/llm`'s public API for a concern that + is not the seam's — durability belongs to the engine, and [ADR-0011](0011-internal-llm-abstraction.md) keeps + the seam a provider contract, not an execution-host one. It would let a durable store write block a live + provider stream, and an observer that hangs would stall the chain with no timeout of our own. And it would + give the two money events emitted from one callback two different durability mechanisms, which is precisely + the drift that made this correction necessary. +- **Emit from the turn loop after the chain call returns**, where an `await` is genuinely available. Rejected + because it is strictly worse than the chosen mechanism on the failure this ADR is about: the write would not + even *start* until the whole chain call finished, so a crash mid-chain loses every failover attempt inside + it. Chaining starts the write at the settle instant and only *joins* later, which is a narrower window, not a + wider one. +- **Leave §1 as written and implement something else.** Rejected outright. An ADR whose mechanism paragraph + describes something the code does not do is the corpus drift this project has repeatedly paid for; the + remedy is an append-only correction, not a quiet reinterpretation. + +### The session path needs no barrier, and the reason is structural + +ADR-0076 scoped its event to the run path on the ground that "the session path already has this ledger". A +review of this ADR asked whether that leaves a session resume losing realized cost the same way. It does not, +and the reason is worth recording once so the question stops recurring: the session write is **synchronous and +already committed** when the handler returns. + +`persister.ts` handles `cost:updated` through `persistDurably(() => store.recordSessionCost(...))`, and +`recordSessionCost` is declared `(entry: SessionCostEntry) => void` — a synchronous `better-sqlite3` +transaction, not a promise. `persistDurably` calls it inline and re-throws, so the money is on disk before the +next line of the handler runs, and the handler itself runs on the delivery of an event emitted from the +synchronous `onAttempt`. There is no in-flight window for a crash to land in. + +So the asymmetry this ADR institutionalises is not "run path protected, session path forgotten". It is that +the run path's store is `persistEvent: (event) => Promise` — asynchronous by seam design, because a +cloud store must plug in — and an asynchronous write is exactly what needs a barrier. The session path is +stronger here, not weaker, and stays so until its store becomes asynchronous; at that point it inherits this +decision rather than needing a new one. + +### What ADR-0076 keeps + +Unchanged and not reopened: the event exists, its name, its meaning, its shape's canonical home, the +idempotency boundary and the crash-after-commit case it explicitly does NOT cover, the no-double-count +arithmetic against `node:completed`'s telescoping delta, the run-path-only scope and why the session path needs +no arm, the four rejected alternatives, and the reason it lands after +[ADR-0075](0075-fail-closed-resume-on-an-unreadable-event-log.md). This ADR replaces one paragraph. + +## Consequences + +### Positive + +- **One mechanism for both money events, emitted from one callback.** They cannot drift apart, and a future + reader who finds one finds the other. The alternative left two durability shapes three lines apart in the + same function. +- **The guarantee ADR-0076 wanted is actually obtainable**, and the tool-dispatch barrier makes it stronger + than §2's: realized spend is durable before the run mutates the world, not merely before it spends again. +- **No seam change.** `@relavium/llm` keeps its provider contract, and a slow disk cannot stall a provider + stream. +- The write starts earlier than any barrier-only design would allow, so the crash window is the narrowest of + the three mechanisms considered. + +### Negative + +- **A third barrier on the hot path.** Mitigation: §2's outstanding-count guard applies unchanged — the barrier + costs nothing when nothing is in flight, which is the common case, and an unconditional await would change + observable interleaving that existing tests legitimately pin. +- **Two chained in-flight promises now exist** (the governor's conservative chain and the ledger's realized + chain), so a barrier that joined only one would be silently half-safe. Mitigation is structural rather than + advisory — Decision §4: the barriers join through a single entry point that owns both chains and both + retained failures, so "await the wrong one" is not an expressible mistake. What a reviewer still has to + catch is a new barrier that calls neither. +- **The tool-dispatch barrier is new engine surface**, on the hottest path a run has, and it lands before the + effect journal that will build on it. If it is placed wrong, both this ledger and `CR-12` inherit the same + crash window. Mitigation: it goes in at `agent-turn.ts`'s single `await dispatchToolCalls(...)` — one call + site, before any tool runs — rather than inside `ToolRegistry.dispatch`, where per-call placement would have + to be re-proven for every dispatch path. +- **A durability failure now fails a turn at the tool boundary too**, where previously it would have surfaced + only at the next egress or the terminal. Deliberate, and the same posture §2 accepted: a run that cannot + record what it spent must not go on to mutate anything. +- **ADR-0076 §1 must be read together with this ADR**, since the corpus is append-only and that paragraph + stays on the page. Mitigation: the title, the `Related` line and the "What ADR-0076 keeps" section above make + the scope of the correction unambiguous — one paragraph, not a section. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 4b1b1143..e224c685 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -120,6 +120,7 @@ flowchart TD | 0074 | [Durable conservative budget commitments across run and session resume (amends ADR-0028, ADR-0036, ADR-0045, ADR-0070)](0074-durable-conservative-budget-commitments.md) | Accepted | 2026-07-30 | | 0075 | [A resume fails closed on an unreadable event log (amends ADR-0074 §5)](0075-fail-closed-resume-on-an-unreadable-event-log.md) | Accepted | 2026-08-09 | | 0076 | [A durable per-attempt realized-cost ledger (amends ADR-0070, extends ADR-0074)](0076-durable-per-attempt-realized-cost-ledger.md) | Accepted | 2026-08-09 | +| 0077 | [The realized-cost ledger uses ADR-0074 §2's barrier mechanism (amends ADR-0076 §1)](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) | Accepted | 2026-08-10 | ## Creating a new ADR diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 1d9ca1bd..a25657eb 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -279,9 +279,16 @@ Ordered by whether the repo currently states something untrue, then by blast rad > transaction as its event. The `node:completed` delta then telescopes to zero on its own; a test must pin > that `SUM(run_costs) == runs.total_cost_microcents` still holds, since that is the invariant carrying the > no-double-count claim. -> 4. `packages/core/src/engine/engine.ts` — emit through `#emitDurable` at the attempt boundary, AWAITED before -> the next tool side effect, the next egress and the node terminal. This is the barrier; without the await -> the event is just another observation. +> 4. The attempt boundary and its barriers — **corrected by +> [ADR-0077](../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)**. ADR-0076 +> §1 asked for an `await` at the settle point; there is no `await` to be had there. The seam's observer is +> `onAttempt?: (record) => void` — synchronous, void — and BOTH money events are emitted from that one +> callback a few lines apart, which `budget-governor.ts` already states outright ("which cannot await"). So +> the ledger takes ADR-0074 §2's shape: **start** the write at the settle instant on a chained in-flight +> promise, **join** it at three barriers — before the next egress admission, **before tool dispatch** +> (`agent-turn.ts`'s `await dispatchToolCalls(...)`, the barrier this adds beyond §2), and at the turn/node +> terminal. Every barrier awaits AND observes the failure, because `#emitDurable` is total for store faults +> and resolves. > 5. `packages/core/src/engine/checkpoint.ts` — fold it as a SUM of deltas, never a last-wins snapshot (ADR-0074 > §2's reasoning: concurrent events under a `fan_out` have no canonical `seq` order). > diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 454d0432..4b4ce4d5 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -18,8 +18,10 @@ export type SchemaVersion = typeof SCHEMA_VERSION; * and the per-event ordinal is always `sequenceNumber`, never `seqNo`. Order mirrors the * `RunEvent` union in the spec: `agent:reasoning` sits immediately after `agent:token` (the reasoning * host-emit, EA6/2.5.H amending [ADR-0036]); `agent:approval_requested` + `agent:file_patch_proposed` - * sit after `agent:tool_result`, and the five governance events close the list: `run:paused`, `run:timeout`, + * sit after `agent:tool_result`, then the five governance events: `run:paused`, `run:timeout`, * `budget:warning`, `budget:paused` (ADR-0028) and `budget:estimate_committed` ([ADR-0074], dual-envelope). + * `cost:attempt_settled` ([ADR-0076]) closes the list — the realized twin of that last one, and the only + * durable member of the `cost:` namespace (`cost:updated` is streamed). */ export const RUN_EVENT_TYPES = [ 'run:started', @@ -56,6 +58,11 @@ export const RUN_EVENT_TYPES = [ // rides 'runId' on a run and 'sessionId' on a session, so it is listed here (with the run types) and reused // on the session envelope rather than duplicated into SESSION_EVENT_TYPES. NEVER realized spend. 'budget:estimate_committed', + // One settled provider attempt's REALIZED charge, made durable (ADR-0076) — the realized twin of the + // estimate above, emitted from the same `onAttempt` callback and joined at the same barriers (ADR-0077). + // RUN-ONLY, deliberately: the session path already records per-attempt realized spend into `session_costs` + // on every 'cost:updated' (ADR-0070), so it needs no session arm and is NOT in SESSION_EVENT_TYPES. + 'cost:attempt_settled', ] as const; export type RunEventType = (typeof RUN_EVENT_TYPES)[number]; diff --git a/packages/shared/src/run-event.test.ts b/packages/shared/src/run-event.test.ts index ffc37563..1c29e5cc 100644 --- a/packages/shared/src/run-event.test.ts +++ b/packages/shared/src/run-event.test.ts @@ -175,6 +175,18 @@ const valid: Record> = { estimateMicrocents: 400, cumulativeConservativeMicrocents: 400, }, + 'cost:attempt_settled': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, }; /** One targeted invalid payload per variant (a missing/invalid required field). */ @@ -219,6 +231,90 @@ const reject: Record> = { estimateMicrocents: 400, cumulativeConservativeMicrocents: 400, }, + // ADR-0076. The first two pin the DIVERGENCES from this event's two siblings: `attemptNumber` and `priced` are + // optional on `cost:updated` / `budget:estimate_committed` (they had historical rows) and REQUIRED here (it has + // none). Loosening later stays additive; tightening later would be the one-way door `parseStoredRunEvent` + // describes — so a test that lets either slip through optional is what makes the door close silently. + 'cost:attempt_settled (no attemptNumber)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (no priced)': { + // Without the flag, `costMicrocents: 0` with real tokens is ambiguous between "unpriced" and "free" — the + // exact ambiguity a ledger row exists to resolve. + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + }, + 'cost:attempt_settled (cumulative below this attempt)': { + // The same producer bug the conservative twin pins: reading the run-wide counter BEFORE folding this + // attempt into it. Reconstruction sums the deltas so the RESTORE survives it, which is precisely why + // nothing else would complain — every display reading the snapshot just under-reports. + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 500, + cumulativeCostMicrocents: 0, + // `priced` is REQUIRED on this event, so omitting it here would reject the fixture on the MISSING FIELD and + // never reach the refinement — a test that passes with the refinement deleted. It did, until a break-verify + // caught it. Every fixture below carries a complete payload for the same reason: exactly one thing wrong. + priced: true, + }, + 'cost:attempt_settled (attemptNumber 0)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 0, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (fractional cost)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 12.5, + // 13, not 12.5 — a fractional cumulative would reject this fixture on the CUMULATIVE field and leave + // `costMicrocents`'s integer bound untested. One thing wrong per fixture, and it has to be the named one. + cumulativeCostMicrocents: 13, + priced: true, + }, + 'cost:attempt_settled (negative cost)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: -1, + cumulativeCostMicrocents: 0, + priced: true, + }, 'run:started (bad executionMode)': { type: 'run:started', ...env, @@ -556,7 +652,7 @@ describe('RunEvent union — every variant', () => { } }); - it('covers exactly the 24 canonical colon-namespaced names, pinned to a literal list', () => { + it('covers exactly the 25 canonical colon-namespaced names, pinned to a literal list', () => { // A hardcoded contract list — independent of RUN_EVENT_TYPES — so the union and the // constant cannot silently drift together. const CONTRACT_NAMES = [ @@ -584,6 +680,7 @@ describe('RunEvent union — every variant', () => { 'budget:warning', 'budget:paused', 'budget:estimate_committed', // ADR-0074 §2 — a durable conservative commitment; an ESTIMATE, not spend + 'cost:attempt_settled', // ADR-0076 — the realized twin of the line above; the only DURABLE cost: event ]; // The matrix above proves each canonical name's valid payload parses (so a // renamed/missing variant fails there); the union member count catches an *extra* @@ -591,7 +688,7 @@ describe('RunEvent union — every variant', () => { // RunEventSchema wraps the union in the correlation-key refinement; reach the raw union. expect(RunEventSchema.innerType().options).toHaveLength(CONTRACT_NAMES.length); expect(new Set(RUN_EVENT_TYPES)).toEqual(new Set(CONTRACT_NAMES)); - expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 24 + expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 25 // STRUCTURAL, not a comment: the §5 forward-compat fixtures stand in for "a type a newer binary wrote" using // the `test:` prefix. Step B first used `budget:estimate_committed` for that and ADR-0074 §2 then made it // real, silently inverting three fixtures. A `test:`-prefixed name must never become a canonical event. diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index 03ba4368..c5d95766 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -590,6 +590,93 @@ export const BudgetEstimateCommittedEventSchema = z.object({ }); export type BudgetEstimateCommittedEvent = z.infer; +/** + * One SETTLED provider attempt's REALIZED charge, made durable + * ([ADR-0076](../../../docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md)). + * + * The realized twin of {@link BudgetEstimateCommittedEventSchema}, and the pairing is the point: + * `budget:estimate_committed` records money that MIGHT have been billed, this records money that WAS. Both are + * emitted from the SAME synchronous `onAttempt` callback a few lines apart (`agent-turn.ts`), which is why + * [ADR-0077](../../../docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) + * gives them the same barrier mechanism rather than two. + * + * **Why it exists at all.** `cost:updated` is the only other carrier of a realized per-attempt charge, and it + * is streamed, never persisted. The durable record is otherwise reconstructed from a LATER boundary — + * `node:completed.cumulativeCostMicrocents`. An agent turn is a loop, so a node can make many paid calls + * before it completes, and a crash mid-loop discarded every one of them. Worse than forgetting: the resumed + * run SPENT IT AGAIN, because the cap it re-evaluated against was understated by exactly the amount already + * charged. + * + * **RUN PATH ONLY — `...runBase`, deliberately not `...dualBase`.** `cost:updated` is dual-envelope, so the + * absence of a session arm is a choice, not an oversight: the session path ALREADY has this ledger. + * `persister.ts` writes `recordSessionCost` on every `cost:updated`, carrying the PER-ATTEMPT increment (not a + * cumulative snapshot) into `session_costs`, and since `#W15-4` that write is latched so a failure gates + * further egress. This event closes the RUN path's equivalent gap. + * + * **Scope, so the omission is not read as a gap.** This covers the provider attempts of an AGENT TURN. A media + * job's realized cost (ADR-0045 §5) does NOT emit one and does not need to: it is already durable through the + * `node:completed` / `node:failed` / `run:*` cumulative snapshots `#W15-6` added for exactly that reason. And + * no cost event of any kind makes a TOOL EFFECT idempotent — that is the durable effect journal, a different + * decision about a different failure (ADR-0076 scopes it out by name). + */ +export const CostAttemptSettledEventSchema = z.object({ + type: z.literal('cost:attempt_settled'), + ...runBase, + /** The agent node that owned the attempt. Required — on the run path every attempt has an owning vertex. */ + nodeId: nonEmptyString, + /** The canonical model id this attempt actually ran on — the per-model attribution key. */ + model: nonEmptyString, + /** + * 1-based WITHIN-CHAIN (`FallbackChain`) attempt, matching `cost:updated` — it resets to 1 on each node-retry + * re-dispatch, and does NOT join `node:*.attemptNumber` (the node-retry dispatch index). To attribute a + * charge to a node-retry attempt, partition the `sequenceNumber`-ordered stream at the + * `node:started` / `node:retrying` boundaries. + * + * **REQUIRED here, unlike on `cost:updated` and `budget:estimate_committed`, and the divergence is + * deliberate.** Those two carry it optionally because they predate this event and their historical rows may + * lack it. This type is new, so it has no historical rows — and a per-ATTEMPT ledger whose row cannot say + * which attempt it belongs to is not a ledger. The emitter always has it (`onAttempt`'s `nonSkippedAttempts`). + * Requiring it now is free; requiring it LATER would be the one-way door {@link parseStoredRunEvent} + * describes — every historical row without it becomes a known type with an invalid body, which must throw. + */ + attemptNumber: positiveInt, + inputTokens: nonNegativeInt, + outputTokens: nonNegativeInt, + /** + * THIS attempt's realized charge — **the field checkpoint reconstruction adds up** (see + * {@link cumulativeCostMicrocents}). + * + * `nonNegativeInt`, NOT the `positiveInt` its estimate twin uses, and the asymmetry is real rather than an + * oversight. A conservative commitment of zero is unreachable by construction (`BudgetGovernor#admit` + * refuses `estimateMicrocents <= 0`). A realized charge of zero is routinely reachable: an UNPRICED model + * (the chain swallows the cost tracker's `UnknownModelError`) and a genuinely FREE one both settle here at + * zero with real tokens. {@link priced} is what separates those two — which is precisely why it is required. + */ + costMicrocents: nonNegativeInt, + /** + * The run-wide realized running total after this attempt — a producer-side snapshot for DISPLAY and + * cross-checking. **Not the restore source**, for the same reason ADR-0074 §2 gave for its twin: under a + * `fan_out` concurrent events have no canonical `seq` order, so a last-wins read of this field can restore a + * LOWER total and hand already-spent money back to the cap as headroom. Reconstruction sums + * {@link costMicrocents} instead, and takes the greater of that sum and the durable node-boundary snapshots + * — a sum within one family, a max across families, neither of which can over-count. + * + * (`cost:updated` gets away with a last-wins cumulative only because it is streamed and never persisted, so + * it is not the precedent for a durable event.) + */ + cumulativeCostMicrocents: nonNegativeInt, + /** + * Whether this egress could be PRICED (ADR-0070 §6). **REQUIRED here, unlike on `cost:updated`.** + * + * On `cost:updated` the flag is additive-and-optional because it was added to an event that already had + * historical rows. This type has none. And without it `costMicrocents: 0` with real tokens is ambiguous + * between "we could not price this" and "this model is genuinely free" — an ambiguity a LEDGER must not + * carry, since the whole reason to write the row is to answer "what did this run cost, and do we know". + */ + priced: z.boolean(), +}); +export type CostAttemptSettledEvent = z.infer; + /** The run-event variants, discriminated on `type` (exposed via `RunEventSchema.innerType()`). */ const RunEventUnionSchema = z.discriminatedUnion('type', [ RunStartedEventSchema, @@ -616,6 +703,7 @@ const RunEventUnionSchema = z.discriminatedUnion('type', [ BudgetWarningEventSchema, BudgetPausedEventSchema, BudgetEstimateCommittedEventSchema, + CostAttemptSettledEventSchema, ]); /** The pre-refinement union value — the input every cross-field refinement helper below receives. */ @@ -663,6 +751,29 @@ function refineBudgetEstimateCommitted(event: RunEventUnion, ctx: z.RefinementCt } } +/** + * A settled attempt's cumulative total must already INCLUDE its own charge (ADR-0076). + * + * The same invariant, and the same single most likely emit-site bug, as {@link refineBudgetEstimateCommitted}: + * reading the run-wide counter BEFORE folding this attempt into it. That mistake emits + * `{ cost: 500, cumulative: 0 }`, and while reconstruction sums the deltas (so the RESTORE survives it), every + * display and cross-check that reads the snapshot silently under-reports. Pinned here because nothing else + * anywhere would complain. + */ +function refineCostAttemptSettled(event: RunEventUnion, ctx: z.RefinementCtx): void { + if ( + event.type === 'cost:attempt_settled' && + event.cumulativeCostMicrocents < event.costMicrocents + ) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: + 'cumulativeCostMicrocents must already include costMicrocents (read the counter AFTER folding this attempt)', + path: ['cumulativeCostMicrocents'], + }); + } +} + /** A gate's on-timeout policy only has meaning when a timeout is configured. */ function refineHumanGateTimeout(event: RunEventUnion, ctx: z.RefinementCtx): void { if ( @@ -795,6 +906,7 @@ export const RunEventSchema = RunEventUnionSchema.superRefine((event, ctx) => { refineMediaJobFrozenBasis(event, ctx); refineApprovalPreview(event, ctx); refineBudgetEstimateCommitted(event, ctx); + refineCostAttemptSettled(event, ctx); }); export type RunEvent = z.infer; From abc8a6e565be5488c0f6562847eb5648e363573f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 19:30:40 +0300 Subject: [PATCH 05/26] =?UTF-8?q?docs(decisions):=20fold=20the=20Opus=20re?= =?UTF-8?q?view=20of=20ADR-0077=20=E2=80=94=20the=20barrier=20had=20no=20h?= =?UTF-8?q?ome?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six findings, three High. All verified against code before folding; the two that mattered are ones I had not reached. **The ledger cannot live on `BudgetGovernor`, and every signpost said it should.** ADR-0077 §1 read "exactly as §2's `#commitmentsInFlight`", and `node-executor.ts`'s comment literally says "add it to the governor's emit type instead". Following that would have been silently wrong, because the governor is conditional and the ledger is not: - `engine.ts:412` builds one only `if (params.plan.budget !== undefined)`; `budget` is optional in the workflow schema. An unbudgeted run spends real money and would have recorded nothing. - `engine.ts:2109` returns early with no governor → barrier B3 is a no-op. - `engine.ts:1190` returns `undefined` → the turn core never installs the `preAttempt` wrapper → barrier B1 does not exist. - `engine.ts:1214` — `budgetApproved ? undefined : this.#makePreEgressHook()` drops the hook for an approved re-dispatch, so B1 vanishes on the exact path the user just authorised more money on. An unbudgeted run would have got one barrier of three; one with no tool calls, none — and every test written against a budgeted fixture would have passed. New Decision §5 makes the owner the engine, unconditionally, names all three holes, and requires the regression to be an UNBUDGETED run. The correct composition already exists one surface over: `session-host.ts:382` installs a preEgress hook that is "ALWAYS present, cap or no cap", reads the durability probe FIRST, then delegates to `governor?.preEgress(info)`. This ADR's own "the session path is stronger here" section had the answer in it. **ADR-0076 §1 now carries the amendment marker**, following the precedent `f1b7773` set when ADR-0075 landed — it edited ADR-0074 §5's heading in place to add "(amended by ADR-0075)". Without it a reader arriving from `current.md`, `constants.ts` or the `run-event.ts` docblock — the likely paths — reads the withdrawn mechanism with nothing on the page saying so. Append-only-legal, and the mitigation ADR-0077 claimed (its own title and Related line) only ever reached someone who had already found ADR-0077. **Two implementation traps recorded**, both silent, both would review as correct: - The ledger emit must FOLLOW the cumulative fold. Emitting first leaves the cumulative stale, `refineCostAttemptSettled` rejects at the producer gate, and that gate runs in `#bus.next` — OUTSIDE `#emitDurable`'s try. So the wrong order does not degrade: `#emitDurable` REJECTS in the one place the design assumes it cannot. - The durable emit must not route through `#nodeEmit`; it returns `void`, so the promise is unawaitable — the unbarriered shape this ADR exists to prevent. Also: "Concretely, three things" listed four (now five); the hook into the turn core is stated as an optional `AgentTurnParams` field like `preEgress`, since `agent-turn.ts` is the boundary `AgentSession` shares. Verified clean by the review and left alone: ADR-0077's three central claims all hold; both required-field divergences are sound and every emit site can supply them; the run-only justification is STRONGER than ADR-0076 states (`event-bus.ts` delivers synchronously, so the session persister's write runs inside `onAttempt`'s own stack, and `session-host.ts:382` gates per-attempt, not per-turn); and all six reject fixtures are honest — one issue each, each the reason its name claims. Correction to `8d7ffcf`'s message: the bare `onAttempt` invocation is `fallback-chain.ts:903`, not `:902` (`:902` is the enclosing `#emit` signature). Still open and next: `sse-event-schema.md` has not been updated, so the canonical home still presents a closed union that no longer is. That is step 2 and lands before step 3. pnpm run ci EXIT=0. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- ...urable-per-attempt-realized-cost-ledger.md | 12 +++-- ...ses-the-conservative-commitment-barrier.md | 53 +++++++++++++++++-- 2 files changed, 59 insertions(+), 6 deletions(-) diff --git a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md index 26b87cf4..fc22d83b 100644 --- a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md +++ b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md @@ -30,9 +30,15 @@ The event carries the attempt's identity (node, model, the within-chain attempt Three properties make it a ledger rather than another observation: -1. **Written before the next thing that can spend or mutate.** It goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call — before the next tool side effect, before the next egress, and before the node/turn terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. - - **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** §2 needs +1. **Written before the next thing that can spend or mutate.** *(The MECHANISM below is **amended by + [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)** — the guarantee stands, + the "awaited emit" shape does not: `onAttempt` is synchronous, so there is no `await` to place there.)* It + goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call — before the + next tool side effect, before the next egress, and before the node/turn terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. + + **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** + **← AMENDED BY [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md); the + paragraph below is superseded and kept only because the corpus is append-only.** §2 needs `flushBudgetCommitments` because a conservative commitment is emitted fire-and-forget from inside the governor, so something has to join the outstanding writes at the turn boundary. A settled attempt is emitted by the engine at the point it settles, on the path that is about to continue, so no queue is diff --git a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md index 683d5091..bb564627 100644 --- a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md +++ b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md @@ -50,11 +50,12 @@ section: synchronously at the settle point onto a chained in-flight promise, and JOINED at every barrier that precedes spending or mutating. It is not a bare awaited emit.** -Concretely, three things: +Concretely, five things: 1. **Start the write at the settle instant.** `onAttempt` cannot await, but it can *begin*. The emit is chained - onto a per-owner in-flight promise exactly as §2's `#commitmentsInFlight` is — which also serializes the - writes, so two settles in the same tick cannot interleave their persists. + onto a per-owner in-flight promise, copying the shape of §2's `#commitmentsInFlight` — which also serializes + the writes, so two settles in the same tick cannot interleave their persists. **The shape is copied; the + owner is not** — see §5. 2. **Join at three barriers, not two.** §2 joins at the pre-egress check and at the turn/node boundary. ADR-0076's guarantee names a third thing to precede — the next **tool side effect** — so the ledger's barrier set is: before the next egress admission, **before tool dispatch**, and at the turn/node terminal. @@ -71,6 +72,47 @@ Concretely, three things: chains and both retained failures; there is no supported way to await half the money. This is what turns the "a future barrier joins only one chain" hazard from a review item into something a caller cannot express. +5. **The ledger and its join are owned by the ENGINE, unconditionally — never by `BudgetGovernor`.** Reading + §1's "as §2's `#commitmentsInFlight`" as "put it on the governor" would silently omit most runs, and every + signpost in the codebase points that way (`node-executor.ts`'s comment even says "add it to the governor's + emit type instead"). It is wrong here, because the governor is **conditional and the ledger is not**: + + - `engine.ts` constructs a `BudgetGovernor` only `if (params.plan.budget !== undefined)`, and `budget` is + optional in the workflow schema. An unbudgeted run spends real money and would get no ledger at all. + - `#flushBudgetCommitments` returns immediately when there is no governor, so **barrier B3 becomes a + no-op**. + - `#makePreEgressHook()` returns `undefined` without a governor, so the turn core never installs the + `preAttempt` wrapper and **barrier B1 does not exist**. + - Even on a BUDGETED run, `const preEgress = budgetApproved ? undefined : this.#makePreEgressHook()` drops + the hook for an approved `pause_for_approval` re-dispatch — so B1 disappears on the exact path the user + just authorised more money on. + + Implemented literally, an unbudgeted run would get one barrier of three, and an unbudgeted run with no tool + calls would get none — the fire-and-forget state this ADR exists to remove, passing every test written + against a budgeted fixture. + + **The correct shape already exists one surface over.** `session-host.ts` installs an `preEgress` hook that + is *"ALWAYS present, cap or no cap"*, reads the durability probe FIRST, and only then delegates to + `governor?.preEgress(info)` — *"It composes with the governor rather than replacing it, and it runs + FIRST."* The run path never got that. So this ADR's own "the session path is stronger here" section had the + answer to this flaw in it; the run path adopts the same composition. + +### Two implementation traps this decision creates + +Named because both are silent, and both would look correct in review: + +- **The ledger emit must follow the cumulative fold, not precede it.** The engine advances its run-wide total + in `#nodeEmit`'s `cost:updated` arm. Emitting the ledger draft *before* that leaves the cumulative stale, and + `refineCostAttemptSettled` rejects `cumulative < cost` at the producer gate — which runs in `#bus.next`, + **outside** `#emitDurable`'s `try`. So the wrong order does not degrade: `#emitDurable` **rejects** instead + of resolving, in the one place the whole design assumes it cannot. Nothing pins that order today. +- **The durable emit must not be routed through `#nodeEmit`.** It returns `void`, so the promise would be + unawaitable — the exact unbarriered shape this ADR exists to prevent, reached through the door + `node-executor.ts`'s in-node-event-union comment already warns about. + +The hook that carries the emit into the turn core is an OPTIONAL `AgentTurnParams` field, modelled on +`preEgress`, because `agent-turn.ts` is the boundary `AgentSession` shares. Leaving it unset on the session +path is what makes this event run-only as a runtime fact rather than a comment. ### Why the tool-dispatch barrier matters beyond this ADR @@ -149,6 +191,11 @@ no arm, the four rejected alternatives, and the reason it lands after advisory — Decision §4: the barriers join through a single entry point that owns both chains and both retained failures, so "await the wrong one" is not an expressible mistake. What a reviewer still has to catch is a new barrier that calls neither. +- **Three barrier holes have to close for this to be true at all**, and each is a line that reads as correct + today: the governor-conditional `#flushBudgetCommitments` early return, the `undefined` from + `#makePreEgressHook`, and `budgetApproved ? undefined`. Mitigation: Decision §5 names all three, and the + regression that proves them is an **unbudgeted** run — a fixture with a budget passes even when every + barrier is missing. - **The tool-dispatch barrier is new engine surface**, on the hottest path a run has, and it lands before the effect journal that will build on it. If it is placed wrong, both this ledger and `CR-12` inherit the same crash window. Mitigation: it goes in at `agent-turn.ts`'s single `await dispatchToolCalls(...)` — one call From 762c06f051bb415b427c79805ba6032fd36e2404 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 19:55:16 +0300 Subject: [PATCH 06/26] docs(reference): specify cost:attempt_settled in its canonical home MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 2 of 5. The Opus review flagged that step 1 shipped the Zod implementation while the canonical contract still presented a CLOSED union that no longer is — which inverts CLAUDE.md rule 8 and ADR-0076's own sentence that `run-event.ts` is "the Zod validation that implements it, not a second specification". This closes that. Seven insertion points, but two of them are RECONCILIATIONS, not additions — this event makes two existing sentences untrue and appending without fixing them would leave the spec contradicting itself: - The `attemptNumber`-families note said "Run totals are unaffected: `cost:updated.cumulativeCostMicrocents` is the engine's authoritative running total." True only for a LIVE reader. `cost:updated` is never persisted, so a reader reconstructing from the durable log cannot use it at all. - `node:completed.cumulativeCostMicrocents` was "THE durable cost source checkpoint/resume restores from, since `cost:updated` is streamed-only". It is now one of two — and it remains the ONLY carrier of media spend, which emits no attempt row. That is why the restore is a max and not a replacement. So the doc now states the three-way split once, as ADR-0076's last Negative requires and did not have: `cost:updated` = the live observation; `cost:attempt_settled` = the durable record of what was charged; the RESTORE = the greater of (SUM of attempt deltas) and (the node-boundary snapshots). Sum within the family, max across families. Adding them would double-count, because a completed node's snapshot already contains its own attempts. The row goes in the MAIN union table, not the governance table: that section hard-codes "adds four run events", repeats "These four", and declares all of them non-terminal governance/suspension signals. A realized-cost ledger row is neither. Placed last, mirroring its position in the union and in RUN_EVENT_TYPES. Forward-compatibility gets a new paragraph, because this is the FIRST event to exercise ADR-0075's replay carve-out and that is precisely why ADR-0075 landed first: adding it is additive for every rendering consumer and deliberately NOT additive for a replay. "Adding a new event type is never a breaking change" is true of the stream and false of the resume, and the two halves now have to be read together. `run_costs` is cited, not restated — it has its home in database-schema.md (rule 8). The dual-envelope correlation-key paragraph is deliberately untouched: this event is run-only. pnpm run ci EXIT=0. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- README.md | 311 +++++++++++-------- assets/readme/agent-to-workflow.svg | 70 +++++ assets/readme/relavium-architecture.svg | 100 ++++++ assets/readme/relavium-hero.svg | 99 ++++++ docs/reference/contracts/sse-event-schema.md | 29 +- 5 files changed, 476 insertions(+), 133 deletions(-) create mode 100644 assets/readme/agent-to-workflow.svg create mode 100644 assets/readme/relavium-architecture.svg create mode 100644 assets/readme/relavium-hero.svg diff --git a/README.md b/README.md index 13bf182a..9283d547 100644 --- a/README.md +++ b/README.md @@ -1,158 +1,213 @@ # Relavium -> **Start as an agent. Ship the workflow. Own every run.** -> A multi-surface, local-first AI agent workflow platform — a product of [HodeTech](https://github.com/HodeTech). - -Relavium meets you where you already work — in conversation — and gives that -conversation somewhere to go. You **start as an agent**: a multi-turn session in your -terminal, in VS Code, or in a desktop chat panel. When a flow proves itself, you **ship -the workflow**: export the session to a git-committable, multi-agent, multi-model -`.relavium.yaml` pipeline that runs identically in your editor, your terminal, and your -CI. Or author workflows directly. Either way you **own every run** — every step -debuggable, every token and dollar tracked, every artifact yours, nothing leaving your -machine unless you choose it. - -## Why Relavium? - -- **Four surfaces, one engine.** Desktop (Tauri), CLI, VS Code, and (planned) the web - portal run the _identical_ pure-TypeScript engine. No Python sidecar, no single-tool - lock-in — every surface is a first-class execution target. -- **A chat-to-workflow continuum.** Other tools make every session ephemeral. Relavium - sessions are persistent, resumable, and one-click exportable into a reviewed, - committed workflow. -- **You own your LLM seam.** Multi-provider routing with fallback chains - (`[claude → gpt-4o → gemini]`) is first-class through Relavium's own `@relavium/llm` - abstraction over the official provider SDKs — no Vercel AI SDK, no LangChain. -- **Local-first by design.** Zero cloud, no account required. Your API keys live in your - OS keychain — never in plaintext, never in logs. Optional managed inference and cloud - execution are planned extensions on the same engine. -- **Workflows are git objects.** `.relavium.yaml` files are diffable, reviewable, - PR-able, and shareable — team infrastructure, not a proprietary JSON blob or buried - Python. -- **Multimodal, end-to-end.** Image / audio / video as input and output — including - rule-driven media generation — flow through the same seam and engine. - -## Highlights - -- **Chat-to-workflow export** — turn a proven session into a reusable `.relavium.yaml`. -- **Persistent, resumable agent sessions** — no run is ever ephemeral. -- **Live execution** — tokens stream as the run progresses; parallel branches run together. -- **Multi-model fallback chains** — runs survive provider outages and rate limits. -- **Checkpoint & resume** — pause and resume at any node boundary, even across processes. -- **Human gates with timeout policy** — pause for an approve / reject / input decision. -- **Per-node cost waterfall** — token and dollar attribution per node, per model. -- **Interactive Home** — a bare `relavium` invocation opens a management center: start agents, - monitor runs, browse history, manage providers. -- **MCP client** — agents consume tools from external MCP servers over stdio, HTTP, SSE, and - WebSocket, with secrets in the keychain. -- **Live model catalog** — browse and switch models mid-session; per-model cost tracking. -- **Local-first, zero-install posture** — BYOK, OS keychain, no sign-up. - -## Getting started - -The CLI is the first usable surface. It ships as a single npm binary — `npm install -g relavium` -(the public npm publish is the final maintainer step of the **v0.1.1** release; until it lands, build from -source per [local dev setup](docs/runbooks/local-dev-setup.md)). Then **start as an agent → ship the workflow -→ own every run**: +

+ Relavium — start as an agent, ship the workflow, own every run +

+ +

+ Start as an agent. Ship the workflow. Own every run. +

+ +

+ A local-first AI agent platform that turns productive conversations into
+ version-controlled, multi-agent workflows — on one pure-TypeScript engine.
+ A product of HodeTech. +

+ +

+ npm version + CI status + Node.js 22 or newer + Strict TypeScript + Local-first BYOK + Proprietary license +

+ +

+ Get started · + Why Relavium · + Architecture · + Status · + Documentation +

+ +Relavium begins where agent work naturally begins: in conversation. Explore a task in a +persistent session, keep the flow that proves useful, and graduate it into a reviewable +`.relavium.yaml` workflow. The conversation and the workflow are not separate products — +they are two entry points into the same engine, tool registry, model seam, and event stream. + +## From conversation to infrastructure + +

+ A Relavium agent session becoming a committed workflow and observable runs +

+ +Most agent tools make you choose between a flexible chat and an operable workflow. Relavium +treats them as a continuum: + +1. **Explore in an agent session.** Work conversationally with streaming output, tools, + model controls, persistent history, and human approval. +2. **Promote what works.** Export the proven flow into git-native YAML that can be reviewed, + changed in a PR, and shared without embedding provider keys. +3. **Run it deliberately.** Execute locally or in CI with typed events, durable history, + human gates, checkpoints, and cost controls. + +You can also author the workflow directly; conversation is an on-ramp, not a requirement. + +## Get started + +The published CLI is the fastest way to use Relavium. It requires **Node.js 22 or newer**. + +```bash +npm install -g relavium +``` + +1. Connect a provider. The key is read from stdin and stored in the OS keychain — + never passed through argv. ```bash -# 1. Point Relavium at a provider — your key goes to the OS keychain, never a file relavium provider add anthropic -echo "$ANTHROPIC_API_KEY" | relavium provider set-key anthropic # the key is read from stdin, never argv +printf '%s\n' "$ANTHROPIC_API_KEY" | relavium provider set-key anthropic +``` + +2. Start as an agent. -# 2. Start as an agent — a multi-turn session in your terminal +```bash relavium chat -# …converse until a flow proves itself, then run /export inside the REPL -# to ship the session to a git-committable .relavium.yaml +``` + +3. Run `/export` inside the session to create a git-committable workflow. Execute it + interactively, or stream NDJSON for CI with `--json`. -# 3. Own every run — execute the workflow and stream every event (CI-friendly with --json) +```bash relavium run ./my-workflow.relavium.yaml --json ``` -Prefer to author directly? `relavium create` scaffolds an agent or a minimal single-agent workflow, and -`relavium import` / `relavium export` move them between projects. The full surface is the -[CLI command reference](docs/reference/cli/commands.md). +Prefer to author first? `relavium create` scaffolds an agent or a minimal workflow; +`relavium import` and `relavium export` move validated artifacts between projects. See the +[CLI command reference](docs/reference/cli/commands.md) for the complete surface and the +[local development runbook](docs/runbooks/local-dev-setup.md) to build from source. + +## Why Relavium + +| | | +|---|---| +| **Conversation becomes infrastructure** | A useful session can graduate into a durable workflow instead of disappearing into chat history. | +| **Git-native by construction** | Workflows are diffable `.relavium.yaml` files — reviewable in pull requests and owned by the team that runs them. | +| **Multi-model without framework lock-in** | Relavium owns its `LLMProvider` seam and routes across Anthropic, OpenAI-compatible providers, and Gemini without LangChain or the Vercel AI SDK. | +| **Local-first control** | Local BYOK is the default, no Relavium account is required, and provider keys are stored in the OS keychain rather than workflow files. | +| **Execution you can inspect** | Typed event streams, local run history, human gates, checkpoints, retries, fallback chains, and cost controls make a run observable. | +| **One engine, multiple surfaces** | `AgentSession` and `WorkflowEngine` share one platform-pure core designed for the CLI, desktop, VS Code, and future cloud workers. | ## Architecture -```mermaid -flowchart TD - subgraph Surfaces - D[Desktop · Tauri] - C[CLI] - V[VS Code extension] - P[Web portal · planned] - end - subgraph Engine["@relavium/core — one pure-TypeScript engine"] - WE[WorkflowEngine] - AS[AgentSession] - BUS[(RunEventBus · ToolRegistry)] - WE --- BUS - AS --- BUS - end - SEAM["@relavium/llm seam"] - PROV[Anthropic · OpenAI/DeepSeek · Gemini] - D --> Engine - C --> Engine - V --> Engine - P --> Engine - Engine --> SEAM --> PROV -``` +

+ Relavium architecture: multiple surfaces over one engine with AgentSession, WorkflowEngine, the LLM seam, MCP, and durable storage +

+ +The center of Relavium is `@relavium/core`, a strict TypeScript engine with **zero +platform-specific imports**. It exposes two co-equal entry points: -One engine, **two co-equal entry points** — `WorkflowEngine` (runs YAML pipelines) and -`AgentSession` (runs conversational chat) — sharing the same tool registry, the same -`@relavium/llm` multi-provider seam, and the same event bus. The engine has **zero -platform-specific imports**, so the same source runs in the Tauri WebView, the VS Code -host, the Node CLI, and (planned) a Bun server. Supporting packages: `@relavium/shared` -(Zod contracts), `@relavium/db` (Drizzle — SQLite locally, PostgreSQL planned), and -`@relavium/ui` (ReactFlow canvas + shadcn). See [docs/architecture/](docs/architecture/). +- **`AgentSession`** for conversational, multi-turn work. +- **`WorkflowEngine`** for declarative `.relavium.yaml` execution. -## Execution modes +Both reuse the same `ToolRegistry`, typed event substrate, and Relavium-owned +`@relavium/llm` abstraction. Official provider SDKs are confined to thin adapters; no +vendor SDK type crosses the seam. Host packages supply persistence, MCP connections, +keychain access, files, processes, and network I/O without making the engine +platform-specific. Read the [architecture overview](docs/architecture/) or the +[decision records](docs/decisions/) for the reasoning behind those boundaries. -One engine, three modes behind the one `LLMProvider` seam: +> The CLI is the currently published product surface. Desktop and VS Code integrations, +> plus managed inference and cloud execution, are under development or planned. The +> diagram shows the shared-engine topology, not equal release availability. -- **Local (BYOK)** — the default. Your keys, your machine, zero Relavium data. -- **Managed inference** — planned. Relavium's metered keys; the engine still runs locally. -- **Cloud execution** — planned. Run workflows on cloud workers for 24/7 automation and - team sharing. +## What ships today -## Status +| Capability | Available in `relavium@0.1.1` | +|---|---| +| Conversational agents | Streaming multi-turn chat, persisted sessions, resume, model reseat, context compaction, and workflow export | +| Workflow runtime | YAML parse and validation, DAG execution, parallel branches, retries, model fallback, checkpoints, and typed live events | +| Human control | Per-tool approval modes plus durable workflow gates that can pause and resume out of process | +| Operations | Interactive Home, run status and history, event-log replay, deterministic exit codes, and NDJSON output for CI | +| Providers and tools | Anthropic, OpenAI-compatible, Gemini, an inbound MCP client, built-in tools, and a live/offline model catalog | +| Local ownership | BYOK, OS-keychain storage, project-local git artifacts, and local run/session history | -The engine is complete and the CLI is feature-complete (cut as **v0.1.1**, npm publish pending). -What's shipped: +For exact command behavior and contracts, use the [reference documentation](docs/reference/) +rather than this overview. -- **Agent sessions** — `relavium chat` with persistent, resumable, exportable multi-turn sessions. -- **Workflow engine** — `relavium run` executes `.relavium.yaml` pipelines with live streaming, - checkpoint/resume, multi-model fallback, cost governance, and human gates. -- **Interactive Home** — the bare `relavium` invocation opens a management center with a - slash-command system, per-tool approval modes (ask/plan/accept-edits/auto), and context compaction. -- **MCP client** — agents consume tools from external MCP servers over stdio + network - transports, with secrets in the OS keychain. -- **Live model catalog** — onboard with a wizard, browse models, switch mid-session, track - per-model cost. -- **YAML authoring** — `relavium create` (wizard), `import`, and share-safe `export`. +## Local-first, precisely + +- **No account is required for local BYOK.** The CLI runs the engine and stores history on + your machine. +- **Provider keys do not belong in workflows or committed configuration.** Interactive + setup stores them in the OS keychain; a documented environment fallback exists for + automation. +- **Workflows remain ordinary files.** They can be reviewed, branched, reverted, and moved + without exporting from a proprietary database. +- **Network use is explicit.** LLM requests go to the provider you configure; optional + catalog refreshes and future managed/cloud modes are not hidden prerequisites for local + execution. + +The binding guarantees live in the [product constraints](docs/product-constraints.md) and +[security standard](docs/standards/security-review.md). + +## Project status + +Relavium is under active development. The **CLI is published as v0.1.1**; the pure engine, +agent-session entry point, workflow runtime, inbound MCP client, and CLI management surface +are implemented. The current engineering focus is **Phase 2.6.5 — Core Reliability +Remediation**, which hardens the execution core before the next product wave opens. + +Status changes quickly, so this README intentionally stays high-level. The canonical source +for the exact active wave, completed work, and open reliability obligations is +[docs/roadmap/current.md](docs/roadmap/current.md). + +## Repository map + +| Path | Responsibility | +|---|---| +| [`packages/core`](packages/core/) | Platform-pure agent-session and workflow engine | +| [`packages/llm`](packages/llm/) | Relavium model seam, adapters, fallback, usage, and cost logic | +| [`packages/shared`](packages/shared/) | Zod schemas and inferred types — the contract source of truth | +| [`packages/db`](packages/db/) | Local SQLite persistence with a Postgres-compatible schema and migrations | +| [`packages/mcp`](packages/mcp/) | SDK-confined inbound MCP client and schema validation | +| [`apps/cli`](apps/cli/) | Published terminal product and integration harness | +| [`apps/desktop`](apps/desktop/) | Tauri desktop surface under development | +| [`apps/vscode-extension`](apps/vscode-extension/) | VS Code surface under development | + +The full dependency graph and ownership rules live in +[docs/project-structure.md](docs/project-structure.md). + +## Development + +Relavium is a pnpm + Turborepo monorepo. For a first local verification: + +```bash +corepack enable +pnpm install --frozen-lockfile +pnpm run ci +``` -**Next: Phase 2.6 (Conversational Authoring and the First-Class CLI)** — a full-screen -Home-managed CLI with conversational workflow authoring, management browsers, competitor-breadth -tools, settings/theming, and `en`/`tr` localization. For live status and the full roadmap, see -[docs/roadmap/current.md](docs/roadmap/current.md) and the [roadmap](docs/roadmap/README.md). +Use pnpm only — never npm or yarn for workspace development. Start with the +[local development setup](docs/runbooks/local-dev-setup.md), then read +[`CLAUDE.md`](CLAUDE.md) or [`AGENTS.md`](AGENTS.md) before making changes. ## Documentation -The canonical documentation lives in [`docs/`](docs/) — start at -[docs/README.md](docs/README.md), which is organized by _the kind of question each -section answers_. +The canonical documentation is organized by the question you are trying to answer: -| Start here | | -|------------|---| -| [Vision](docs/vision.md) · [Product constraints](docs/product-constraints.md) · [UVP](docs/uvp.md) | What and why | -| [Tech stack](docs/tech-stack.md) · [Project structure](docs/project-structure.md) | What it's built with | -| [Architecture](docs/architecture/) · [Decisions (ADRs)](docs/decisions/) · [Reference](docs/reference/) | How it works | -| [Roadmap](docs/roadmap/README.md) · [Standards](docs/standards/) | Where it's going, and the rules | +| Start here | Answers | +|---|---| +| [Vision](docs/vision.md) · [Product constraints](docs/product-constraints.md) · [UVP](docs/uvp.md) | What is Relavium, and why does it exist? | +| [Architecture](docs/architecture/) · [ADRs](docs/decisions/) | How is it built, and why these boundaries? | +| [Reference](docs/reference/) | What are the exact YAML, event, CLI, database, and integration contracts? | +| [Roadmap](docs/roadmap/README.md) · [Current state](docs/roadmap/current.md) | What is shipped, active, and next? | +| [Standards](docs/standards/) · [Runbooks](docs/runbooks/) | How should the project be changed and operated? | ## License Relavium is **proprietary software** — © 2026 HodeTech, all rights reserved. It is -**not** open source and grants no rights except as expressly stated. See -[LICENSE](LICENSE) for the full terms. For licensing inquiries, written permission, or -commercial-use agreements, contact [HodeTech](https://github.com/HodeTech). +not open source and grants no rights except as expressly stated. See [LICENSE](LICENSE) +for the full terms. For licensing inquiries or commercial-use agreements, contact +[HodeTech](https://github.com/HodeTech). diff --git a/assets/readme/agent-to-workflow.svg b/assets/readme/agent-to-workflow.svg new file mode 100644 index 00000000..9fc0fa6d --- /dev/null +++ b/assets/readme/agent-to-workflow.svg @@ -0,0 +1,70 @@ + + From agent session to owned workflow runs + Four stages show an agent session becoming a proven flow, a committed Relavium YAML workflow, and observable runs. + + + + + + + + + + + + + + + + + + + + + + + THE RELAVIUM CONTINUUM + A conversation is the beginning, not the dead end. + + + + + + + 1 + AGENT SESSION + explore in conversation + + persistent + tool-aware + + + + + 2 + PROVEN FLOW + keep what actually works + + decisions + structure + + + + + 3 + .RELAVIUM.YAML + review, diff, commit + + portable + git-native + + + + + 4 + OWN EVERY RUN + observe and operate + + events + history + gates + + + ONE ENGINE · TWO ENTRY POINTS · REPEATABLE EXECUTION + + diff --git a/assets/readme/relavium-architecture.svg b/assets/readme/relavium-architecture.svg new file mode 100644 index 00000000..8a8ffc85 --- /dev/null +++ b/assets/readme/relavium-architecture.svg @@ -0,0 +1,100 @@ + + Relavium shared-engine architecture + CLI, desktop, VS Code, and cloud surfaces connect to one pure TypeScript engine with AgentSession and WorkflowEngine entry points, backed by shared contracts, an LLM seam, MCP, and durable storage. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SHARED-ENGINE ARCHITECTURE + + SURFACES + + + + + + + + CLI + terminal + CI + + Desktop + Tauri management center + + VS Code + authoring + execution + + Cloud host + future workers + API + + + + + + + + @RELAVIUM/CORE + PURE TYPESCRIPT · ZERO PLATFORM IMPORTS + + + + + AgentSession + conversational, multi-turn entry point + + + + + WorkflowEngine + declarative DAG execution entry point + + + TOOL REGISTRY · TYPED EVENT BUS · CHECKPOINT / RESUME · POLICY + + + + + HOST-INJECTED CAPABILITIES + SHARED CONTRACTS + + + + + + + @relavium/llm + provider-neutral model seam + @relavium/shared + Zod contracts + types + @relavium/mcp + inbound tools + SDK fence + @relavium/db + durable local / cloud stores + + POLICY IN CORE · MECHANISM AT THE HOST · PROVIDERS BEHIND THE SEAM + + diff --git a/assets/readme/relavium-hero.svg b/assets/readme/relavium-hero.svg new file mode 100644 index 00000000..4335142a --- /dev/null +++ b/assets/readme/relavium-hero.svg @@ -0,0 +1,99 @@ + + Relavium + A local-first agent platform: start as an agent, ship the workflow, own every run. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + LOCAL-FIRST · MULTI-MODEL · GIT-NATIVE + + + Start as an agent. + Ship the workflow. + Own every run. + Conversation becomes version-controlled automation + on one pure-TypeScript engine. + + + + + + AGENT SESSION + explore · refine · persist + + + + + + + + + .RELAVIUM.YAML + diff · review · commit + + GIT-NATIVE + + + + + + + + + + RUN 01 + local · completed + + + + RUN 02 + CI · streaming + + + RELAVIUM · A HODETECH PRODUCT + + diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index b88baf35..fc73ddb8 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -64,7 +64,8 @@ export type RunEvent = | RunTimeoutEvent | BudgetWarningEvent | BudgetPausedEvent - | BudgetEstimateCommittedEvent; // dual-envelope (ADR-0074 §2) — a conservative ESTIMATE, never realized spend + | BudgetEstimateCommittedEvent // dual-envelope (ADR-0074 §2) — a conservative ESTIMATE, never realized spend + | CostAttemptSettledEvent; // RUN-ONLY (ADR-0076) — the REALIZED twin of the line above; the only DURABLE cost: event ``` > `RunPausedEvent` is the multi-gate aggregate (below); `RunTimeoutEvent` / `BudgetWarningEvent` / `BudgetPausedEvent` / `BudgetEstimateCommittedEvent` are the resource-governance events defined in [Workflow governance and reserved events](#workflow-governance-and-reserved-events). @@ -80,7 +81,7 @@ export type RunEvent = | `agent:approval_requested` | A side-effecting tool dispatch is awaiting an **interactive per-tool approval** decision (ADR-0057 EA3/EA5). The engine's `confirmDispatch` emits it — for **every** governed dispatch reaching the gate, whether the host then prompts a human or auto-decides — just before invoking the host's `ConfirmActionHook`; the registry then awaits the verdict (approve ⇒ dispatch, reject ⇒ a fatal `tool_denied`). A **dual-envelope** event (`runId`/`sessionId`), like `agent:tool_call` — in Phase 2.5 emitted only on the chat session path (the approval regime), and **carried on the session stream** (not run-only — it is **not** dropped like `agent:file_patch_proposed`). | `nodeId`, `toolId`, `action: 'fs_write' \| 'process' \| 'egress' \| 'os'` (the governed side-effect class — [tool-registry.md](../shared-core/tool-registry.md)), `preview` (**secret-free, display-only**: `{ path? }` for a write, `{ command? }` for a process, `{ host? }` for egress, `{}` for an `os` action like `read_clipboard`/`notify` — never a full URL/query, never a secret. Secret-freedom is **enforced** by the registry's redaction, not asserted, so a field **may contain the literal `[redacted]` marker and is not guaranteed to be a resolvable path/command/host — a machine consumer must never parse it**; see [tool-registry.md](../shared-core/tool-registry.md) §Preview redaction), `attemptNumber?` | | `agent:file_patch_proposed` | An agent proposed a file change (**gated — no write until the user accepts**; e.g. the VS Code inline-diff review). | `nodeId`, `patches: [{ uri, unifiedDiff }]` (≥1 — an empty proposal is meaningless), `attemptNumber?` | | `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note), `priced?` (**ADR-0070** — additive + optional, so an older reader ignores it: `false` when the egress could **not be priced**. An unpriced model still emits this event with its **real tokens** and `costMicrocents: 0`, which makes "cost 0 + tokens > 0" ambiguous between *could not price* and *genuinely free* — an ambiguity nothing else in the event resolves. The durable `session_costs` row records it as an `unpriced_calls` **counter**, not a boolean, because a model can be priced **mid-session**). **Generative-node variant (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5):** a `media_surface: 'generative'` agent node emits **exactly one** `cost:updated` with `inputTokens` / `outputTokens` **= 0** (no token billing — the spend rides entirely in `costMicrocents` as the per-modality media addend) and **no `attemptNumber`** (no FallbackChain on the generative path — one provider, no failover). | -| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — the durable cost source checkpoint/resume restores from, since `cost:updated` is streamed-only; the engine always populates it. `node:failed` mirrors this field, 2.S/D-GC) | +| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — **one of the two durable cost sources** checkpoint/resume restores from, the other being the SUM of `cost:attempt_settled` deltas since [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md); the restore takes the greater of the two, and this snapshot is still the ONLY carrier of media spend, which emits no attempt row. `cost:updated` is streamed-only and is not a durable source at all. The engine always populates this field. `node:failed` mirrors it, 2.S/D-GC) | | `node:failed` | A node failed (TERMINAL — exactly one per node; emitted when the node-retry budget is exhausted, on a fatal / `retry_on`-excluded failure, **or** when a pending retry is abandoned by a cancel or a sibling abort — see 1.S). | `nodeId`, `error: {code, message, retryable, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `correlationId` is a secret-free id joined to the internal log — ADR-0036), `attemptNumber?` (the last attempt, when a retry budget was spent — 1.S), `cumulativeCostMicrocents?` (the run-wide running total snapshotted AT this node boundary — the durable fail-cost so a billed-but-failed **paid media job**'s realized spend survives the transient `cost:updated`, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `node:completed`) | | `node:retrying` | A retryable node attempt failed and the engine will re-dispatch the whole node (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)) — **non-terminal** (the node continues; `node:failed` is the terminal). | `nodeId`, `attemptNumber` (the attempt that just failed, 1-based), `error: {code, message, retryable}` (the `NodeFailure` shape — **no** `correlationId`; that anchors the terminal failure), `delayMs` (backoff before the next attempt) | | `node:skipped` | A node was skip-propagated (never ran). | `nodeId`, `reason: 'branch_not_taken' \| 'upstream_unreachable'` (`branch_not_taken` = a `condition` routed away from it; `upstream_unreachable` = every in-edge is dead because an upstream was skipped/failed). Emitted so the event log is a **complete, replayable** record — checkpoint/resume reconstructs a skipped vertex from it ([run-plan.md](../shared-core/run-plan.md)) and a surface can render the dimmed path instead of the node silently vanishing. | @@ -91,6 +92,9 @@ export type RunEvent = | `run:completed` | The run finished. | `outputs` (a record **keyed by each terminal `output` vertex's node id**, the value being that vertex's captured output — see [run-plan.md §output capture](../shared-core/run-plan.md)), `totalTokensUsed`, `totalCostMicrocents` (integer micro-cents closing total for the whole run), `durationMs` | | `run:failed` | The run failed. | `error: {code, message, retryable, nodeId?, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `nodeId` is the root-cause node; `correlationId` joins to the internal log — ADR-0036), `partialOutputs`, `cumulativeCostMicrocents?` (the run-wide running total at failure — the durable fail-cost for a **paid media job** a sibling node's failure abandoned, whose lone estimate addend is folded just before this terminal _after_ the root-cause `node:failed` snapshot, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `run:cancelled` and the `run:completed` counterpart `totalCostMicrocents`) | | `run:cancelled` | The run was cancelled. | `cumulativeCostMicrocents?` (the run-wide running total at cancellation — the durable fail-cost for a **paid media job** pending at the cancel, whose lone estimate addend is folded just before this terminal, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; the `run:completed` counterpart is `totalCostMicrocents`) | +| `cost:attempt_settled` | **One settled provider attempt's REALIZED charge, made durable** ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) — the realized twin of `budget:estimate_committed`, emitted from the same callback and joined at the same barriers ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). It exists because `cost:updated` is streamed and never persisted, so before this event a crash mid-agent-loop discarded every charge since the last node boundary — and the resumed run spent it AGAIN, against a cap understated by exactly that amount. **RUN-ONLY** (`runId` always, never `sessionId`), unlike `cost:updated` and unlike its estimate twin: the session path already records the same per-attempt increment into `session_costs` synchronously (ADR-0070 + `#W15-4`), so it needs no arm here. **Scope:** it covers the provider attempts of an AGENT TURN. A media job's realized cost (ADR-0045 §5) does **not** emit one and does not need to — it is already durable through the `node:completed` / `node:failed` / `run:*` cumulative snapshots. And no cost event of any kind makes a TOOL EFFECT idempotent; that is a separate decision about a separate failure. | `nodeId`, `model`, `attemptNumber` (**required**, 1-based within-chain — optional on `cost:updated` only because that event predates this one and has historical rows; a per-attempt ledger row that cannot say which attempt it is, is not a ledger), `inputTokens`, `outputTokens`, `costMicrocents` (**this attempt's delta**, not a cumulative — `0` is reachable for an unpriced or genuinely free model), `cumulativeCostMicrocents` (the run-wide total **after** this attempt; a display snapshot, **not** the restore source — see the fold rule below), `priced` (**required**, same reason as `attemptNumber`: without it `costMicrocents: 0` with real tokens cannot distinguish *could not price* from *genuinely free*, which is the ambiguity a ledger exists to remove) | + +> **Which cost event is authoritative for what — the three-way split, stated once.** `cost:updated` is the **live observation**: streamed, never persisted, last-wins, and it is what a surface renders while a run is moving. `cost:attempt_settled` is the **durable record of what was charged**: one row per settled attempt, and the only cost event a reader can trust after a restart. **Restoring** a run's realized total is neither of those alone — it is the **greater of** (the SUM of `cost:attempt_settled.costMicrocents`) and (the node-boundary snapshots on `node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`). A sum WITHIN the attempt family, a max ACROSS families: the sum is order-independent (which last-wins over `cumulativeCostMicrocents` is not — see the conservative twin's fold rule for the `fan_out` ordering argument), the snapshots still carry media spend that emits no attempt row, and neither source can over-count, so the larger is the better lower bound. Adding them would double-count, because a completed node's snapshot already includes its own attempts. The derived `run_costs` / `runs` rows this folds into are specified in [database-schema.md](../shared-core/database-schema.md), not here. > **The money basis is FROZEN at submit time** ([ADR-0074](../../decisions/0074-durable-conservative-budget-commitments.md) §3). `units` is the authored billed volume the submission was priced on, and `acceptedCostMicrocents` is what the admission actually reserved. A resume restores the reservation from `acceptedCostMicrocents` with **no pricing lookup**, so neither a workflow edit nor a user-price/catalog change between submission and resume can move a commitment the provider has already accepted, or rewrite the job's historical cost. `acceptedCostMicrocents: 0` is meaningful and distinct from absent — it says the submission was priced and reserved nothing, which is what an unpriced model's allow-degrade path does. **`acceptedCostMicrocents` present ⇒ `units` required** (schema-enforced): a frozen cost on an unfrozen basis would restore the old reservation while re-deriving the volume, which is the drift §3 exists to prevent. The converse does NOT hold — `units` alone is legitimate, and is exactly what the approved-bypass path writes (it freezes the volume and omits the cost, because no pricing hook ran and `0` would freeze "priced at zero" for a job that was never priced). **Both absent means the row is LEGACY** (written before §3): resume must re-derive the volume from the workflow definition and re-price from today's catalog, which may under-reserve if the price has fallen, so with a cap configured the governor **fails closed**: new egress is **held** — awaited, not refused — until that job settles and its realized charge replaces the guess. Holding rather than throwing is deliberate: `budget_exceeded` is not in `RETRYABLE_ERROR_CODES` and `retry_on` cannot widen it, so a thrown refusal would kill a sibling node and abort the run, abandoning the very job it was waiting for. The hold is bounded by the job's own `deadlineAt` and is always broken by an abort, and it announces itself once through the governor's hold notice (routed to stderr by `relavium gate`) — that notice is the observability §3 requires, so a resume is never a silent stall. @@ -99,9 +103,9 @@ export type RunEvent = `attemptNumber` appears on two **independent** counter families that must not be conflated (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)): - **Node-retry dispatch attempt** — on `node:started` / `node:completed` / `node:failed` / `node:retrying`. The engine's **above-chain** whole-node re-dispatch index. Absent ⇒ attempt 1; present + >1 ⇒ a re-dispatch (distinguishes "attempt N starting" from a replay). -- **Within-chain attempt** — on `cost:updated` / `agent:tool_call` / `agent:tool_result` / `agent:file_patch_proposed`. The **within-chain** `FallbackChain` attempt index inside a *single* node dispatch; it **resets to 1 on every node-retry re-dispatch** (a fresh chain runs each time). +- **Within-chain attempt** — on `cost:updated` / `cost:attempt_settled` / `agent:tool_call` / `agent:tool_result` / `agent:file_patch_proposed` (and `budget:estimate_committed`). The **within-chain** `FallbackChain` attempt index inside a *single* node dispatch; it **resets to 1 on every node-retry re-dispatch** (a fresh chain runs each time). It is optional on all of them except `cost:attempt_settled`, where it is required — that event has no historical rows to accommodate, and it is the one whose identity *is* the attempt. -The two do **not** join: on a node the budget retried, `node:completed.attemptNumber` may be `2` while the accompanying `cost:updated.attemptNumber` is `1`. To attribute cost to a node-retry attempt, **partition the `sequenceNumber`-ordered stream at each `node:started` / `node:retrying` boundary** — do not key by `(nodeId, attemptNumber)` across families. (Run totals are unaffected: `cost:updated.cumulativeCostMicrocents` is the engine's authoritative running total.) +The two do **not** join: on a node the budget retried, `node:completed.attemptNumber` may be `2` while the accompanying `cost:updated.attemptNumber` is `1`. To attribute cost to a node-retry attempt, **partition the `sequenceNumber`-ordered stream at each `node:started` / `node:retrying` boundary** — do not key by `(nodeId, attemptNumber)` across families. (Run totals are unaffected: `cost:updated.cumulativeCostMicrocents` is the engine's authoritative running total **for a LIVE reader**. A reader reconstructing a run's total from the durable log uses the three-way split above instead — `cost:updated` is never persisted, so it is not available to one.) ### Selected definitions @@ -223,6 +227,19 @@ export interface RunTimeoutEvent extends BaseEvent { elapsedMs: number; timeoutMs: number; } + +export interface CostAttemptSettledEvent extends BaseEvent { + type: 'cost:attempt_settled'; + // RUN-ONLY: `runId` always, `sessionId` never — unlike its estimate twin and unlike cost:updated. + nodeId: string; // required — on the run path every attempt has an owning vertex + model: string; // canonical model id this attempt actually ran on + attemptNumber: number; // 1-based WITHIN-CHAIN attempt. REQUIRED here, optional on cost:updated — see below + inputTokens: number; + outputTokens: number; + costMicrocents: number; // THIS attempt's realized charge — the delta a reader SUMS. 0 is reachable (unpriced/free) + cumulativeCostMicrocents: number; // the run-wide total AFTER it — a display snapshot, NOT the restore source + priced: boolean; // REQUIRED here, optional on cost:updated — see below +} ``` ### Security: event payloads never carry secrets @@ -299,7 +316,7 @@ A turn that **fails** (a provider error, a rate limit, an exhausted budget cap) Within a turn, the conversational work reuses the **same** `agent:token` / `agent:reasoning` / `agent:tool_call` / `agent:tool_result` / `cost:updated` event shapes the `AgentRunner` already emits — carried on the session envelope (`sessionId`). The per-turn append of user/assistant/tool messages is persisted as `session_messages` (see [database-schema.md](../shared-core/database-schema.md)); the contract is owned by [agent-session-spec.md](agent-session-spec.md). On every surface session events are produced and consumed **in-process** exactly like run events — only `llm_stream` crosses IPC on the desktop ([ipc-contract.md](ipc-contract.md#run-events-are-webview-side)). So the **complete typed event stream for a session** is the eight `session:*` lifecycle/side events (the `SessionEvent` union above — started / turn_started / turn_completed / cancelled / exported / compacting / compacted / trimmed) **plus** `agent:token` / `agent:reasoning` / `agent:tool_call` / `agent:tool_result` / `cost:updated` (and, on the chat approval path, `agent:approval_requested` — ADR-0057) carrying `sessionId` — plus `budget:estimate_committed` once [ADR-0074](../../decisions/0074-durable-conservative-budget-commitments.md) §4 wires the session's durable budget write; the schema and the sink already carry it, but no session producer reaches it yet — this full set is exactly what `relavium chat --json` emits (`agent:reasoning` included: a `--json` consumer that does not want it filters on `type`). -**The session stream (`SessionHandle`, 1.W).** A session is **long-lived across turns**, so — unlike a run's exactly-one-terminal `RunHandle` — the `SessionHandle.events` async-iterable stays **open across turns**: `session:turn_completed` is a per-turn boundary, **not** a stream terminal. The stream closes **only** on `session:cancelled` (the session's sole terminal); `session:exported` is a side event (1.Z), never a terminal. The bus assigns the **per-session** `sequenceNumber` — a monotonic counter keyed on `sessionId`, independent of any run's `runId` counter on the same shared bus (ADR-0036 "one bus, two namespaces") — with the **same** gap-detection / resync rule as a run. `AgentSession` (1.V) emits *envelope-free* drafts through its injected `SessionEventSink`; 1.W's `createSessionEventSink` attaches the `sessionId` and the bus stamps the `sequenceNumber` + `timestamp` at the one authoritative translation point. The bus's validation gate accepts both families via the combined `RunOrSessionEventSchema` (`@relavium/shared`). `agent:file_patch_proposed` is **run-only** (it carries `runId`, emitted by the `AgentRunner` workflow adapter — not the shared turn core), so it is **not** part of a session stream; `createSessionEventSink` drops it defensively at the seam. By contrast, `agent:token` / `agent:reasoning` (EA6), `agent:approval_requested` (ADR-0057 EA3/EA5) are dual-envelope events the chat session path emits, so the sink **carries** them (they are not run-only). `budget:estimate_committed` (ADR-0074 §2) is dual-envelope and the sink carries it too, but the session producer is §4's work — today only the run path emits it. +**The session stream (`SessionHandle`, 1.W).** A session is **long-lived across turns**, so — unlike a run's exactly-one-terminal `RunHandle` — the `SessionHandle.events` async-iterable stays **open across turns**: `session:turn_completed` is a per-turn boundary, **not** a stream terminal. The stream closes **only** on `session:cancelled` (the session's sole terminal); `session:exported` is a side event (1.Z), never a terminal. The bus assigns the **per-session** `sequenceNumber` — a monotonic counter keyed on `sessionId`, independent of any run's `runId` counter on the same shared bus (ADR-0036 "one bus, two namespaces") — with the **same** gap-detection / resync rule as a run. `AgentSession` (1.V) emits *envelope-free* drafts through its injected `SessionEventSink`; 1.W's `createSessionEventSink` attaches the `sessionId` and the bus stamps the `sequenceNumber` + `timestamp` at the one authoritative translation point. The bus's validation gate accepts both families via the combined `RunOrSessionEventSchema` (`@relavium/shared`). `agent:file_patch_proposed` is **run-only** (it carries `runId`, emitted by the `AgentRunner` workflow adapter — not the shared turn core), so it is **not** part of a session stream; `createSessionEventSink` drops it defensively at the seam. By contrast, `agent:token` / `agent:reasoning` (EA6), `agent:approval_requested` (ADR-0057 EA3/EA5) are dual-envelope events the chat session path emits, so the sink **carries** them (they are not run-only). `budget:estimate_committed` (ADR-0074 §2) is dual-envelope and the sink carries it too, but the session producer is §4's work — today only the run path emits it. `cost:attempt_settled` ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) is **run-only like `agent:file_patch_proposed`**, and for a stronger reason than a sink-level drop: the hook that produces it is an optional turn-core parameter the session path never sets ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)), so nothing is emitted there to drop. The session's realized spend is recorded per attempt into `session_costs` instead (ADR-0070), synchronously and before the next egress (`#W15-4`). ## Workflow governance and reserved events @@ -343,6 +360,8 @@ Removing or repurposing an existing field/type is a breaking change and is not d **Where the promise does NOT apply: a REPLAY.** "Ignore unknown `type`s" is addressed to consumers that RENDER the stream. A caller that reconstructs authoritative state in order to *do* something — `checkpointer.ts` seeding `engine.resumeFromCheckpoint` — cannot safely ignore a row it does not understand: it has no way to know whether that row was a node terminal, an async job submission, a gate decision or a cost commitment, so tolerating the hole means re-running completed work or re-submitting an already-billed job, silently. [ADR-0075](../../decisions/0075-fail-closed-resume-on-an-unreadable-event-log.md) narrows ADR-0074 §5 accordingly: the replay read (`loadRunEventLogForReplay`) **refuses** when any row was skipped, and every display read stays tolerant. There is no session counterpart because no session resume reads a stored event log — a session's durable state is typed rows, not events. +**`cost:attempt_settled` is the first event to exercise that carve-out, and it is why ADR-0075 landed first.** Adding it is additive and v1.0-legal for every RENDERING consumer — an older `relavium logs` drops the row and shows the rest of the run. It is deliberately **not** additive for a REPLAY: an older binary resuming a log that contains it would re-run paid work against a cap missing the very charges the event exists to record, which is the failure that makes the fix self-defeating on a downgrade. The refusal is the correct behaviour, and the remedy is an upgrade. Read the two halves together before adding the next durable type — "adding a new event `type` is never a breaking change" is true of the stream and false of the resume. + ## Transport notes ### Phase 1 — local (in-process on every surface) From ff7617a3a66a69b929ff9e9764a48e0dc9729d3a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 20:03:07 +0300 Subject: [PATCH 07/26] feat(db): fold cost:attempt_settled into run_costs, and fix the step-cost regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 3 of 5. Two things, and the second was not in the plan. **The new `applyDerived` arm.** One settled attempt becomes a `run_costs` row in the SAME transaction as its event — which is what makes the ledger idempotent with no second uniqueness key, since `UNIQUE(run_id, seq)` already bars a duplicate event and the derived row cannot outlive it. The row is a TELESCOPING delta off the event's cumulative, not the event's raw `costMicrocents` — and I had it the other way round until a break-verify refused to go red. My fixtures all had `cumulative - prev == costMicrocents`, so the two were indistinguishable; the mutation that should have failed passed. Chasing why found the case where they differ, and it is not hypothetical: `#emitDurable` starts every `persistEvent` immediately and serializes only DELIVERY — "persists stay concurrent", in its own comment. So a sibling's `node:completed` can commit FIRST carrying a cumulative that already includes this attempt, and a raw-delta row then adds that money a second time. The new test proves it: raw delta yields 1_400 where the truth is 1_000. Telescoping cannot double-count, because every money event carries an ABSOLUTE cumulative, so the total converges on the largest one seen regardless of commit order — and `SUM(run_costs) == total` holds by construction rather than by hoping the log is ordered. The event still carries the exact per-attempt charge; that is what a reader and the step-5 checkpoint fold sum. Only the derived row telescopes. Per-attempt attribution degrades under a `fan_out` exactly as the per-node delta already does, and now says so. **The step-cost regression ADR-0076 did not see.** `node:completed` computed `nodeCost` once and used it TWICE — the `run_costs` row and `step_executions.costMicrocents`. ADR-0076 property 3 celebrates that delta telescoping to zero and says nothing about the step row. So every agent node's step cost would have become 0, and that column is user-visible through `relavium status --json`. No test pinned it, so it would have shipped silently. `node:failed` shared the bug plus an `if (nodeCost > 0)` guard that would have skipped the write on exactly the failures whose cost matters most. Both now read `nodeSettledCost` — the sum of the node's rows — with the node-retry attribution caveat written down rather than left to be discovered. Also: `run_costs.input_tokens`/`output_tokens` carry the REAL per-attempt tokens (nothing sums those columns; only cost carries ADR-0070's invariant), the run's token totals are NOT bumped here (`node:completed` already adds the sum across attempts), and `modelId` stays unwritten — it is an FK to a catalog UUID that `schema.ts` documents as a dead column, and `event.model` is a raw provider string this store cannot resolve. The `default` arm's comment now says plainly that it is NOT an exhaustiveness guard: `RunEvent` has no `assertNever` anywhere, so the next durable money event added without an arm lands there silently — a row, no derived write, no compile error, no failing test. Seven tests, six break-verify mutations, each with the mutation confirmed applied before trusting the red: drop the `runs` bump; drop the `run_costs` insert; raw delta instead of telescoping; step cost back to `nodeCost` on completed; on failed; and bumping the run token totals. pnpm run ci EXIT=0. Refs: ADR-0070, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- packages/db/src/run-history-store.test.ts | 240 ++++++++++++++++++++++ packages/db/src/run-history-store.ts | 116 +++++++++-- 2 files changed, 342 insertions(+), 14 deletions(-) diff --git a/packages/db/src/run-history-store.test.ts b/packages/db/src/run-history-store.test.ts index 56c0d274..a28e55ed 100644 --- a/packages/db/src/run-history-store.test.ts +++ b/packages/db/src/run-history-store.test.ts @@ -1367,4 +1367,244 @@ describe('createRunHistoryReader', () => { expect(reader.loadRun('run-z')?.totalCostMicrocents).toBe(0); }); }); + + describe('the realized-cost ledger (ADR-0076 / #W15-1)', () => { + const ts = TS; + + const sumRunCosts = (runId: string): number => + client.db + .select({ c: runCosts.costMicrocents }) + .from(runCosts) + .where(eq(runCosts.runId, runId)) + .all() + .reduce((total, row) => total + row.c, 0); + + /** One settled attempt. `cumulative` is the run-wide total AFTER it (the schema refinement pins that). */ + const settled = ( + runId: string, + seq: number, + nodeId: string, + cost: number, + cumulative: number, + attemptNumber = 1, + tokens = { input: 10, output: 5 }, + ): RunEvent => + evRun( + runId, + 'cost:attempt_settled', + seq, + { + nodeId, + model: 'claude-opus-4-8', + attemptNumber, + inputTokens: tokens.input, + outputTokens: tokens.output, + costMicrocents: cost, + cumulativeCostMicrocents: cumulative, + priced: true, + }, + ts, + ); + + const startRun = async (runId: string): Promise> => { + const store = storeFor('wf'); + const workflowId = await store.resolveWorkflowId('wf'); + await store.persistEvent( + evRun(runId, 'run:started', 0, { workflowId, inputs: {}, executionMode: 'local' }, ts), + ); + await store.persistEvent( + evRun(runId, 'node:started', 1, { nodeId: 'n1', nodeType: 'agent' }, ts), + ); + return store; + }; + + it('writes a run_costs row AND advances the run total — the pair the invariant rests on', async () => { + const store = await startRun('run-l1'); + await store.persistEvent(settled('run-l1', 2, 'n1', 400, 400)); + + expect(reader.loadRun('run-l1')?.totalCostMicrocents).toBe(400); + expect(sumRunCosts('run-l1')).toBe(400); + const rows = client.db.select().from(runCosts).where(eq(runCosts.runId, 'run-l1')).all(); + expect(rows).toHaveLength(1); + // The REAL per-attempt tokens, unlike `foldCumulative`'s money-only addend. + expect(rows[0]).toMatchObject({ nodeId: 'n1', inputTokens: 10, outputTokens: 5 }); + }); + + it('is the INVERSE of budget:estimate_committed — the realized twin DOES touch the money of record', async () => { + // The estimate falls through `applyDerived`'s default and must never reach `runs`/`run_costs` + // (ADR-0074's central negative guarantee). Its realized twin must, or the ledger records nothing. + const store = await startRun('run-l2'); + await store.persistEvent( + evRun( + 'run-l2', + 'budget:estimate_committed', + 2, + { + nodeId: 'n1', + attemptNumber: 1, + model: 'claude-opus-4-8', + estimateMicrocents: 999, + cumulativeConservativeMicrocents: 999, + }, + ts, + ), + ); + expect(reader.loadRun('run-l2')?.totalCostMicrocents).toBe(0); + + await store.persistEvent(settled('run-l2', 3, 'n1', 400, 400)); + expect(reader.loadRun('run-l2')?.totalCostMicrocents).toBe(400); + expect(sumRunCosts('run-l2')).toBe(400); + }); + + it('telescopes: attempt rows advance the total, so node:completed contributes ZERO', async () => { + // ADR-0076 property 3, the load-bearing one. The terminal row is still written (it carries the node's + // token totals); its COST is zero by arithmetic, and the SUM invariant holds with no special case. + const store = await startRun('run-l3'); + await store.persistEvent(settled('run-l3', 2, 'n1', 400, 400, 1)); + await store.persistEvent(settled('run-l3', 3, 'n1', 600, 1_000, 2)); + await store.persistEvent( + evRun( + 'run-l3', + 'node:completed', + 4, + { + nodeId: 'n1', + output: { ok: true }, + tokensUsed: { input: 20, output: 10 }, + durationMs: 5, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + + const rows = client.db.select().from(runCosts).where(eq(runCosts.runId, 'run-l3')).all(); + expect(rows).toHaveLength(3); // two attempts + the terminal + expect(rows[2]?.costMicrocents).toBe(0); // the terminal's delta telescoped away + expect(reader.loadRun('run-l3')?.totalCostMicrocents).toBe(1_000); + expect(sumRunCosts('run-l3')).toBe(1_000); // ADR-0070's invariant, unchanged + }); + + it('does NOT double-count when a higher cumulative committed first (persists are concurrent)', async () => { + // `#emitDurable` starts every `persistEvent` immediately and serializes only DELIVERY — "persists stay + // concurrent", in its own words. So a sibling's terminal can land BEFORE an attempt row whose charge it + // already includes. Writing the event's raw `costMicrocents` would add that money twice; the telescoping + // fold cannot, because every money event carries an ABSOLUTE cumulative. + const store = await startRun('run-l7'); + await store.persistEvent( + evRun('run-l7', 'node:started', 2, { nodeId: 'n2', nodeType: 'agent' }, ts), + ); + // The sibling commits first with a cumulative that ALREADY contains n1's 400. + await store.persistEvent( + evRun( + 'run-l7', + 'node:completed', + 3, + { + nodeId: 'n2', + output: {}, + tokensUsed: { input: 1, output: 1 }, + durationMs: 1, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + expect(reader.loadRun('run-l7')?.totalCostMicrocents).toBe(1_000); + + // n1's attempt row lands late, carrying its own true charge of 400 and a cumulative of 400. + await store.persistEvent(settled('run-l7', 4, 'n1', 400, 400)); + + // 1_000, not 1_400. The money was already banked; this row contributes nothing and says so. + expect(reader.loadRun('run-l7')?.totalCostMicrocents).toBe(1_000); + expect(sumRunCosts('run-l7')).toBe(1_000); + }); + + it('keeps the step row at the node TRUE cost after telescoping, not the zeroed delta', async () => { + // The regression the telescoping would otherwise ship silently: `node:completed` used its own delta for + // `step_executions.costMicrocents`, and that delta is now routinely 0. The column is user-visible + // (`relavium status --json`), and nothing pinned it before this test. + const store = await startRun('run-l4'); + await store.persistEvent(settled('run-l4', 2, 'n1', 400, 400, 1)); + await store.persistEvent(settled('run-l4', 3, 'n1', 600, 1_000, 2)); + await store.persistEvent( + evRun( + 'run-l4', + 'node:completed', + 4, + { + nodeId: 'n1', + output: { ok: true }, + tokensUsed: { input: 20, output: 10 }, + durationMs: 5, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + + const step = client.db + .select() + .from(stepExecutions) + .where(and(eq(stepExecutions.runId, 'run-l4'), eq(stepExecutions.nodeId, 'n1'))) + .get(); + expect(step?.costMicrocents).toBe(1_000); + }); + + it('keeps the step row at the node TRUE cost on a FAILED node too', async () => { + // `node:failed` shared the same `nodeCost`-for-two-purposes bug, plus an `if (nodeCost > 0)` guard that + // would have skipped the write entirely on exactly the failures whose cost matters most. + const store = await startRun('run-l5'); + await store.persistEvent(settled('run-l5', 2, 'n1', 700, 700, 1)); + await store.persistEvent( + evRun( + 'run-l5', + 'node:failed', + 3, + { + nodeId: 'n1', + error: { code: 'provider_unavailable', message: 'boom', retryable: false }, + cumulativeCostMicrocents: 700, + }, + ts, + ), + ); + + const step = client.db + .select() + .from(stepExecutions) + .where(and(eq(stepExecutions.runId, 'run-l5'), eq(stepExecutions.nodeId, 'n1'))) + .get(); + expect(step?.costMicrocents).toBe(700); + expect(reader.loadRun('run-l5')?.totalCostMicrocents).toBe(700); + expect(sumRunCosts('run-l5')).toBe(700); + }); + + it('counts run-level TOKENS once — the attempt row must not bump the run totals', async () => { + // `node:completed.tokensUsed` is already the sum across this node's attempts, so an attempt arm that + // also bumped `runs.total_*_tokens` would double them. The `run_costs` row still carries per-attempt + // tokens for attribution; nothing sums that column. + const store = await startRun('run-l6'); + await store.persistEvent(settled('run-l6', 2, 'n1', 400, 400, 1, { input: 12, output: 6 })); + await store.persistEvent( + evRun( + 'run-l6', + 'node:completed', + 3, + { + nodeId: 'n1', + output: {}, + tokensUsed: { input: 12, output: 6 }, + durationMs: 5, + cumulativeCostMicrocents: 400, + }, + ts, + ), + ); + + const run = reader.loadRun('run-l6'); + expect(run?.totalInputTokens).toBe(12); + expect(run?.totalOutputTokens).toBe(6); + }); + }); }); diff --git a/packages/db/src/run-history-store.ts b/packages/db/src/run-history-store.ts index 28f0d478..a6842c21 100644 --- a/packages/db/src/run-history-store.ts +++ b/packages/db/src/run-history-store.ts @@ -525,6 +525,29 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis return delta; }; + /** + * Every micro-cent `run_costs` holds for one node — the step row's TRUE cost once a node's money can arrive + * in more than one row (ADR-0076). + * + * Before the realized-cost ledger, a node's whole charge landed in exactly one `run_costs` row written at its + * terminal, so the terminal's telescoping delta and the node's cost were the same number and the terminal arms + * used one variable for both. They are no longer the same number: `cost:attempt_settled` rows advance + * `runs.total_cost_microcents` as each attempt settles, so by the time the terminal folds its snapshot the + * delta is **zero by arithmetic** — which is correct for the run total and wrong for `step_executions`, a + * user-visible column (`relavium status --json`). Read the node's rows instead. + * + * **Attribution caveat, stated rather than discovered.** Under a node-RETRY there are several + * `step_executions` rows for one `nodeId` (keyed by attempt), while `run_costs` has no attempt column — so an + * earlier attempt's money lands on the final step row. That is the same class of approximation the per-node + * fan-out delta already carries (see the `node:completed` arm); the run-level SUM stays exact either way. + */ + const nodeSettledCost = (tx: TxDb, runId: string, nodeId: string): number => + tx + .select({ c: sql`coalesce(sum(${runCosts.costMicrocents}), 0)` }) + .from(runCosts) + .where(and(eq(runCosts.runId, runId), eq(runCosts.nodeId, nodeId))) + .get()?.c ?? 0; + /** Apply an event's derived `runs`/`step_executions`/`run_costs` writes (`run:started` inserts the runs row). */ const applyDerived = (tx: TxDb, event: RunEvent, runId: string, ts: number): void => { switch (event.type) { @@ -588,7 +611,11 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis outputJson: JSON.stringify(event.output), inputTokens: event.tokensUsed.input, outputTokens: event.tokensUsed.output, - costMicrocents: nodeCost, + // The node's TRUE cost, NOT `nodeCost` (ADR-0076). `nodeCost` is the terminal's telescoping delta, + // and once `cost:attempt_settled` rows have advanced the run total it is zero by arithmetic — + // correct for `runs`, and a silent regression here, because this column is user-visible. Read after + // the insert above so this node's terminal row is included. See `nodeSettledCost`. + costMicrocents: nodeSettledCost(tx, runId, event.nodeId), durationMs: event.durationMs, completedAt: ts, updatedAt: ts, @@ -623,19 +650,74 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis // persisted, so this snapshot is the only durable carrier, and dropping it left the run total (and // `sum(run_costs)`) short of money that was really charged. failStepRow(tx, event, runId, ts); - const nodeCost = foldCumulative( - tx, - runId, - event.cumulativeCostMicrocents, - ts, - event.nodeId, - ); - if (nodeCost > 0) { - tx.update(stepExecutions) - .set({ costMicrocents: nodeCost, updatedAt: ts }) - .where(stepMatch(runId, event.nodeId, event.attemptNumber)) - .run(); - } + foldCumulative(tx, runId, event.cumulativeCostMicrocents, ts, event.nodeId); + // Same correction as `node:completed` (ADR-0076): the step row carries the node's TRUE cost, not the + // terminal's telescoping delta. The old `if (nodeCost > 0)` guard went with it — with attempt rows in + // play the delta is routinely zero on a node that really spent, so the guard would skip the write on + // exactly the failures whose cost matters most. `nodeSettledCost` returns 0 for a node that spent + // nothing, which writes a truthful 0 rather than leaving a stale value. + tx.update(stepExecutions) + .set({ costMicrocents: nodeSettledCost(tx, runId, event.nodeId), updatedAt: ts }) + .where(stepMatch(runId, event.nodeId, event.attemptNumber)) + .run(); + return; + } + case 'cost:attempt_settled': { + // ADR-0076: one settled provider attempt's REALIZED charge becomes a `run_costs` row in the SAME + // transaction as its event, which is what makes the ledger idempotent without a second uniqueness key — + // `UNIQUE(run_id, seq)` already bars a duplicate event, and the derived row cannot outlive it. + // + // **The row is a TELESCOPING delta off the event's cumulative — NOT the raw `costMicrocents`.** The + // raw value is this attempt's true charge and the event keeps carrying it (that is what a reader and + // the checkpoint fold sum); what goes in `run_costs` is `max(0, cumulative - currentRunCost)`, the same + // arithmetic every other money arm here uses. The reason is ordering, and it is not hypothetical: + // `#emitDurable` starts each `persistEvent` immediately and only serializes DELIVERY — its own comment + // says "persists stay concurrent". So a later event's write can commit FIRST. Writing the raw delta + // then DOUBLE-COUNTS: a sibling's `node:completed` lands with a cumulative that already includes this + // attempt, and this row adds it a second time. Telescoping cannot — every money event carries an + // ABSOLUTE cumulative, so the running total converges on the largest one seen regardless of the order + // the rows commit in, and `SUM(run_costs) == runs.total_cost_microcents` holds by construction. + // + // In the ordered case (the normal one) the two are identical: `cumulative - prev == costMicrocents` + // exactly. The divergence is a `fan_out`/out-of-order case, where per-ATTEMPT attribution degrades to + // an approximation — precisely the caveat `node:completed`'s per-node delta already carries and + // documents. The event remains the exact per-attempt record; this row is the money of record. + // + // **The two writes are a pair.** ADR-0076 property 3 says the terminal's fold telescopes to zero + // because the attempt rows advanced `sum(run_costs)`. What the fold actually subtracts is + // `runs.total_cost_microcents` (`currentRunCost`), and the two are equal only because every writer + // bumps `runs` alongside its `run_costs` insert. Dropping either write here would keep compiling, keep + // a test that checks only one table green, and corrupt every later delta. + const prev = currentRunCost(tx, runId); + const attemptCost = Math.max(0, event.cumulativeCostMicrocents - prev); + tx.insert(runCosts) + .values({ + id: deps.uuid(), + runId, + nodeId: event.nodeId, + // The REAL per-attempt tokens, unlike `foldCumulative`'s money-only addend — per-attempt token + // attribution is half of why this ledger exists. The row is written even when `attemptCost` is 0 + // (an unpriced or free attempt, or one whose money a concurrent sibling already banked), because + // the tokens are still real; `foldCumulative` skips a zero row only because it has none to carry. + // Only the COST column carries ADR-0070's `SUM(run_costs) == runs.total_cost_microcents` + // invariant; nothing sums these token columns, and `node:completed` remains the single writer of + // the run-level token totals (see below). + inputTokens: event.inputTokens, + outputTokens: event.outputTokens, + costMicrocents: attemptCost, + createdAt: ts, + // `modelId` is deliberately NOT written. It is an FK to `model_catalog` (a UUID) and `schema.ts` + // documents it as a dead column; `event.model` is a raw provider string this store cannot resolve. + // Per-model attribution on the run path comes from the durable EVENT, not from this row. + } satisfies NewRunCostRow) + .run(); + tx.update(runs) + // Money only. `runs.total_input_tokens` / `total_output_tokens` are NOT bumped here: `node:completed` + // already adds the turn's full `tokensUsed`, which is the sum across these attempts, so adding them + // here would double the run's token totals. + .set({ totalCostMicrocents: prev + attemptCost, updatedAt: ts }) + .where(eq(runs.id, runId)) + .run(); return; } case 'human_gate:paused': @@ -702,6 +784,12 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis // must never reach `runs.total_cost_microcents` or `run_costs`. Folding it into either would present an // upper bound as an invoice and break ADR-0070's `SUM(run_costs) == runs.total_cost_microcents`. Falling // through here is what keeps that true — asserted, not assumed, in this package's tests. + // + // Its realized twin `cost:attempt_settled` (ADR-0076) does the OPPOSITE and has its own arm above. Note + // what that means for anyone adding the next durable type: this `default` is not an exhaustiveness + // guard. `RunEvent` has no `assertNever` anywhere in the repo, so a new money event added without an + // arm lands here silently — a `run_events` row, no derived write, no compile error, no failing test. + // Decide deliberately which side of this line a new type belongs on. return; } }; From cd8c97175ebb2afeb7238272ff4574b81f11c177 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 20:05:23 +0300 Subject: [PATCH 08/26] docs(shared,decisions): fold the Sonnet review of #W15-1 step 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second pass, deliberately aimed where the Opus round did not look. Four findings, all valid. **The required-field defence was half-built.** `attemptNumber` and `priced` got reject fixtures because their required-ness is a DIVERGENCE worth arguing for. `nodeId` and `model` are required too — and had none, so loosening either to `.optional()` reddened nothing. That is the same silent one-way door those other two fixtures exist to hold shut, one field over, and the file's own convention already defends it: `budget:estimate_committed` carries `(no model)`, `budget:paused` carries `(missing/empty nodeId)`. Four fixtures added covering both the missing and the empty-string form; each break-verified with the mutation confirmed applied, and each reddens exactly its own named case. **ADR-0077 used B1/B2/B3 without ever defining them.** The Opus fold introduced the shorthand in the new Decision §5 — the most load-bearing part of the ADR — while §2 listed the three barriers in prose only, and B2 appeared nowhere. §2 now names them where they are introduced. **ADR-0076 §1's amendment marker was scoped too narrowly to do its job.** Two sentences past the marker, the same unmarked paragraph still asserts "the attempt boundary AWAITS that call" as fact, and Consequences > Negative still says "each awaited". A reader who reads past the parenthetical lands on the retracted mechanism stated plainly — which is exactly the failure the marker was added to prevent. Both now carry an inline pointer. Correction to `abc8a6e`'s message: the two guard lines are `engine.ts:1192` (`#makePreEgressHook`'s `return undefined`) and `:2111` (`#flushBudgetCommitments`'s `if (governor === undefined) return`); `:1190` and `:2109` are the enclosing signatures. The ADR body itself cites no line numbers, per the corpus convention, so only the commit message was off. Verified clean by this pass and left alone: ADR-0077's H1 matches its index row; every code citation in it checks out, including all four governor-conditional holes; every factual claim in the schema docblock is true against `persister.ts`, `session_costs`, ADR-0045 §5 and `parseStoredRunEvent`; the accept-boundary (`cumulative == cost`) is already covered by the valid fixture; and all four `RunEvent` consumers are true no-ops on the new type today — none renders a wrong number, which is what made deferring them to steps 4-5 safe rather than lucky. pnpm run ci EXIT=0. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- ...urable-per-attempt-realized-cost-ledger.md | 9 ++-- ...ses-the-conservative-commitment-barrier.md | 13 +++-- packages/shared/src/run-event.test.ts | 50 +++++++++++++++++++ 3 files changed, 65 insertions(+), 7 deletions(-) diff --git a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md index fc22d83b..f93a1d69 100644 --- a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md +++ b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md @@ -33,8 +33,10 @@ Three properties make it a ledger rather than another observation: 1. **Written before the next thing that can spend or mutate.** *(The MECHANISM below is **amended by [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)** — the guarantee stands, the "awaited emit" shape does not: `onAttempt` is synchronous, so there is no `await` to place there.)* It - goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call — before the - next tool side effect, before the next egress, and before the node/turn terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. + goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call *(← the + bare inline await is the part ADR-0077 supersedes: the write is STARTED at the settle instant and JOINED at + each barrier)* — before the next tool side effect, before the next egress, and before the node/turn + terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** **← AMENDED BY [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md); the @@ -104,7 +106,8 @@ A new durable event `type` is precisely the input ADR-0075 governs. Before it, a ### Negative -- **More durable writes on the hot path**, one per settled provider attempt, each awaited. That is the cost of the barrier and it is the same cost ADR-0074 §2 accepted for estimates; the volume is per-attempt, not per-token, so a long run adds rows in the hundreds, not the millions. Mitigation: the write shares the single `BEGIN IMMEDIATE` its event already takes, so it is not an additional transaction. +- **More durable writes on the hot path**, one per settled provider attempt, each awaited *(→ per ADR-0077: + each STARTED at its settle instant and JOINED at the next barrier, not awaited inline)*. That is the cost of the barrier and it is the same cost ADR-0074 §2 accepted for estimates; the volume is per-attempt, not per-token, so a long run adds rows in the hundreds, not the millions. Mitigation: the write shares the single `BEGIN IMMEDIATE` its event already takes, so it is not an additional transaction. - **A durability failure now fails a turn that would previously have completed.** Deliberate, and the same posture as §2: a run that cannot record what it spent must not keep spending. Mitigation: the failure is classified and names the attempt, rather than surfacing later as an unexplained cap refusal. - **An older binary cannot replay a log containing this event.** Inherited from ADR-0075 and stated there; the remedy is an upgrade, and every read-only surface still shows the run. - **`cost:updated` remains, and now has a durable sibling.** Two events describing one charge is a real risk of drift. Mitigation: the streamed one keeps its documented role as the live observation and the durable one is the record; the spec says which is authoritative, and the engine emits both from one place so they cannot disagree about the amount. diff --git a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md index bb564627..961c4c50 100644 --- a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md +++ b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md @@ -58,10 +58,15 @@ Concretely, five things: owner is not** — see §5. 2. **Join at three barriers, not two.** §2 joins at the pre-egress check and at the turn/node boundary. ADR-0076's guarantee names a third thing to precede — the next **tool side effect** — so the ledger's - barrier set is: before the next egress admission, **before tool dispatch**, and at the turn/node terminal. - The tool-dispatch barrier is the one this ADR adds to the §2 shape; without it the ledger would repeat - §2's coverage rather than extend it, and the duplicate-effect window ADR-0076 exists to narrow would stay - open on the path that mutates the world. + barrier set is, and these three names are used throughout this ADR: + + - **B1** — before the next egress admission (§2 has this one); + - **B2** — **before tool dispatch** (new here); + - **B3** — at the turn/node terminal (§2 has this one). + + **B2** is what this ADR adds to the §2 shape; without it the ledger would repeat §2's coverage rather than + extend it, and the duplicate-effect window ADR-0076 exists to narrow would stay open on the path that + mutates the world. 3. **Every barrier awaits AND observes.** Because `#emitDurable` is total, a barrier that only awaits proceeds on a run whose ledger write did not land. Each barrier must therefore also read the failure state and refuse to continue, and §2's retained-failure pattern (`#commitmentFailure`, re-thrown at the next barrier) is the diff --git a/packages/shared/src/run-event.test.ts b/packages/shared/src/run-event.test.ts index 1c29e5cc..881ba066 100644 --- a/packages/shared/src/run-event.test.ts +++ b/packages/shared/src/run-event.test.ts @@ -277,6 +277,56 @@ const reject: Record> = { // caught it. Every fixture below carries a complete payload for the same reason: exactly one thing wrong. priced: true, }, + // `nodeId` and `model` are required too, and the file's own convention is to defend a required field with + // a fixture — `budget:estimate_committed` has `(no model)`, `budget:paused` has `(missing/empty nodeId)`. + // Without these, loosening either to `.optional()` reddens nothing, which is the same silent one-way door + // the `attemptNumber`/`priced` fixtures exist to hold shut. + 'cost:attempt_settled (no nodeId)': { + type: 'cost:attempt_settled', + ...env, + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (empty nodeId)': { + type: 'cost:attempt_settled', + ...env, + nodeId: '', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (no model)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (empty model)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: '', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, 'cost:attempt_settled (attemptNumber 0)': { type: 'cost:attempt_settled', ...env, From 83f7990ba5853f736a836d2d39518ad89b52e830 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 20:12:25 +0300 Subject: [PATCH 09/26] =?UTF-8?q?feat(core,shared,docs):=20restore=20reali?= =?UTF-8?q?zed=20cost=20from=20the=20ledger=20=E2=80=94=20Math.max,=20not?= =?UTF-8?q?=20a=20sum?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 5 of 5. The staged plan said "fold it as a SUM of deltas, never a last-wins snapshot", borrowing ADR-0074 §2's reasoning. Half right, and the wrong half cost two rewrites — both caught by a test, not by review. **First attempt: sum the attempt deltas into the realized total.** Wrong — a node terminal's snapshot already contains the attempts it covers, and both write that field, so it double-counts. **Second attempt: sum into a SEPARATE accumulator, then max the two families.** This is what I wrote, and the motivating-scenario test refused to go green. The two families cover DIFFERENT money: a media node writes a cumulative snapshot and emits no attempt row at all. So on a run where media spend is the larger figure, `max(snapshots, sum(attempts))` silently drops every attempt made after the last node boundary — which is exactly the crash-mid-agent-loop case the ledger exists to fix. The test now pins it explicitly: node `a` is media at 1_000, node `b` then spends 800 and dies before its terminal; the two-family max restores 1_000 and a resume re-spends b's 800. **What is correct is the fold already two arms above it:** `Math.max` over every durable ABSOLUTE total — the node snapshots, `budget:paused.spentMicrocents`, and now `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so each is a true run-wide total and the largest is the engine's real one regardless of commit order. ADR-0074 §2 rejected `Math.max` for the CONSERVATIVE total, and that is why the plan said "sum". The reasoning does not carry: it was rejected only because a future deliberate release would DECREASE that total. Realized spend has no release and is monotonic, which is the one property that makes maxing safe. `sse-event-schema.md` and the schema docblock said "a sum within one family, a max across families" — my own sentence from step 2, now false. Both corrected, each naming BOTH wrong routes so the next reader does not re-derive them. Five tests. Break-verify with the mutation confirmed applied each time: last-wins instead of max, and dropping the arm, both redden; adding the node snapshot instead of maxing it reddens two. **One mutation reddens nothing, and the test says so rather than implying coverage.** Replacing the attempt arm's `Math.max` with `+= costMicrocents` leaves every test green, because on a well-formed log the two are equivalent — a node's attempts always carry a lower `sequenceNumber` than its own terminal, so the delta is summed exactly once before the snapshot maxes over it. `Math.max` is chosen for robustness and for matching the sibling arms, not because a fixture can tell them apart. The gap is named in the test, with the log shape that would close it (a terminal ordered before the attempts it covers) and the note that the engine does not produce one. pnpm run ci EXIT=0. Refs: ADR-0074, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- docs/reference/contracts/sse-event-schema.md | 2 +- packages/core/src/engine/checkpoint.test.ts | 101 +++++++++++++++++++ packages/core/src/engine/checkpoint.ts | 56 +++++++++- packages/shared/src/run-event.ts | 28 +++-- 4 files changed, 174 insertions(+), 13 deletions(-) diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index fc73ddb8..9d7ad07b 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -94,7 +94,7 @@ export type RunEvent = | `run:cancelled` | The run was cancelled. | `cumulativeCostMicrocents?` (the run-wide running total at cancellation — the durable fail-cost for a **paid media job** pending at the cancel, whose lone estimate addend is folded just before this terminal, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; the `run:completed` counterpart is `totalCostMicrocents`) | | `cost:attempt_settled` | **One settled provider attempt's REALIZED charge, made durable** ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) — the realized twin of `budget:estimate_committed`, emitted from the same callback and joined at the same barriers ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). It exists because `cost:updated` is streamed and never persisted, so before this event a crash mid-agent-loop discarded every charge since the last node boundary — and the resumed run spent it AGAIN, against a cap understated by exactly that amount. **RUN-ONLY** (`runId` always, never `sessionId`), unlike `cost:updated` and unlike its estimate twin: the session path already records the same per-attempt increment into `session_costs` synchronously (ADR-0070 + `#W15-4`), so it needs no arm here. **Scope:** it covers the provider attempts of an AGENT TURN. A media job's realized cost (ADR-0045 §5) does **not** emit one and does not need to — it is already durable through the `node:completed` / `node:failed` / `run:*` cumulative snapshots. And no cost event of any kind makes a TOOL EFFECT idempotent; that is a separate decision about a separate failure. | `nodeId`, `model`, `attemptNumber` (**required**, 1-based within-chain — optional on `cost:updated` only because that event predates this one and has historical rows; a per-attempt ledger row that cannot say which attempt it is, is not a ledger), `inputTokens`, `outputTokens`, `costMicrocents` (**this attempt's delta**, not a cumulative — `0` is reachable for an unpriced or genuinely free model), `cumulativeCostMicrocents` (the run-wide total **after** this attempt; a display snapshot, **not** the restore source — see the fold rule below), `priced` (**required**, same reason as `attemptNumber`: without it `costMicrocents: 0` with real tokens cannot distinguish *could not price* from *genuinely free*, which is the ambiguity a ledger exists to remove) | -> **Which cost event is authoritative for what — the three-way split, stated once.** `cost:updated` is the **live observation**: streamed, never persisted, last-wins, and it is what a surface renders while a run is moving. `cost:attempt_settled` is the **durable record of what was charged**: one row per settled attempt, and the only cost event a reader can trust after a restart. **Restoring** a run's realized total is neither of those alone — it is the **greater of** (the SUM of `cost:attempt_settled.costMicrocents`) and (the node-boundary snapshots on `node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`). A sum WITHIN the attempt family, a max ACROSS families: the sum is order-independent (which last-wins over `cumulativeCostMicrocents` is not — see the conservative twin's fold rule for the `fan_out` ordering argument), the snapshots still carry media spend that emits no attempt row, and neither source can over-count, so the larger is the better lower bound. Adding them would double-count, because a completed node's snapshot already includes its own attempts. The derived `run_costs` / `runs` rows this folds into are specified in [database-schema.md](../shared-core/database-schema.md), not here. +> **Which cost event is authoritative for what — the three-way split, stated once.** `cost:updated` is the **live observation**: streamed, never persisted, last-wins, and it is what a surface renders while a run is moving. `cost:attempt_settled` is the **durable record of what was charged**: one row per settled attempt, and the only cost event a reader can trust after a restart. **Restoring** a run's realized total is `Math.max` over every durable ABSOLUTE total in the log — the node-boundary snapshots (`node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`), `budget:paused.spentMicrocents`, and `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so each is a true run-wide total at that instant and the largest is the engine's real total whatever order the rows landed in. **Do not sum the attempt deltas**, by either obvious route: adding `costMicrocents` into the same total double-counts against a node snapshot that already includes those attempts, and summing them into a *separate* accumulator to max against the snapshots **under-counts** — the two sources cover different money, since a media node writes a snapshot and emits no attempt row at all, so every attempt made after the last node boundary vanishes whenever earlier media spend is the larger figure. (`Math.max` is right here and wrong for the conservative twin below for one reason only: realized spend is monotonic, while a conservative commitment can be deliberately released.) The derived `run_costs` / `runs` rows this folds into are specified in [database-schema.md](../shared-core/database-schema.md), not here. > **The money basis is FROZEN at submit time** ([ADR-0074](../../decisions/0074-durable-conservative-budget-commitments.md) §3). `units` is the authored billed volume the submission was priced on, and `acceptedCostMicrocents` is what the admission actually reserved. A resume restores the reservation from `acceptedCostMicrocents` with **no pricing lookup**, so neither a workflow edit nor a user-price/catalog change between submission and resume can move a commitment the provider has already accepted, or rewrite the job's historical cost. `acceptedCostMicrocents: 0` is meaningful and distinct from absent — it says the submission was priced and reserved nothing, which is what an unpriced model's allow-degrade path does. **`acceptedCostMicrocents` present ⇒ `units` required** (schema-enforced): a frozen cost on an unfrozen basis would restore the old reservation while re-deriving the volume, which is the drift §3 exists to prevent. The converse does NOT hold — `units` alone is legitimate, and is exactly what the approved-bypass path writes (it freezes the volume and omits the cost, because no pricing hook ran and `0` would freeze "priced at zero" for a job that was never priced). **Both absent means the row is LEGACY** (written before §3): resume must re-derive the volume from the workflow definition and re-price from today's catalog, which may under-reserve if the price has fallen, so with a cap configured the governor **fails closed**: new egress is **held** — awaited, not refused — until that job settles and its realized charge replaces the guess. Holding rather than throwing is deliberate: `budget_exceeded` is not in `RETRYABLE_ERROR_CODES` and `retry_on` cannot widen it, so a thrown refusal would kill a sibling node and abort the run, abandoning the very job it was waiting for. The hold is bounded by the job's own `deadlineAt` and is always broken by an abort, and it announces itself once through the governor's hold notice (routed to stderr by `relavium gate`) — that notice is the observability §3 requires, so a resume is never a silent stall. diff --git a/packages/core/src/engine/checkpoint.test.ts b/packages/core/src/engine/checkpoint.test.ts index 017f2a6c..73db6a02 100644 --- a/packages/core/src/engine/checkpoint.test.ts +++ b/packages/core/src/engine/checkpoint.test.ts @@ -616,6 +616,107 @@ describe('reconstructCheckpointState', () => { error: { code: 'tool_failed', message: 'boom', retryable: false }, }); }); + + describe('the realized-cost ledger (ADR-0076)', () => { + /** One settled attempt. `cost` is THIS attempt's delta; `cumulative` is the producer's snapshot after it. */ + const settled = (seq: number, nodeId: string, cost: number, cumulative: number): RunEvent => ({ + type: 'cost:attempt_settled', + ...base(seq), + nodeId, + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents: cost, + cumulativeCostMicrocents: cumulative, + priced: true, + }); + + /** A node terminal carrying the durable run-wide snapshot — the OTHER family feeding the same field. */ + const completedAt = (seq: number, nodeId: string, cumulative: number): RunEvent => ({ + type: 'node:completed', + ...base(seq), + nodeId, + output: null, + tokensUsed: { input: 0, output: 0 }, + durationMs: 1, + cumulativeCostMicrocents: cumulative, + }); + + it('restores from the attempt events, taking the highest absolute cumulative', () => { + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + settled(2, 'a', 600, 1_000), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('does NOT double-count a node snapshot against the attempts it already covers', () => { + // A completed node's snapshot ALREADY contains its attempts. This pins that the snapshot is MAXED in, + // never added: `acc.cumulativeCostMicrocents += event.cumulativeCostMicrocents` on the `node:completed` + // arm restores 2_000 for a run that spent 1_000, and this test reddens on it. + // + // **What it does NOT pin, stated rather than implied.** Replacing the attempt arm's `Math.max` with + // `+= event.costMicrocents` leaves this — and every other test here — green, because on a WELL-FORMED + // log the two folds are equivalent: a node's attempts always carry a lower `sequenceNumber` than its + // own terminal, so the delta is summed exactly once before the snapshot maxes over it. `Math.max` is + // chosen for robustness (it needs no assumption about ordering or about an event being folded once) and + // for matching the sibling arms, not because a fixture can tell them apart. No honest test can, so none + // is written; the mutation that WOULD close the gap is a log with a node terminal ordered before the + // attempts it covers, which the engine does not produce. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + settled(2, 'a', 600, 1_000), + completedAt(3, 'a', 1_000), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('is ORDER-INDEPENDENT, which a last-wins read of the event snapshot is not', () => { + // Concurrent events under a `fan_out` have no canonical `seq` order, so the LOWER seq can carry the + // HIGHER cumulative. Last-wins would restore 400 here and hand 600 back to the cap as headroom. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 600, 1_000), + settled(2, 'b', 400, 400), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('restores the HIGHER total when a crash landed mid-loop, past the last node terminal', () => { + // The motivating scenario, and the one that killed the max-of-two-families design. Node `a` is a MEDIA + // node: it writes a 1_000 snapshot and emits no attempt row at all. Node `b` then makes two paid calls + // and the process dies before its terminal. Summing attempts into a separate accumulator gives 800, + // `max(1_000, 800)` gives 1_000, and node b's 800 vanishes — a resume would spend it again, which is + // the exact bug the ledger exists to close. Maxing over ABSOLUTE cumulatives gives 1_800. + const state = reconstructCheckpointState([ + started, + completedAt(1, 'a', 1_000), + settled(2, 'b', 500, 1_500), + settled(3, 'b', 300, 1_800), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_800); + }); + + it('keeps realized and conservative money apart', () => { + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + { + type: 'budget:estimate_committed', + ...base(2), + nodeId: 'a', + model: 'claude-opus-4-8', + estimateMicrocents: 900, + cumulativeConservativeMicrocents: 900, + }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(400); + expect(state?.conservativeCostMicrocents).toBe(900); + }); + }); }); describe('createInMemoryCheckpointer', () => { diff --git a/packages/core/src/engine/checkpoint.ts b/packages/core/src/engine/checkpoint.ts index b7d1fa10..8c147340 100644 --- a/packages/core/src/engine/checkpoint.ts +++ b/packages/core/src/engine/checkpoint.ts @@ -94,10 +94,34 @@ export interface CheckpointState { /** Running token totals (summed from `node:completed`), restored so a resumed run's `run:completed` totals stay correct. */ readonly totalInputTokens: number; readonly totalOutputTokens: number; - /** The run-wide cumulative cost (integer micro-cents), restored on resume from the durable - * `node:completed.cumulativeCostMicrocents` snapshot and/or `budget:paused.spentMicrocents` — the higher - * wins (`Math.max`, order-independent). (`cost:updated` is also folded when present, but it is streamed, - * not persisted, so it never appears in a real durable log.) Keeps post-resume cost cumulative. */ + /** + * The run-wide realized cost (integer micro-cents), restored on resume so post-resume spend keeps + * accumulating against a cap that remembers what already went out. + * + * **The fold is `Math.max` over every durable ABSOLUTE total**, and there are now three carriers of one: + * the node-boundary snapshots (`node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`), + * `budget:paused.spentMicrocents`, and — since + * [ADR-0076](../../../../docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md) — + * `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so + * each is a true run-wide total at that instant and the largest is the engine's real total, whatever order + * the rows landed in. + * + * **Neither obvious alternative works, and one of them under-counts silently:** + * + * - SUMMING `cost:attempt_settled.costMicrocents` into this field double-counts, because a node terminal's + * snapshot already contains the attempts it covers. + * - Summing the attempts into a SEPARATE accumulator and taking the max of the two families under-counts. + * The families cover different money — a media node writes a snapshot and emits no attempt row at all — + * so whenever earlier media spend is the larger figure, every attempt made after the last node boundary + * is silently dropped. That is exactly the crash-mid-agent-loop case the ledger exists for. + * + * The `Math.max` reasoning {@link conservativeCostMicrocents} rejects does not carry here: it was rejected + * there only because a future deliberate release would DECREASE the conservative total. Realized spend has + * no release and is monotonic by construction. + * + * (`cost:updated` is also folded when present, but it is streamed, not persisted, so it never appears in a + * real durable log.) + */ readonly cumulativeCostMicrocents: number; /** * The run-wide durable **conservative** total (integer micro-cents) — money a provider may already have billed @@ -350,6 +374,30 @@ export function reconstructCheckpointState( // §1's future release. See the field's doc for the full comparison.) acc.conservativeCostMicrocents += event.estimateMicrocents; } + if (event.type === 'cost:attempt_settled') { + // `Math.max` over the event's ABSOLUTE cumulative — the same fold `node:completed` / `node:failed` use + // two arms above, and deliberately NOT the sum-of-deltas its conservative sibling uses. Three things + // make that the right choice, and two plausible alternatives are wrong: + // + // - SUMMING `costMicrocents` into this field DOUBLE-COUNTS: a node terminal's snapshot already contains + // the attempts it covers, and both write here. + // - Summing into a SEPARATE accumulator and taking the max at the end UNDER-counts, which is subtler. + // The two families cover different money: a media node writes a snapshot and emits no attempt row at + // all, so `max(snapshots, sum(attempts))` silently drops every attempt made after the last boundary + // whenever earlier media spend is the larger number. + // - `Math.max` over absolute totals has neither failure. The producer reads this counter AFTER folding + // this attempt into it, so the value is a true run-wide total at that instant, exactly like a node + // boundary's — and the largest one seen is the engine's real total no matter what order the rows land + // in. + // + // ADR-0074 §2 rejected `Math.max` for the CONSERVATIVE total, and that reasoning does not carry: it was + // rejected only because a future deliberate release would DECREASE that total. Realized spend has no + // release and is monotonic by construction. + acc.cumulativeCostMicrocents = Math.max( + acc.cumulativeCostMicrocents, + event.cumulativeCostMicrocents, + ); + } applyRunEvent(acc, event); applyNodeEvent(acc, event); applyMediaJobEvent(acc, event); diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index c5d95766..f516186a 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -654,15 +654,27 @@ export const CostAttemptSettledEventSchema = z.object({ */ costMicrocents: nonNegativeInt, /** - * The run-wide realized running total after this attempt — a producer-side snapshot for DISPLAY and - * cross-checking. **Not the restore source**, for the same reason ADR-0074 §2 gave for its twin: under a - * `fan_out` concurrent events have no canonical `seq` order, so a last-wins read of this field can restore a - * LOWER total and hand already-spent money back to the cap as headroom. Reconstruction sums - * {@link costMicrocents} instead, and takes the greater of that sum and the durable node-boundary snapshots - * — a sum within one family, a max across families, neither of which can over-count. + * The run-wide realized running total after this attempt — an ABSOLUTE total, read immediately after this + * attempt was folded into the counter, and **this is the field reconstruction restores from**, via + * `Math.max` against every other absolute total in the log (the node-boundary snapshots and + * `budget:paused.spentMicrocents`). * - * (`cost:updated` gets away with a last-wins cumulative only because it is streamed and never persisted, so - * it is not the precedent for a durable event.) + * `Math.max`, never LAST-WINS: under a `fan_out` concurrent events have no canonical `seq` order, so the + * lower `seq` can carry the higher total and a last-wins read would hand already-spent money back to the cap + * as headroom. Maxing over absolutes is order-independent because realized spend is monotonic — which is + * exactly the property `budget:estimate_committed` lacks, and the whole reason its fold sums signed deltas + * instead. + * + * **Do not restore by summing {@link costMicrocents}.** Into this same total it double-counts, because a + * node terminal's snapshot already contains the attempts it covers. Into a separate accumulator maxed + * against the snapshots it UNDER-counts, which is the subtler failure: the two sources cover different + * money — a media node writes a snapshot and emits no attempt row at all — so every attempt after the last + * node boundary disappears whenever earlier media spend is the larger figure, which is precisely the + * crash-mid-agent-loop case this event exists for. `costMicrocents` remains the exact per-attempt record for + * accounting and for the derived `run_costs` row; it is not the restore path. + * + * (`cost:updated` carries the same figure but is streamed and never persisted, so it is not a restore source + * at all.) */ cumulativeCostMicrocents: nonNegativeInt, /** From 351f3afc38590ef10f6d04602c8f1cd39b8c0723 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 20:20:02 +0300 Subject: [PATCH 10/26] =?UTF-8?q?feat(core):=20wire=20the=20money-durabili?= =?UTF-8?q?ty=20barrier=20=E2=80=94=20engine-owned,=20three=20joins?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 4 of 5, implementing ADR-0077 rather than ADR-0076 §1's withdrawn "awaited emit at the attempt boundary", which cannot be written: `onAttempt` is `(record) => void`. **New `money-durability.ts`, owned by the engine and constructed UNCONDITIONALLY.** Every signpost pointed at `BudgetGovernor` — ADR-0077 §1's own "exactly as §2's `#commitmentsInFlight`", and `node-executor.ts`'s "add it to the governor's emit type instead" — and following any of them would have been silently wrong: the governor exists only `if (params.plan.budget !== undefined)`, and an unbudgeted run spends real money. It copies §2's shape field for field (chained in-flight promise, pending guard, retained failure, sticky broken flag, `Promise.resolve().then(...)` so a synchronously-throwing store cannot brick the chain) and adds the thing §4 requires: ONE `join()` that fronts both money chains, so "await the wrong one" is not expressible. **Three barrier holes closed, all of which read as correct code:** - `#flushBudgetCommitments`'s `if (governor === undefined) return` — B3 was a no-op on every unbudgeted run. It is now the single join. - the `preAttempt` wrapper gated on `preEgress === undefined` — B1 did not exist without a governor. Now installed when either the hook or the money port is present. - `budgetApproved ? undefined : #makePreEgressHook()` — B1 vanished on an approved re-dispatch. The money port is threaded unconditionally, because an approved node is the last place to stop recording what the user just authorised. **B2 is new**: `await params.money?.join()` immediately before `dispatchToolCalls`. This is what makes the ledger extend §2's guarantee rather than repeat it — realized spend durable before the run MUTATES THE WORLD, not merely before it spends again. One join per tool turn, before the loop. The emit sits in `onAttempt` strictly AFTER `params.emit({type:'cost:updated'})`, and the order is load-bearing: `#nodeEmit`'s `cost:updated` arm advances `#cumulativeCostMicrocents`, which the engine then stamps onto the ledger draft. Recording first would stamp a stale total, `refineCostAttemptSettled` would reject it at the producer gate, and that gate runs in `#bus.next` — OUTSIDE `#emitDurable`'s try — so `#emitDurable` would REJECT in the one place the whole design assumes it cannot. Run-only is now a runtime fact, not a comment: `money` is an optional `AgentTurnParams` field modelled on `preEgress`, `AgentSession` never sets it, and `agent-runner.ts` reads it from the ctx with no `deps` fallback — a host wiring a runner directly has no run to record against. Eight unit tests, all green: serialization, retained-failure-thrown-once, the sticky flag, secret-free message with the cause on `cause`, the sync-throw brick, the conservative join, a conservative failure surfacing through the same join, and the zero-egress no-op. **The wiring is NOT pinned, and I am naming that rather than implying it is.** Deleting all three joins leaves 1096 of 1097 core tests green; the one failure is ADR-0074 §2's existing conservative-ordering e2e, which sees B3 only because B3 now fronts its flush. So nothing today pins B1, B2, or the realized ledger's ordering at all. Three tests close it, and they belong on the `m2-e2e-harness.e2e.test.ts` delaying-store harness that already exists for §2: (1) `cost:attempt_settled` persists before the next attempt's egress, (2) before the tool registry's dispatch — asserted on a dispatch spy, since that is B2's whole point, (3) before `node:completed`. Plus ADR-0077's required regression: a rejecting `persistEvent` must leave the dispatch spy UNCALLED — the mutation that matters most, because `#emitDurable` resolves on a store fault, so an await-only barrier passes every ordering test above while observing nothing. pnpm run ci EXIT=0. Refs: ADR-0074, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- packages/core/src/engine/agent-runner.ts | 4 + packages/core/src/engine/agent-turn.ts | 48 ++++- packages/core/src/engine/engine.ts | 52 +++++- .../core/src/engine/money-durability.test.ts | 144 +++++++++++++++ packages/core/src/engine/money-durability.ts | 168 ++++++++++++++++++ packages/core/src/engine/node-executor.ts | 9 + 6 files changed, 416 insertions(+), 9 deletions(-) create mode 100644 packages/core/src/engine/money-durability.test.ts create mode 100644 packages/core/src/engine/money-durability.ts diff --git a/packages/core/src/engine/agent-runner.ts b/packages/core/src/engine/agent-runner.ts index 99a3e992..2fc2be8c 100644 --- a/packages/core/src/engine/agent-runner.ts +++ b/packages/core/src/engine/agent-runner.ts @@ -396,6 +396,10 @@ async function executeAgent( dispatchContext, limits: deps.limits ?? DEFAULT_AGENT_TURN_LIMITS, ...(preEgress === undefined ? {} : { preEgress }), + // Straight from the ctx, with no `deps` fallback — the ledger belongs to a RUN and only the run loop can + // supply it. A host wiring a runner directly gets no ledger, which is correct: there is no run to + // record against (ADR-0076 / ADR-0077). + ...(ctx.money === undefined ? {} : { money: ctx.money }), ...(deps.resolvePrice === undefined ? {} : { resolvePrice: deps.resolvePrice }), // user-pricing overlay (S10) // Media cost governance (1.AF/D17): forward the node's requested output modalities + a per-modality // unit estimate so the budget governor prices a media-output turn pre-egress. Both omitted for a diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index e9c78db1..5175db2f 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -62,6 +62,7 @@ import { CommitmentDurabilityError, type BudgetAdmission, } from './budget-governor.js'; +import type { TurnMoneyPort } from './money-durability.js'; import type { NodeStreamEvent } from './node-executor.js'; /** @@ -163,6 +164,16 @@ export interface AgentTurnParams { readonly limits: AgentTurnLimits; /** Pre-egress budget hook (default no-op; 1.AC fills it). */ readonly preEgress?: PreEgressHook; + /** + * The run's money-durability port (ADR-0076 / ADR-0077) — RUN-PATH ONLY, and that is what makes + * `cost:attempt_settled` a run-only event as a runtime fact rather than a comment: `AgentSession` never sets + * this, so nothing is recorded on the session path and there is nothing for a sink to drop. (The session's + * realized spend is already recorded synchronously into `session_costs`, ADR-0070 + `#W15-4`.) + * + * Optional exactly like {@link preEgress}, and threaded the same way, because this is the boundary the + * session shares. + */ + readonly money?: TurnMoneyPort; /** * The non-text output the node requested (1.AF/D17) — forwarded to {@link PreEgressHook} so the budget * governor knows this is a media-output turn. The AgentRunner lowers it from the node's `output_modalities`. @@ -654,6 +665,13 @@ async function dispatchToolUseTurn( false, ); } + // **Barrier B2 (ADR-0077)** — the one this ADR adds beyond ADR-0074 §2's pair, and the reason the ledger + // extends the guarantee rather than repeating it: realized spend must be durable before the run MUTATES THE + // WORLD, not merely before it spends again. Before the loop, not per call — one join per tool turn. + // + // It awaits AND observes: `join()` throws the retained failure, which propagates as a turn failure. Awaiting + // alone would not be a barrier, since `#emitDurable` absorbs a store fault and resolves. + await params.money?.join(); // A reached `tool_use` stop always followed a successful (non-skipped) attempt, so `nonSkippedAttempts >= 1`. const dispatched = await dispatchToolCalls(toolCalls, params, activeModel, nonSkippedAttempts); let next = corrections; @@ -844,6 +862,24 @@ async function driveAgentTurn( // between "unpriced" and "genuinely free" — and a free-LOOKING row in the /cost breakdown would be a lie. priced: record.cost !== undefined, }); + // ADR-0076's durable ledger row, STARTED here and joined at the next barrier (ADR-0077) — this callback + // cannot await, which is the whole reason the mechanism is a chain plus barriers rather than an inline + // await. + // + // **Strictly AFTER `params.emit` above, and the order is load-bearing.** The engine advances its run-wide + // `#cumulativeCostMicrocents` inside `#nodeEmit`'s `cost:updated` arm, and stamps that counter onto this + // draft. Recording FIRST would stamp a stale total, which `refineCostAttemptSettled` rejects at the + // producer gate — and that gate runs in `#bus.next`, OUTSIDE `#emitDurable`'s try, so the wrong order does + // not degrade quietly: it makes `#emitDurable` REJECT in the one place the design assumes it cannot. + params.money?.record({ + nodeId: params.nodeId, + model: record.model, + attemptNumber: nonSkippedAttempts, + inputTokens: record.usage.inputTokens, + outputTokens: record.usage.outputTokens, + costMicrocents: record.cost?.costMicrocents ?? 0, + priced: record.cost !== undefined, + }); }; const chainCapabilities: ChainCapabilities = @@ -873,7 +909,11 @@ async function driveAgentTurn( // maxTokens }` — `provider` is THIS attempt's routing provider (review M2) — so wrap the hook to also carry // the turn-static media estimate (1.AF/D17); otherwise the failover-attempt check would silently drop the // media addend (ADR-0044 §3). `...info` forwards `provider` to the governor's endpoint estimate unchanged. - ...(preEgress === undefined + // Installed when EITHER a budget hook or the money port is present. Gating it on `preEgress` alone — as it + // was — is one of the three barrier holes ADR-0077 §5 names: `#makePreEgressHook()` returns `undefined` + // without a governor, and `budgetApproved` drops it even WITH one, so an unbudgeted run (or an approved + // re-dispatch) would have got no B1 at all while still spending real money. + ...(preEgress === undefined && params.money === undefined ? {} : { preAttempt: async (info: { @@ -881,6 +921,12 @@ async function driveAgentTurn( readonly provider: ProviderId; readonly maxTokens?: number; }) => { + // **Barrier B1 (ADR-0077)** — before the next egress admission, and before the governor call, so a + // run whose ledger write did not land admits nothing further. It awaits AND observes: `join()` + // throws the retained failure rather than returning, which is the only way a caller here can see + // it (`#emitDurable` absorbs a store fault and resolves). + await params.money?.join(); + if (preEgress === undefined) return; // This is the only admitting boundary. Check cancellation on BOTH sides of the awaited governor call: // a cancellation landing while warning durability/admission is pending must not reach key resolution // or provider egress, and any just-acquired lease is released before the cancellation propagates. diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index bed05d9d..2b4336df 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -68,6 +68,7 @@ import { type BudgetAdmission, } from './budget-governor.js'; import type { CheckpointPendingMediaJob, CheckpointState } from './checkpoint.js'; +import { MoneyDurability, isLedgerDurabilityError } from './money-durability.js'; import type { AbortControllerLike, ExecutionHost } from './execution-host.js'; import type { GateRequest, @@ -347,6 +348,8 @@ class RunExecution { #runTimeoutDisarm: (() => void) | undefined; /** The pre-egress budget governor, when a workflow `budget` is configured (ADR-0028, 1.AC). */ readonly #budgetGovernor: BudgetGovernor | undefined; + /** The money-durability barrier for BOTH chains — always present, cap or no cap (ADR-0077 §5). */ + readonly #money: MoneyDurability; /** Vertices whose budget gate was APPROVED — their next re-dispatch (and all its node-retry attempts) skips * the pre-egress check so the deferred LLM call actually issues (H3). Consumed once per dispatch in * `#dispatch` and cleared on `#settle`. */ @@ -409,6 +412,26 @@ class RunExecution { this.#maskedInputs = maskInputs(params.inputs, secretNames); this.#maxTokensEstimate = params.maxTokensEstimate ?? DEFAULT_MAX_TOKENS_ESTIMATE; this.#resolvePrice = params.resolvePrice; + // UNCONDITIONAL, unlike the governor below (ADR-0077 §5). The conservative half is inherently + // budget-scoped — no cap, nothing to reserve — but a run without a budget still spends real money, so a + // ledger that only existed alongside a governor would silently skip every unbudgeted run. It fronts the + // join for BOTH chains, so `flushConservative` is wired to the governor once that exists. + this.#money = new MoneyDurability({ + emit: (draft) => + this.#emitDurable({ + ...draft, + type: 'cost:attempt_settled', + runId: this.runId, + // Stamped HERE because the engine owns the run-wide counter — `#nodeEmit`'s `cost:updated` arm has + // already folded this attempt into it (the turn core records strictly after that emit), so reading + // it now satisfies `refineCostAttemptSettled`'s "cumulative already includes this charge" by + // construction, the same way the governor reads its counter after `+=`. + cumulativeCostMicrocents: this.#cumulativeCostMicrocents, + }), + ...(params.plan.budget === undefined + ? {} + : { flushConservative: async () => this.#budgetGovernor?.flushCommitments() }), + }); if (params.plan.budget !== undefined) { this.#budgetGovernor = new BudgetGovernor({ budget: params.plan.budget, @@ -1225,6 +1248,10 @@ class RunExecution { signal: this.#abort.signal, attemptNumber, ...(preEgress === undefined ? {} : { preEgress }), + // Unconditional, unlike `preEgress` above — which `budgetApproved` deliberately drops for an approved + // re-dispatch. The ledger must not be dropped with it: an approved node is the one the user just + // authorised MORE money on, so it is the last place to stop recording what that money was. + money: this.#money.turnPort(), }; // After the executor completes, an `output` node with `save_to` writes its produced media to the // host (1.AF/D16). A write failure FAILS the node (→ run:failed) — save_to is a real deliverable. @@ -2107,23 +2134,32 @@ class RunExecution { * reservation, always a loud failure. */ async #flushBudgetCommitments(nodeId: string): Promise { - const governor = this.#budgetGovernor; - if (governor === undefined) return; + // **Barrier B3 (ADR-0077)** — and it is now the SINGLE join for both money chains. The old + // `if (governor === undefined) return;` is gone: it was one of §5's three barrier holes, because a run + // without a budget has no conservative commitments but does have a realized ledger, and returning early + // left that ledger started and never joined — exactly the fire-and-forget state ADR-0076 exists to remove. + // `MoneyDurability.join()` fronts both, so there is no supported way to await half the money. try { - await governor.flushCommitments(); + await this.#money.join(); } catch (error) { - // Attribute it to the node whose commitment actually failed, not to whichever node reached this barrier + // Attribute it to the node whose write actually failed, not to whichever node reached this barrier // first — under a `fan_out` both branches await the same chain link, so the first to flush may have made no // commitment at all. Falls back to this node when the error carries no owner. - const owner = error instanceof CommitmentDurabilityError ? (error.nodeId ?? nodeId) : nodeId; + const ledger = isLedgerDurabilityError(error); + const owner = + error instanceof CommitmentDurabilityError || ledger ? (error.nodeId ?? nodeId) : nodeId; this.#failure ??= { nodeId: owner, error: { code: 'internal', // The cause is deliberately NOT in the message: a durable-write failure can carry a filesystem path, and - // a user-facing `run:failed` message must not. It survives on `CommitmentDurabilityError.cause` for a - // host that narrows on the class — which is the only carrier, since this path has no store to log it. - message: 'a conservative budget commitment could not be made durable', + // a user-facing `run:failed` message must not. It survives on the error's `cause` for a host that + // narrows on the class — which is the only carrier, since this path has no store to log it. The two + // messages are distinct because the remedies differ: an estimate that could not be recorded leaves the + // cap conservative, while a realized charge that could not be recorded leaves it UNDERSTATED. + message: ledger + ? 'a realized provider charge could not be made durable' + : 'a conservative budget commitment could not be made durable', retryable: false, }, }; diff --git a/packages/core/src/engine/money-durability.test.ts b/packages/core/src/engine/money-durability.test.ts new file mode 100644 index 00000000..b03d27eb --- /dev/null +++ b/packages/core/src/engine/money-durability.test.ts @@ -0,0 +1,144 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { + MoneyDurability, + isLedgerDurabilityError, + type SettledAttemptDraft, +} from './money-durability.js'; + +const draft = (nodeId = 'n1', costMicrocents = 400): SettledAttemptDraft => ({ + nodeId, + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents, + priced: true, +}); + +describe('MoneyDurability', () => { + it('SERIALIZES writes chained in the same tick', async () => { + // Chaining rather than a set is what stops two attempts settling in one tick from interleaving their + // persists — the same reason ADR-0074 §2 chains its estimates. + const order: string[] = []; + let releaseFirst: (() => void) | undefined; + const money = new MoneyDurability({ + emit: async (d) => { + order.push(`start:${d.nodeId}`); + if (d.nodeId === 'a') { + await new Promise((resolve) => { + releaseFirst = resolve; + }); + } + order.push(`end:${d.nodeId}`); + }, + }); + + money.record(draft('a')); + money.record(draft('b')); + await vi.waitFor(() => { + expect(releaseFirst).toBeDefined(); + }); + expect(order).toEqual(['start:a']); // b has not started while a is in flight + releaseFirst?.(); + await money.join(); + expect(order).toEqual(['start:a', 'end:a', 'start:b', 'end:b']); + }); + + it('re-throws a failed write at the barrier, ONCE, and stays sticky', async () => { + // Nobody awaits the write at its call site, so the rejection has to be retained and surfaced here or it + // is unhandled. Surfaced once — a later join must not re-report the same broken write as new — while the + // sticky flag keeps the barrier meaningful for the rest of the run. + const money = new MoneyDurability({ + emit: () => Promise.reject(new Error('disk full')), + }); + money.record(draft('a')); + + await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); + expect(money.durabilityBroken).toBe(true); + await expect(money.join()).resolves.toBeUndefined(); // not re-reported + expect(money.durabilityBroken).toBe(true); // but still broken + }); + + it('names the owning node and keeps the cause OFF the message', async () => { + // A durable-write failure can carry a filesystem path, and this message reaches a user-facing + // `run:failed`. The cause rides `cause` for a host that narrows on the class. + const cause = new Error('ENOENT: /Users/someone/.relavium/history.db'); + const money = new MoneyDurability({ emit: () => Promise.reject(cause) }); + money.record(draft('the-node')); + + await money.join().then( + () => expect.unreachable('join must throw'), + (error: unknown) => { + if (!isLedgerDurabilityError(error)) expect.unreachable('wrong error class'); + expect(error.nodeId).toBe('the-node'); + expect(error.cause).toBe(cause); + expect(error.message).not.toContain('.relavium'); + }, + ); + }); + + it('survives a SYNCHRONOUSLY throwing sink instead of bricking the chain', async () => { + // A `better-sqlite3`-backed store throws synchronously. A bare `emit(draft)` inside the `.then` would + // escape before `.catch`/`.finally` attach: the pending count leaks at >= 1 forever and the chain is left + // permanently rejected, so every later join throws with no way to clear it. + const money = new MoneyDurability({ + emit: () => { + throw new Error('sync boom'); + }, + }); + money.record(draft('a')); + await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); + + // The chain is still usable: a later successful write joins cleanly rather than re-throwing forever. + const seen: string[] = []; + const healthy = new MoneyDurability({ + emit: (d) => { + seen.push(d.nodeId); + }, + }); + healthy.record(draft('b')); + await expect(healthy.join()).resolves.toBeUndefined(); + expect(seen).toEqual(['b']); + }); + + it('joins the CONSERVATIVE chain too — one join, not two (ADR-0077 §4)', async () => { + const flushConservative = vi.fn(async () => {}); + const money = new MoneyDurability({ emit: () => {}, flushConservative }); + + await money.join(); + expect(flushConservative).toHaveBeenCalledTimes(1); + }); + + it('surfaces a CONSERVATIVE failure through the same join', async () => { + // The estimate half throws its own retained failure from `flushCommitments`. A caller awaiting the single + // join must see it — otherwise "one join" would silently swallow half the money's failures. + const money = new MoneyDurability({ + emit: () => {}, + flushConservative: () => Promise.reject(new Error('commitment write failed')), + }); + money.record(draft('a')); + + await expect(money.join()).rejects.toThrow('commitment write failed'); + }); + + it('is a no-op join when nothing was ever recorded — an unbudgeted, zero-egress run', async () => { + const money = new MoneyDurability({ emit: () => expect.unreachable('nothing to emit') }); + await expect(money.join()).resolves.toBeUndefined(); + expect(money.durabilityBroken).toBe(false); + }); + + it('exposes a turn port that records and joins without leaking the class', async () => { + const seen: SettledAttemptDraft[] = []; + const money = new MoneyDurability({ + emit: (d) => { + seen.push(d); + }, + }); + // Destructured, because the turn core passes `params.money` around by value. + const { record, join } = money.turnPort(); + record(draft('a', 700)); + await join(); + expect(seen).toEqual([expect.objectContaining({ nodeId: 'a', costMicrocents: 700 })]); + }); +}); diff --git a/packages/core/src/engine/money-durability.ts b/packages/core/src/engine/money-durability.ts new file mode 100644 index 00000000..2a7f6ebd --- /dev/null +++ b/packages/core/src/engine/money-durability.ts @@ -0,0 +1,168 @@ +/** + * The run's money-durability barrier — the single place a caller waits for BOTH kinds of money write + * ([ADR-0077](../../../../docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). + * + * Two events record money and both are emitted from the SAME synchronous `onAttempt` callback, a few lines + * apart in `agent-turn.ts`: `budget:estimate_committed` (a conservative ESTIMATE, ADR-0074 §2) and + * `cost:attempt_settled` (the REALIZED charge, ADR-0076). Neither can be awaited at its emit site — the seam's + * observer is `(record: AttemptRecord) => void` — so each STARTS its durable write there and something joins it + * later. This owns the realized chain and fronts the join for both. + * + * **Why it is not part of `BudgetGovernor`, which is where every signpost points.** The governor owns the + * conservative chain and `node-executor.ts`'s comment literally says "add it to the governor's emit type + * instead". But the engine constructs a governor only when the workflow declares a `budget`, and the realized + * ledger is not optional: an unbudgeted run spends real money and must still record it. Hosting it there would + * have silently skipped every unbudgeted run while passing any test written against a budgeted fixture. + * + * **One join, not two** (ADR-0077 §4). `join()` awaits the realized chain AND the conservative one, and + * reports whichever failure it finds. There is deliberately no public way to await half the money. + */ + +/** A realized-cost ledger write that did not reach the store. Carries the owning node for attribution. */ +export class LedgerDurabilityError extends Error { + readonly nodeId: string | undefined; + + constructor(cause: unknown, nodeId?: string) { + // Secret-free by construction: the cause can carry a filesystem path, so it rides `cause` and never the + // message — the same posture `CommitmentDurabilityError` takes for its estimate twin. + super('a realized-cost ledger write could not be made durable', { cause }); + this.name = 'LedgerDurabilityError'; + this.nodeId = nodeId; + } +} + +export function isLedgerDurabilityError(error: unknown): error is LedgerDurabilityError { + return error instanceof LedgerDurabilityError; +} + +/** The per-attempt ledger draft, minus the run-wide total the ENGINE stamps (it owns that counter). */ +export interface SettledAttemptDraft { + readonly nodeId: string; + readonly model: string; + readonly attemptNumber: number; + readonly inputTokens: number; + readonly outputTokens: number; + readonly costMicrocents: number; + readonly priced: boolean; +} + +export interface MoneyDurabilityOptions { + /** + * Start the durable write for one settled attempt. The engine wires this to `#emitDurable`, stamping the + * run-wide cumulative it alone owns. **May throw synchronously** — a `better-sqlite3` store does — which is + * why {@link MoneyDurability.record} never calls it bare. + */ + readonly emit: (draft: SettledAttemptDraft) => Promise | void; + /** + * Join the CONSERVATIVE chain, when a budget governor exists. Wired to `BudgetGovernor.flushCommitments`, + * which throws its own retained failure. Absent on an unbudgeted run — the realized half still applies. + */ + readonly flushConservative?: () => Promise; +} + +/** + * The port `agent-turn.ts` sees. Narrower than the class on purpose: the turn core may START a write and JOIN + * the barrier, and has no business inspecting or clearing durability state. + */ +export interface TurnMoneyPort { + readonly record: (draft: SettledAttemptDraft) => void; + readonly join: () => Promise; +} + +export class MoneyDurability { + readonly #options: MoneyDurabilityOptions; + + /** + * The realized chain. Chaining (rather than a set) serializes the writes, so two attempts settling in the + * same tick cannot interleave their persists — the same reason ADR-0074 §2 chains its estimates. + */ + #inFlight: Promise = Promise.resolve(); + + /** + * How many ledger writes are outstanding, so the barrier costs NOTHING when there is nothing to wait for. + * Awaiting an already-resolved promise still burns a microtask, and the barriers sit on the hot path; worse + * than the cost, an unconditional await changes observable interleaving that existing tests legitimately + * pin. Copied from `#pendingCommitments` for exactly those reasons. + */ + #pending = 0; + + /** + * A ledger write that FAILED, retained until someone joins the barrier. + * + * Nobody awaits the write at its call site, so without this the rejection would be unhandled. It is + * surfaced ONCE — a later join must not re-report the same broken write as if it were new — while + * {@link #broken} stays set, so the barrier is still entered and a subsequent failure is still reported. + */ + #failure: LedgerDurabilityError | undefined; + + /** Sticky: a run whose ledger has ever failed keeps paying the barrier for the rest of the process. */ + #broken = false; + + constructor(options: MoneyDurabilityOptions) { + this.#options = options; + } + + /** Whether a ledger write has ever failed on this run — the state a caller checks alongside the await. */ + get durabilityBroken(): boolean { + return this.#broken; + } + + /** + * START one settled attempt's durable write. Synchronous by necessity (the caller is a chain observer that + * cannot await) and never awaited here, so a rejection is captured and re-thrown at the next barrier. + */ + record(draft: SettledAttemptDraft): void { + this.#pending += 1; + this.#inFlight = this.#inFlight.then(() => + // `Promise.resolve().then(…)`, NOT a bare `emit(draft)`. A sink that throws SYNCHRONOUSLY — which a + // `better-sqlite3`-backed store does — would escape before `.catch`/`.finally` attach: no typed error, + // `#pending` leaked at >= 1 forever, and `#inFlight` left permanently REJECTED so every later join + // throws with no way to clear it. The governor's estimate twin documents the same brick at length. + // The inner chain therefore ALWAYS resolves, which is why there is no onRejected arm here. + Promise.resolve() + .then(() => this.#options.emit(draft)) + .catch((error: unknown) => { + // Keep the FIRST failure: it is the one that broke durability, and later writes may well fail for + // the same reason. + this.#failure ??= new LedgerDurabilityError(error, draft.nodeId); + this.#broken = true; + }) + .finally(() => { + this.#pending -= 1; + }), + ); + } + + /** + * The barrier — ADR-0077's B1 / B2 / B3, all three of them this one call. + * + * Awaits the realized chain and the conservative one, then throws the first retained failure. Both halves + * matter: awaiting alone is NOT a barrier, because the engine's `#emitDurable` is total for store faults — + * it absorbs a `persistEvent` rejection into the run's failure state and RESOLVES — so a caller that only + * awaits proceeds on a run whose write did not land. The throw is how a caller in the turn core, which has + * no access to the engine's own failure state, observes it. + */ + async join(): Promise { + if (this.#pending > 0 || this.#failure !== undefined) { + await this.#inFlight; + } + // The conservative half is joined unconditionally when a governor exists — its own barrier is cheap when + // nothing is outstanding, and skipping it here is how "await the wrong one" would creep back in. + await this.#options.flushConservative?.(); + const failure = this.#failure; + if (failure !== undefined) { + this.#failure = undefined; + throw failure; + } + } + + /** The narrow port handed to the turn core. Bound so a destructured `record` still works. */ + turnPort(): TurnMoneyPort { + return { + record: (draft) => { + this.record(draft); + }, + join: () => this.join(), + }; + } +} diff --git a/packages/core/src/engine/node-executor.ts b/packages/core/src/engine/node-executor.ts index 843354b1..d18cea34 100644 --- a/packages/core/src/engine/node-executor.ts +++ b/packages/core/src/engine/node-executor.ts @@ -191,6 +191,15 @@ export interface NodeExecContext { * attempt is gated before egress. */ readonly preEgress?: import('./agent-turn.js').PreEgressHook; + /** + * The run's money-durability port (ADR-0076 / ADR-0077) — the realized-cost ledger's START hook and the + * single barrier the turn core joins at. + * + * Optional on this seam so a stub executor and the session path stay unchanged, but the run loop supplies it + * UNCONDITIONALLY — unlike {@link preEgress}, which is budget-scoped and is deliberately dropped for an + * approved re-dispatch. A run with no `budget` still spends real money. + */ + readonly money?: import('./money-durability.js').TurnMoneyPort; } /** The injected per-vertex executor. 1.O (`AgentRunner`) and 1.P (node handlers) implement it. */ From 50c60bd737192616375a5a2511d3d0992ea1e664 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 20:52:19 +0300 Subject: [PATCH 11/26] =?UTF-8?q?test(core):=20four=20ledger-barrier=20e2e?= =?UTF-8?q?=20tests=20=E2=80=94=20and=20what=20they=20do=20NOT=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #W15-1 step 4's ordering tests, on the delaying-store harness ADR-0074 §2 already established. All four run on HAPPY_PATH, which declares NO `budget` — deliberately, since ADR-0077 §5's finding is that a budgeted fixture passes even with every barrier hole open. Also fixes a real defect the tests exposed in `351f3af`. **The observe half did not work, and the ADR had already named why.** `MoneyDurability`'s `.catch` never fires for a store fault, because `#emitDurable` is TOTAL: it absorbs a `persistEvent` rejection into `#failure` and RESOLVES. So the ledger saw a successful write. The engine's emit wrapper now compares `#failure` across the await and throws when it moved — turning the absorbed fault back into something a barrier can act on. It can over-trigger when a sibling fails in the same window; that direction is fail-closed, which is the right way to be wrong at a money barrier. Two of my four assertions were also wrong about real behaviour, and the tests now say what actually happens: - The terminal message is `#emitDurable`'s "a durable run-event write failed", not the barrier's. Correct, not a miss: `#emitDurable` sets `#failure` first and `??=` keeps the first cause. The barrier's job here is to stop the dispatch, not to rename the failure. - `persistOrder` records `type:nodeId`, not the bare type. HAPPY_PATH's `input` node completes long before the agent spends anything, so the first version of the B3 test asserted on the WRONG node's terminal and would have passed for the wrong reason. **What these four tests actually pin — stated plainly, because it is less than their names suggest.** Deleting B1, B2 or B3 individually reddens NOTHING. Only removing the `record` call in `onAttempt` reddens, and it reddens all four. So they pin that the ledger event is emitted and that a stalled ledger write stalls the run; they do NOT pin that any of the three joins is what stalls it. The likely mechanism is `#emitDurable`'s `#deliveryTail`, which already serializes delivery behind a blocked persist — the same pre-existing serialization that made ADR-0074 §2's own barrier look correct before its flush was removed. That leaves an open question worth answering before this is called done: whether B1/B2/B3 add a guarantee the delivery tail does not already provide, or whether the tail is doing the work and the joins are belt-and-braces. Either answer is fine; shipping without knowing which is not. The instrument that would settle it is a store that resolves the ledger persist but delays only its DELIVERY, so the tail and the joins can be told apart — which the current harness cannot express. Recorded rather than asserted, per this phase's own discipline: a test that implies coverage it does not have is worse than none. pnpm run ci EXIT=0. Refs: ADR-0074, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- packages/core/src/engine/engine.ts | 18 +- .../src/engine/m2-e2e-harness.e2e.test.ts | 177 +++++++++++++++++- 2 files changed, 191 insertions(+), 4 deletions(-) diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index 2b4336df..c205ccb2 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -417,8 +417,16 @@ class RunExecution { // ledger that only existed alongside a governor would silently skip every unbudgeted run. It fronts the // join for BOTH chains, so `flushConservative` is wired to the governor once that exists. this.#money = new MoneyDurability({ - emit: (draft) => - this.#emitDurable({ + emit: async (draft) => { + // **The observe half, and it has to be here rather than in `MoneyDurability`.** `#emitDurable` is + // TOTAL for store faults: it absorbs a `persistEvent` rejection into `#failure` and RESOLVES. So the + // ledger's own `.catch` never fires for the failure mode it exists to catch, and a barrier that only + // awaited would sail straight past a run whose money write did not land — exactly the trap ADR-0076 + // §1 named and ADR-0077 kept. Comparing `#failure` across the await is what turns the absorbed fault + // back into something the barrier can throw. It can over-trigger when a SIBLING fails in the same + // window; that direction is fail-closed and correct at a money barrier. + const failureBefore = this.#failure; + await this.#emitDurable({ ...draft, type: 'cost:attempt_settled', runId: this.runId, @@ -427,7 +435,11 @@ class RunExecution { // it now satisfies `refineCostAttemptSettled`'s "cumulative already includes this charge" by // construction, the same way the governor reads its counter after `+=`. cumulativeCostMicrocents: this.#cumulativeCostMicrocents, - }), + }); + if (this.#failure !== failureBefore) { + throw new Error('the run failed while this realized charge was being made durable'); + } + }, ...(params.plan.budget === undefined ? {} : { flushConservative: async () => this.#budgetGovernor?.flushCommitments() }), diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 7d71ba40..fe82a193 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -653,6 +653,10 @@ function buildEngine( resolveProvider: (id: ProviderId) => LlmProvider | undefined, resolveMediaSurface?: (model: string) => 'chat' | 'generative' | undefined, resolvePrice?: PricingOverlay, + // Overridable ONLY so the ADR-0077 barrier tests can observe tool dispatch directly. B2's whole claim is + // "the ledger write lands before the run mutates the world", and the only honest witness to that is the + // dispatch itself — not an event, which would just be re-reading the thing under test. + registry: ToolRegistry = echoRegistry, ): WorkflowEngine { return new WorkflowEngine({ host, @@ -663,7 +667,7 @@ function buildEngine( resolveProvider, ...(resolveMediaSurface === undefined ? {} : { resolveMediaSurface }), ...(resolvePrice === undefined ? {} : { resolvePrice }), - registry: echoRegistry, + registry, tools: [echoToolDef], keyFor: () => 'k', sleep: () => Promise.resolve(), @@ -1059,6 +1063,177 @@ describe('M2 — end-to-end Node harness (1.U)', () => { ); }); + // --- The realized-cost ledger's barriers (ADR-0076 / ADR-0077) ------------------------------------ + // + // Every test here runs on HAPPY_PATH, which declares NO `budget` — deliberately. ADR-0077 §5's whole + // finding is that the barriers used to live behind a governor that only exists when a workflow declares a + // cap, so a budgeted fixture would pass even with all three holes open. An unbudgeted run is the regression. + + /** A registry that records every dispatch, so a barrier can be witnessed by the side effect it gates. */ + function spyingRegistry(): { registry: ToolRegistry; calls: string[] } { + const calls: string[] = []; + return { + calls, + registry: { + has: (id) => echoRegistry.has(id), + list: () => echoRegistry.list(), + dispatch: (call, ctx) => { + calls.push(call.name); + return echoRegistry.dispatch(call, ctx); + }, + }, + }; + } + + /** A store that blocks `cost:attempt_settled`'s write until released, recording persist order. */ + function blockingLedgerStore() { + const inner = new InMemoryRunStore(); + const persistOrder: string[] = []; + let resolveWrite: (() => void) | undefined; + return { + persistOrder, + blocked: () => resolveWrite !== undefined, + release: () => resolveWrite?.(), + store: { + resolveWorkflowId: (slug: string) => inner.resolveWorkflowId(slug), + listInterruptedRuns: () => inner.listInterruptedRuns(), + eventsFor: (runId: string) => inner.eventsFor(runId), + persistEvent: async (event: RunEvent): Promise => { + if (event.type === 'cost:attempt_settled' && resolveWrite === undefined) { + await new Promise((resolve) => { + resolveWrite = resolve; + }); + } + // `type:nodeId`, not just the type. HAPPY_PATH's `input` node completes long before the agent + // spends anything, so a bare `node:completed` check would fire on the wrong node and pass for the + // wrong reason — which it did, on the first run of this test. + persistOrder.push( + 'nodeId' in event && typeof event.nodeId === 'string' + ? `${event.type}:${event.nodeId}` + : event.type, + ); + await inner.persistEvent(event); + }, + }, + }; + } + + it('ledger B2: a TOOL is not dispatched until the attempt`s charge is durable (ADR-0077)', async () => { + // The barrier ADR-0077 adds beyond ADR-0074 §2's pair, and the one that makes this ledger EXTEND the + // guarantee rather than repeat it: realized spend must be durable before the run mutates the world, not + // merely before it spends again. Witnessed on the dispatch itself. + const { registry, calls } = spyingRegistry(); + const ledger = blockingLedgerStore(); + const provider = scriptedProvider([toolUseTurn('t1'), textTurn('done')]); + const host = createInMemoryHost({ store: ledger.store }); + const engine = buildEngine(host, () => provider, undefined, undefined, registry); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + // Spin until the ledger write blocks. HAPPY_PATH runs input → agent → output and the charge only settles + // after a full successful attempt, so this needs more turns of the microtask queue than §2's fixture, + // whose commitment fires on a usage-less stream almost immediately. + for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); + expect(ledger.blocked()).toBe(true); + // THE assertion: the model asked for a tool, the charge is not durable, so nothing ran. + expect(calls).toEqual([]); + + ledger.release(); + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + expect(calls).toEqual(['echo']); // it did run, once the write landed + expect(events.at(-1)?.type).toBe('run:completed'); + }); + + it('ledger B3: the node terminal is not durable until the attempt`s charge is (ADR-0077)', async () => { + const ledger = blockingLedgerStore(); + const provider = scriptedProvider([textTurn('done')]); + const host = createInMemoryHost({ store: ledger.store }); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + // Spin until the ledger write blocks. HAPPY_PATH runs input → agent → output and the charge only settles + // after a full successful attempt, so this needs more turns of the microtask queue than §2's fixture, + // whose commitment fires on a usage-less stream almost immediately. + for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); + expect(ledger.blocked()).toBe(true); + // The AGENT node's own terminal must not be durable — a crash here records progress the money log lacks. + // (`in`'s terminal is legitimately already there; it spent nothing.) + expect(ledger.persistOrder).not.toContain('node:completed:work'); + + ledger.release(); + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + expect(ledger.persistOrder.indexOf('cost:attempt_settled:work')).toBeLessThan( + ledger.persistOrder.indexOf('node:completed:work'), + ); + }); + + it('ledger: a REJECTED write dispatches no tool — the observe half, not just the await (ADR-0077)', async () => { + // The mutation that matters most. `#emitDurable` is TOTAL for store faults: it absorbs a `persistEvent` + // rejection into the run's failure state and RESOLVES. So a barrier that only awaits passes both ordering + // tests above while observing nothing, and the run would mutate the world on a charge it never recorded. + // `MoneyDurability.join()` re-throws the retained failure, which is what stops the dispatch here. + const { registry, calls } = spyingRegistry(); + const inner = new InMemoryRunStore(); + const store = { + resolveWorkflowId: (slug: string) => inner.resolveWorkflowId(slug), + listInterruptedRuns: () => inner.listInterruptedRuns(), + eventsFor: (runId: string) => inner.eventsFor(runId), + persistEvent: async (event: RunEvent): Promise => { + // Reject ASYNCHRONOUSLY, after a tick — a synchronous throw would let the engine's own abort win the + // race and make this pass without any barrier at all. + if (event.type === 'cost:attempt_settled') { + await Promise.resolve(); + throw new Error('ledger write failed'); + } + await inner.persistEvent(event); + }, + }; + const provider = scriptedProvider([toolUseTurn('t1'), textTurn('done')]); + const host = createInMemoryHost({ store }); + const engine = buildEngine(host, () => provider, undefined, undefined, registry); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + + expect(calls).toEqual([]); // the world was not mutated on an unrecorded charge + const terminal = events.at(-1); + expect(terminal?.type).toBe('run:failed'); + if (terminal?.type === 'run:failed') { + // `#emitDurable`'s own message, not the barrier's — and that is correct, not a miss. It absorbs the + // rejection and sets `#failure` FIRST, and `#failure ??=` keeps the first cause. The barrier's job here + // is to stop the dispatch, not to rename the failure; its own message reaches the terminal only when a + // ledger write fails somewhere `#emitDurable` does not already absorb. + expect(terminal.error.message).toBe('a durable run-event write failed'); + // Secret-free either way: a store error can carry a filesystem path and must never reach the user. + expect(terminal.error.message).not.toContain('ledger write failed'); + } + }); + + it('ledger: an UNBUDGETED run records its realized spend at all (ADR-0077 §5)', async () => { + // The plainest regression for §5's finding. Before it, the ledger's barriers hung off `BudgetGovernor`, + // which the engine builds only when a workflow declares a `budget` — so this run, which spends real + // money, would have started a write nobody ever joined. + const inner = new InMemoryRunStore(); + const provider = scriptedProvider([textTurn('done')]); + const host = createInMemoryHost({ store: inner }); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + const { events } = await drive(handle, host); + + expect(events.at(-1)?.type).toBe('run:completed'); + const persisted = inner.eventsFor(events[0]?.runId ?? ''); + const ledgerRows = persisted.filter((e) => e.type === 'cost:attempt_settled'); + expect(ledgerRows).toHaveLength(1); + const row = ledgerRows[0]; + if (row?.type !== 'cost:attempt_settled') expect.unreachable('missing ledger row'); + expect(row.attemptNumber).toBe(1); + expect(row.priced).toBe(true); // the built-in pricing table knows this model, so the charge is real + // The cumulative must already include this charge, which is what the schema refinement pins. + expect(row.cumulativeCostMicrocents).toBeGreaterThanOrEqual(row.costMicrocents); + }); + it('conservative commitment: a RESUMED run still cannot spend past the cap it committed against (ADR-0074 §2)', async () => { // The half the previous test cannot reach: `#seedFromCheckpoint` must hand the folded total to the resumed // governor. Remove that one call and the fold still computes the right number, the log still contains the From 9308798a8fe81da38eae70c77573456c51949283 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 21:17:47 +0300 Subject: [PATCH 12/26] =?UTF-8?q?fix(core,shared,docs):=20fold=20the=20Opu?= =?UTF-8?q?s=20review=20of=20#W15-1=20=E2=80=94=20the=20barriers=20were=20?= =?UTF-8?q?fine,=20the=20tests=20were=20not?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review settled `50c60bd`'s open question empirically, in a sandbox, and the answer is the opposite of my hypothesis: B1/B2/B3 each add a guarantee the delivery tail does NOT provide. `#emitDurable` awaits `persistEvent` BEFORE `await prior`, so the tail serializes delivery only — a later event's persist still commits while an earlier one is blocked. And the tool dispatch never touches `#emitDurable` at all: `#nodeEmit` routes in-node events through `#bus.emit`, which neither persists nor chains onto the tail. There is no awaited durable emit between `onAttempt` and `dispatchToolCalls`. **The defect was in my tests, not the design.** They asserted one tick after `ledger.blocked()` flipped — which happens INSIDE `persistEvent`, several turns before the run would have reached the guarded action. The dispatch was still sitting in the queue, so the assertion passed with or without a barrier. Draining to quiescence first (microtasks plus a macrotask turn, repeatedly) is the whole fix, and `50c60bd`'s claim that settling this "needs a store that delays only DELIVERY, which the current harness cannot express" was false — the harness was fine, it was sampled too early. Verified: deleting B2 now reddens the tool-dispatch test; deleting B3 reddens the node-terminal test. B1 alone still reddens nothing, and the reason is now known rather than assumed — on this fixture the second egress only comes after the tool, so B2 masks it. Named, not implied. **A money-losing bug found next door.** `checkpoint.ts`'s `budget:paused` arm ASSIGNED `spentMicrocents` while the comment two lines above already claimed the sources "reconcile via that `Math.max` fold". `spentMicrocents` is captured at `checkPreEgress` and the event is emitted much later; under a `fan_out` a sibling attempt settles a HIGHER cumulative in that window and the assign clobbers it — handing a resumed cap headroom for money already spent, the exact bypass ADR-0074/ADR-0076 exist to close. Latent before this work (node boundaries are rare); the ledger writes per attempt and multiplies the high-water marks available to clobber. Now a `Math.max`, with a break-verified regression. **Three false claims in canonical homes, all mine, all corrected:** - `checkpoint.ts`'s field doc listed `run:*` and `budget:paused` as carriers the fold maxes. `run:*` is not folded at all (the engine says so at the emit site), and `budget:paused` was the assign above. The same sentence was duplicated verbatim into `run-event.ts` and `sse-event-schema.md`. - `run-event.ts` still called `costMicrocents` "the field checkpoint reconstruction adds up" twenty lines above the field that says not to sum it — a contradiction inside one Zod object. `83f7990` claimed the docblock was corrected; only the second half was. - `sse-event-schema.md`'s `node:completed` row still described the max-of-two- families design that the same file's fold rule says under-counts, thirteen lines apart in the one canonical file. **The rejected-write test's comment was false and now says so.** What stops the dispatch there is the engine's own abort, not the barrier: deleting the observe half OR B2 leaves it green. ADR-0077's stated required regression is therefore still unmet, and the shape that would meet it is named in the test — reject the ledger write while a sibling has already set `#failure`, so `#emitDurable`'s `this.#failure === undefined` guard skips the abort and only the barrier is left. `current.md`'s staged plan no longer describes the retracted SUM fold, and now records that all five steps are landed. Carried forward, not fixed here, each verified by the review: the observe half uses `#failure` (a "run started failing" signal) where `#emitDurable` could return a per-event outcome — it under-triggers during a cancel and on an already-failed run, and loses the store error as `cause`; `MoneyDurability`'s `#broken` flag is dead and has no production reader, so a broken ledger has no surface, unlike the estimate twin's; `join()` clears `#failure` after throwing, so under a `fan_out` a sibling can consume another node's failure; `agent-turn`'s `throwMappedChainError` rescues `CommitmentDurabilityError` but not its realized twin, so a B1 failure loses its `nodeId` and reports the conservative message; `run:completed` assigns the run total without a matching `run_costs` row, so the "holds by construction" claim is broader than the code; and no test covers the node-retry attribution the `nodeSettledCost` docblock spends a paragraph on. pnpm run ci EXIT=0. Refs: ADR-0074, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- docs/reference/contracts/sse-event-schema.md | 2 +- docs/roadmap/current.md | 17 +++++++--- packages/core/src/engine/checkpoint.test.ts | 20 ++++++++++++ packages/core/src/engine/checkpoint.ts | 27 +++++++++++----- .../src/engine/m2-e2e-harness.e2e.test.ts | 31 ++++++++++++++++--- packages/shared/src/run-event.ts | 6 ++-- 6 files changed, 85 insertions(+), 18 deletions(-) diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index 9d7ad07b..5857b2df 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -81,7 +81,7 @@ export type RunEvent = | `agent:approval_requested` | A side-effecting tool dispatch is awaiting an **interactive per-tool approval** decision (ADR-0057 EA3/EA5). The engine's `confirmDispatch` emits it — for **every** governed dispatch reaching the gate, whether the host then prompts a human or auto-decides — just before invoking the host's `ConfirmActionHook`; the registry then awaits the verdict (approve ⇒ dispatch, reject ⇒ a fatal `tool_denied`). A **dual-envelope** event (`runId`/`sessionId`), like `agent:tool_call` — in Phase 2.5 emitted only on the chat session path (the approval regime), and **carried on the session stream** (not run-only — it is **not** dropped like `agent:file_patch_proposed`). | `nodeId`, `toolId`, `action: 'fs_write' \| 'process' \| 'egress' \| 'os'` (the governed side-effect class — [tool-registry.md](../shared-core/tool-registry.md)), `preview` (**secret-free, display-only**: `{ path? }` for a write, `{ command? }` for a process, `{ host? }` for egress, `{}` for an `os` action like `read_clipboard`/`notify` — never a full URL/query, never a secret. Secret-freedom is **enforced** by the registry's redaction, not asserted, so a field **may contain the literal `[redacted]` marker and is not guaranteed to be a resolvable path/command/host — a machine consumer must never parse it**; see [tool-registry.md](../shared-core/tool-registry.md) §Preview redaction), `attemptNumber?` | | `agent:file_patch_proposed` | An agent proposed a file change (**gated — no write until the user accepts**; e.g. the VS Code inline-diff review). | `nodeId`, `patches: [{ uri, unifiedDiff }]` (≥1 — an empty proposal is meaningless), `attemptNumber?` | | `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note), `priced?` (**ADR-0070** — additive + optional, so an older reader ignores it: `false` when the egress could **not be priced**. An unpriced model still emits this event with its **real tokens** and `costMicrocents: 0`, which makes "cost 0 + tokens > 0" ambiguous between *could not price* and *genuinely free* — an ambiguity nothing else in the event resolves. The durable `session_costs` row records it as an `unpriced_calls` **counter**, not a boolean, because a model can be priced **mid-session**). **Generative-node variant (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5):** a `media_surface: 'generative'` agent node emits **exactly one** `cost:updated` with `inputTokens` / `outputTokens` **= 0** (no token billing — the spend rides entirely in `costMicrocents` as the per-modality media addend) and **no `attemptNumber`** (no FallbackChain on the generative path — one provider, no failover). | -| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — **one of the two durable cost sources** checkpoint/resume restores from, the other being the SUM of `cost:attempt_settled` deltas since [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md); the restore takes the greater of the two, and this snapshot is still the ONLY carrier of media spend, which emits no attempt row. `cost:updated` is streamed-only and is not a durable source at all. The engine always populates this field. `node:failed` mirrors it, 2.S/D-GC) | +| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — **one of the durable absolute totals** checkpoint/resume maxes over, alongside `node:failed`'s, `budget:paused.spentMicrocents`, and `cost:attempt_settled.cumulativeCostMicrocents` since [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md); see the fold rule below for why it is a max and never a sum. This snapshot is still the ONLY carrier of media spend, which emits no attempt row. `cost:updated` is streamed-only and is not a durable source at all. The engine always populates this field. `node:failed` mirrors it, 2.S/D-GC) | | `node:failed` | A node failed (TERMINAL — exactly one per node; emitted when the node-retry budget is exhausted, on a fatal / `retry_on`-excluded failure, **or** when a pending retry is abandoned by a cancel or a sibling abort — see 1.S). | `nodeId`, `error: {code, message, retryable, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `correlationId` is a secret-free id joined to the internal log — ADR-0036), `attemptNumber?` (the last attempt, when a retry budget was spent — 1.S), `cumulativeCostMicrocents?` (the run-wide running total snapshotted AT this node boundary — the durable fail-cost so a billed-but-failed **paid media job**'s realized spend survives the transient `cost:updated`, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `node:completed`) | | `node:retrying` | A retryable node attempt failed and the engine will re-dispatch the whole node (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)) — **non-terminal** (the node continues; `node:failed` is the terminal). | `nodeId`, `attemptNumber` (the attempt that just failed, 1-based), `error: {code, message, retryable}` (the `NodeFailure` shape — **no** `correlationId`; that anchors the terminal failure), `delayMs` (backoff before the next attempt) | | `node:skipped` | A node was skip-propagated (never ran). | `nodeId`, `reason: 'branch_not_taken' \| 'upstream_unreachable'` (`branch_not_taken` = a `condition` routed away from it; `upstream_unreachable` = every in-edge is dead because an upstream was skipped/failed). Emitted so the event log is a **complete, replayable** record — checkpoint/resume reconstructs a skipped vertex from it ([run-plan.md](../shared-core/run-plan.md)) and a surface can render the dimmed path instead of the node silently vanishing. | diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index a25657eb..c7c242aa 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -289,11 +289,20 @@ Ordered by whether the repo currently states something untrue, then by blast rad > (`agent-turn.ts`'s `await dispatchToolCalls(...)`, the barrier this adds beyond §2), and at the turn/node > terminal. Every barrier awaits AND observes the failure, because `#emitDurable` is total for store faults > and resolves. -> 5. `packages/core/src/engine/checkpoint.ts` — fold it as a SUM of deltas, never a last-wins snapshot (ADR-0074 -> §2's reasoning: concurrent events under a `fan_out` have no canonical `seq` order). +> 5. `packages/core/src/engine/checkpoint.ts` — **corrected during implementation**. The instruction here was +> "fold it as a SUM of deltas", borrowing ADR-0074 §2's reasoning, and the wrong half of that cost two +> rewrites. Summing into the realized total double-counts (a node terminal's snapshot already contains its +> attempts); summing into a separate accumulator and maxing the two families UNDER-counts (a media node +> writes a snapshot and emits no attempt row, so every attempt after the last boundary vanishes). What is +> correct is the fold already two arms above it: `Math.max` over every durable ABSOLUTE total. §2's +> rejection of `Math.max` does not carry — it was rejected there only because a release can DECREASE a +> conservative total, and realized spend is monotonic. > -> The break-verify that matters most is step 4's ordering: deleting the `await` must redden a test, or the -> ledger records the charge without the guarantee that makes it one. +> **STATUS: all five steps are landed** (`8d7ffcf` … `50c60bd`), each with an Opus and a Sonnet round folded. +> The break-verify that matters most was step 4's ordering, and it now holds: with the tests draining to +> quiescence before asserting, deleting B2 reddens the tool-dispatch test and deleting B3 reddens the node- +> terminal test. B1 is masked by B2 on the current fixture (the second egress only comes after the tool), so +> it is covered by construction rather than by a red — named rather than implied. > > **This is now the prerequisite of Phase 2.6.5, not a parallel track.** Its five steps touch the same four > files `CR-10` (ordered append tail) and `CR-12` (effect journal) restructure, so it lands first — see the diff --git a/packages/core/src/engine/checkpoint.test.ts b/packages/core/src/engine/checkpoint.test.ts index 73db6a02..60a6622b 100644 --- a/packages/core/src/engine/checkpoint.test.ts +++ b/packages/core/src/engine/checkpoint.test.ts @@ -700,6 +700,26 @@ describe('reconstructCheckpointState', () => { expect(state?.cumulativeCostMicrocents).toBe(1_800); }); + it('a stale budget:paused cannot CLOBBER a higher ledger total', () => { + // `budget:paused.spentMicrocents` is captured at the pre-egress check and the event is emitted much + // later, after the outcome propagates. Under a `fan_out` a sibling attempt settles a HIGHER cumulative + // in that window — and this arm used to ASSIGN, handing the resumed cap headroom for money already + // spent. Latent before the ledger, because node boundaries are rare; the ledger writes per attempt. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 5_000, 5_000), + { + type: 'budget:paused', + ...base(2), + nodeId: 'b', + spentMicrocents: 3_000, + limitMicrocents: 10_000, + gateId: 'g1', + }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(5_000); + }); + it('keeps realized and conservative money apart', () => { const state = reconstructCheckpointState([ started, diff --git a/packages/core/src/engine/checkpoint.ts b/packages/core/src/engine/checkpoint.ts index 8c147340..95aa79da 100644 --- a/packages/core/src/engine/checkpoint.ts +++ b/packages/core/src/engine/checkpoint.ts @@ -98,13 +98,18 @@ export interface CheckpointState { * The run-wide realized cost (integer micro-cents), restored on resume so post-resume spend keeps * accumulating against a cap that remembers what already went out. * - * **The fold is `Math.max` over every durable ABSOLUTE total**, and there are now three carriers of one: - * the node-boundary snapshots (`node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`), - * `budget:paused.spentMicrocents`, and — since + * **The fold is `Math.max` over every durable ABSOLUTE total that is folded here**, and there are exactly + * three: `node:completed.cumulativeCostMicrocents`, `node:failed.cumulativeCostMicrocents`, and — since * [ADR-0076](../../../../docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md) — - * `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so - * each is a true run-wide total at that instant and the largest is the engine's real total, whatever order - * the rows landed in. + * `cost:attempt_settled.cumulativeCostMicrocents`. Plus `budget:paused.spentMicrocents` in + * `applyGateEvent`, maxed the same way. Each is read immediately after its own increment, so each is a true + * run-wide total at that instant and the largest is the engine's real total, whatever order the rows landed + * in. + * + * **`run:failed` / `run:cancelled` carry the field but are NOT folded here**, and that is deliberate rather + * than an oversight — the engine states it at the emit site. A run terminal is the last event of the run, so + * a restored total that omitted it would only ever be read by a resume that cannot happen. Named because the + * field list above reads like it should include them. * * **Neither obvious alternative works, and one of them under-counts silently:** * @@ -314,8 +319,16 @@ function applyGateEvent(acc: ReconAccumulator, event: RunEvent): void { // keeps its spend and the re-seeded governor blocks correctly (H2). A plain-human-gate / crash resume now // recovers the same total from the durable `node:completed.cumulativeCostMicrocents` (applyNodeEvent above) — // the two durable sources reconcile via that `Math.max` fold (cost-event persistence is no longer deferred). + // + // `Math.max`, not an ASSIGN — which is what this was, while the comment above already claimed otherwise. + // `spentMicrocents` is captured at `checkPreEgress` and the event is emitted much later, after the + // outcome propagates through `#onOutcome`/`#settlePaused`. Under a `fan_out` a sibling attempt can settle + // a HIGHER cumulative in that window, and an assign then clobbers it — handing the resumed cap headroom + // for money already spent, which is the exact bypass ADR-0074/ADR-0076 exist to close. Latent before the + // realized ledger, because `node:completed` was the only other writer and node boundaries are rarer; + // `cost:attempt_settled` multiplies the high-water marks available to clobber. if (event.type === 'budget:paused') { - acc.cumulativeCostMicrocents = event.spentMicrocents; + acc.cumulativeCostMicrocents = Math.max(acc.cumulativeCostMicrocents, event.spentMicrocents); } return; } diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index fe82a193..6fd3c0ce 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -1085,6 +1085,21 @@ describe('M2 — end-to-end Node harness (1.U)', () => { }; } + /** + * Drain the queue to QUIESCENCE — microtasks and a macrotask turn, repeatedly. + * + * Load-bearing, and its absence made the first version of every test below hollow. `ledger.blocked()` flips + * INSIDE `persistEvent`, several turns before the run would have reached the action a barrier guards. Assert + * at that instant and the dispatch is still sitting in the queue, so the assertion passes whether or not the + * barrier exists. Draining first is what makes deleting a join go red. + */ + async function settle(): Promise { + for (let i = 0; i < 50; i += 1) { + for (let j = 0; j < 200; j += 1) await Promise.resolve(); + await new Promise((resolve) => setTimeout(resolve, 0)); + } + } + /** A store that blocks `cost:attempt_settled`'s write until released, recording persist order. */ function blockingLedgerStore() { const inner = new InMemoryRunStore(); @@ -1134,6 +1149,7 @@ describe('M2 — end-to-end Node harness (1.U)', () => { // whose commitment fires on a usage-less stream almost immediately. for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); expect(ledger.blocked()).toBe(true); + await settle(); // give the guarded action every chance to run before asserting it did not // THE assertion: the model asked for a tool, the charge is not durable, so nothing ran. expect(calls).toEqual([]); @@ -1156,6 +1172,7 @@ describe('M2 — end-to-end Node harness (1.U)', () => { // whose commitment fires on a usage-less stream almost immediately. for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); expect(ledger.blocked()).toBe(true); + await settle(); // give the guarded action every chance to run before asserting it did not // The AGENT node's own terminal must not be durable — a crash here records progress the money log lacks. // (`in`'s terminal is legitimately already there; it spent nothing.) expect(ledger.persistOrder).not.toContain('node:completed:work'); @@ -1169,10 +1186,16 @@ describe('M2 — end-to-end Node harness (1.U)', () => { }); it('ledger: a REJECTED write dispatches no tool — the observe half, not just the await (ADR-0077)', async () => { - // The mutation that matters most. `#emitDurable` is TOTAL for store faults: it absorbs a `persistEvent` - // rejection into the run's failure state and RESOLVES. So a barrier that only awaits passes both ordering - // tests above while observing nothing, and the run would mutate the world on a charge it never recorded. - // `MoneyDurability.join()` re-throws the retained failure, which is what stops the dispatch here. + // **What stops the dispatch here is the ENGINE'S ABORT, not the barrier — stated because the obvious + // reading of this test is wrong.** `#emitDurable` absorbs the rejection, sets `#failure` and calls + // `#abort.abort()`, and `dispatchToolUseTurn`'s `throwIfAborted` ends the turn. Deleting the observe half + // OR B2 leaves this test green, so it does NOT pin either. + // + // What it does pin is still worth having: a rejected ledger write ends the run without mutating the + // world, and the terminal message stays secret-free. ADR-0077's stated required regression — the barrier + // itself refusing — needs a case that DEFEATS the abort: reject the ledger write while a concurrent + // sibling has already set `#failure`, so `#emitDurable`'s `this.#failure === undefined` guard skips the + // abort and only the barrier is left to stop the dispatch. Not built; named so it is not lost. const { registry, calls } = spyingRegistry(); const inner = new InMemoryRunStore(); const store = { diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index f516186a..f4fc34fa 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -643,8 +643,10 @@ export const CostAttemptSettledEventSchema = z.object({ inputTokens: nonNegativeInt, outputTokens: nonNegativeInt, /** - * THIS attempt's realized charge — **the field checkpoint reconstruction adds up** (see - * {@link cumulativeCostMicrocents}). + * THIS attempt's realized charge — the exact per-attempt record, and what the derived `run_costs` row and + * any "why did this run cost that" answer are built from. **It is NOT the restore path**: reconstruction + * maxes {@link cumulativeCostMicrocents} instead, and that field's doc explains why summing this one both + * double-counts and under-counts depending on which route you take. * * `nonNegativeInt`, NOT the `positiveInt` its estimate twin uses, and the asymmetry is real rather than an * oversight. A conservative commitment of zero is unreachable by construction (`BudgetGovernor#admit` From 5b82da09da632f8d55f9fa04c816124c60926ef6 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 21:40:31 +0300 Subject: [PATCH 13/26] =?UTF-8?q?fix(core,shared,docs):=20fold=20the=20Son?= =?UTF-8?q?net=20review=20of=20#W15-1=20=E2=80=94=20a=20fourth=20false=20c?= =?UTF-8?q?laim,=20one=20file=20down?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The second pass found what the first fold missed, in the file the first fold was correcting three other instances in. **A fourth false claim, and it is the same drift one function lower.** `refineCostAttemptSettled`'s docblock still said "reconstruction sums the deltas (so the RESTORE survives it)" — the design step 5 retracted — twenty lines below the field doc that `9308798` had just corrected to say the opposite. The test fixture repeated it verbatim. So `run-event.ts` contradicted itself internally while a commit message claimed all such claims were corrected. Both now say the cumulative IS the restore path, and record which revision was wrong so the next reader does not re-derive it. **Two standards violations closed:** - `engine.ts`'s observe half threw a bare, uncaused `Error`, against error-handling.md's typed-discriminated rule. Now a `LedgerDurabilityError` carrying the node — which matters more than the class name, because of the fix below. - `agent-turn.ts`'s `throwMappedChainError` had an arm for `CommitmentDurabilityError` and none for its realized twin, so a barrier-B1 failure was flattened into a generic `AgentTurnError`. That arm exists to keep the `nodeId` identifying WHOSE write broke — under a `fan_out` the observing node is not the owner — and to keep `isLedgerDurabilityError` narrowing at the engine's B3 catch. Both were being lost. Correction to `9308798`'s carried-forward list: I described this as "reports the conservative message". The review checked and it does not — the message was already the realized one. What is lost is the typed identity and the node attribution. **B1 is untested and now says so, with the reason rather than a hope.** Deleting it leaves all 1102 core tests green, and the review confirmed no fixture reddens it anywhere. The reason is structural, not a fixture gap: on today's engine every path reaching a SECOND egress after a settled attempt passes through B2 or B3 first — within one chain a settled attempt ends it, so a second egress means another tool round or another node dispatch. B1 is defence in depth against a future path that crosses neither. Kept deliberately, and the comment says that if a later reader proves it unreachable the honest move is to delete it, not to leave an untestable line with a hopeful comment. **Roadmap staleness closed**, both files the review named: `current.md` said "23 of 24 closed" while its own status line said all five steps landed, and `#W15-1` had no ✅ unlike its sibling. The 2.6.5 phase doc was untouched by all nine commits and still called its own prerequisite "staged". It now records the prerequisite satisfied — and, more usefully, the two things `W1` inherits: `CR-10` gains a concrete adversary (the telescoping `run_costs` delta exists because out-of-order commit is real today, not a cloud hypothetical), and `CR-11`/`CR-92` gain a known-open seam (the observe half reads `#failure` because `#emitDurable` discards the store error, which under-triggers on cancel and on an already-failing run — a per-event durable outcome would close it, at the same choke point `CR-10` restructures). Still carried forward, unchanged: `MoneyDurability#broken` is dead with no production reader, so a broken ledger has no surface unlike the estimate twin; `join()` clears the retained failure, so a fan-out sibling can consume another node's; a simultaneous conservative+ledger failure surfaces the conservative one and delays the ledger one by exactly one barrier; `run:completed` assigns the run total with no matching `run_costs` row and no test pins the SUM invariant there; no node-retry test covers the `nodeSettledCost` attribution caveat; and ADR-0077's required regression (the barrier, not the abort, refusing the dispatch) is still unmet — its shape is named in the test. pnpm run ci EXIT=0. Core 1102 green. Refs: ADR-0074, ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- docs/roadmap/current.md | 8 +++++-- ...hase-2.6.5-core-reliability-remediation.md | 23 +++++++++++++++---- packages/core/src/engine/agent-turn.ts | 20 +++++++++++++++- packages/core/src/engine/engine.ts | 15 ++++++++++-- packages/shared/src/run-event.test.ts | 4 ++-- packages/shared/src/run-event.ts | 10 +++++--- 6 files changed, 66 insertions(+), 14 deletions(-) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index c7c242aa..97a54ef6 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -245,7 +245,8 @@ Ordered by whether the repo currently states something untrue, then by blast rad **Every item names the check that would close it, not just the defect.** -> **Status, 2026-08-09 — 23 of 24 closed.** Everything marked ✅ below is fixed, +> **Status, 2026-08-10 — 24 of 24 closed.** `#W15-1`, the last one, landed 2026-08-10 behind ADR-0076 + +> ADR-0077 (see §A). Everything marked ✅ below is fixed, > break-verified with the mutation confirmed applied, and **merged to `main` via PR #81** (2026-08-09). §E's six coverage gaps > are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort listener is removed, which > is the unkillable run reproduced exactly rather than an assertion standing in for it. @@ -323,7 +324,10 @@ Ordered by whether the repo currently states something untrue, then by blast rad #### A. Needs an ADR first — the only part that is NOT PR #81's to close -- **`#W15-1` (blocker) — a workflow's REALIZED cost is not durable at the provider-attempt boundary.** +- ✅ **`#W15-1` (blocker) — a workflow's REALIZED cost is not durable at the provider-attempt boundary.** + *Closed 2026-08-10 by ADR-0076 + ADR-0077 and the five staged steps; the barrier is a chain-and-join, not + the inline await ADR-0076 §1 first specified. See the staged plan below for what each step landed and the + three carried-forward gaps.* `engine.ts` folds `cost:updated` into memory and streams it; the store documents that it is never persisted, and the checkpoint can only recover cost from a LATER `node:completed`/gate snapshot. So: a paid call succeeds → the model asks for a tool → the process dies during the tool → the realized cost is gone and the diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index b8c767a6..b113ca4e 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -3,7 +3,7 @@ - **Status**: planned - **Opened**: 2026-08-09 · **Plan corrected**: 2026-08-10 - **Predecessor**: Wave 1 of the 2.5.5 remediation (complete — PR #81), then the `#W15-1` realized-cost - ledger implementation (an accepted ADR with a staged implementation — see [Prerequisite](#prerequisite)) + ledger implementation (**complete 2026-08-10**, ADR-0076 + ADR-0077 — see [Prerequisite](#prerequisite)) - **Successor**: Wave 2 of the 2.5.5 remediation, **reduced** — this phase absorbs Wave 2's hostile-MCP threat class (see [Phase boundary](#phase-boundary)); Wave 2 keeps the filesystem jail, the secrets layer, persistence-security and the 2.5.5 certification @@ -74,11 +74,26 @@ rejected: a phase whose exit criteria cannot be evaluated until a later phase is ## Prerequisite -**`#W15-1` — the durable per-attempt realized-cost ledger — lands before `W1` starts.** It is not part of this -phase, but it is not independent of it either. +**`#W15-1` — the durable per-attempt realized-cost ledger — lands before `W1` starts. ✅ SATISFIED +(2026-08-10).** All five steps are merged, each with an Opus and a Sonnet round folded, and the decision +acquired a correction on the way: +[ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) amends +ADR-0076 §1, whose stated mechanism (an inline `await` at the attempt boundary) is unimplementable — the +seam's attempt observer is synchronous. The ledger uses ADR-0074 §2's chain-and-join shape instead, with a +third barrier before tool dispatch that `CR-12`'s effect journal is expected to extend rather than re-thread. + +Two things `W1` inherits from it. **`CR-10` gains a concrete adversary:** the ledger's derived `run_costs` row +had to be written as a telescoping delta rather than the event's raw charge, precisely because +`#emitDurable` starts every persist immediately and serializes only delivery — out-of-order commit is real +today, not a cloud-store hypothetical. And **`CR-11`/`CR-92` gain a known-open seam:** the ledger's observe +half reads the run's `#failure` because `#emitDurable` discards the store error, which under-triggers during +a cancel and on an already-failing run. A per-event durable outcome would close it, and that is a change to +the same choke point `CR-10` restructures. + +The original reasoning for the ordering, kept because it is what made it right: [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md) is Accepted and its implementation -is staged in five steps in [current.md](../current.md). Those five steps touch +was staged in five steps in [current.md](../current.md). Those five steps touch `packages/shared/src/run-event.ts`, `packages/core/src/engine/engine.ts`, `packages/core/src/engine/checkpoint.ts` and `packages/db/src/run-history-store.ts` — the same four files `CR-10` and `CR-12` restructure. Landing it after the durability spine means writing its barrier against a diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index 5175db2f..30398ba0 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -62,7 +62,7 @@ import { CommitmentDurabilityError, type BudgetAdmission, } from './budget-governor.js'; -import type { TurnMoneyPort } from './money-durability.js'; +import { LedgerDurabilityError, type TurnMoneyPort } from './money-durability.js'; import type { NodeStreamEvent } from './node-executor.js'; /** @@ -460,6 +460,14 @@ function throwMappedChainError(error: LlmError): never { if (error.cause instanceof CommitmentDurabilityError) { throw error.cause; } + // ADR-0076/ADR-0077's realized twin, for the SAME two reasons — barrier B1 joins the money chain inside + // `preAttempt`, so a ledger failure arrives here wrapped exactly like a commitment failure. Without this + // arm the class is flattened away: `isLedgerDurabilityError` stops narrowing at the engine's B3 catch, and + // the `nodeId` identifying WHOSE write broke is replaced by whichever node's turn happened to observe the + // barrier — the fan-out misattribution `CommitmentDurabilityError` got this arm to prevent. + if (error.cause instanceof LedgerDurabilityError) { + throw error.cause; + } throw new AgentTurnError(codeForLlmError(error), error.message, error.retryable); } @@ -925,6 +933,16 @@ async function driveAgentTurn( // run whose ledger write did not land admits nothing further. It awaits AND observes: `join()` // throws the retained failure rather than returning, which is the only way a caller here can see // it (`#emitDurable` absorbs a store fault and resolves). + // + // **UNTESTED, and the reason is worth knowing rather than guessing.** Deleting this line leaves + // the entire core suite green, and no fixture reddens it because on today's engine every path + // that reaches a SECOND egress after a settled attempt passes through B2 or B3 first: within one + // chain, a settled attempt ends it (a post-content failure surfaces rather than failing over), so + // a second egress means either another tool round (B2) or another node dispatch (B3). B1 is + // therefore defence in depth against a future path that reaches egress without crossing either — + // a chain that continues past a settled attempt, or a turn core that stops routing tools through + // `dispatchToolUseTurn`. Kept deliberately; if a later reader finds it genuinely unreachable, the + // honest move is to delete it and say so, not to leave an untestable line with a hopeful comment. await params.money?.join(); if (preEgress === undefined) return; // This is the only admitting boundary. Check cancellation on BOTH sides of the awaited governor call: diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index c205ccb2..fe157c50 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -68,7 +68,11 @@ import { type BudgetAdmission, } from './budget-governor.js'; import type { CheckpointPendingMediaJob, CheckpointState } from './checkpoint.js'; -import { MoneyDurability, isLedgerDurabilityError } from './money-durability.js'; +import { + LedgerDurabilityError, + MoneyDurability, + isLedgerDurabilityError, +} from './money-durability.js'; import type { AbortControllerLike, ExecutionHost } from './execution-host.js'; import type { GateRequest, @@ -437,7 +441,14 @@ class RunExecution { cumulativeCostMicrocents: this.#cumulativeCostMicrocents, }); if (this.#failure !== failureBefore) { - throw new Error('the run failed while this realized charge was being made durable'); + // Typed, not a bare `Error` (error-handling.md). `#emitDurable` discards the store error in its own + // catch, so there is no `cause` left to preserve — the run's `#failure` carries the user-facing + // reason instead, and this class exists to keep the node attribution that would otherwise be lost + // when the chain flattens a `preAttempt` throw. + throw new LedgerDurabilityError( + new Error('the run failed while this realized charge was being made durable'), + draft.nodeId, + ); } }, ...(params.plan.budget === undefined diff --git a/packages/shared/src/run-event.test.ts b/packages/shared/src/run-event.test.ts index 881ba066..c0fd6dfb 100644 --- a/packages/shared/src/run-event.test.ts +++ b/packages/shared/src/run-event.test.ts @@ -261,8 +261,8 @@ const reject: Record> = { }, 'cost:attempt_settled (cumulative below this attempt)': { // The same producer bug the conservative twin pins: reading the run-wide counter BEFORE folding this - // attempt into it. Reconstruction sums the deltas so the RESTORE survives it, which is precisely why - // nothing else would complain — every display reading the snapshot just under-reports. + // attempt into it. The cumulative IS the restore path (reconstruction maxes it), so such a row restores a + // total missing this charge — nothing else anywhere would complain, which is why this fixture exists. type: 'cost:attempt_settled', ...env, nodeId: 'n', diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index f4fc34fa..db14cb19 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -770,9 +770,13 @@ function refineBudgetEstimateCommitted(event: RunEventUnion, ctx: z.RefinementCt * * The same invariant, and the same single most likely emit-site bug, as {@link refineBudgetEstimateCommitted}: * reading the run-wide counter BEFORE folding this attempt into it. That mistake emits - * `{ cost: 500, cumulative: 0 }`, and while reconstruction sums the deltas (so the RESTORE survives it), every - * display and cross-check that reads the snapshot silently under-reports. Pinned here because nothing else - * anywhere would complain. + * `{ cost: 500, cumulative: 0 }` — and since the cumulative IS the restore path here (reconstruction maxes it; + * see the field's own doc), such a row would restore a total missing this charge and hand the cap headroom for + * money already spent. Pinned here because nothing else anywhere would complain. + * + * (An earlier revision of this comment said reconstruction "sums the deltas, so the RESTORE survives it". + * That described the design ADR-0076 originally staged and step 5 retracted — summing double-counts against a + * node terminal's snapshot. The restore has never summed since the code landed.) */ function refineCostAttemptSettled(event: RunEventUnion, ctx: z.RefinementCtx): void { if ( From 5f69ddc46a580fcb04d0cf289b2193a59afb2809 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 22:34:25 +0300 Subject: [PATCH 14/26] fix(tooling): stop a repo-local checkout leaking into the root test run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CR-90`, the first item of phase 2.6.5. The defect was live in this working tree, not a stale note: a root `vitest list` discovered **472** test files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling checkout of this same repo, complete with its own sources. So a repo-wide run was executing a foreign tree's suites, and `pnpm coverage` was counting that tree's sources as 0%-covered files against the engine floor. It hid unusually well. `.claude/worktrees/` is excluded through `.git/info/exclude` — LOCAL and untracked — so `git status` stays clean, a fresh clone does not carry the rule at all, and Vitest never consulted git in the first place. After the fix: 238 files, exactly this repo's own. Both halves of the exclusion are needed and they fail differently. `test.exclude` stops the foreign TESTS from running; `coverage.exclude` stops its SOURCES from being counted — without the second, a nested checkout's `packages/*/src/**` matches the coverage `include` and lands in the report as 0%-covered files, dragging a per-package threshold under its floor for a reason invisible in any diff. `defaultExclude` is SPREAD rather than replaced, since setting `exclude` overrides Vitest's own list and dropping `**/node_modules/**` would be a far worse collection bug than the one being fixed. **A list of named locations, deliberately, not a structural predicate.** The obvious general rule — "a workspace root nested inside another one", `'**/*/{packages,apps,tools}/**'` — is wrong, and I checked rather than assumed: it also matches this repo's own `packages/core/src/tools/*.test.ts`, because `**` eats `packages/core` and leaves `src` for the `*`. A rule that silently drops real tests is worse than the leak it closes. **The regression is `tools/test-isolation/check.mjs`, wired into `pnpm run ci`.** It is a tools check rather than a `*.test.ts` for a structural reason: a test running inside Vitest cannot observe which files Vitest chose to collect — only a subprocess can. It plants a SYNTHETIC checkout fixture (per the phase doc: not a real `git worktree`, which would make the check depend on git state and on the cwd being a repo at all — the exact non-determinism this item is about), and plants it under `.worktrees/` rather than `.claude/`, so it exercises a pattern that is NOT the one that actually bit us and the list stays honest rather than tautological. Two assertions, and the second is what keeps the fix honest: the fixture must NOT be collected, and a canary real test file must still BE collected. Without the canary the cheapest way to pass is an over-broad exclude that drops the repo's own suites — going green by not running. Break-verified, mutation confirmed applied, real exit codes checked: emptying `REPO_LOCAL_CHECKOUTS` exits 1 naming the leaked fixture; adding `**/shared/**` exits 1 naming the missing canary and reporting 226 collected instead of 238. `pnpm coverage` now measures only our sources — 96.95% lines, 93.32% branches — and still exits 0. pnpm run ci EXIT=0. Refs: CR-90 Co-Authored-By: Claude Opus 5 --- package.json | 3 +- tools/test-isolation/check.mjs | 108 +++++++++++++++++++++++++++++++++ vitest.config.ts | 32 +++++++++- 3 files changed, 140 insertions(+), 3 deletions(-) create mode 100644 tools/test-isolation/check.mjs diff --git a/package.json b/package.json index dd881b40..8a78b10f 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,8 @@ "test": "turbo run test", "coverage": "vitest run --coverage", "coverage:enforced": "node tools/coverage-gate/run.mjs", - "ci": "turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm smoke:cli", + "ci": "turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm lint:test-isolation && pnpm smoke:cli", + "lint:test-isolation": "node tools/test-isolation/check.mjs", "lint:fence-check": "node tools/lint-fixtures/assert-fence.mjs", "lint:engine-deps": "node tools/engine-deps/check.mjs", "lint:bundle-closure": "node tools/bundle-closure/check.mjs", diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs new file mode 100644 index 00000000..872f2dc7 --- /dev/null +++ b/tools/test-isolation/check.mjs @@ -0,0 +1,108 @@ +/** + * Test-isolation guard — a repo-local SECOND checkout must never be collected by a root run (`CR-90`). + * + * The defect this closes was live, not theoretical. Measured 2026-08-10: a root `vitest list` discovered + * **472** test files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling + * checkout of this same repo. A repo-wide run was therefore executing a foreign tree's tests, and + * `pnpm coverage` was counting that tree's sources as 0%-covered files against the engine floor. It hid well: + * `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, so `git status` + * stayed clean and a fresh clone never even carried the rule. + * + * The fix is `REPO_LOCAL_CHECKOUTS` in `vitest.config.ts`. This is its regression, and it exists as a tools + * check rather than a `*.test.ts` for a reason worth stating: a test running INSIDE Vitest cannot observe + * which files Vitest chose to collect. Only a subprocess can. + * + * **Two assertions, and the second is the one that keeps the fix honest.** A fixture placed in an excluded + * location must NOT be collected — and every real test file must still BE collected. Without the second, the + * cheapest way to pass is an over-broad exclude that quietly drops the repo's own suites, which is a far worse + * failure than the one being prevented. + * + * Exits non-zero so CI fails loudly. Run from the repo root: + * node tools/test-isolation/check.mjs + */ +import { execFileSync } from 'node:child_process'; +import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +/** + * Where the synthetic checkout is planted. A SYNTHETIC fixture, deliberately, not a real `git worktree`: + * a real one depends on git state and on the working directory being a repo at all, which is exactly the + * non-determinism this check is about. What matters is the SHAPE — a nested directory holding a workspace + * layout with its own test and source file — and that is reproducible anywhere. + * + * `.worktrees/` is one of the excluded names; the fixture goes there so the check exercises a pattern that is + * NOT the one that actually bit us (`.claude/`), which keeps the list honest rather than tautological. + */ +const FIXTURE_ROOT = '.worktrees/__test_isolation_fixture__'; +const FIXTURE_PKG = join(FIXTURE_ROOT, 'packages', 'probe', 'src'); + +/** A file every real run must find, so an over-broad exclude cannot pass this check. */ +const CANARY = 'packages/shared/src/run-event.test.ts'; + +function plantFixture() { + mkdirSync(FIXTURE_PKG, { recursive: true }); + writeFileSync(join(FIXTURE_PKG, 'probe.ts'), 'export const probe = 1;\n'); + writeFileSync( + join(FIXTURE_PKG, 'probe.test.ts'), + [ + "import { describe, expect, it } from 'vitest';", + "import { probe } from './probe.js';", + '', + // If the exclusion regresses, this runs in the root suite and its message says why. + "describe('__test_isolation_fixture__', () => {", + " it('MUST NOT be collected by a root run — see tools/test-isolation', () => {", + ' expect(probe).toBe(1);', + ' });', + '});', + '', + ].join('\n'), + ); +} + +function collectedFiles() { + // `--filesOnly` keeps this cheap: Vitest resolves the file set without importing a single test module. + const out = execFileSync('npx', ['vitest', 'list', '--filesOnly'], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }); + return out + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.endsWith('.test.ts') || line.endsWith('.test.tsx')); +} + +let files; +try { + plantFixture(); + files = collectedFiles(); +} finally { + // Always, even on a throw — a lingering fixture would be collected by the NEXT run and turn a clean tree + // into a confusing red. + rmSync(FIXTURE_ROOT, { recursive: true, force: true }); +} + +const leaked = files.filter((f) => f.includes('__test_isolation_fixture__')); +if (leaked.length > 0) { + console.error( + `✖ a repo-local checkout leaked into the root test run (${leaked.length} file(s)):\n` + + leaked.map((f) => ` ${f}`).join('\n') + + `\n Add its directory to REPO_LOCAL_CHECKOUTS in vitest.config.ts.` + + `\n A root run must see this repo's tests and nothing else — a second checkout's suites are not ours` + + `\n to run, and its sources are not ours to count against the coverage floor.`, + ); + process.exit(1); +} + +if (!files.includes(CANARY)) { + console.error( + `✖ the canary test file was NOT collected: ${CANARY}` + + `\n REPO_LOCAL_CHECKOUTS (vitest.config.ts) is excluding real tests. That is worse than the leak it` + + `\n prevents: the suite goes green by not running. Narrow the pattern.` + + `\n Collected ${files.length} file(s).`, + ); + process.exit(1); +} + +console.log( + `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout.`, +); diff --git a/vitest.config.ts b/vitest.config.ts index bddb4c17..56076065 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,4 +1,25 @@ -import { defineConfig } from 'vitest/config'; +import { defaultExclude, defineConfig } from 'vitest/config'; + +/** + * Directories that hold a SECOND checkout of this repo inside the working tree, and must never be collected + * by a root run (`CR-90`). + * + * This is not hypothetical housekeeping. Measured on 2026-08-10: a root `vitest list` found **472** test + * files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling checkout, + * complete with its own sources. So a repo-wide run was executing a foreign tree's tests, and `pnpm coverage` + * was measuring its sources as 0%-covered files against the engine floor. + * + * It hides well: `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, + * so `git status` stays clean, a fresh clone does not even carry the rule, and Vitest never consulted git in + * the first place. + * + * **A list of named locations, deliberately, not a structural predicate.** The obvious general rule — "a + * workspace root nested inside another one", `'**\/*\/{packages,apps,tools}\/**'` — is wrong: it also matches + * this repo's own `packages/core/src/tools/*.test.ts`, because `**` happily eats `packages/core` and leaves + * `src` for the `*`. A rule that silently drops real tests is worse than the leak it closes. `tools/test-isolation` + * is the regression that proves this list actually works. + */ +const REPO_LOCAL_CHECKOUTS = ['**/.claude/**', '**/.worktrees/**', '**/worktrees/**']; /** * Root, workspace-aware Vitest config. Per-package `test` scripts run `vitest run` @@ -26,6 +47,9 @@ export default defineConfig({ // SAME `.test.` prefix — an additive extension, not the rejected `.spec.` suffix — and stays confined to // apps/cli's renderer layer (the coverage `exclude` below drops it, and apps are coverage-excluded anyway). include: ['**/*.test.ts', '**/*.test.tsx'], + // `defaultExclude` is SPREAD, not replaced — setting `exclude` overrides Vitest's own list, and dropping + // `**/node_modules/**` from it would be a far bigger collection bug than the one being fixed. + exclude: [...defaultExclude, ...REPO_LOCAL_CHECKOUTS], passWithNoTests: true, coverage: { provider: 'v8', @@ -34,7 +58,11 @@ export default defineConfig({ // cwd-tolerant: `**/src/**` matches whether the run is rooted at the repo or a package, so a // package-scoped `--coverage` run no longer reports a false 0%. Apps stay smoke-only (excluded). include: ['**/src/**/*.ts'], - exclude: ['**/*.test.ts', '**/*.test.tsx', '**/apps/**'], + // Both halves matter and they fail differently: `test.exclude` stops a foreign tree's TESTS from + // running, this stops its SOURCES from being counted. Without it a nested checkout's `packages/*/src/**` + // matches the `include` above and lands in the report as 0%-covered files, dragging the per-package + // threshold below its floor for a reason no one can see in the diff. + exclude: ['**/*.test.ts', '**/*.test.tsx', '**/apps/**', ...REPO_LOCAL_CHECKOUTS], // The enforced Phase-1 engine floor, scoped per-glob so it targets only the built engine // package(s) and never the not-yet-90% shared/db or the unbuilt core. // From 3d8f237dc8350252bf5ff8e32e6ac87e2f26e0b9 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 22:56:15 +0300 Subject: [PATCH 15/26] =?UTF-8?q?fix(repo):=20fold=20the=20Opus=20review?= =?UTF-8?q?=20of=20CR-90=20=E2=80=94=20the=20guard=20was=20blind=20to=20th?= =?UTF-8?q?e=20defect=20it=20was=20written=20for?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thirteen findings. The two that mattered were both in what I had claimed was the regression. **BLOCKER — a false mechanism, committed into a permanent config comment.** I wrote that the leaked sources drag "the per-package threshold below its floor". The review ran the counterfactual: with the leak, coverage reports 48.85% lines against 96.95%, and still exits 0. Vitest matches threshold globs ROOT-RELATIVE with no implicit leading `**`, so `.claude/worktrees/x/packages/llm/src/a.ts` matches none of the three per-package globs and lands in the `global` group, which sets no thresholds and is skipped. What the leak destroys is the reported number and the lcov/html artifact — real, but not what I said. Stated precisely now, including that the line becomes genuinely floor-load-bearing the moment anyone adds a global threshold, because that comment is what a future maintainer reads to decide whether the line is still needed. **HIGH — the guard passed green on the exact defect it existed to prevent.** It planted one fixture under `.worktrees/` and never exercised `**/.claude/**`, the one entry that actually bit this repo. I fixed that by deriving fixtures FROM the list — and that fix has the same hole, which the break-verify caught: deleting the entry also deletes its probe, so the guard re-collected all 234 foreign suites and reported `✓ 472 file(s) collected, none from a repo-local checkout`. Fixture-per-entry can only prove every LISTED location is excluded. It cannot prove the list is COMPLETE. So the guard's primary assertion now consults no list at all: it walks the collected paths and fails on any whose ancestor carries its own `pnpm-workspace.yaml` — a second checkout of a pnpm monorepo always does. Verified: dropping `'**/.claude/**'` now exits 1 with `234 collected file(s) belong to a SECOND checkout ... .claude/worktrees/wf_.../`. A structural predicate is safe here precisely because it runs in JS against the filesystem; the same idea as a glob is what wrongly matched `packages/core/src/tools/`. **HIGH — the guard was in `pnpm run ci` and not in `ci.yml`**, recreating incident #312's exact shape, in a file whose own comments cite that incident twice. Added. This is also the honest answer to "does the fix change CI": no. `.claude/worktrees/` is hidden by a LOCAL `.git/info/exclude` and CI checks out clean, so both excludes are no-ops there. The exclusions defend a developer's tree; the guard is the only CI-facing artifact — and it was the one thing not wired in. Also fixed, each verified rather than assumed: - A single canary cannot catch an over-broad exclude that misses its package. The rejected structural rule drops four files under `packages/core/src/tools/` while a `packages/shared` canary stays green. Now every one of the six workspaces must yield tests. - `vitest list` observes `test.exclude` only, so nothing checked the `coverage.exclude` half of the acceptance criterion. Checked as config text. - `execFileSync('npx', …)` contradicted the convention `tools/coverage-gate/run.mjs` documents in so many words, and breaks on Windows. Now `process.execPath` + a module-resolved vitest entry, `shell: false`, `cwd: repoRoot`. - No cwd pinning; stdout parsing that a `projects` config or a colorized run would break; a POSIX-separator canary compared against `path.relative` output on Windows. All three closed by `--json` + `relative(repoRoot, file)`. - The fixture left an empty `.worktrees/` behind and cleanup was not signal-safe. Now removes the parent it created, and clears before planting so a Ctrl-C'd run self-heals. - `.gitignore` gained the tracked entries the local `.git/info/exclude` was standing in for — the commit diagnosed that gap and then did not fix it. - `**/worktrees/**` root-anchored to `worktrees/**`: this product's CLI manages git checkouts, so `apps/cli/src/worktrees/` is plausible, and the undotted pattern would have deleted its tests from collection and its sources from coverage. - `lint:test-isolation` moved to the FRONT of the `ci` chain — a collection regression should not be reported after the whole suite already ran under the wrong file set. It costs 0.4s. - Scope is `repo`, not `tooling`; `commit-style.md` enumerates the scopes and `tooling` is not one. `5f69ddc` used it. `docs/standards/testing.md`'s CI-gate section now names the guard, and the phase doc marks CR-90 closed with three corrections to its own recorded evidence: the re-measured 472 → 238, that the floor was never at risk, and that CI never saw it. Break-verified, mutation confirmed applied, real exit codes: dropping `.claude` exits 1 naming the checkout; emptying the list exits 1; removing it from `coverage.exclude` only exits 1; the over-broad structural rule exits 1 as an unplantable pattern. Guard runs identically from any cwd and leaves no litter. pnpm run ci EXIT=0. Refs: CR-90 Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 8 + .gitignore | 8 + ...hase-2.6.5-core-reliability-remediation.md | 26 +- docs/standards/testing.md | 8 + package.json | 2 +- tools/test-isolation/check.mjs | 265 ++++++++++++++---- vitest.config.ts | 35 ++- 7 files changed, 287 insertions(+), 65 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a8602a4c..7b61c4f5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -138,6 +138,14 @@ jobs: - name: Lint the tooling scripts run: pnpm lint:tools + # A repo-local SECOND checkout must never be collected by a root run (CR-90). This is the one check + # whose defect CI could not have caught on its own: the leak lives in `.claude/worktrees/`, hidden by a + # LOCAL `.git/info/exclude` entry, so a clean CI checkout never has it. The vitest exclusions defend a + # developer's tree; this step is what stops them being weakened — without it the guard would be the only + # root-`ci` check missing here, which is the #312 divergence the two steps above exist to close. + - name: Test isolation (no repo-local checkout in the run) + run: pnpm lint:test-isolation + # RUN the artifact this job just built, through the SAME `pnpm smoke:cli` script the root `ci` script # calls — a check that exists in only one of the two is exactly the #312 divergence this change closes. # Until now nothing in the required gate executed diff --git a/.gitignore b/.gitignore index 4822ff15..2afc7fb6 100644 --- a/.gitignore +++ b/.gitignore @@ -53,6 +53,14 @@ coverage-tmp/ # Claude Code transient state (skills/agents ARE committed; runtime locks are not) .claude/scheduled_tasks.lock .claude/**/*.lock +# A repo-local SECOND checkout (CR-90). `.claude/worktrees/` was only ever hidden by `.git/info/exclude`, +# which is local and untracked — so a fresh clone did not carry the rule and nobody could see the reason. +# Tracked here so it travels. The vitest side of this is `REPO_LOCAL_CHECKOUTS` in vitest.config.ts; the two +# lists are kept in step by tools/test-isolation, which fails on any collected file that turns out to live in +# a directory carrying its own pnpm-workspace.yaml. +.claude/worktrees/ +.worktrees/ +/worktrees/ # Private analysis — never commit, never reference from tracked files docs/analysis/private/ diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index b113ca4e..6299ddcc 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -783,7 +783,7 @@ Carried from our own backlog and re-confirmed by the reviews. `CR-90` and `CR-91` execute **first** (see [Execution order](#execution-order)); `CR-92` executes inside the durability spine. They are grouped here by theme only. -### CR-90 — Root coverage collects in-repo worktrees · Medium +### CR-90 — Root coverage collects in-repo worktrees · Medium · ✅ CLOSED 2026-08-10 The root Vitest include patterns have no exclude for repo-local checkout directories, so a stale worktree adds test files and 0%-covered sources to the run. In one working tree this produced 471 discovered test files and 23 suite failures; excluding the worktree path produced 237 files and a clean run. @@ -792,6 +792,30 @@ self-test uses a **synthetic repo-local checkout fixture** — a directory tree source file — not a real `git worktree`; a real worktree makes the test depend on git state and the working directory, which is exactly the non-determinism this item is about. +**Closed by `5f69ddc` + its review fold.** `REPO_LOCAL_CHECKOUTS` in `vitest.config.ts` (spread into both +excludes), the guard at `tools/test-isolation/check.mjs`, wired into `pnpm run ci` and `.github/workflows/ci.yml`, +plus the tracked `.gitignore` entries the local `.git/info/exclude` was standing in for. + +Three corrections to the record above, all measured rather than recalled: + +- **Re-measured 2026-08-10: 472 → 238**, not 471 → 237; the tree grew one test file. The foreign 234 were a + full agent-tooling checkout under `.claude/worktrees//`. +- **It did NOT threaten the coverage floor**, which the original evidence implied and an earlier version of + this fix asserted. Vitest matches threshold globs ROOT-RELATIVE with no implicit leading `**`, so the foreign + sources match none of the three per-package globs and land in the unset `global` group. The counterfactual + exits 0 at 48.85% lines. What the leak destroys is the reported number and the lcov/html artifact — and it + becomes floor-load-bearing the moment a global threshold is added. +- **CI never saw any of it.** `.claude/worktrees/` was hidden by a LOCAL `.git/info/exclude` entry and CI checks + out clean, so both excludes are no-ops there. The exclusions defend a developer's tree; the guard is the part + that defends CI, by failing if they are ever weakened. + +The acceptance criterion above names a fixture, and a fixture alone is not sufficient — recorded because the +first attempt shipped exactly that gap. Deriving fixtures from the exclusion list proves every LISTED location +is excluded, and cannot prove the list is COMPLETE: removing the one entry that mattered also removes its +probe, and the guard reports green while re-collecting all 234 foreign suites. The guard's primary assertion is +therefore structural and consults no list — any collected file whose ancestor carries its own +`pnpm-workspace.yaml` is in a second checkout. + ### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium It pins that the live stream reported success; it does not prove durable truth after a restart. **Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same diff --git a/docs/standards/testing.md b/docs/standards/testing.md index f1daf09b..0a9feef3 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -135,3 +135,11 @@ journeys, not exhaustive logic — exhaustive logic belongs in engine unit tests PRs must pass: typecheck, lint ([code-style-typescript.md](code-style-typescript.md)), all unit tests, and the fixture-mode conformance suite. The live conformance suite and the desktop e2e suite run nightly and on release branches. A red required check blocks merge. + +**Test isolation is part of that gate** (`pnpm lint:test-isolation`, `tools/test-isolation`). A root run must +collect this repo's tests and nothing else. A second checkout of the repo inside the working tree — an agent +worktree, a scratch clone — is otherwise collected in full: measured once at 234 foreign test files out of 472, +with its sources in the coverage denominator. The guard fails on any collected file whose ancestor carries its +own `pnpm-workspace.yaml`, so it stays true even when the exclusion list in `vitest.config.ts` is the thing +that is wrong. It also asserts every workspace still yields tests, because the cheapest way to pass an +isolation check is an over-broad exclude that goes green by not running. diff --git a/package.json b/package.json index 8a78b10f..370fda27 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,7 @@ "test": "turbo run test", "coverage": "vitest run --coverage", "coverage:enforced": "node tools/coverage-gate/run.mjs", - "ci": "turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm lint:test-isolation && pnpm smoke:cli", + "ci": "pnpm lint:test-isolation && turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm smoke:cli", "lint:test-isolation": "node tools/test-isolation/check.mjs", "lint:fence-check": "node tools/lint-fixtures/assert-fence.mjs", "lint:engine-deps": "node tools/engine-deps/check.mjs", diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs index 872f2dc7..40bd9d70 100644 --- a/tools/test-isolation/check.mjs +++ b/tools/test-isolation/check.mjs @@ -3,53 +3,138 @@ * * The defect this closes was live, not theoretical. Measured 2026-08-10: a root `vitest list` discovered * **472** test files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling - * checkout of this same repo. A repo-wide run was therefore executing a foreign tree's tests, and - * `pnpm coverage` was counting that tree's sources as 0%-covered files against the engine floor. It hid well: - * `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, so `git status` - * stayed clean and a fresh clone never even carried the rule. + * checkout of this same repo. A repo-wide run was therefore executing a foreign tree's tests, and the coverage + * report was counting its sources: 48.85% lines against 96.95% once excluded. * - * The fix is `REPO_LOCAL_CHECKOUTS` in `vitest.config.ts`. This is its regression, and it exists as a tools - * check rather than a `*.test.ts` for a reason worth stating: a test running INSIDE Vitest cannot observe - * which files Vitest chose to collect. Only a subprocess can. + * It hid well: `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, so + * `git status` stayed clean and a fresh clone never even carried the rule. That is also why CI never saw it — + * CI checks out clean. The `vitest.config.ts` exclusions defend a developer's tree; **this guard is what + * defends CI**, by failing if those exclusions are ever weakened or removed. * - * **Two assertions, and the second is the one that keeps the fix honest.** A fixture placed in an excluded - * location must NOT be collected — and every real test file must still BE collected. Without the second, the - * cheapest way to pass is an over-broad exclude that quietly drops the repo's own suites, which is a far worse - * failure than the one being prevented. + * It is a tools check rather than a `*.test.ts` for a structural reason: a test running INSIDE Vitest cannot + * observe which files Vitest chose to collect. Only a subprocess can. * - * Exits non-zero so CI fails loudly. Run from the repo root: + * ## What it asserts, and why each one is load-bearing + * + * 1. **PRIMARY — no collected file lives in a second checkout, listed or not.** Consults no list: it walks the + * collected paths and fails on any whose ancestor carries its own `pnpm-workspace.yaml`. This is the only + * assertion that stays true when the exclusion list is itself the thing that is wrong, and it is here + * because assertion 2 alone was not enough — see below. + * 2. **One fixture per `REPO_LOCAL_CHECKOUTS` entry, none collected.** The list is read out of + * `vitest.config.ts` (as text — this file is `.mjs` and the config is TypeScript, so it cannot be + * imported), so a new entry is probed the moment it is added. **But it proves only that every LISTED + * location is excluded, never that the list is COMPLETE**: because the fixtures are derived from the list, + * deleting `'**\/.claude\/**'` also deletes its probe, and the guard re-collects all 234 foreign suites + * while printing a green line. Measured, on the first version of this file. Hence assertion 1. + * 3. **Every workspace still yields tests.** A single canary is not enough: the structural rule this repo + * rejected (`'**\/*\/{packages,apps,tools}\/**'`) drops exactly four real files under + * `packages/core/src/tools/`, and a canary in `packages/shared` never notices. Passing by not running is a + * worse failure than the leak. + * 4. **The list reaches BOTH excludes.** `vitest list` observes `test.exclude` only, so every assertion above + * is blind to `coverage.exclude` — the half that keeps a foreign tree out of the coverage denominator. The + * phase doc's acceptance criterion names both, so the config text is checked directly. + * + * Exits non-zero so CI fails loudly. Run from anywhere: * node tools/test-isolation/check.mjs */ -import { execFileSync } from 'node:child_process'; -import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; -import { join } from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { createRequire } from 'node:module'; +import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..'); +const require = createRequire(join(repoRoot, 'package.json')); + +const CONFIG_PATH = join(repoRoot, 'vitest.config.ts'); +const FIXTURE_DIR = '__test_isolation_fixture__'; + +/** Every workspace that must still yield at least one test file — see assertion 2. */ +const WORKSPACES = [ + 'packages/core', + 'packages/llm', + 'packages/mcp', + 'packages/shared', + 'packages/db', + 'apps/cli', +]; + +function fail(message) { + console.error(`✖ ${message}`); + process.exit(1); +} /** - * Where the synthetic checkout is planted. A SYNTHETIC fixture, deliberately, not a real `git worktree`: - * a real one depends on git state and on the working directory being a repo at all, which is exactly the - * non-determinism this check is about. What matters is the SHAPE — a nested directory holding a workspace - * layout with its own test and source file — and that is reproducible anywhere. - * - * `.worktrees/` is one of the excluded names; the fixture goes there so the check exercises a pattern that is - * NOT the one that actually bit us (`.claude/`), which keeps the list honest rather than tautological. + * The plantable directory for one exclusion pattern: `**\/.claude\/**` → `.claude`, `worktrees/**` → + * `worktrees`. A pattern that does not reduce to a plain path is rejected rather than skipped — silently + * dropping an entry is exactly how this guard went blind the first time. */ -const FIXTURE_ROOT = '.worktrees/__test_isolation_fixture__'; -const FIXTURE_PKG = join(FIXTURE_ROOT, 'packages', 'probe', 'src'); +function plantableDir(pattern) { + const stripped = pattern.replace(/^\*\*\//, '').replace(/\/\*\*$/, ''); + if (/[*?[\]{}]/.test(stripped) || stripped.length === 0) { + fail( + `REPO_LOCAL_CHECKOUTS entry ${JSON.stringify(pattern)} is not a plantable directory path, so this ` + + `guard cannot prove it works.\n Use a form like '**//**' or '/**', or add an explicit ` + + `fixture location here.`, + ); + } + return stripped; +} + +// --- 1. Read the list from the config itself, so the two cannot drift ----------------------------- + +const configText = readFileSync(CONFIG_PATH, 'utf8'); +const listMatch = /export const REPO_LOCAL_CHECKOUTS = \[([^\]]*)\]/.exec(configText); +if (listMatch === null) { + fail(`cannot find \`export const REPO_LOCAL_CHECKOUTS\` in ${relative(repoRoot, CONFIG_PATH)}`); +} +const patterns = [...listMatch[1].matchAll(/'([^']+)'/g)].map((m) => m[1]); +if (patterns.length === 0) { + fail( + `REPO_LOCAL_CHECKOUTS is EMPTY in ${relative(repoRoot, CONFIG_PATH)}.\n` + + ` A repo-local second checkout would be collected by every root run and counted in coverage.`, + ); +} + +// --- 2. The list must reach BOTH excludes (assertion 3) -------------------------------------------- + +const spreadCount = (configText.match(/\.\.\.REPO_LOCAL_CHECKOUTS/g) ?? []).length; +if (spreadCount < 2) { + fail( + `REPO_LOCAL_CHECKOUTS is spread ${spreadCount} time(s) in ${relative(repoRoot, CONFIG_PATH)}; both ` + + `\`test.exclude\` and \`coverage.exclude\` need it.\n` + + ` \`test.exclude\` stops a foreign tree's TESTS from running; \`coverage.exclude\` stops its SOURCES ` + + `from entering the coverage denominator.\n \`vitest list\` cannot see the second one, which is why it ` + + `is checked here as text.`, + ); +} + +// --- 3. Plant one fixture per entry, collect, assert ------------------------------------------------ -/** A file every real run must find, so an over-broad exclude cannot pass this check. */ -const CANARY = 'packages/shared/src/run-event.test.ts'; +const fixtures = patterns.map((pattern) => ({ + pattern, + root: join(repoRoot, plantableDir(pattern), FIXTURE_DIR), + parent: join(repoRoot, plantableDir(pattern)), + parentExisted: existsSync(join(repoRoot, plantableDir(pattern))), +})); -function plantFixture() { - mkdirSync(FIXTURE_PKG, { recursive: true }); - writeFileSync(join(FIXTURE_PKG, 'probe.ts'), 'export const probe = 1;\n'); +/** + * A SYNTHETIC checkout, deliberately, not a real `git worktree`: a real one depends on git state and on the + * working directory being a repo at all, which is the exact non-determinism `CR-90` is about. What matters is + * the SHAPE — a nested workspace layout with its own test and source file — and that is reproducible anywhere. + */ +function plant(fixture) { + rmSync(fixture.root, { recursive: true, force: true }); // self-healing after a Ctrl-C'd earlier run + const src = join(fixture.root, 'packages', 'probe', 'src'); + mkdirSync(src, { recursive: true }); + writeFileSync(join(src, 'probe.ts'), 'export const probe = 1;\n'); writeFileSync( - join(FIXTURE_PKG, 'probe.test.ts'), + join(src, 'probe.test.ts'), [ "import { describe, expect, it } from 'vitest';", "import { probe } from './probe.js';", '', - // If the exclusion regresses, this runs in the root suite and its message says why. - "describe('__test_isolation_fixture__', () => {", + `describe('${FIXTURE_DIR}', () => {`, " it('MUST NOT be collected by a root run — see tools/test-isolation', () => {", ' expect(probe).toBe(1);', ' });', @@ -59,50 +144,122 @@ function plantFixture() { ); } +function clear(fixture) { + rmSync(fixture.root, { recursive: true, force: true }); + // Do not leave an empty `worktrees/` or `.worktrees/` behind that this guard itself created. + if ( + !fixture.parentExisted && + existsSync(fixture.parent) && + readdirSync(fixture.parent).length === 0 + ) { + rmSync(fixture.parent, { recursive: true, force: true }); + } +} + +/** + * Resolve vitest through the module graph and spawn it with `process.execPath`, never a bare `npx`. Spawning + * `npx` would search a writeable `PATH`, need `shell: true` for the Windows `.cmd` shim, and can reach for the + * registry when it cannot resolve locally — the convention `tools/coverage-gate/run.mjs` documents. + */ function collectedFiles() { - // `--filesOnly` keeps this cheap: Vitest resolves the file set without importing a single test module. - const out = execFileSync('npx', ['vitest', 'list', '--filesOnly'], { + let vitestEntry; + try { + const pkgPath = require.resolve('vitest/package.json'); + vitestEntry = join(dirname(pkgPath), require('vitest/package.json').bin.vitest); + } catch (error) { + fail(`cannot resolve the vitest entry point: ${error.message}`); + } + // `--json` returns absolute `file` paths, so nothing depends on the reporter's text layout (a `projects` + // config prefixes `[name] `) or on the platform separator. `--filesOnly` keeps it cheap: the file set is + // resolved without importing a single test module. + const result = spawnSync(process.execPath, [vitestEntry, 'list', '--filesOnly', '--json'], { + cwd: repoRoot, encoding: 'utf8', - stdio: ['ignore', 'pipe', 'pipe'], + shell: false, }); - return out - .split('\n') - .map((line) => line.trim()) - .filter((line) => line.endsWith('.test.ts') || line.endsWith('.test.tsx')); + if (result.status !== 0) { + fail(`\`vitest list\` failed (exit ${result.status}):\n${result.stderr ?? ''}`); + } + let parsed; + try { + parsed = JSON.parse(result.stdout); + } catch { + fail(`\`vitest list --json\` did not return JSON:\n${result.stdout.slice(0, 400)}`); + } + return parsed.map((entry) => relative(repoRoot, entry.file).split(sep).join('/')); } let files; try { - plantFixture(); + for (const fixture of fixtures) plant(fixture); files = collectedFiles(); } finally { - // Always, even on a throw — a lingering fixture would be collected by the NEXT run and turn a clean tree - // into a confusing red. - rmSync(FIXTURE_ROOT, { recursive: true, force: true }); + for (const fixture of fixtures) clear(fixture); } -const leaked = files.filter((f) => f.includes('__test_isolation_fixture__')); +// --- 4. The PRIMARY assertion: no collected file may live in a second checkout, listed or not ------ +// +// The fixture probes above prove that every LISTED location is excluded. They cannot prove the list is +// COMPLETE — and that distinction is not academic. Because the fixtures are derived from the list, deleting +// `'**/.claude/**'` also deletes its probe: the guard then re-collects all 234 foreign suites and reports +// green. Measured, on the first version of this file. +// +// So the real test is structural and runs against the filesystem rather than a glob (which is what makes it +// safe here — `vitest.config.ts` cannot express this, a script can): a second checkout of a pnpm monorepo +// carries its own `pnpm-workspace.yaml`. Any collected file with such an ancestor below the repo root is in a +// tree that is not ours, whatever the exclusion list happens to say. +function nestedCheckoutOf(fileRel) { + const parts = fileRel.split('/'); + for (let i = 1; i < parts.length; i += 1) { + const candidate = parts.slice(0, i).join('/'); + if (existsSync(join(repoRoot, candidate, 'pnpm-workspace.yaml'))) return candidate; + } + return undefined; +} + +const foreign = new Map(); +for (const file of files) { + const root = nestedCheckoutOf(file); + if (root !== undefined) foreign.set(root, (foreign.get(root) ?? 0) + 1); +} +if (foreign.size > 0) { + fail( + `${[...foreign.values()].reduce((a, b) => a + b, 0)} collected file(s) belong to a SECOND checkout of ` + + `this repo:\n` + + [...foreign] + .map(([root, n]) => ` ${root}/ (${n} file(s), has its own pnpm-workspace.yaml)`) + .join('\n') + + `\n Add it to REPO_LOCAL_CHECKOUTS in vitest.config.ts.` + + `\n This check does not consult that list — it walks the collected paths — so it stays true even when` + + `\n the list is the thing that is wrong.`, + ); +} + +const leaked = files.filter((f) => f.includes(FIXTURE_DIR)); if (leaked.length > 0) { - console.error( - `✖ a repo-local checkout leaked into the root test run (${leaked.length} file(s)):\n` + + const missed = fixtures + .filter((fx) => leaked.some((f) => f.startsWith(`${plantableDir(fx.pattern)}/`))) + .map((fx) => fx.pattern); + fail( + `a repo-local checkout leaked into the root test run (${leaked.length} file(s)):\n` + leaked.map((f) => ` ${f}`).join('\n') + - `\n Add its directory to REPO_LOCAL_CHECKOUTS in vitest.config.ts.` + + `\n Not excluded despite being listed: ${missed.map((p) => JSON.stringify(p)).join(', ')}` + `\n A root run must see this repo's tests and nothing else — a second checkout's suites are not ours` + `\n to run, and its sources are not ours to count against the coverage floor.`, ); - process.exit(1); } -if (!files.includes(CANARY)) { - console.error( - `✖ the canary test file was NOT collected: ${CANARY}` + +const silent = WORKSPACES.filter((ws) => !files.some((f) => f.startsWith(`${ws}/`))); +if (silent.length > 0) { + fail( + `these workspaces yielded NO test files: ${silent.join(', ')}` + `\n REPO_LOCAL_CHECKOUTS (vitest.config.ts) is excluding real tests. That is worse than the leak it` + `\n prevents: the suite goes green by not running. Narrow the pattern.` + - `\n Collected ${files.length} file(s).`, + `\n Collected ${files.length} file(s) in total.`, ); - process.exit(1); } console.log( - `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout.`, + `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout ` + + `(${patterns.length} location(s) probed, ${WORKSPACES.length} workspaces verified non-empty).`, ); diff --git a/vitest.config.ts b/vitest.config.ts index 56076065..22d45cf1 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -6,20 +6,28 @@ import { defaultExclude, defineConfig } from 'vitest/config'; * * This is not hypothetical housekeeping. Measured on 2026-08-10: a root `vitest list` found **472** test * files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling checkout, - * complete with its own sources. So a repo-wide run was executing a foreign tree's tests, and `pnpm coverage` - * was measuring its sources as 0%-covered files against the engine floor. + * complete with its own sources. So a repo-wide run was executing a foreign tree's tests, and the coverage + * report was counting its sources (48.85% lines, against 96.95% once excluded). * * It hides well: `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, * so `git status` stays clean, a fresh clone does not even carry the rule, and Vitest never consulted git in - * the first place. + * the first place. That also means CI, which checks out clean, never saw any of it — this list defends the + * DEVELOPER's run, and the guard in `tools/test-isolation` is the part that defends CI. * * **A list of named locations, deliberately, not a structural predicate.** The obvious general rule — "a * workspace root nested inside another one", `'**\/*\/{packages,apps,tools}\/**'` — is wrong: it also matches * this repo's own `packages/core/src/tools/*.test.ts`, because `**` happily eats `packages/core` and leaves - * `src` for the `*`. A rule that silently drops real tests is worse than the leak it closes. `tools/test-isolation` - * is the regression that proves this list actually works. + * `src` for the `*`. Verified: it drops exactly four real test files. A rule that silently drops real tests is + * worse than the leak it closes. + * + * **Each entry must be a plantable directory path**, not an arbitrary glob: `tools/test-isolation` imports + * this list and plants one fixture per entry, so a new entry is guarded automatically and the guard cannot + * drift out of step with the list. The two dotted names are matched anywhere (`**\/`); the undotted one is + * ROOT-ANCHORED on purpose — a product whose CLI manages git checkouts could plausibly grow + * `apps/cli/src/worktrees/`, and `**\/worktrees\/**` would silently delete its tests from collection and its + * sources from coverage. */ -const REPO_LOCAL_CHECKOUTS = ['**/.claude/**', '**/.worktrees/**', '**/worktrees/**']; +export const REPO_LOCAL_CHECKOUTS = ['**/.claude/**', '**/.worktrees/**', 'worktrees/**']; /** * Root, workspace-aware Vitest config. Per-package `test` scripts run `vitest run` @@ -59,9 +67,18 @@ export default defineConfig({ // package-scoped `--coverage` run no longer reports a false 0%. Apps stay smoke-only (excluded). include: ['**/src/**/*.ts'], // Both halves matter and they fail differently: `test.exclude` stops a foreign tree's TESTS from - // running, this stops its SOURCES from being counted. Without it a nested checkout's `packages/*/src/**` - // matches the `include` above and lands in the report as 0%-covered files, dragging the per-package - // threshold below its floor for a reason no one can see in the diff. + // running, this stops its SOURCES from being counted. `test-exclude` globs with `dot: true`, so the + // `include` above reaches straight into `.claude/` and a nested checkout's `packages/*/src/**` lands in + // the report as 0%-covered files. Measured: 48.85% lines with the leak, 96.95% without. + // + // **It corrupts the reported number and the html/lcov artifact — it does NOT trip a threshold, and an + // earlier version of this comment wrongly said it did.** Verified against vitest's own matcher: the + // per-package thresholds below are matched ROOT-RELATIVE with no implicit leading `**`, so + // `.claude/worktrees/x/packages/llm/src/a.ts` matches none of them and falls into the `global` group, + // which sets no thresholds and is skipped. The counterfactual run exits 0 at 48.85%. Stated precisely + // because this line becomes genuinely threshold-load-bearing the moment anyone adds a global threshold, + // and a maintainer deciding whether it is still needed should not be reading a mechanism that does not + // exist. exclude: ['**/*.test.ts', '**/*.test.tsx', '**/apps/**', ...REPO_LOCAL_CHECKOUTS], // The enforced Phase-1 engine floor, scoped per-glob so it targets only the built engine // package(s) and never the not-yet-90% shared/db or the unbuilt core. From 03ac4627979339c74b0b04476bc659ca09e9cb0f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 23:17:16 +0300 Subject: [PATCH 16/26] =?UTF-8?q?fix(repo):=20fold=20the=20Sonnet=20review?= =?UTF-8?q?=20of=20CR-90=20=E2=80=94=20the=20primary=20assertion=20was=20n?= =?UTF-8?q?ever=20exercised?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two HIGH findings, both in the thing I had just called the fix. **The primary assertion had zero coverage of its own detection logic.** The fold in `3d8f237` replaced the list-based check with a structural one — any collected file whose ancestor carries its own `pnpm-workspace.yaml` is in a second checkout — and then never ran it. `plant()` wrote only `packages/probe/src/{probe.ts, probe.test.ts}`, no workspace marker, so the fixtures did not shape-match a real checkout for THAT assertion; and the listed fixtures are (correctly) excluded from collection, so the walk only ever saw this repo's own paths and returned undefined every time. The one demonstration that it worked was a manual check against a real worktree that happened to be sitting on this machine — not repeatable, absent on a fresh clone and in CI. A refactor breaking the walk would have shipped green forever. Closed with a pure-function self-test that runs BEFORE the guard trusts itself: it plants a positive control carrying a `pnpm-workspace.yaml` at a deliberately UNLISTED path, asserts the detector flags it, and asserts it does not flag `packages/core/src/dag.test.ts`. Three break-verified mutations now redden it — `i = 0` instead of `1`, the wrong marker filename, and a detector that never matches. The listed fixtures also carry the marker now, so a broken exclusion trips both assertions and the error names the cause twice. **`process.exit(1)` from inside the try skipped the `finally`, leaking fixtures.** Node does not unwind a `finally` on `process.exit`, and `collectedFiles()` called `fail()` in three places — a broken lockfile, a vitest crash, unparseable JSON. All three left every fixture on disk. Two parents were gitignored; the third was not: `plantableDir('**/.claude/**')` strips to `.claude`, so the fixture lands as a SIBLING of `.claude/worktrees/`, which `.claude/worktrees/` does not cover. The next plain `pnpm test` would then have collected it — CR-90's own defect, self-inflicted by its guard's crash path. Those paths now throw; a single top-level catch cleans up and then reports. Verified with a forced mid-run failure: exit 1, no fixtures left behind. The two scratch directory names are also gitignored now, as belt-and-braces. **Two overclaims corrected, per the phase's "record rather than assert" clause:** - "A second checkout ALWAYS carries its own `pnpm-workspace.yaml`" — true for `git worktree add`, `git clone` and a full copy, false for a sparse checkout, a `--no-checkout` worktree, or a partial rsync of `packages/**`. Such a tree would pass the primary assertion in silence. Recorded as a limitation in both the guard and the phase doc, with the second marker that would close it. - `WORKSPACES` was a hand-maintained six-entry array — the exact shape the primary assertion was rebuilt to escape. It is now DERIVED by walking the tree for `*.test.ts`, which is non-circular: the filesystem says which workspaces have tests, and Vitest must then have found them. `packages/ui`'s first test will be checked without anyone remembering to add it. Confirmed clean by the review and left alone: `i = 1` is deliberate and right (`i = 0` resolves to the repo's OWN marker and would flag every file); the 48.85/96.95, four-files-under-`core/src/tools`, 0.4s and "`vitest list` observes `test.exclude` only" claims all reproduce exactly; no tracked file is shadowed by the new `.gitignore` entries. Two LOWs acknowledged and not changed: `.gitignore`'s `.claude/worktrees/` is root-anchored while its vitest counterpart is not (`.claude/` only exists at the root); and `spreadCount < 2` proves two spreads exist, not that they are in the two right places — a pathological double-mutation is absorbed by the fixture check, which the documented break-verify confirms. pnpm run ci EXIT=0. Refs: CR-90 Co-Authored-By: Claude Opus 5 --- .gitignore | 5 + ...hase-2.6.5-core-reliability-remediation.md | 7 + tools/test-isolation/check.mjs | 149 ++++++++++++++---- 3 files changed, 134 insertions(+), 27 deletions(-) diff --git a/.gitignore b/.gitignore index 2afc7fb6..84b48485 100644 --- a/.gitignore +++ b/.gitignore @@ -61,6 +61,11 @@ coverage-tmp/ .claude/worktrees/ .worktrees/ /worktrees/ +# The guard's own scratch directories. It clears them explicitly on every exit path, including a failure — +# but `.claude/__test_isolation_fixture__/` is a SIBLING of `.claude/worktrees/`, so the entry above would not +# have covered it if a crash ever did leak one, and a leaked fixture is CR-90's defect self-inflicted. +__test_isolation_fixture__/ +__test_isolation_detector_probe__/ # Private analysis — never commit, never reference from tracked files docs/analysis/private/ diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index 6299ddcc..077e1af7 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -816,6 +816,13 @@ probe, and the guard reports green while re-collecting all 234 foreign suites. T therefore structural and consults no list — any collected file whose ancestor carries its own `pnpm-workspace.yaml` is in a second checkout. +**One recorded limitation, not a property.** "A second checkout always carries its own `pnpm-workspace.yaml`" +holds for `git worktree add`, `git clone` and a full copy — every case seen here — but not for a sparse +checkout, a `--no-checkout` worktree, or a partial rsync of `packages/**`. Such a tree would be collected with +nothing in its ancestry to detect, and the primary assertion would pass in silence, leaving only the +list-based checks. Nothing in this repo produces one today; if that changes, the predicate needs a second +marker. Stated here and in the guard's own docblock because the first version asserted it as an absolute. + ### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium It pins that the live stream reported success; it does not prove durable truth after a restart. **Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs index 40bd9d70..a5194d48 100644 --- a/tools/test-isolation/check.mjs +++ b/tools/test-isolation/check.mjs @@ -49,15 +49,42 @@ const require = createRequire(join(repoRoot, 'package.json')); const CONFIG_PATH = join(repoRoot, 'vitest.config.ts'); const FIXTURE_DIR = '__test_isolation_fixture__'; -/** Every workspace that must still yield at least one test file — see assertion 2. */ -const WORKSPACES = [ - 'packages/core', - 'packages/llm', - 'packages/mcp', - 'packages/shared', - 'packages/db', - 'apps/cli', -]; +/** Where the detector's own self-test plants its positive control. Deliberately NOT an excluded location. */ +const PROBE_DIR = '__test_isolation_detector_probe__'; + +/** + * Every workspace that holds at least one `*.test.ts` ON DISK — DERIVED, never hand-listed. + * + * A hardcoded array is the same shape the primary assertion was rebuilt to escape: it can only prove the + * places someone remembered. If `packages/ui` gains its first test tomorrow, a static list silently stops + * checking it. Walking the tree is non-circular — the filesystem says which workspaces have tests, and Vitest + * must then have found them. + */ +function workspacesWithTests() { + const roots = ['packages', 'apps']; + const found = []; + for (const root of roots) { + const rootDir = join(repoRoot, root); + if (!existsSync(rootDir)) continue; + for (const name of readdirSync(rootDir)) { + const src = join(rootDir, name, 'src'); + if (existsSync(src) && hasTestFile(src)) found.push(`${root}/${name}`); + } + } + return found; +} + +function hasTestFile(dir) { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.name === 'node_modules' || entry.name === 'dist') continue; + if (entry.isDirectory()) { + if (hasTestFile(join(dir, entry.name))) return true; + } else if (entry.name.endsWith('.test.ts') || entry.name.endsWith('.test.tsx')) { + return true; + } + } + return false; +} function fail(message) { console.error(`✖ ${message}`); @@ -127,6 +154,9 @@ function plant(fixture) { rmSync(fixture.root, { recursive: true, force: true }); // self-healing after a Ctrl-C'd earlier run const src = join(fixture.root, 'packages', 'probe', 'src'); mkdirSync(src, { recursive: true }); + // The workspace marker too, so the fixture shape-matches a real second checkout for BOTH assertions: if an + // exclusion breaks, the structural check fires alongside the leak check and the error names the cause twice. + writeFileSync(join(fixture.root, 'pnpm-workspace.yaml'), "packages:\n - 'packages/*'\n"); writeFileSync(join(src, 'probe.ts'), 'export const probe = 1;\n'); writeFileSync( join(src, 'probe.test.ts'), @@ -167,7 +197,12 @@ function collectedFiles() { const pkgPath = require.resolve('vitest/package.json'); vitestEntry = join(dirname(pkgPath), require('vitest/package.json').bin.vitest); } catch (error) { - fail(`cannot resolve the vitest entry point: ${error.message}`); + // THROW, never `process.exit`, from anywhere inside the try/finally below. `process.exit` does not unwind + // a `finally` in Node, so exiting here would leave all three fixtures on disk — and the `.claude/` one is + // a SIBLING of `.claude/worktrees/`, so the `.gitignore` entries do not cover it. The next plain + // `pnpm test` would then collect the leftover fixture: CR-90's own defect, self-inflicted by its guard's + // crash path. Verified: `process.exit` inside a `try` skips the `finally` body. + throw new Error(`cannot resolve the vitest entry point: ${error.message}`); } // `--json` returns absolute `file` paths, so nothing depends on the reporter's text layout (a `projects` // config prefixes `[name] `) or on the platform separator. `--filesOnly` keeps it cheap: the file set is @@ -178,24 +213,92 @@ function collectedFiles() { shell: false, }); if (result.status !== 0) { - fail(`\`vitest list\` failed (exit ${result.status}):\n${result.stderr ?? ''}`); + throw new Error(`\`vitest list\` failed (exit ${result.status}):\n${result.stderr ?? ''}`); } let parsed; try { parsed = JSON.parse(result.stdout); } catch { - fail(`\`vitest list --json\` did not return JSON:\n${result.stdout.slice(0, 400)}`); + throw new Error(`\`vitest list --json\` did not return JSON:\n${result.stdout.slice(0, 400)}`); } return parsed.map((entry) => relative(repoRoot, entry.file).split(sep).join('/')); } +/** + * A second checkout of a pnpm monorepo carries its own `pnpm-workspace.yaml` at its root. Any collected file + * with such an ancestor BELOW the repo root is in a tree that is not ours. + * + * `i = 1`, not `0`, and that is load-bearing: `i = 0` yields `candidate = ''`, whose `pnpm-workspace.yaml` is + * the repo's OWN — every file would report as nested inside itself. + * + * **Recorded limitation, not a property.** "A second checkout always carries one" holds for `git worktree + * add`, `git clone` and a full copy — every case seen here — but NOT for a sparse checkout, a + * `--no-checkout` worktree, or a partial rsync of `packages/**`. Those would be collected with nothing in + * their ancestry to detect, and this assertion would pass in silence, leaving only the list-based checks + * below. Nothing in this repo produces such a tree today; if that changes, this predicate needs a second + * marker (a `package.json` whose `name` matches the root's would be the obvious one). + */ +function nestedCheckoutOf(fileRel) { + const parts = fileRel.split('/'); + for (let i = 1; i < parts.length; i += 1) { + const candidate = parts.slice(0, i).join('/'); + if (existsSync(join(repoRoot, candidate, 'pnpm-workspace.yaml'))) return candidate; + } + return undefined; +} + +/** + * Prove the detector above actually fires, BEFORE trusting it to pass. + * + * Without this, `nestedCheckoutOf`'s match branch is never executed by the guard: the listed fixtures are + * (correctly) excluded from collection, so the structural walk only ever sees this repo's own paths and + * returns `undefined` every time. The one demonstration that it works was a manual check against a real + * worktree that happened to be present on one developer's machine — not repeatable, and absent on a fresh + * clone or in CI. A refactor that broke the ancestor walk would have shipped green forever. + * + * A pure-function self-test rather than a second `vitest list`: it exercises the same walk, the same `i = 1` + * boundary and the same marker file, for none of the cost. + */ +function selfTestDetector() { + const probe = join(repoRoot, PROBE_DIR); + try { + mkdirSync(join(probe, 'packages', 'probe', 'src'), { recursive: true }); + writeFileSync(join(probe, 'pnpm-workspace.yaml'), "packages:\n - 'packages/*'\n"); + const inside = `${PROBE_DIR}/packages/probe/src/probe.test.ts`; + if (nestedCheckoutOf(inside) !== PROBE_DIR) { + throw new Error( + `the nested-checkout detector FAILED its own self-test: it did not flag ${inside}, which sits under a` + + `\n directory carrying its own pnpm-workspace.yaml. The primary assertion is not working, so a` + + `\n green result from this guard would mean nothing.`, + ); + } + // And it must not fire on our own tree — an over-eager detector would fail every run for the wrong reason. + if (nestedCheckoutOf('packages/core/src/dag.test.ts') !== undefined) { + throw new Error( + "the nested-checkout detector flags this repo's OWN files. Check the `i = 1` loop bound.", + ); + } + } finally { + rmSync(probe, { recursive: true, force: true }); + } +} + +const cleanupAll = () => { + for (const fixture of fixtures) clear(fixture); +}; + let files; try { + selfTestDetector(); for (const fixture of fixtures) plant(fixture); files = collectedFiles(); -} finally { - for (const fixture of fixtures) clear(fixture); +} catch (error) { + // Explicit cleanup BEFORE reporting, rather than a `finally` plus a `process.exit` inside the try — Node + // does not unwind a `finally` on `process.exit`, and a leaked fixture is CR-90's own defect self-inflicted. + cleanupAll(); + fail(error instanceof Error ? error.message : String(error)); } +cleanupAll(); // --- 4. The PRIMARY assertion: no collected file may live in a second checkout, listed or not ------ // @@ -205,17 +308,8 @@ try { // green. Measured, on the first version of this file. // // So the real test is structural and runs against the filesystem rather than a glob (which is what makes it -// safe here — `vitest.config.ts` cannot express this, a script can): a second checkout of a pnpm monorepo -// carries its own `pnpm-workspace.yaml`. Any collected file with such an ancestor below the repo root is in a -// tree that is not ours, whatever the exclusion list happens to say. -function nestedCheckoutOf(fileRel) { - const parts = fileRel.split('/'); - for (let i = 1; i < parts.length; i += 1) { - const candidate = parts.slice(0, i).join('/'); - if (existsSync(join(repoRoot, candidate, 'pnpm-workspace.yaml'))) return candidate; - } - return undefined; -} +// safe here — `vitest.config.ts` cannot express this, a script can). `nestedCheckoutOf` is defined above, +// beside the self-test that proves it fires. const foreign = new Map(); for (const file of files) { @@ -249,7 +343,8 @@ if (leaked.length > 0) { ); } -const silent = WORKSPACES.filter((ws) => !files.some((f) => f.startsWith(`${ws}/`))); +const expected = workspacesWithTests(); +const silent = expected.filter((ws) => !files.some((f) => f.startsWith(`${ws}/`))); if (silent.length > 0) { fail( `these workspaces yielded NO test files: ${silent.join(', ')}` + @@ -261,5 +356,5 @@ if (silent.length > 0) { console.log( `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout ` + - `(${patterns.length} location(s) probed, ${WORKSPACES.length} workspaces verified non-empty).`, + `(${patterns.length} location(s) probed, ${expected.length} workspaces with tests on disk all collected).`, ); From 6a9eec9a3ffd2d49266783a945d80c78a2b4256e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 23:25:46 +0300 Subject: [PATCH 17/26] =?UTF-8?q?feat(core):=20add=20the=20durable-truth?= =?UTF-8?q?=20oracle=20=E2=80=94=20four=20views=20of=20one=20run=20must=20?= =?UTF-8?q?agree?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CR-91`, the second half of phase 2.6.5's instrument. The e2e harness could prove a run STREAMED a terminal and that it said what the test expected. That is one view, taken in-process while the engine that produced it is still alive, and it cannot see the failure class this phase exists for: a caller told `completed` while the durable log says `failed`, a restart reconciling to a different terminal, or a checkpoint fold that disagrees with the terminal so a resume does the wrong work. All three are invisible to a live assertion. `checkDurableTruth` compares four views — the live terminal, the durable history, the history a FRESH engine leaves behind after `reconcile()`, and the status `reconstructCheckpointState` derives from the log alone — and returns a structured verdict. `formatDurableTruth` renders the diff. **A verdict rather than a throw**, so callers can assert on the specific disagreement and the oracle can have its own tests. A thrower would force every caller into one message and would itself be awkward to test — which matters here, because this is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` will be proven with, and an unproven instrument proves nothing. Two design points, both deliberate: - **Envelope fields are excluded from the comparison.** `timestamp` and `sequenceNumber` move on every restart; comparing them would make every run look like a disagreement while catching nothing. What must agree is the terminal's TYPE and the payload a consumer acts on — outputs, error code, cost. One test pins that envelope drift is NOT a disagreement, so a later "make it stricter" refactor has to argue with a red test. - **The checkpoint view is not decoration.** A resume seeds itself from the fold, so a fold disagreeing with the durable terminal means a resumed run does the wrong work even when every event on disk is intact. Ten unit tests cover the detection logic: the headline live-says-completed / history-says-failed case, a SAME-TYPE terminal whose payload differs (a type-only comparison would pass it), a restart that changes the terminal, a reconcile that adds a second terminal to a closed run, a fold that disagrees, a run that streamed a terminal and persisted none, and the async-store shape. Three e2e tests apply it to real engine runs on all three terminals, restarting a fresh engine over the same store to reconcile — which also pins that `reconcile()` produces NOTHING for a run that already closed, since repairing a run that died WITHOUT a terminal is its job and touching one that landed is the defect. Break-verified: stopping the engine from persisting terminals reddens all three e2e tests with the four-view diff. Coverage 98.36% lines on the new module. pnpm run ci EXIT=0, pnpm coverage EXIT=0. Refs: CR-91 Co-Authored-By: Claude Opus 5 --- ...hase-2.6.5-core-reliability-remediation.md | 20 +- .../core/src/engine/durable-truth.test.ts | 179 ++++++++++++++ packages/core/src/engine/durable-truth.ts | 219 ++++++++++++++++++ .../src/engine/m2-e2e-harness.e2e.test.ts | 88 +++++++ 4 files changed, 505 insertions(+), 1 deletion(-) create mode 100644 packages/core/src/engine/durable-truth.test.ts create mode 100644 packages/core/src/engine/durable-truth.ts diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index 077e1af7..2104f253 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -823,12 +823,30 @@ nothing in its ancestry to detect, and the primary assertion would pass in silen list-based checks. Nothing in this repo produces one today; if that changes, the predicate needs a second marker. Stated here and in the guard's own docblock because the first version asserted it as an absolute. -### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium +### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium · ✅ CLOSED 2026-08-10 It pins that the live stream reported success; it does not prove durable truth after a restart. **Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same terminal type and payload. **This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with**, so it lands before them: a spine asserted with the harness the reviews already found insufficient is not asserted. +**Closed by `packages/core/src/engine/durable-truth.ts`** — `checkDurableTruth` compares four views (live +terminal, durable history, the history a FRESH engine leaves after `reconcile()`, and the status the +checkpoint fold derives) and returns a structured verdict rather than throwing, so callers can assert on the +specific disagreement and the oracle can have its own tests. `formatDurableTruth` renders the diff. + +Two design points worth carrying into the spine work: + +- **Envelope fields are excluded from the comparison on purpose.** `timestamp` and `sequenceNumber` move on + every restart; comparing them would make every run look like a disagreement while catching nothing. What + must agree is the terminal's TYPE and the payload a consumer acts on — outputs, error code, cost. +- **The checkpoint view is not decoration.** A resume seeds itself from the fold, so a fold that disagrees + with the durable terminal means a resumed run does the wrong work even when every event on disk is intact. + That is the `CR-92` shape the live stream cannot see. + +Ten unit tests cover the detection logic (including the headline case: live says `completed`, history says +`failed`), and three e2e tests apply the oracle to real engine runs on all three terminals. Break-verified: +stopping the engine from persisting terminals reddens all three e2e tests with a readable four-view diff. + ### CR-92 — Terminal persistence failure lets live and durable truth diverge · High · in the durability spine The engine can complete the delivery chain for a terminal event even when its persistence failed, and reconciliation may later produce a different terminal than the original. Media reclaim can run before the diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts new file mode 100644 index 00000000..812c0e08 --- /dev/null +++ b/packages/core/src/engine/durable-truth.test.ts @@ -0,0 +1,179 @@ +import type { RunEvent } from '@relavium/shared'; +import { describe, expect, it } from 'vitest'; + +import { checkDurableTruth, formatDurableTruth } from './durable-truth.js'; + +const TS = '2026-01-01T00:00:00.000Z'; +const base = (sequenceNumber: number) => ({ runId: 'r1', sequenceNumber, timestamp: TS }); + +const started: RunEvent = { + type: 'run:started', + ...base(0), + workflowId: '00000000-0000-4000-8000-000000000001', + inputs: {}, + executionMode: 'local', +}; + +const completed = (seq: number, outputs: unknown = { a: 1 }, cost = 500): RunEvent => ({ + type: 'run:completed', + ...base(seq), + outputs: outputs as Record, + totalTokensUsed: { input: 1, output: 1 }, + totalCostMicrocents: cost, + durationMs: 10, +}); + +const failed = (seq: number, code: 'internal' | 'tool_failed' = 'internal'): RunEvent => ({ + type: 'run:failed', + ...base(seq), + error: { code, message: 'boom', retryable: false }, + partialOutputs: {}, +}); + +const log = (...events: RunEvent[]) => { + const stored = [started, ...events]; + return { eventsFor: () => stored, stored }; +}; + +describe('checkDurableTruth', () => { + it('AGREES when live, history, reconcile and the checkpoint all say the same thing', async () => { + const terminal = completed(1); + const { eventsFor } = log(terminal); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + reconcile: () => Promise.resolve([]), + }); + + expect(verdict.agrees).toBe(true); + expect(verdict.disagreements).toEqual([]); + expect(verdict.durableTerminalCount).toBe(1); + expect(verdict.checkpointStatus).toBe('completed'); + }); + + it('catches the headline defect: a caller told `completed` while history says `failed`', async () => { + // CR-92's exact shape. The live stream is the only thing an e2e assertion used to read, so this shipped + // invisible: the API returns success and outputs, and the durable record disagrees. + const { eventsFor } = log(failed(1)); + const verdict = await checkDurableTruth({ runId: 'r1', live: completed(1), eventsFor }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements[0]).toContain('live and durable history disagree'); + expect(formatDurableTruth(verdict)).toContain('run:completed'); + expect(formatDurableTruth(verdict)).toContain('run:failed'); + }); + + it('catches a SAME-TYPE terminal whose payload differs', async () => { + // Type-only comparison would pass this. Two runs both `completed`, different outputs and different money + // — which is the reconciliation-rewrote-the-answer case, not a crash. + const { eventsFor } = log(completed(1, { a: 1 }, 500)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: 2 }, 900), + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements[0]).toContain('live and durable history disagree'); + }); + + it('IGNORES envelope drift — a restart moves the clock, and that is not a disagreement', async () => { + const stored: RunEvent[] = [ + started, + { ...completed(1), timestamp: '2026-02-02T00:00:00.000Z' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(7), // a different sequenceNumber too + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(true); + }); + + it('catches a restart that CHANGES the terminal', async () => { + const stored: RunEvent[] = [started, completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + reconcile: () => { + // A reconcile that rewrites an already-settled run — the "may later produce a different terminal" + // half of CR-92. + stored.splice(1, 1, failed(2)); + return Promise.resolve([failed(2)]); + }, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('a restart CHANGED the terminal'))).toBe( + true, + ); + }); + + it('catches a reconcile that adds a SECOND terminal to an already-closed run', async () => { + const stored: RunEvent[] = [started, completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + reconcile: () => { + stored.push(failed(2)); + return Promise.resolve([failed(2)]); + }, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.durableTerminalCount).toBe(2); + expect(verdict.disagreements.some((d) => d.includes('exactly one closes a run'))).toBe(true); + expect(verdict.disagreements.some((d) => d.includes('must not add a second'))).toBe(true); + }); + + it('catches a checkpoint fold that disagrees with the durable terminal', async () => { + // The fold is what a resume seeds itself from, so a disagreement here means a resumed run does the wrong + // work even though every event on disk is intact. Constructed by omitting `run:started`, which makes the + // fold return `undefined` while the terminal is plainly there. + const stored: RunEvent[] = [completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.checkpointStatus).toBeUndefined(); + expect(verdict.disagreements.some((d) => d.includes('checkpoint fold'))).toBe(true); + }); + + it('catches a run that streamed a terminal and persisted NONE — the crash window', async () => { + const { eventsFor } = log(); + const verdict = await checkDurableTruth({ runId: 'r1', live: completed(1), eventsFor }); + + expect(verdict.agrees).toBe(false); + expect(verdict.durableTerminalCount).toBe(0); + expect(verdict.history).toBeUndefined(); + }); + + it('accepts an async eventsFor — the real store returns a promise', async () => { + const terminal = completed(1); + const stored = [started, terminal]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor: () => Promise.resolve(stored), + }); + + expect(verdict.agrees).toBe(true); + }); + + it('skips view 3 when no reconcile is supplied, and says so in the counts', async () => { + const terminal = failed(1, 'tool_failed'); + const { eventsFor } = log(terminal); + const verdict = await checkDurableTruth({ runId: 'r1', live: terminal, eventsFor }); + + expect(verdict.agrees).toBe(true); + expect(verdict.reconciledCount).toBe(0); + expect(verdict.afterReconcile).toEqual(verdict.history); + }); +}); diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts new file mode 100644 index 00000000..b010d217 --- /dev/null +++ b/packages/core/src/engine/durable-truth.ts @@ -0,0 +1,219 @@ +/** + * The durable-truth oracle (`CR-91`) — does a run's LIVE answer survive a restart? + * + * The e2e harness could already prove that a run streamed a terminal and that the terminal said what the test + * expected. That is one view, taken in-process, while the engine that produced it is still alive. It cannot + * catch the failure class phase 2.6.5 exists for: a caller receives `run:completed` with outputs while the + * durable log says the run failed, or a restart reconciles the same run to a DIFFERENT terminal, or a resume + * re-runs work the log already records as done. Every one of those is invisible to a live assertion. + * + * So this compares FOUR independent views of the same run and reports where they disagree: + * + * 1. **live** — the terminal the caller drained from `RunHandle.events`. + * 2. **history** — the terminal as it exists in the store, read back. + * 3. **after reconcile** — the history a *fresh* engine leaves behind after `reconcile()` on restart. It must + * be the SAME terminal: reconciliation repairs a run that died without one, and must never overwrite or + * duplicate one that already landed. + * 4. **checkpoint** — the status `reconstructCheckpointState` derives from the durable log alone. A resume + * seeds itself from this, so if it disagrees with the terminal, a resumed run does the wrong work. + * + * **It returns a verdict rather than throwing.** A thrower is hard to test and forces every caller into the + * same message; a structured verdict lets the callers that need one assert on the specific disagreement, and + * lets this module have its own tests. {@link formatDurableTruth} renders it when a caller just wants to fail + * with a readable diff. + * + * This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with, which is why it lands before + * them — a spine asserted with a harness that only reads the live stream is not asserted. + */ + +import type { RunEvent, RunStatus } from '@relavium/shared'; + +import { reconstructCheckpointState } from './checkpoint.js'; + +/** The run terminals. Exactly one closes a run (ADR-0036); the oracle's whole job is to check they agree. */ +const TERMINALS = new Set(['run:completed', 'run:failed', 'run:cancelled']); + +/** The run status each terminal implies — the checkpoint fold must arrive at the same one. */ +const STATUS_FOR: Partial> = { + 'run:completed': 'completed', + 'run:failed': 'failed', + 'run:cancelled': 'cancelled', +}; + +/** + * A terminal reduced to what must agree across views: its type, and the payload a consumer acts on. + * + * Deliberately NOT the whole event. `timestamp` and `sequenceNumber` are envelope fields the bus stamps, and + * comparing them would make every restart look like a disagreement while catching nothing — the failure this + * oracle is for is a terminal whose TYPE or OUTPUT changed, not one whose clock moved. + */ +export interface TerminalView { + readonly type: RunEvent['type']; + /** `run:completed.outputs` / `run:failed.partialOutputs`, JSON-normalized. Absent for `run:cancelled`. */ + readonly outputs?: string; + /** `run:failed.error.code`, when the terminal carries one. */ + readonly errorCode?: string; + /** The run-wide realized total the terminal reports, when it carries one. */ + readonly costMicrocents?: number; +} + +export interface DurableTruthVerdict { + readonly agrees: boolean; + readonly runId: string; + readonly live: TerminalView | undefined; + readonly history: TerminalView | undefined; + readonly afterReconcile: TerminalView | undefined; + readonly checkpointStatus: RunStatus | undefined; + /** How many terminals the durable log holds. Anything but 1 breaks exactly-one-terminal (ADR-0036). */ + readonly durableTerminalCount: number; + /** Events `reconcile()` produced. Non-empty for a run that already had a durable terminal is a defect. */ + readonly reconciledCount: number; + /** One sentence per disagreement, in the order checked. Empty iff `agrees`. */ + readonly disagreements: readonly string[]; +} + +function viewOf(event: RunEvent | undefined): TerminalView | undefined { + if (event === undefined) return undefined; + switch (event.type) { + case 'run:completed': + return { + type: event.type, + outputs: JSON.stringify(event.outputs), + costMicrocents: event.totalCostMicrocents, + }; + case 'run:failed': + return { + type: event.type, + outputs: JSON.stringify(event.partialOutputs), + errorCode: event.error.code, + ...(event.cumulativeCostMicrocents === undefined + ? {} + : { costMicrocents: event.cumulativeCostMicrocents }), + }; + case 'run:cancelled': + return { + type: event.type, + ...(event.cumulativeCostMicrocents === undefined + ? {} + : { costMicrocents: event.cumulativeCostMicrocents }), + }; + default: + return undefined; + } +} + +function terminalIn(events: readonly RunEvent[]): RunEvent | undefined { + return events.find((event) => TERMINALS.has(event.type)); +} + +function describe(view: TerminalView | undefined): string { + if (view === undefined) return 'none'; + const parts: string[] = [view.type]; + if (view.errorCode !== undefined) parts.push(`code=${view.errorCode}`); + if (view.costMicrocents !== undefined) parts.push(`cost=${view.costMicrocents}`); + if (view.outputs !== undefined) parts.push(`outputs=${view.outputs}`); + return parts.join(' '); +} + +function sameView(a: TerminalView | undefined, b: TerminalView | undefined): boolean { + if (a === undefined || b === undefined) return a === b; + return ( + a.type === b.type && + a.outputs === b.outputs && + a.errorCode === b.errorCode && + a.costMicrocents === b.costMicrocents + ); +} + +export interface DurableTruthInput { + readonly runId: string; + /** The terminal the caller drained from the live stream, or `undefined` if the stream produced none. */ + readonly live: RunEvent | undefined; + /** Read the run's persisted events — the same store the run was executed against. */ + readonly eventsFor: (runId: string) => readonly RunEvent[] | Promise; + /** + * Run `reconcile()` as a FRESH engine over that same store, exactly as a restarted process would. Omit only + * when the caller has already restarted for its own reasons; then view 3 is skipped and the verdict says so. + */ + readonly reconcile?: () => Promise; +} + +/** + * Compare the four views. Cheap enough to call at the end of every e2e run; it reads the log twice and runs + * one reconcile. + */ +export async function checkDurableTruth(input: DurableTruthInput): Promise { + const disagreements: string[] = []; + + const before = [...(await input.eventsFor(input.runId))]; + const live = viewOf(input.live); + const history = viewOf(terminalIn(before)); + + if (!sameView(live, history)) { + disagreements.push( + `live and durable history disagree: live=${describe(live)} history=${describe(history)}`, + ); + } + + let reconciledCount = 0; + let afterReconcile = history; + if (input.reconcile !== undefined) { + const reconciled = await input.reconcile(); + reconciledCount = reconciled.length; + const after = [...(await input.eventsFor(input.runId))]; + afterReconcile = viewOf(terminalIn(after)); + if (!sameView(history, afterReconcile)) { + disagreements.push( + `a restart CHANGED the terminal: before=${describe(history)} after=${describe(afterReconcile)}`, + ); + } + if (history !== undefined && reconciledCount > 0) { + disagreements.push( + `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + + `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, it must not add a second`, + ); + } + } + + const finalEvents = [...(await input.eventsFor(input.runId))]; + const terminalCount = finalEvents.filter((event) => TERMINALS.has(event.type)).length; + if (terminalCount !== 1) { + disagreements.push( + `the durable log holds ${terminalCount} terminal(s); exactly one closes a run (ADR-0036)`, + ); + } + + const checkpointStatus = reconstructCheckpointState(finalEvents)?.runStatus; + const expectedStatus = afterReconcile === undefined ? undefined : STATUS_FOR[afterReconcile.type]; + if (expectedStatus !== undefined && checkpointStatus !== expectedStatus) { + disagreements.push( + `the checkpoint fold says '${String(checkpointStatus)}' while the durable terminal says ` + + `'${expectedStatus}' — a resume seeds itself from the fold, so it would do the wrong work`, + ); + } + + return { + agrees: disagreements.length === 0, + runId: input.runId, + live, + history, + afterReconcile, + checkpointStatus, + durableTerminalCount: terminalCount, + reconciledCount, + disagreements, + }; +} + +/** Render a verdict for an assertion message — every view on its own line, so the diff reads at a glance. */ +export function formatDurableTruth(verdict: DurableTruthVerdict): string { + return [ + `durable truth for run ${verdict.runId}: ${verdict.agrees ? 'AGREES' : 'DISAGREES'}`, + ` live : ${describe(verdict.live)}`, + ` history : ${describe(verdict.history)}`, + ` after reconcile : ${describe(verdict.afterReconcile)}`, + ` checkpoint : ${String(verdict.checkpointStatus)}`, + ` terminals in log: ${verdict.durableTerminalCount}`, + ...verdict.disagreements.map((d) => ` ✖ ${d}`), + ].join('\n'); +} diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 6fd3c0ce..11638e26 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -50,6 +50,7 @@ import type { ToolDef as CoreToolDef, ToolRegistry, ToolResultPart } from '../to import { markUntrusted } from '../tools/untrusted.js'; import { reconstructCheckpointState } from './checkpoint.js'; import { WorkflowEngine } from './engine.js'; +import { checkDurableTruth, formatDurableTruth } from './durable-truth.js'; import { createInMemoryHost, InMemoryRunStore } from './execution-host.js'; import { createStandardNodeExecutor } from './node-handlers/dispatcher.js'; import type { RunHandle } from './run-handle.js'; @@ -1063,6 +1064,93 @@ describe('M2 — end-to-end Node harness (1.U)', () => { ); }); + // --- The durable-truth oracle (CR-91) -------------------------------------------------------------- + // + // Everything above this point asserts on the LIVE stream. That is one view, taken in-process while the + // engine that produced it is still alive, and it cannot see the failure class this phase exists for: a + // caller told `completed` while the log says `failed`, a restart reconciling to a different terminal, or a + // checkpoint fold that disagrees with the terminal and would make a resume do the wrong work. + // + // `checkDurableTruth` compares four views. These tests apply it to the three terminals a run can reach, on + // real engine runs rather than synthetic logs — `durable-truth.test.ts` covers the oracle's own detection + // logic, this covers the engine actually satisfying it. + + /** Run the oracle over a finished run, restarting a FRESH engine over the same store to reconcile. */ + async function assertDurableTruth( + host: Host, + store: InMemoryRunStore, + events: readonly RunEvent[], + resolveProvider: (id: ProviderId) => LlmProvider | undefined, + ) { + const runId = events[0]?.runId; + if (runId === undefined) expect.unreachable('no run:started'); + const verdict = await checkDurableTruth({ + runId, + live: events.at(-1), + eventsFor: (id) => store.eventsFor(id), + // A fresh engine over the SAME store, which is what a restarted process is. + reconcile: () => buildEngine(host, resolveProvider).reconcile(), + }); + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + return verdict; + } + + it('durable truth: a COMPLETED run survives a restart unchanged (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + const provider = scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]); + const { events } = await drive( + buildEngine(host, () => provider).start({ workflow: HAPPY_PATH, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:completed'); + const verdict = await assertDurableTruth(host, store, events, () => provider); + // A run that already closed must give reconcile() nothing to do — repairing a run that DIED without a + // terminal is its job; touching one that landed is the defect. + expect(verdict.reconciledCount).toBe(0); + expect(verdict.durableTerminalCount).toBe(1); + }); + + it('durable truth: a FAILED run agrees across all four views (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + // A fatal provider error — no retry budget on HAPPY_PATH, so the node fails and the run follows. + const provider = scriptedProvider([ + [ + { + type: 'error', + error: { kind: 'auth', retryable: false, provider: 'anthropic', message: 'nope' }, + }, + ], + ]); + const { events } = await drive( + buildEngine(host, () => provider).start({ workflow: HAPPY_PATH, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:failed'); + const verdict = await assertDurableTruth(host, store, events, () => provider); + expect(verdict.checkpointStatus).toBe('failed'); + expect(verdict.history?.errorCode).toBe(verdict.live?.errorCode); + }); + + it('durable truth: a CANCELLED run agrees across all four views (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + const provider = scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + const events: RunEvent[] = []; + for await (const event of handle.events) { + events.push(event); + if (event.type === 'node:started' && event.nodeId === 'work') engine.cancel(handle.runId); + } + + expect(events.at(-1)?.type).toBe('run:cancelled'); + await assertDurableTruth(host, store, events, () => provider); + }); + // --- The realized-cost ledger's barriers (ADR-0076 / ADR-0077) ------------------------------------ // // Every test here runs on HAPPY_PATH, which declares NO `budget` — deliberately. ADR-0077 §5's whole From cc58dbac43667fd0ded25853f16031228db50acc Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 10 Aug 2026 23:46:11 +0300 Subject: [PATCH 18/26] =?UTF-8?q?fix(core):=20fold=20the=20Opus=20review?= =?UTF-8?q?=20of=20CR-91=20=E2=80=94=20the=20instrument=20could=20not=20in?= =?UTF-8?q?strument=20the=20spine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourteen findings. The central one invalidated the claim this item exists for. **It could not express CR-10, CR-11 or CR-12, and three places said it could.** Measured by the reviewer against the built module: a durable log with a hole, committed out of order, or with an event AFTER its terminal all verdicted `agrees=true`. CR-10 IS "the log is not a gap-free ordered prefix". CR-11 has no ownership concept in the oracle at all; CR-12 is about external effects and needs a journal view this module does not model. The docblock, the commit message and the phase doc each claimed all four. They now say what is true — CR-92 fully, CR-10 partly, CR-11 and CR-12 not at all, budget for their own predicates. **CR-10's property turns out to be only half-checkable, and finding that out cost a wrong assertion.** I added a `0..n-1` gap-free check; it failed a perfectly healthy engine, because the durable log's sequence numbers are a strictly increasing SUBSEQUENCE — streamed events take numbers and are deliberately never persisted. A real completed run reads `[0,1,2,3,5,10,11,12,13,14]`. So "no persisted event is missing from the middle" is NOT expressible from the log alone: a streamed event's absence is indistinguishable from a lost one. CR-10's acceptance will need a store harness that records what it was ASKED to persist — worth knowing now rather than when that test is written. What the log proves unaided is that it starts at seq 0 and only moves forward, which does catch the out-of-order commit `#emitDurable` currently permits. **Four soundness holes, each measured, each closed:** - **No `runId` binding.** `eventsFor` returning another run's log verdicted `agrees=true`, and `durableTerminalCount === 1` did not catch it — the other run has a terminal too. The oracle's own tests all used `eventsFor: () => stored`, which ignores its argument, so nothing pinned that the id was honoured. CR-11's scenario is two owners over one store, i.e. exactly this. - **`reconciledCount` blamed this run for every run reconcile touched.** `reconcile()` repairs all interrupted runs and returns their events; attributing the raw array produced a factually wrong message and a false failure. Now filtered per run. - **A CORRECT crash repair was reported as a disagreement** — `a restart CHANGED the terminal: before=none after=run:failed` — so no crash test could use the oracle, which is most of what CR-91 is for. `expect: 'repaired'` inverts the pass condition for a run that died mid-flight. - **`terminalIn` took the FIRST terminal** while `reconstructCheckpointState` folds status last-wins, so two views read different terminals on a duplicated log — and a reconcile that APPENDED a second was invisible to the terminal-changed check. Now the last, with the count owning duplication. **The comparison was too narrow and too fragile.** `TerminalView` dropped `error.retryable`, `error.nodeId`, `correlationId` and `totalTokensUsed`: a live `{tool_failed, nodeId:'A', retryable:false}` compared EQUAL to a durable `{tool_failed, nodeId:'B', retryable:true}`, and a flipped `retryable` changes whether a surface offers a retry. `correlationId` is the best discriminator of the lot — it identifies THIS terminal event. And `JSON.stringify` made `{a:1,b:2}` disagree with `{b:2,a:1}` (a false failure the first time either side round-trips through a store, which is the module's whole purpose) and THREW out of the oracle on a circular or bigint payload, so the instrument became the failure with no verdict. Both fixed by a canonical, non-throwing stringify. **The resume view the acceptance criterion names still is not the checkpoint fold**, and the closure note had substituted one for the other silently. A real resume goes through the host's `Checkpointer` — the CLI's uses ADR-0075's strict read that REFUSES a log with an uninterpretable row, which a local fold reads happily. `loadCheckpoint` now takes the real port and the verdict reports `checkpointSource`, so a reader can tell which was used. Wiring it is CR-92's job, and the phase doc says so instead of implying it is done. Also: exported from `packages/core/src/index.ts` as a supported testing API. It is engine-pure and ships in `dist` either way, and the surface that most needs it — `apps/cli`'s harness against the real `history.db`, where CR-92 has to be certified — is outside this package. The `apps/cli/src/test-support.ts` precedent does not transfer: that lives in an app, never built as a library. And the CANCELLED e2e now uses `handle.cancel()`, which swallows `run_already_terminal`; `engine.cancel(runId)` throws it, and the stream is a buffered adapter, so a lagging consumer would have thrown inside the `for await` rather than failing an assertion. Latent, not live — green 5/5 — but the file already uses the safe idiom three times elsewhere. Twenty-two unit tests (up from ten), three e2e. Coverage 94.11%. pnpm run ci EXIT=0, pnpm coverage EXIT=0. Refs: CR-91 Co-Authored-By: Claude Opus 5 --- ...hase-2.6.5-core-reliability-remediation.md | 56 ++-- .../core/src/engine/durable-truth.test.ts | 214 ++++++++++++++- packages/core/src/engine/durable-truth.ts | 248 +++++++++++++++--- .../src/engine/m2-e2e-harness.e2e.test.ts | 6 +- packages/core/src/index.ts | 10 + 5 files changed, 482 insertions(+), 52 deletions(-) diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index 2104f253..423fa5c5 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -829,23 +829,49 @@ It pins that the live stream reported success; it does not prove durable truth a terminal type and payload. **This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with**, so it lands before them: a spine asserted with the harness the reviews already found insufficient is not asserted. -**Closed by `packages/core/src/engine/durable-truth.ts`** — `checkDurableTruth` compares four views (live -terminal, durable history, the history a FRESH engine leaves after `reconcile()`, and the status the -checkpoint fold derives) and returns a structured verdict rather than throwing, so callers can assert on the -specific disagreement and the oracle can have its own tests. `formatDurableTruth` renders the diff. - -Two design points worth carrying into the spine work: +**Closed by `packages/core/src/engine/durable-truth.ts`** (exported from the package index — the surfaces that +most need it, notably `apps/cli`'s harness against the real `history.db`, are outside this package). +`checkDurableTruth` compares the live terminal, the durable history, the history a FRESH engine leaves after +`reconcile()`, the checkpoint status, and the log's own order, returning a structured verdict rather than +throwing so callers can assert on the specific disagreement and the oracle can have its own tests. + +**What it instruments, corrected — the first version of this note claimed all four spine items and that was +false:** + +| item | expressible | why | +|---|---|---| +| `CR-92` | yes | terminal agreement across live / history / restart is exactly what it compares | +| `CR-10` | **partly** | see the limitation below | +| `CR-11` | **no** | it has no concept of run ownership or a fencing token | +| `CR-12` | **no** | external effects need an effect-journal view and a side-effect counter it does not model | + +`CR-11` and `CR-12` need their own predicates. Budget for that rather than assuming this covers them. + +**`CR-10`'s property is only half-checkable from the log, and finding that out cost a wrong assertion.** The +durable log's sequence numbers are a strictly increasing SUBSEQUENCE of the run's, never `0..n-1` — streamed +events (`agent:token`, `cost:updated`, …) take numbers and are deliberately never persisted. A real completed +run reads `[0,1,2,3,5,10,11,12,13,14]`, and asserting `0..n-1` failed a perfectly healthy engine. So "no +persisted event is missing from the middle" is **not** expressible from the log alone: a streamed event's +absence is indistinguishable from a lost one. **`CR-10`'s acceptance needs a store harness that records which +events it was ASKED to persist.** What the log proves unaided is that it starts at seq 0 and only moves +forward — which does catch out-of-order commit, the thing `#emitDurable` currently permits. + +Three design points worth carrying into the spine work: - **Envelope fields are excluded from the comparison on purpose.** `timestamp` and `sequenceNumber` move on - every restart; comparing them would make every run look like a disagreement while catching nothing. What - must agree is the terminal's TYPE and the payload a consumer acts on — outputs, error code, cost. -- **The checkpoint view is not decoration.** A resume seeds itself from the fold, so a fold that disagrees - with the durable terminal means a resumed run does the wrong work even when every event on disk is intact. - That is the `CR-92` shape the live stream cannot see. - -Ten unit tests cover the detection logic (including the headline case: live says `completed`, history says -`failed`), and three e2e tests apply the oracle to real engine runs on all three terminals. Break-verified: -stopping the engine from persisting terminals reddens all three e2e tests with a readable four-view diff. + every restart; comparing them would make every run look like a disagreement while catching nothing. +- **`expect: 'repaired'` exists because a CORRECT crash repair was a false failure.** A run that died without + a terminal and was reconciled on restart verdicted `a restart CHANGED the terminal: before=none + after=run:failed` — so no crash test could use the oracle at all, which is most of what it is for. +- **The resume view the acceptance criterion names is still not the checkpoint fold.** A real resume goes + through the host's `Checkpointer`, and the CLI's uses ADR-0075's STRICT read that REFUSES a log with an + uninterpretable row; a local fold reads it happily. `loadCheckpoint` takes the real port, and the verdict + reports `checkpointSource` so a reader can tell which was used. Supplying it is `CR-92`'s job. + +Twenty unit tests cover the detection logic (the headline live-`completed`/history-`failed` case, a +same-type terminal whose payload differs, a cross-run log, out-of-order and headless logs, a correct repair, +key-order and circular payloads), and three e2e tests apply the oracle to real engine runs on all three +terminals. Break-verified: stopping the engine from persisting terminals reddens all three e2e tests. ### CR-92 — Terminal persistence failure lets live and durable truth diverge · High · in the durability spine The engine can complete the delivery chain for a terminal event even when its persistence failed, and diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts index 812c0e08..3984ae01 100644 --- a/packages/core/src/engine/durable-truth.test.ts +++ b/packages/core/src/engine/durable-truth.test.ts @@ -127,7 +127,12 @@ describe('checkDurableTruth', () => { expect(verdict.agrees).toBe(false); expect(verdict.durableTerminalCount).toBe(2); expect(verdict.disagreements.some((d) => d.includes('exactly one closes a run'))).toBe(true); - expect(verdict.disagreements.some((d) => d.includes('must not add a second'))).toBe(true); + expect(verdict.disagreements.some((d) => d.includes('never one that landed'))).toBe(true); + // And the terminal-changed check fires too, now that `terminalIn` reads the LAST terminal — with `find` + // it read the first, so a reconcile that APPENDED a second was invisible to this check. + expect(verdict.disagreements.some((d) => d.includes('a restart CHANGED the terminal'))).toBe( + true, + ); }); it('catches a checkpoint fold that disagrees with the durable terminal', async () => { @@ -167,6 +172,213 @@ describe('checkDurableTruth', () => { expect(verdict.agrees).toBe(true); }); + it('catches a log belonging to ANOTHER run', async () => { + // The store ignored its id argument, or the caller crossed two runs. `durableTerminalCount === 1` does + // not catch it — the other run has a terminal too — so the oracle used to report agreement on the wrong + // log entirely. CR-11's scenario is two owners over one store, which is exactly this shape. + const other: RunEvent[] = [ + { ...started, runId: 'rOTHER' }, + { ...completed(1), runId: 'rOTHER' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => other, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('belonging to another run'))).toBe(true); + }); + + it('catches a live terminal carrying a different runId', async () => { + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), runId: 'rOTHER' }, + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('not r1'))).toBe(true); + }); + + it('accepts a durable log whose seqs SKIP — streamed events take numbers and are never persisted', async () => { + // A real completed run reads [0,1,2,3,5,10,11,12,13,14]. Asserting `0..n-1` here failed a healthy engine, + // which is how this limitation was found: "no persisted event is missing from the middle" is not + // expressible from the log alone, because a streamed event's absence looks identical to a lost one. + const stored: RunEvent[] = [started, { ...completed(1), sequenceNumber: 7 }]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 7 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + }); + + it('catches an OUT-OF-ORDER durable log — CR-10s checkable half', async () => { + const stored: RunEvent[] = [ + { ...started, sequenceNumber: 0 }, + { ...failed(9) }, + { ...completed(1), sequenceNumber: 4 }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 4 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('is not ordered'))).toBe(true); + }); + + it('catches a durable log missing its head', async () => { + const stored: RunEvent[] = [{ ...completed(1), sequenceNumber: 4 }]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 4 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('not 0'))).toBe(true); + }); + + it('catches an event persisted AFTER the terminal', async () => { + const stored: RunEvent[] = [ + started, + completed(1), + { type: 'node:skipped', ...base(2), nodeId: 'x', reason: 'branch_not_taken' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('continues past its terminal'))).toBe(true); + }); + + it('does NOT blame this run for events reconcile() wrote for ANOTHER one', async () => { + // `reconcile()` repairs every interrupted run in the store and returns all of their events. Attributing + // the raw count here produced a factually wrong message and a false failure. + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor, + reconcile: () => Promise.resolve([{ ...failed(9), runId: 'rOTHER' }]), + }); + + expect(verdict.agrees).toBe(true); + expect(verdict.reconciledCount).toBe(0); + }); + + it('accepts a CORRECT crash repair under expect:"repaired" — the case this phase exists for', async () => { + // Before the mode existed, a run that died without a terminal and was repaired on restart verdicted + // `a restart CHANGED the terminal: before=none after=run:failed` — so no crash test could use the oracle + // at all, which is most of what CR-91 is for. + const stored: RunEvent[] = [started]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor: () => stored, + reconcile: () => { + stored.push(failed(1)); + return Promise.resolve([failed(1)]); + }, + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + expect(verdict.reconciledCount).toBe(1); + expect(verdict.afterReconcile?.type).toBe('run:failed'); + }); + + it('under expect:"repaired", a restart that repairs NOTHING is the failure', async () => { + const stored: RunEvent[] = [started]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor: () => stored, + reconcile: () => Promise.resolve([]), + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('NO durable terminal'))).toBe(true); + }); + + it('catches a failure whose retryable / nodeId differ while the CODE matches', async () => { + // Dropped from the first TerminalView, and a flipped `retryable` changes whether a surface offers a retry. + const durable: RunEvent = { + type: 'run:failed', + ...base(1), + error: { code: 'tool_failed', message: 'boom', retryable: true, nodeId: 'B' }, + partialOutputs: {}, + }; + const liveEvent: RunEvent = { + type: 'run:failed', + ...base(1), + error: { code: 'tool_failed', message: 'boom', retryable: false, nodeId: 'A' }, + partialOutputs: {}, + }; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: liveEvent, + eventsFor: () => [started, durable], + }); + + expect(verdict.agrees).toBe(false); + }); + + it('compares outputs key-order-insensitively, and never throws on an uncomparable payload', async () => { + const reordered = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: 1, b: 2 }), + eventsFor: () => [started, completed(1, { b: 2, a: 1 })], + }); + expect(reordered.agrees, 'key order is not a disagreement').toBe(true); + + // A circular payload used to throw OUT of the oracle, so the instrument became the failure. + const circular: Record = {}; + circular['self'] = circular; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, circular), + eventsFor: () => [started, completed(1, circular)], + }); + expect(verdict.agrees).toBe(true); + }); + + it('reports WHERE the checkpoint status came from — a local fold is not the resume path', async () => { + const terminal = completed(1); + const { eventsFor } = log(terminal); + + const local = await checkDurableTruth({ runId: 'r1', live: terminal, eventsFor }); + expect(local.checkpointSource).toBe('local-fold'); + + const viaPort = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + loadCheckpoint: () => Promise.resolve({ runStatus: 'completed' as const }), + }); + expect(viaPort.checkpointSource).toBe('checkpointer'); + + // And the port is what is BELIEVED when supplied: a checkpointer that refuses (undefined) disagrees with + // a durable terminal, which a local fold would have read happily. + const refusing = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + loadCheckpoint: () => Promise.resolve(undefined), + }); + expect(refusing.agrees).toBe(false); + expect(refusing.disagreements.some((d) => d.includes('checkpoint port'))).toBe(true); + }); + it('skips view 3 when no reconcile is supplied, and says so in the counts', async () => { const terminal = failed(1, 'tool_failed'); const { eventsFor } = log(terminal); diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts index b010d217..24d22243 100644 --- a/packages/core/src/engine/durable-truth.ts +++ b/packages/core/src/engine/durable-truth.ts @@ -7,23 +7,35 @@ * durable log says the run failed, or a restart reconciles the same run to a DIFFERENT terminal, or a resume * re-runs work the log already records as done. Every one of those is invisible to a live assertion. * - * So this compares FOUR independent views of the same run and reports where they disagree: + * So this compares several views of the same run and reports where they disagree: * * 1. **live** — the terminal the caller drained from `RunHandle.events`. * 2. **history** — the terminal as it exists in the store, read back. - * 3. **after reconcile** — the history a *fresh* engine leaves behind after `reconcile()` on restart. It must - * be the SAME terminal: reconciliation repairs a run that died without one, and must never overwrite or - * duplicate one that already landed. + * 3. **after reconcile** — the history a *fresh* engine leaves behind after `reconcile()` on restart. * 4. **checkpoint** — the status `reconstructCheckpointState` derives from the durable log alone. A resume * seeds itself from this, so if it disagrees with the terminal, a resumed run does the wrong work. + * 5. **the durable prefix itself** — sequence numbers gap-free and ordered, exactly one terminal, and the + * terminal last. Not a "view" of the terminal, but the property every other view silently assumes. * - * **It returns a verdict rather than throwing.** A thrower is hard to test and forces every caller into the - * same message; a structured verdict lets the callers that need one assert on the specific disagreement, and - * lets this module have its own tests. {@link formatDurableTruth} renders it when a caller just wants to fail - * with a readable diff. + * They are four PROJECTIONS of two sources (the live stream and the store), not four independent + * observations — worth saying because an earlier version of this docblock called them independent, and in an + * in-memory harness views 1 and 2 are literally the same object. + * + * **What it CAN and CANNOT instrument, stated because the phase doc once claimed more.** It expresses + * `CR-92` (terminal durable truth) and `CR-10` (the durable log is an ordered, gap-free prefix). It does NOT + * express `CR-11` — it has no concept of run ownership or a fencing token — and it does NOT express `CR-12`, + * which is about external effects and needs an effect-journal view plus a side-effect counter this module + * does not model. Those two need their own predicates; this one is not the instrument for them. * - * This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with, which is why it lands before - * them — a spine asserted with a harness that only reads the live stream is not asserted. + * It also does not yet cover the RESUME view that `CR-91`'s acceptance criterion names. View 4 folds the log + * locally, which is close but not the same thing: a real resume goes through the host's `Checkpointer`, and + * the CLI's implementation uses ADR-0075's STRICT read that refuses a log with an uninterpretable row. A + * resume that would refuse therefore reads here as a clean fold. `loadCheckpoint` exists so a caller can + * supply the real port; when it is absent the verdict says the fold was local. + * + * **It returns a verdict rather than throwing.** A thrower is hard to test and forces every caller into the + * same message; a structured verdict lets callers assert on the specific disagreement, and lets this module + * have its own tests. {@link formatDurableTruth} renders it when a caller just wants a readable diff. */ import type { RunEvent, RunStatus } from '@relavium/shared'; @@ -49,12 +61,29 @@ const STATUS_FOR: Partial> = { */ export interface TerminalView { readonly type: RunEvent['type']; - /** `run:completed.outputs` / `run:failed.partialOutputs`, JSON-normalized. Absent for `run:cancelled`. */ + /** `run:completed.outputs` / `run:failed.partialOutputs`, canonically stringified. `run:cancelled` has none. */ readonly outputs?: string; /** `run:failed.error.code`, when the terminal carries one. */ readonly errorCode?: string; + /** + * The rest of `run:failed.error`'s identity. Dropped in the first version, and the omission let a real + * defect through: a live `{tool_failed, nodeId:'A', retryable:false}` compared equal to a durable + * `{tool_failed, nodeId:'B', retryable:true}`. A flipped `retryable` changes whether a surface offers a + * retry, and `nodeId` names the root cause. + */ + readonly errorRetryable?: boolean; + readonly errorNodeId?: string; + /** + * The secret-free id joined to the internal log (ADR-0036) — the single best discriminator here, because it + * identifies THIS terminal event. It differs across views only when the terminal was genuinely rewritten, + * which is exactly the defect. + */ + readonly correlationId?: string; /** The run-wide realized total the terminal reports, when it carries one. */ readonly costMicrocents?: number; + /** `run:completed.totalTokensUsed`, restored by the same fold as cost — comparing one and not the other + * was an inconsistency, not a decision. */ + readonly tokens?: string; } export interface DurableTruthVerdict { @@ -64,28 +93,63 @@ export interface DurableTruthVerdict { readonly history: TerminalView | undefined; readonly afterReconcile: TerminalView | undefined; readonly checkpointStatus: RunStatus | undefined; + /** + * Where {@link checkpointStatus} came from. `'local-fold'` means this module folded the log itself, which + * is NOT the resume path — a real resume goes through the host's `Checkpointer` and may refuse a log this + * fold reads happily (ADR-0075). Supply `loadCheckpoint` to get `'checkpointer'`. + */ + readonly checkpointSource: 'local-fold' | 'checkpointer'; /** How many terminals the durable log holds. Anything but 1 breaks exactly-one-terminal (ADR-0036). */ readonly durableTerminalCount: number; - /** Events `reconcile()` produced. Non-empty for a run that already had a durable terminal is a defect. */ + /** Events `reconcile()` produced FOR THIS RUN — it repairs every interrupted run and returns them all. */ readonly reconciledCount: number; /** One sentence per disagreement, in the order checked. Empty iff `agrees`. */ readonly disagreements: readonly string[]; } +/** + * Stringify a payload for comparison, key-order-insensitively and WITHOUT ever throwing out of the oracle. + * + * A bare `JSON.stringify` fails twice: `{a:1,b:2}` and `{b:2,a:1}` compare unequal, which is a false failure + * the moment either side has been through a store round-trip — the whole point of this module; and a circular + * or `bigint` payload throws, turning the instrument itself into the failure with no verdict at all. + */ +function canonical(value: unknown): string { + try { + const seen = new WeakSet(); + return JSON.stringify(value, function replace(_key, val: unknown): unknown { + if (typeof val === 'bigint') return `${val.toString()}n`; + if (typeof val !== 'object' || val === null) return val; + if (seen.has(val)) return '[circular]'; + seen.add(val); + if (Array.isArray(val)) return val; + return Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b))); + }); + } catch (error) { + return `[uncomparable: ${error instanceof Error ? error.name : 'unknown'}]`; + } +} + function viewOf(event: RunEvent | undefined): TerminalView | undefined { if (event === undefined) return undefined; switch (event.type) { case 'run:completed': return { type: event.type, - outputs: JSON.stringify(event.outputs), + outputs: canonical(event.outputs), costMicrocents: event.totalCostMicrocents, + tokens: canonical(event.totalTokensUsed), }; case 'run:failed': return { type: event.type, - outputs: JSON.stringify(event.partialOutputs), + outputs: canonical(event.partialOutputs), errorCode: event.error.code, + errorRetryable: event.error.retryable, + ...(event.error.nodeId === undefined ? {} : { errorNodeId: event.error.nodeId }), + ...(event.error.correlationId === undefined + ? {} + : { correlationId: event.error.correlationId }), ...(event.cumulativeCostMicrocents === undefined ? {} : { costMicrocents: event.cumulativeCostMicrocents }), @@ -102,8 +166,15 @@ function viewOf(event: RunEvent | undefined): TerminalView | undefined { } } +/** + * The LAST terminal, not the first. + * + * A duplicated log is caught by the count check either way, but `find` made two views disagree by + * construction: `reconstructCheckpointState` folds run status last-wins, so view 4 read the last terminal + * while views 2 and 3 read the first. Taking the last aligns them and leaves duplication to the count. + */ function terminalIn(events: readonly RunEvent[]): RunEvent | undefined { - return events.find((event) => TERMINALS.has(event.type)); + return [...events].reverse().find((event) => TERMINALS.has(event.type)); } function describe(view: TerminalView | undefined): string { @@ -121,7 +192,11 @@ function sameView(a: TerminalView | undefined, b: TerminalView | undefined): boo a.type === b.type && a.outputs === b.outputs && a.errorCode === b.errorCode && - a.costMicrocents === b.costMicrocents + a.errorRetryable === b.errorRetryable && + a.errorNodeId === b.errorNodeId && + a.correlationId === b.correlationId && + a.costMicrocents === b.costMicrocents && + a.tokens === b.tokens ); } @@ -136,6 +211,24 @@ export interface DurableTruthInput { * when the caller has already restarted for its own reasons; then view 3 is skipped and the verdict says so. */ readonly reconcile?: () => Promise; + /** + * The host's real `Checkpointer.load` — what a resume actually goes through. Supply it and view 4 observes + * the resume path (including ADR-0075's strict read, which REFUSES a log with an uninterpretable row); + * omit it and the verdict folds the log locally and records that it did. + */ + readonly loadCheckpoint?: (runId: string) => Promise<{ runStatus: RunStatus } | undefined>; + /** + * What the run is expected to look like on disk, and it changes what counts as a disagreement. + * + * - `'settled'` (default): the run closed under its own power. `reconcile()` must find nothing to do, and a + * changed terminal is a defect. + * - `'repaired'`: the run DIED without a terminal and a restart is expected to write one. Then a durable + * terminal appearing where there was none is the PASS condition, not a failure. + * + * Without this, a correct crash repair — the exact scenario this phase exists for — reported as + * `a restart CHANGED the terminal: before=none after=run:failed`, so no crash test could use the oracle. + */ + readonly expect?: 'settled' | 'repaired'; } /** @@ -144,12 +237,41 @@ export interface DurableTruthInput { */ export async function checkDurableTruth(input: DurableTruthInput): Promise { const disagreements: string[] = []; + const expectRepaired = input.expect === 'repaired'; const before = [...(await input.eventsFor(input.runId))]; + + // Bind everything to THIS run before comparing anything. Without it the oracle happily reports agreement on + // another run's log — measured — and `durableTerminalCount === 1` does not catch it, because the other run + // has a terminal too. A store that ignores its id argument, or a caller that crosses two runs, both land + // here; and `CR-11`'s whole scenario is two owners over one store. + const foreign = before.filter((event) => event.runId !== input.runId); + if (foreign.length > 0) { + disagreements.push( + `eventsFor(${input.runId}) returned ${foreign.length} event(s) belonging to another run ` + + `(e.g. ${String(foreign[0]?.runId)}) — every view below would be comparing the wrong log`, + ); + } + if (input.live !== undefined && input.live.runId !== input.runId) { + disagreements.push(`the live terminal carries runId ${input.live.runId}, not ${input.runId}`); + } + const live = viewOf(input.live); const history = viewOf(terminalIn(before)); - if (!sameView(live, history)) { + if (expectRepaired) { + // A run that died mid-flight: nothing durable yet, and nothing live either. + if (history !== undefined) { + disagreements.push( + `expected a run needing repair, but the log already holds a terminal (${describe(history)})`, + ); + } + if (live !== undefined) { + disagreements.push( + `expected a run that died without a terminal, but the live stream produced ${describe(live)}`, + ); + } + } else if (!sameView(live, history)) { disagreements.push( `live and durable history disagree: live=${describe(live)} history=${describe(history)}`, ); @@ -157,25 +279,42 @@ export async function checkDurableTruth(input: DurableTruthInput): Promise 0) { - disagreements.push( - `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + - `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, it must not add a second`, - ); + // Per-RUN, not the whole array. `reconcile()` repairs every interrupted run in the store and returns all + // of their events, so attributing the raw count to this run produced a factually wrong message and a + // false failure the moment a second run existed — which is, again, `CR-11`'s exact shape. + reconciledCount = reconciled.filter((event) => event.runId === input.runId).length; + finalEvents = [...(await input.eventsFor(input.runId))]; + afterReconcile = viewOf(terminalIn(finalEvents)); + + if (expectRepaired) { + if (afterReconcile === undefined) { + disagreements.push( + 'a restart left the run with NO durable terminal — a crashed run must reconcile to one', + ); + } + if (reconciledCount === 0) { + disagreements.push('reconcile() produced no event for this run, so nothing repaired it'); + } + } else { + if (!sameView(history, afterReconcile)) { + disagreements.push( + `a restart CHANGED the terminal: before=${describe(history)} after=${describe(afterReconcile)}`, + ); + } + if (history !== undefined && reconciledCount > 0) { + disagreements.push( + `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + + `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, never one that landed`, + ); + } } + } else { + finalEvents = before; } - const finalEvents = [...(await input.eventsFor(input.runId))]; const terminalCount = finalEvents.filter((event) => TERMINALS.has(event.type)).length; if (terminalCount !== 1) { disagreements.push( @@ -183,12 +322,50 @@ export async function checkDurableTruth(input: DurableTruthInput): Promise event.runId === input.runId); + const seqs = ours.map((event) => event.sequenceNumber); + const outOfOrder = seqs.some((seq, index) => index > 0 && seq <= (seqs[index - 1] ?? -1)); + if (outOfOrder) { + disagreements.push( + `the durable log is not ordered: sequenceNumbers ${JSON.stringify(seqs)} — a higher seq committed ` + + `before a lower one, so the checkpoint fold can read a state its causal predecessor never reached`, + ); + } + if (seqs.length > 0 && seqs[0] !== 0) { + disagreements.push( + `the durable log starts at sequenceNumber ${String(seqs[0])}, not 0 — \`run:started\` is always ` + + `persisted and always first, so the head of the log is missing`, + ); + } + const last = ours.at(-1); + if (last !== undefined && terminalCount === 1 && !TERMINALS.has(last.type)) { + disagreements.push( + `the durable log continues past its terminal (last event is '${last.type}') — exactly one terminal ` + + `CLOSES a run, so nothing may follow it`, + ); + } + + const localFold = reconstructCheckpointState(finalEvents)?.runStatus; + const loaded = await input.loadCheckpoint?.(input.runId); + const checkpointStatus = input.loadCheckpoint === undefined ? localFold : loaded?.runStatus; const expectedStatus = afterReconcile === undefined ? undefined : STATUS_FOR[afterReconcile.type]; if (expectedStatus !== undefined && checkpointStatus !== expectedStatus) { disagreements.push( - `the checkpoint fold says '${String(checkpointStatus)}' while the durable terminal says ` + - `'${expectedStatus}' — a resume seeds itself from the fold, so it would do the wrong work`, + `the checkpoint ${input.loadCheckpoint === undefined ? 'fold' : 'port'} says ` + + `'${String(checkpointStatus)}' while the durable terminal says '${expectedStatus}' — a resume seeds ` + + `itself from it, so it would do the wrong work`, ); } @@ -199,6 +376,7 @@ export async function checkDurableTruth(input: DurableTruthInput): Promise ` ✖ ${d}`), ].join('\n'); diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 11638e26..51ca5e11 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -1144,7 +1144,11 @@ describe('M2 — end-to-end Node harness (1.U)', () => { const events: RunEvent[] = []; for await (const event of handle.events) { events.push(event); - if (event.type === 'node:started' && event.nodeId === 'work') engine.cancel(handle.runId); + // `handle.cancel()`, not `engine.cancel(runId)`: the latter THROWS `run_already_terminal` once the run + // settled, and this stream is a buffered push/pull adapter — the consumer's position relative to engine + // progress is a microtask-interleaving property, not a guarantee. `work` is a real node that finishes on + // its own, so a lagging consumer would throw inside this `for await` instead of failing an assertion. + if (event.type === 'node:started' && event.nodeId === 'work') handle.cancel(); } expect(events.at(-1)?.type).toBe('run:cancelled'); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 09cc287a..04866875 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -126,6 +126,16 @@ export type { CheckpointPendingGate, CheckpointPendingMediaJob, } from './engine/checkpoint.js'; +// The durable-truth oracle (CR-91) — a SUPPORTED testing API, exported deliberately. It is engine-pure and +// ships in `dist` either way (the build excludes only `*.test.ts`), and the surfaces that most need it are +// outside this package: `apps/cli`'s regression harness runs against the real `history.db` store, which is +// where CR-92's acceptance ("live, history, resume and reconcile agree") actually has to be certified. +export { checkDurableTruth, formatDurableTruth } from './engine/durable-truth.js'; +export type { + DurableTruthInput, + DurableTruthVerdict, + TerminalView, +} from './engine/durable-truth.js'; export type { ExecutionHost, RunStore, From f5750a3445ab5ca7d88ce89ba568d6f6e77373aa Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 00:06:53 +0300 Subject: [PATCH 19/26] =?UTF-8?q?fix(core,docs):=20fold=20the=20Sonnet=20r?= =?UTF-8?q?eview=20of=20CR-91=20=E2=80=94=20the=20cycle=20guard=20caused?= =?UTF-8?q?=20the=20failure=20it=20prevented?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nine findings. The two HIGH ones were both introduced by the previous fold. **`canonical()`'s cycle guard flagged SHARED references, not circular ones.** The `WeakSet` was never un-marked on the way back up, so it accumulated every object seen anywhere in the tree rather than the current lineage. Measured: `{ a: shared, b: shared }` serialized as `{"a":{"x":1},"b":"[circular]"}` while a structurally identical payload built from two separate objects serialized normally — two equal outputs disagreeing. A `fan_in` echoing one value into two keys is enough to hit it. That is precisely the false-failure-from-a- serialization-quirk class the function's own docblock says it exists to eliminate. Worse, it was unverified: deleting the whole mechanism left all tests green, so it was both wrong and unexercised. Native `JSON.stringify` already tracks ancestry correctly, so the fix is to delete the guard and catch its throw — simpler and right. Pinned by a shared-but-non-circular test that reddens when the flat `WeakSet` is put back. **`formatDurableTruth` never rendered the fields the previous fold added.** A pair differing only in `errorRetryable` or `correlationId` printed BYTE-IDENTICALLY on both sides of a `DISAGREES` verdict — no visible reason, for exactly the fields that commit called "the best discriminator of the lot" and "changes whether a surface offers a retry". `describe()` now renders every compared field, pinned by a test that asserts both sides appear. **Three tests proved less than their names.** The retryable/nodeId test flipped both fields at once, so dropping either comparison alone left it green — it only proved "at least one is compared". And `correlationId` and `tokens` had NO test at all: removing either comparison from `sameView` left all 22 green. Now one test per field via `it.each`, each flipping exactly one, plus a token test. All six mutations verified red: drop retryable / nodeId / correlationId / tokens compare, strip `describe()`'s new fields, restore the flat `WeakSet`. **The phase doc's overclaim survived in the two places the last fold did not reach** — `:170` (the phase-wide execution-order rationale) and `:829` (inside CR-91's own acceptance paragraph), both still saying the oracle is what proves CR-10/11/12/92, ten lines above the corrected table. Both now qualified, with the original wording preserved and dated so the correction is legible rather than silent. Test count corrected 20 → 27. **And the review answered a question I had left open: what CR-92 still needs.** Recorded in the phase doc rather than discovered later — `live: RunEvent | undefined` structurally cannot carry "a distinct typed result that is not a RunEvent", which is what CR-92's acceptance requires the API to return when durability is uncertain; the resume leg can only ever compare `RunStatus` because `CheckpointState` carries no payload, so "type and payload" is half-checkable there; and `expect` has no mode for "durability uncertain, outbox retry pending". Verified TRUE by the review and left alone: the `[0,1,2,3,5,10,11,12,13,14]` seq sequence on a real run, `handle.cancel()` swallowing `run_already_terminal` while `engine.cancel(runId)` throws it, engine purity under the purity gate, and every other branch of the previous fold (runId binding, per-run reconcile counting, `expect:'repaired'`, last-terminal, order/head/count checks) mutation-clean with no double-reporting. 27 unit tests, 3 e2e. Coverage 95.04%; the uncovered lines are the two guards against MISUSING `expect:'repaired'` on a run that did not crash. pnpm run ci EXIT=0, pnpm coverage EXIT=0. Refs: CR-91 Co-Authored-By: Claude Opus 5 --- ...hase-2.6.5-core-reliability-remediation.md | 32 ++++++-- .../core/src/engine/durable-truth.test.ts | 81 ++++++++++++++++--- packages/core/src/engine/durable-truth.ts | 25 ++++-- 3 files changed, 110 insertions(+), 28 deletions(-) diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index 423fa5c5..db251f12 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -167,8 +167,10 @@ flowchart TD `CR-92` sits inside the spine, not in `W9`: an API that returns `completed` while durable history says `failed` is a runtime correctness defect, not a harness concern. `CR-90`/`CR-91` come before the spine because a crash -and durable-truth oracle is what proves `CR-10`/`CR-11`/`CR-12`/`CR-92` landed — building it last would mean -asserting the spine with the harness the reviews already found insufficient. +and durable-truth oracle is what proves `CR-92` landed, and half of `CR-10` — building it last would mean +asserting the spine with the harness the reviews already found insufficient. It does **not** instrument +`CR-11` or `CR-12`; that was an early overclaim, corrected once the oracle was built and measured (see +`CR-91`). Budget those two their own predicates. - **W0 — finish the halves.** Cheap, already half-done. Leaving them half-done is the exact failure this phase is correcting. @@ -826,8 +828,10 @@ marker. Stated here and in the guard's own docblock because the first version as ### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium · ✅ CLOSED 2026-08-10 It pins that the live stream reported success; it does not prove durable truth after a restart. **Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same -terminal type and payload. **This is the instrument `CR-10`, `CR-11`, `CR-12` and `CR-92` are proven with**, so -it lands before them: a spine asserted with the harness the reviews already found insufficient is not asserted. +terminal type and payload. It lands before the spine because a spine asserted with the harness the reviews +already found insufficient is not asserted. *(The original wording here called it "the instrument `CR-10`, +`CR-11`, `CR-12` and `CR-92` are proven with". That was written before the oracle existed and it is not true +of `CR-11` or `CR-12` — see the table below.)* **Closed by `packages/core/src/engine/durable-truth.ts`** (exported from the package index — the surfaces that most need it, notably `apps/cli`'s harness against the real `history.db`, are outside this package). @@ -868,10 +872,22 @@ Three design points worth carrying into the spine work: uninterpretable row; a local fold reads it happily. `loadCheckpoint` takes the real port, and the verdict reports `checkpointSource` so a reader can tell which was used. Supplying it is `CR-92`'s job. -Twenty unit tests cover the detection logic (the headline live-`completed`/history-`failed` case, a -same-type terminal whose payload differs, a cross-run log, out-of-order and headless logs, a correct repair, -key-order and circular payloads), and three e2e tests apply the oracle to real engine runs on all three -terminals. Break-verified: stopping the engine from persisting terminals reddens all three e2e tests. +Twenty-seven unit tests cover the detection logic (the headline live-`completed`/history-`failed` case, a +same-type terminal whose payload differs, each compared field pinned independently, a cross-run log, +out-of-order and headless logs, a correct repair, key-order / circular / shared-reference payloads), and three +e2e tests apply the oracle to real engine runs on all three terminals. Break-verified: stopping the engine +from persisting terminals reddens all three e2e tests. + +**Three things `CR-92` must still add before it can certify itself with this** — measured, not guessed: + +1. `DurableTruthInput.live` is `RunEvent | undefined`, so it structurally cannot carry "a distinct typed + result that is not a `RunEvent`" — which is exactly what `CR-92`'s acceptance says the API must return when + durability is uncertain. That input has to widen before the test can express its own scenario. +2. Even with the real `loadCheckpoint` wired, the resume leg can only compare `RunStatus` — a four-value enum. + `CheckpointState` carries no outputs, error code, `correlationId` or tokens, so `CR-92`'s "agree on the same + terminal type **and payload**" is only checkable for the type half on that leg. +3. `expect` has `'settled' | 'repaired'` and no mode for "durability uncertain, outbox retry pending", which + is `CR-92`'s own state and fits neither. ### CR-92 — Terminal persistence failure lets live and durable truth diverge · High · in the durability spine The engine can complete the delivery chain for a terminal event even when its persistence failed, and diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts index 3984ae01..44916b25 100644 --- a/packages/core/src/engine/durable-truth.test.ts +++ b/packages/core/src/engine/durable-truth.test.ts @@ -310,29 +310,70 @@ describe('checkDurableTruth', () => { expect(verdict.disagreements.some((d) => d.includes('NO durable terminal'))).toBe(true); }); - it('catches a failure whose retryable / nodeId differ while the CODE matches', async () => { - // Dropped from the first TerminalView, and a flipped `retryable` changes whether a surface offers a retry. + /** A `run:failed` differing from the baseline in exactly one error field. */ + const failedWith = (error: Partial>): RunEvent => ({ + type: 'run:failed', + ...base(1), + error: { code: 'tool_failed', message: 'boom', retryable: false, ...error } as never, + partialOutputs: {}, + }); + + // One test per field, each flipping exactly ONE. A single fixture flipping two proved only "at least one + // of them is compared" — dropping either from `sameView` left it green. + it.each([ + ['retryable', failedWith({ nodeId: 'A', correlationId: 'c-1', retryable: true })], + ['nodeId', failedWith({ nodeId: 'B', correlationId: 'c-1' })], + ['correlationId', failedWith({ nodeId: 'A', correlationId: 'c-2' })], + ])('catches a failure whose %s differs while the CODE matches', async (_field, durable) => { + const liveEvent = failedWith({ nodeId: 'A', correlationId: 'c-1' }); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: liveEvent, + eventsFor: () => [started, durable], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + }); + + it('catches a completed run whose TOKEN totals differ', async () => { const durable: RunEvent = { - type: 'run:failed', + type: 'run:completed', ...base(1), - error: { code: 'tool_failed', message: 'boom', retryable: true, nodeId: 'B' }, - partialOutputs: {}, - }; - const liveEvent: RunEvent = { - type: 'run:failed', - ...base(1), - error: { code: 'tool_failed', message: 'boom', retryable: false, nodeId: 'A' }, - partialOutputs: {}, + outputs: { a: 1 }, + totalTokensUsed: { input: 9, output: 9 }, + totalCostMicrocents: 500, + durationMs: 10, }; const verdict = await checkDurableTruth({ runId: 'r1', - live: liveEvent, + live: completed(1), eventsFor: () => [started, durable], }); expect(verdict.agrees).toBe(false); }); + it('RENDERS every compared field, so a DISAGREES verdict shows why', async () => { + // The rendered diff used to print type/code/cost/outputs only — so a pair differing solely in + // `retryable` or `correlationId` printed byte-identically on both sides and told the reader nothing. + const verdict = await checkDurableTruth({ + runId: 'r1', + live: failedWith({ retryable: false, nodeId: 'A', correlationId: 'c-1' }), + eventsFor: () => [ + started, + failedWith({ retryable: true, nodeId: 'B', correlationId: 'c-2' }), + ], + }); + + const rendered = formatDurableTruth(verdict); + expect(rendered).toContain('retryable=false'); + expect(rendered).toContain('retryable=true'); + expect(rendered).toContain('correlationId=c-1'); + expect(rendered).toContain('correlationId=c-2'); + expect(rendered).toContain('node=A'); + expect(rendered).toContain('node=B'); + }); + it('compares outputs key-order-insensitively, and never throws on an uncomparable payload', async () => { const reordered = await checkDurableTruth({ runId: 'r1', @@ -352,6 +393,22 @@ describe('checkDurableTruth', () => { expect(verdict.agrees).toBe(true); }); + it('treats a SHARED reference as shared, not circular — a repeated object is not a cycle', async () => { + // The first cycle guard was a flat `WeakSet` that was never un-marked on the way back up, so it flagged + // every REPEATED reference. `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a + // structurally identical payload built from two separate objects serialized normally — two equal outputs + // disagreeing, which is the exact false failure `canonical` exists to prevent. A `fan_in` echoing one + // value into two keys is enough to hit it. + const shared = { x: 1 }; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: shared, b: shared }), + eventsFor: () => [started, completed(1, { a: { x: 1 }, b: { x: 1 } })], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + }); + it('reports WHERE the checkpoint status came from — a local fold is not the resume path', async () => { const terminal = completed(1); const { eventsFor } = log(terminal); diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts index 24d22243..9c9b7940 100644 --- a/packages/core/src/engine/durable-truth.ts +++ b/packages/core/src/engine/durable-truth.ts @@ -116,16 +116,20 @@ export interface DurableTruthVerdict { */ function canonical(value: unknown): string { try { - const seen = new WeakSet(); - return JSON.stringify(value, function replace(_key, val: unknown): unknown { + return JSON.stringify(value, (_key, val: unknown): unknown => { if (typeof val === 'bigint') return `${val.toString()}n`; - if (typeof val !== 'object' || val === null) return val; - if (seen.has(val)) return '[circular]'; - seen.add(val); - if (Array.isArray(val)) return val; + if (typeof val !== 'object' || val === null || Array.isArray(val)) return val; return Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b))); }); } catch (error) { + // A true cycle (or any other unserializable payload) becomes a single marker rather than escaping. The + // first version tried to be cleverer, with a `WeakSet` marking objects it had already visited — and got + // it WRONG in a way that caused the very failure this function exists to prevent: the set was never + // un-marked on the way back up, so it flagged every REPEATED reference, not every circular one. + // `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a structurally identical + // payload built from two separate objects serialized normally — so two equal outputs disagreed. A + // `fan_in` echoing one value into two keys is enough to hit it. Native `JSON.stringify` already tracks + // the ANCESTRY correctly; letting it throw and catching here is both simpler and right. return `[uncomparable: ${error instanceof Error ? error.name : 'unknown'}]`; } } @@ -179,9 +183,16 @@ function terminalIn(events: readonly RunEvent[]): RunEvent | undefined { function describe(view: TerminalView | undefined): string { if (view === undefined) return 'none'; + // EVERY compared field, not a subset. The first version rendered only type/code/cost/outputs — so a pair + // differing solely in `errorRetryable` or `correlationId` printed BYTE-IDENTICALLY on both sides of a + // `DISAGREES` verdict, leaving a reader with no visible reason for exactly the fields that matter most. const parts: string[] = [view.type]; if (view.errorCode !== undefined) parts.push(`code=${view.errorCode}`); + if (view.errorRetryable !== undefined) parts.push(`retryable=${String(view.errorRetryable)}`); + if (view.errorNodeId !== undefined) parts.push(`node=${view.errorNodeId}`); + if (view.correlationId !== undefined) parts.push(`correlationId=${view.correlationId}`); if (view.costMicrocents !== undefined) parts.push(`cost=${view.costMicrocents}`); + if (view.tokens !== undefined) parts.push(`tokens=${view.tokens}`); if (view.outputs !== undefined) parts.push(`outputs=${view.outputs}`); return parts.join(' '); } @@ -311,8 +322,6 @@ export async function checkDurableTruth(input: DurableTruthInput): Promise TERMINALS.has(event.type)).length; From 08fef4614b395449810e6780ae3db43c1f9730ca Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 00:14:34 +0300 Subject: [PATCH 20/26] =?UTF-8?q?fix(cli,core,docs):=20close=20W0=20?= =?UTF-8?q?=E2=80=94=20the=20three=20half-done=20items,=20and=20a=20fourth?= =?UTF-8?q?=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2.6.5's `W0`. Each was half-closed by Wave 1; leaving them half-done is the exact failure this phase corrects. **`CR-01` — `session:cancelled` was the one arm still calling the store bare.** `RunEventBus` isolates a listener throw from the producer, so a failing terminal write let the cancel path report success while the durability latch never set — and the latch is what refuses the next egress. A session whose terminal row never landed would resume believing it had ended cleanly. Now through `persistDurably`, like every sibling. One correction to the finding: the arm writes through `updateSession`, not `writeTurn` — the terminal marks the session `ended` rather than appending a turn. And the self-detach stays OUTSIDE the wrapper, in a `finally`: `persistDurably` re-throws, and a failed write is exactly when a leaked bus listener is worst, since every later event would re-enter a persister that cannot write. **`CR-02` — the counter stays where it is, and the code now says why.** The decision came out of the code rather than being chosen: both catch paths already apply *count a turn only when a provider ENGAGED*, and by the increment `#runTurn` has resolved. A `flushBudgetCommitments` rejection past that point is a durability failure, not evidence the turn never happened — moving the increment below the flush would hand back a turn the provider billed. Asserted through the CAP, never through `#turnCount`: it is a JS private field, and adding a production accessor to observe it would weaken encapsulation to satisfy a test. The regression drives `maxTurns: 1` with a rejecting flush, then proves the next turn is refused with `turn_limit`. Worth recording for whoever reads it: the rejection SURFACES out of `sendMessage` (ADR-0074 §2 fails the active owner loudly), so the test awaits a rejection before driving the cap. **`CR-03` — it was FOUR paths, not three.** `apps/cli/src/commands/import.ts` writes the same shape and was not in the finding; its payload carries `parsed.slug`, straight out of an imported artifact, which makes it if anything the most attacker-reachable of the set. All four now go through `stringifyJsonLine`, whose escape is lossless so no machine contract changes. The regression asserts at the SOURCE rather than driving four command harnesses. What the finding is about is a call site, and a call site is what a future edit reintroduces — driving harnesses would mostly test the harnesses, and the escaping behaviour is already covered in `sanitize.test.ts`. Its hostile payload is built from code points at runtime rather than typed into the file, because a raw C1/bidi byte in source is a Trojan-Source hazard in its own right. Break-verified, each with the mutation confirmed applied: reverting the cancel arm to the bare store call reddens the latch test; moving the turn increment below the flush reddens the cap test; reverting one `--json` surface reddens its row in the per-surface matrix. pnpm run ci EXIT=0. Refs: CR-01, CR-02, CR-03 Co-Authored-By: Claude Opus 5 --- apps/cli/src/chat/persister.test.ts | 34 ++++++++++ apps/cli/src/chat/persister.ts | 20 +++++- apps/cli/src/commands/agent-run.ts | 3 +- apps/cli/src/commands/chat-export.ts | 3 +- apps/cli/src/commands/chat.ts | 3 +- apps/cli/src/commands/import.ts | 3 +- .../cli/src/render/json-line-surfaces.test.ts | 65 +++++++++++++++++++ ...hase-2.6.5-core-reliability-remediation.md | 28 +++++++- .../core/src/engine/agent-session.test.ts | 27 ++++++++ packages/core/src/engine/agent-session.ts | 6 ++ 10 files changed, 182 insertions(+), 10 deletions(-) create mode 100644 apps/cli/src/render/json-line-surfaces.test.ts diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index e07febc5..f89c7c93 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -250,6 +250,40 @@ describe('createSessionPersister', () => { persister.close(); }); + it('LATCHES a failed session:cancelled write, and still detaches the listener (CR-01)', async () => { + // The one arm that still called the store bare. `RunEventBus` isolates a listener throw from the + // producer, so a failing terminal write let the cancel path report success while the latch never set — + // and the latch is what refuses the next egress. A session whose terminal row never landed would resume + // believing it had ended cleanly. + const { built } = await setup(scriptedResolver([textTurn('hi')])); + const persister = createSessionPersister({ + governor: undefined, + attachDurabilityProbe: built.attachDurabilityProbe, + store, + handle: built.handle, + sessionId: built.sessionId, + agent: built.agent, + context: built.context, + now: () => Date.parse('2026-06-25T00:00:00.000Z'), + uuid: () => 'msg-x', + }); + built.session.start(); + persister.start(); + vi.spyOn(store, 'updateSession').mockImplementation(() => { + throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + }); + + built.session.cancel(); + vi.restoreAllMocks(); + + expect(persister.durabilityFailure?.message).toContain('database is locked'); + // The self-detach is in a `finally`, so a failed write cannot leave the bus listener attached — which is + // exactly when a leak is worst, since every later event would re-enter a persister that cannot write. + expect(() => { + persister.close(); + }).not.toThrow(); + }); + it('does not let a failed cost write advance the in-memory total (#W15-4)', async () => { // `mutableSessionColumns` makes `recordSessionCost` the SINGLE writer of the durable total, so a total // advanced past a write that did not land can never be repaired by a later flush — and a resume then diff --git a/apps/cli/src/chat/persister.ts b/apps/cli/src/chat/persister.ts index f20950da..c70d0d9a 100644 --- a/apps/cli/src/chat/persister.ts +++ b/apps/cli/src/chat/persister.ts @@ -412,9 +412,23 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi case 'session:cancelled': // The session's sole terminal — mark it ended (still resumable from the persisted transcript), then // self-detach so the bus listener does not leak if the REPL's close() is skipped on an early exit. - deps.store.updateSession(record('ended')); - unsubscribe?.(); - unsubscribe = undefined; + // + // Through `persistDurably`, like every other write here (`CR-01`). It was the one arm still calling + // the store bare, and the gap is not cosmetic: `RunEventBus` isolates a listener throw from the + // producer, so a failing write let the cancel path report success while the latch never set — and the + // latch is what refuses the next egress. A session whose terminal row never landed would resume + // believing it had ended cleanly. + // + // The self-detach stays OUTSIDE the wrapper on purpose: `persistDurably` re-throws, and unsubscribing + // is cleanup that must happen either way — a failed write is exactly when a leaked listener is worst. + try { + persistDurably(() => { + deps.store.updateSession(record('ended')); + }); + } finally { + unsubscribe?.(); + unsubscribe = undefined; + } return; default: return; diff --git a/apps/cli/src/commands/agent-run.ts b/apps/cli/src/commands/agent-run.ts index 21ccf674..db6abd0e 100644 --- a/apps/cli/src/commands/agent-run.ts +++ b/apps/cli/src/commands/agent-run.ts @@ -18,6 +18,7 @@ import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; import { createMcpSecretResolver, type McpSecretResolver } from '../secrets/mcp-secret.js'; import { makePlainPrinter } from './chat.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium agent run ` (2.Q) — invoke a single agent **one-shot** (non-interactive) on the same @@ -158,7 +159,7 @@ async function runOneShotTurn( ): Promise { let turnErrorCode: string | undefined; const renderer: (event: SessionStreamHandleEvent) => void = deps.global.json - ? (event) => deps.io.writeOut(`${JSON.stringify(event)}\n`) + ? (event) => deps.io.writeOut(`${stringifyJsonLine(event)}\n`) : // A ONE-SHOT: the session is cancelled in `finally` right after, so suppress the session-continuity recovery // hint (2.5.H) — "the session is still active / resend / `/compact`" would be false with no live REPL. makePlainPrinter(deps.io, false); diff --git a/apps/cli/src/commands/chat-export.ts b/apps/cli/src/commands/chat-export.ts index 17bf4fea..672fa2ca 100644 --- a/apps/cli/src/commands/chat-export.ts +++ b/apps/cli/src/commands/chat-export.ts @@ -6,6 +6,7 @@ import { openSessionStore, type OpenedSessionStore } from '../history/session-op import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium chat-export ` (2.P) — export a persisted session to a `.relavium.yaml` **scaffold** for @@ -87,7 +88,7 @@ export function chatExportCommand( sequenceNumber: result.sequenceNumber, workflowPath: result.path, }); - deps.io.writeOut(`${JSON.stringify(event)}\n`); + deps.io.writeOut(`${stringifyJsonLine(event)}\n`); } else { deps.io.writeOut(`Exported session ${args.sessionId} to ${result.path}\n`); } diff --git a/apps/cli/src/commands/chat.ts b/apps/cli/src/commands/chat.ts index 5abd305c..c06f1b19 100644 --- a/apps/cli/src/commands/chat.ts +++ b/apps/cli/src/commands/chat.ts @@ -109,6 +109,7 @@ import { import { createChatStore, type ChatStoreController } from '../render/tui/chat-store.js'; import { createMentionReader, type MentionReader } from '../render/tui/mention.js'; import { createMcpSecretResolver, type McpSecretResolver } from '../secrets/mcp-secret.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium chat` (2.M) — the agent-first interactive REPL over `@relavium/core`'s `AgentSession`. It binds @@ -2389,7 +2390,7 @@ export async function drivePlain(ctx: ChatDriveContext): Promise { const unsubscribe = ctx.handle.subscribe((event) => - ctx.io.writeOut(`${JSON.stringify(event)}\n`), + ctx.io.writeOut(`${stringifyJsonLine(event)}\n`), ); const rl = createInterface({ input: ctx.io.stdin, terminal: false }); const onSigint = (): void => rl.close(); diff --git a/apps/cli/src/commands/import.ts b/apps/cli/src/commands/import.ts index a71bf5c1..a8d57350 100644 --- a/apps/cli/src/commands/import.ts +++ b/apps/cli/src/commands/import.ts @@ -12,6 +12,7 @@ import { import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; export interface ImportCommandArgs { /** The external workflow/agent YAML to copy into the project (absolute, or relative to cwd). */ @@ -58,7 +59,7 @@ export function importCommand(args: ImportCommandArgs, deps: ImportCommandDeps): writeAuthoredFile(target, rel, serializeAuthored(parsed), args.force); if (deps.global.json) { - deps.io.writeOut(`${JSON.stringify({ id: parsed.slug, kind: parsed.kind, path: rel })}\n`); + deps.io.writeOut(`${stringifyJsonLine({ id: parsed.slug, kind: parsed.kind, path: rel })}\n`); } else { deps.io.writeOut(`Imported ${parsed.kind} '${parsed.slug}' to ${rel}\n`); } diff --git a/apps/cli/src/render/json-line-surfaces.test.ts b/apps/cli/src/render/json-line-surfaces.test.ts new file mode 100644 index 00000000..3d184a65 --- /dev/null +++ b/apps/cli/src/render/json-line-surfaces.test.ts @@ -0,0 +1,65 @@ +/** + * `CR-03` — every `--json` surface goes through the safe serializer, not just the two `#W15-10` reached. + * + * `#W15-10` built `stringifyJsonLine` and wired the workflow renderer and the record writer to it. Four + * sibling paths kept a bare `JSON.stringify`, which escapes `ESC` but leaves `U+009B` (the 8-bit CSI), `DEL` + * and the whole Trojan-Source bidi family RAW — and all four carry model-, tool- or artifact-derived content + * straight to a terminal. A propagation gap: the fix landed, the siblings did not get it. + * + * This is a SOURCE-level assertion rather than four end-to-end drives, deliberately. What the finding is + * about is a call site, and a call site is what a future edit reintroduces; wiring four command harnesses + * would test the harnesses. The escaping behaviour itself is covered in `sanitize.test.ts`. + */ +import { readFileSync } from 'node:fs'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { stringifyJsonLine } from './sanitize.js'; + +const srcRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..'); + +/** Every file that writes NDJSON to stdout under `--json`. Adding a fifth means adding it here. */ +const JSON_SURFACES = [ + 'commands/agent-run.ts', + 'commands/chat.ts', + 'commands/chat-export.ts', + 'commands/import.ts', +]; + +describe('CR-03 — the --json surfaces', () => { + it.each(JSON_SURFACES)( + '%s emits through stringifyJsonLine, never a bare JSON.stringify', + (rel) => { + const source = readFileSync(join(srcRoot, rel), 'utf8'); + // The shape that reaches a terminal: a template-literal NDJSON line written to stdout. + const bare = [...source.matchAll(/write(?:Out|Err)\(`\$\{JSON\.stringify\(/g)]; + expect( + bare, + `${rel} writes a --json line with a bare JSON.stringify. That leaves U+009B, DEL and the bidi ` + + `family raw in content the model or an imported artifact controls. Use stringifyJsonLine — the ` + + `escape is lossless, so no machine contract changes.`, + ).toEqual([]); + expect(source, `${rel} should import the safe serializer`).toContain('stringifyJsonLine'); + }, + ); + + it('the shared serializer neutralizes what a bare JSON.stringify leaves raw', () => { + // Built from code points at RUNTIME, never typed into this file: a raw C1/bidi byte in source is a + // Trojan-Source hazard in its own right, and an editor or a tool in the chain can silently rewrite it. + const CSI_8BIT = String.fromCharCode(0x9b); // the 8-bit CSI — `JSON.stringify` leaves it raw + const DEL = String.fromCharCode(0x7f); + const RLO = String.fromCharCode(0x202e); // right-to-left override + const hostile = { id: 'n1', text: `ok${CSI_8BIT}2J${RLO}evil${DEL}` }; + const safe = stringifyJsonLine(hostile); + const bare = JSON.stringify(hostile); + + for (const raw of [CSI_8BIT, RLO, DEL]) { + expect(bare, 'the bare form is the hazard this item is about').toContain(raw); + expect(safe).not.toContain(raw); + } + // Lossless: `--json` promises to reproduce the data, which is why this escapes rather than strips. + expect(JSON.parse(safe)).toEqual(hostile); + }); +}); diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index db251f12..6c30b3ec 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -237,7 +237,7 @@ to correct. These are partially closed. Each has a verified open half. -### CR-01 — `session:cancelled` bypasses the session durability latch · High +### CR-01 — `session:cancelled` bypasses the session durability latch · High · ✅ CLOSED 2026-08-11 **Evidence.** The CLI chat persister wraps its cost and turn writes in the `persistDurably` failure latch, and since Wave 1 the `session:compacted` and `session:trimmed` arms go through it too. The `session:cancelled` arm @@ -251,7 +251,12 @@ cancel path report success. The durability latch never sets, so nothing gates th **Acceptance.** A failing `writeTurn` on cancel sets `durabilityFailure`, and the next egress is refused. Break-verify by removing the wrapper. -### CR-02 — A failed turn flush leaves the turn counter incremented · Medium +**Closed.** One correction to the finding: the arm writes through `updateSession`, not `writeTurn` — the +terminal marks the session `ended` rather than appending a turn. The self-detach stays OUTSIDE the wrapper, in +a `finally`: `persistDurably` re-throws, and a failed write is exactly when a leaked bus listener is worst, +since every later event would re-enter a persister that cannot write. + +### CR-02 — A failed turn flush leaves the turn counter incremented · Medium · ✅ CLOSED 2026-08-11 **Evidence.** `packages/core/src/engine/agent-session.ts` increments `#turnCount` on the success path *before* it awaits `flushBudgetCommitments()`. Wave 1 fixed the transcript half of this (the flush now precedes the @@ -271,7 +276,14 @@ against the cap. `#turnCount` is a JS private field and **must not** be given a assert through the observable turn-cap behaviour (`maxTurns: 1`, one flushing-failing turn, the next `sendMessage` refused by the cap). Break-verify by moving the increment below the flush. -### CR-03 — Three `--json` paths still bypass the safe serializer · High +**Closed — the counter stays, and the code now says why.** The decision was made from the code rather than +chosen: both catch paths already apply *count a turn only when a provider ENGAGED*, and by the increment +`#runTurn` has resolved. A flush rejection past that point is a durability failure, not evidence the turn +never happened — moving the increment below the flush hands back a turn the provider billed. One detail worth +recording for the test: the rejection SURFACES out of `sendMessage` (ADR-0074 §2 fails the active owner +loudly), so the regression awaits a rejection and then drives the cap. + +### CR-03 — Three `--json` paths still bypass the safe serializer · High · ✅ CLOSED 2026-08-11 **Evidence.** Wave 1 introduced `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), which losslessly escapes C1 controls and Trojan-Source bidi characters, and routed the workflow renderer and the record writer @@ -288,6 +300,16 @@ propagation gap — the fix landed, the siblings did not get it. **Acceptance.** One shared adversarial test drives a C1 + bidi payload through all three surfaces and asserts the raw code points do not survive while `JSON.parse` round-trips to the identical string. +**Closed — and it was FOUR paths, not three.** `apps/cli/src/commands/import.ts` writes the same shape and was +not in the finding; its payload carries `parsed.slug`, which comes straight out of an imported artifact, so it +is if anything the most attacker-reachable of the set. All four now go through `stringifyJsonLine`. + +The regression (`apps/cli/src/render/json-line-surfaces.test.ts`) asserts at the SOURCE, not through four +command harnesses: what the finding is about is a call site, and a call site is what a future edit +reintroduces — driving four harnesses would mostly test the harnesses. The escaping behaviour itself stays +covered in `sanitize.test.ts`. Its hostile payload is built from code points at runtime rather than typed into +the file, because a raw C1/bidi byte in source is a Trojan-Source hazard in its own right. + --- ## W1 — The P0 blockers diff --git a/packages/core/src/engine/agent-session.test.ts b/packages/core/src/engine/agent-session.test.ts index cee01442..8d523fe7 100644 --- a/packages/core/src/engine/agent-session.test.ts +++ b/packages/core/src/engine/agent-session.test.ts @@ -366,6 +366,33 @@ describe('AgentSession (1.V) — multi-turn entry point over the shared turn cor expect(completed.tokensUsed).toEqual({ input: 4, output: 2 }); }); + it('a turn whose durability flush REJECTS still consumes its slot against the cap (CR-02)', async () => { + // `CR-02`'s decision, pinned. The counter is incremented BEFORE `flushBudgetCommitments` is awaited, and + // that ordering is deliberate: the rule both catch paths apply is "count a turn only when a provider + // ENGAGED", and by the increment `#runTurn` has resolved. A flush rejection past that point is a + // durability failure, not evidence the turn never happened — moving the increment below the flush would + // hand back a turn the provider billed. + // + // Asserted through the CAP, not through `#turnCount`. It is a JS private field, and giving it a + // production accessor to satisfy a test would weaken the encapsulation to observe it. + const { deps, events } = harness([textTurn('billed, then unrecordable')], { + maxTurns: 1, + flushBudgetCommitments: () => Promise.reject(new Error('disk full')), + }); + const s = session(deps); + s.start(); + // Turn 1 engages a provider and then fails on the flush. The rejection surfaces (ADR-0074 §2: a + // durability failure fails the ACTIVE owner loudly) — and the turn still counts. + await expect(s.sendMessage('first')).rejects.toThrow('disk full'); + events.length = 0; + await s.sendMessage('second — must be over the cap'); + + const completed = events.find((e) => e.type === 'session:turn_completed'); + expect(completed?.type === 'session:turn_completed' ? completed.error?.code : undefined).toBe( + 'turn_limit', + ); + }); + it('emits session:turn_completed{turn_limit} — loudly, no egress — when driven past the hard cap', async () => { // MANDATORY regression: a turn-loop refactor must not be able to silently drop the cap signal. // maxTurns 1: turn 1 runs; turn 2 is blocked with turn_limit and never reaches the provider. diff --git a/packages/core/src/engine/agent-session.ts b/packages/core/src/engine/agent-session.ts index 8e0d5c13..a04b2716 100644 --- a/packages/core/src/engine/agent-session.ts +++ b/packages/core/src/engine/agent-session.ts @@ -625,6 +625,12 @@ export class AgentSession { // (the reply is kept, the turn is counted). `abort()` interrupts an IN-FLIGHT turn only; a turn the // model already finished is not discarded. This success path **never reads `#abortingTurn`** — that is // precisely what makes a late abort structurally invisible here; the `finally` still clears the marker. + // Counted HERE, BEFORE the durability flush below, and that ordering is a decision rather than an + // accident (`CR-02`). The rule both catch paths already apply is *count a turn against the hard cap + // only when a provider actually ENGAGED* — and by this line `#runTurn` has resolved, so one did. A + // `flushBudgetCommitments` rejection past this point is a durability failure, not evidence the turn + // never happened; moving the increment below the flush would hand back a turn the provider billed. + // Pinned by "a turn whose durability flush REJECTS still consumes its slot against the cap". this.#turnCount += 1; // Append the assistant reply to the cross-turn transcript as TEXT-ONLY. The turn core keeps the // within-turn tool_use/tool_result pairs internal (they never leave runAgentTurn — it returns only the From 1f5d4ecc7798404cd7c55c5336cdb00a8c0e804a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 00:35:59 +0300 Subject: [PATCH 21/26] =?UTF-8?q?fix(cli,core,docs):=20fold=20the=20Opus?= =?UTF-8?q?=20review=20of=20W0=20=E2=80=94=20a=20fifth=20path,=20and=20two?= =?UTF-8?q?=20vacuous=20assertions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ten findings. Three of my own claims were false, and two of the tests I had just break-verified turned out to prove nothing. **`CR-03` was FIVE paths, not four — my headline correction was itself wrong.** `apps/cli/src/commands/export.ts` writes the identical record shape as the `import.ts` I had just added, and `docs/reference/cli/commands.md` pairs the two by that shape in one sentence. I reasoned from the shape and missed the file the spec names. Worse for the threat model: `export`'s `path` derives from the user-typed `--out`, so it genuinely can carry a C1 or bidi byte — which brings me to the second false claim. **`import --json` is the LEAST attacker-reachable of the set, not the most.** I justified adding it by saying `parsed.slug` "comes straight out of an imported artifact". It does — and `kebabIdSchema` rejects the document before that value exists, so it cannot carry a hostile code point at all. Cheap hygiene, wrongly ranked, and the ranking was already recorded in the phase doc as the reason the item's scope grew. **The source-scanning regression was the wrong mechanism and proved it immediately.** It matched `writeOut(`${JSON.stringify(` and missed `export.ts` purely because prettier wrapped the argument onto its own line. A guard that checks only the places someone remembered catches nothing new by construction — which is how this gap reopened twice already (`#W15-10`, then `CR-03`). The call-site half is now an ESLint `no-restricted-syntax` selector firing on the SHAPE anywhere in `apps/cli/src`; break-verified by reverting `export.ts` to the prettier-wrapped form the regex could not see. `render-error.ts` is allowlisted — it pre-strips, so the error envelope is lossy on purpose. The test keeps only the behavioural assertion. **`CR-01`'s stated rationale was false, and the test asserted the wrong half.** The latch cannot refuse a next egress for a TERMINAL event: `session:cancelled` sets `#status = 'cancelled'` and every later entry point is already refused by `#assertSendable`. Nothing reads `durabilityFailure` again. The wrapper is symmetry; the load-bearing half is the `finally`, which stops a throwing write jumping over the unsubscribe and leaving the persister attached to a bus it can no longer serve. (The user was told either way — the throw always escaped into `deliver`'s listener-error sink.) Two attempts at pinning that were vacuous and BOTH passed with the `finally` deleted: `expect(close()).not.toThrow()` (`close()` calls the same idempotent unsubscribe) and driving a second `cancel()` (which emits no event on an already-cancelled session). The working assertion wraps the handle's unsubscribe and counts it — `expected +0 to be 1` under the mutation. **`CR-02`'s decision was right and its accounting stopped one line short.** A flush rejection settles through the unclassified branch, which emitted `{0,0}` — so the turn the decision insists the provider billed consumed its cap slot AND silently dropped its real tokens. That is the mirror of the error the decision refuses to make, and it contradicts EA2/ADR-0055. The success path now captures the usage just before the flush; the unclassified terminal reports it. The mutation for that first "did not apply" — prettier had collapsed the line my anchor matched — and it silently passed. Caught only by asserting the anchor exists, which is the whole reason that discipline is written down. Also: the two comments in `models.ts` / `models-pricing.ts` asserting that "JSON.stringify escapes any control byte on its own" — the exact false premise this item exists to correct, left standing in the same package. And `security-review.md` now carries the machine-output arm of the terminal floor, so the rule has a normative home instead of living in one TSDoc block. pnpm run ci EXIT=0. Refs: CR-01, CR-02, CR-03 Co-Authored-By: Claude Opus 5 --- apps/cli/src/chat/persister.test.ts | 33 ++++++++--- apps/cli/src/chat/persister.ts | 18 +++--- apps/cli/src/commands/export.ts | 12 +++- apps/cli/src/commands/models-pricing.ts | 7 ++- apps/cli/src/commands/models.ts | 2 +- .../cli/src/render/json-line-surfaces.test.ts | 48 +++------------- ...hase-2.6.5-core-reliability-remediation.md | 57 ++++++++++++++----- docs/standards/security-review.md | 10 ++++ eslint.config.mjs | 39 +++++++++++++ .../core/src/engine/agent-session.test.ts | 16 +++++- packages/core/src/engine/agent-session.ts | 33 ++++++++--- 11 files changed, 192 insertions(+), 83 deletions(-) diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index f89c7c93..667743e9 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -267,21 +267,40 @@ describe('createSessionPersister', () => { now: () => Date.parse('2026-06-25T00:00:00.000Z'), uuid: () => 'msg-x', }); + // Observe the DETACH directly by wrapping the unsubscribe the handle hands back. Nothing else can see + // it: a second `cancel()` emits no event on an already-cancelled session, and `close()` calls the same + // idempotent unsubscribe, so both of those pass whether or not the listener leaked. + let unsubscribed = 0; + const realSubscribe = built.handle.subscribe.bind(built.handle); + vi.spyOn(built.handle, 'subscribe').mockImplementation((listener) => { + const off = realSubscribe(listener); + return () => { + unsubscribed += 1; + off(); + }; + }); built.session.start(); persister.start(); - vi.spyOn(store, 'updateSession').mockImplementation(() => { + const updateSpy = vi.spyOn(store, 'updateSession').mockImplementation(() => { throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); }); built.session.cancel(); - vi.restoreAllMocks(); expect(persister.durabilityFailure?.message).toContain('database is locked'); - // The self-detach is in a `finally`, so a failed write cannot leave the bus listener attached — which is - // exactly when a leak is worst, since every later event would re-enter a persister that cannot write. - expect(() => { - persister.close(); - }).not.toThrow(); + // …and the user was still told: latching must not remove the listener-error notice. + expect(listenerNotes.join('\n')).toMatch(/database is locked/); + + // **The half that actually matters, asserted directly.** The self-detach is in a `finally`, so a throwing + // write cannot jump over it and leave this persister attached to the bus. An earlier version of this test + // asserted `close()` does not throw — which is true whether or not the listener leaked, because `close()` + // calls the same idempotent unsubscribe. The real evidence is that a SUBSEQUENT event does not re-enter a + // persister that cannot write, so `updateSession` is never called again. Deliberately WITHOUT `close()`. + expect(updateSpy.mock.calls.length).toBe(1); // the failing write happened exactly once + // THE assertion: the throw did not jump over the unsubscribe. Without the `finally` this is 0, and the + // persister stays on the bus — so every later event re-enters one that cannot write. + expect(unsubscribed).toBe(1); + vi.restoreAllMocks(); }); it('does not let a failed cost write advance the in-memory total (#W15-4)', async () => { diff --git a/apps/cli/src/chat/persister.ts b/apps/cli/src/chat/persister.ts index c70d0d9a..08e8f5ff 100644 --- a/apps/cli/src/chat/persister.ts +++ b/apps/cli/src/chat/persister.ts @@ -413,14 +413,18 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi // The session's sole terminal — mark it ended (still resumable from the persisted transcript), then // self-detach so the bus listener does not leak if the REPL's close() is skipped on an early exit. // - // Through `persistDurably`, like every other write here (`CR-01`). It was the one arm still calling - // the store bare, and the gap is not cosmetic: `RunEventBus` isolates a listener throw from the - // producer, so a failing write let the cancel path report success while the latch never set — and the - // latch is what refuses the next egress. A session whose terminal row never landed would resume - // believing it had ended cleanly. + // Through `persistDurably`, like every other write here (`CR-01`) — but be precise about WHAT that + // buys, because the obvious reading is wrong. The latch half is inert on this arm: `session:cancelled` + // is emitted only by `AgentSession.cancel()`, which sets `#status = 'cancelled'`, and every later + // egress entry point is already refused by `#assertSendable()` with `not_active`. Nothing reads + // `durabilityFailure` again. It is wrapped for SYMMETRY with its siblings — one arm reaching the store + // bare is how the next reader concludes the wrapper is optional. // - // The self-detach stays OUTSIDE the wrapper on purpose: `persistDurably` re-throws, and unsubscribing - // is cleanup that must happen either way — a failed write is exactly when a leaked listener is worst. + // **The `finally` is the load-bearing half.** Before it, a throwing write jumped straight over the two + // unsubscribe lines and left this persister attached to the bus — so every later event re-entered a + // persister that could not write. (The user was told either way: the throw always escaped `onEvent` + // into `deliver`'s listener-error sink.) Cleanup must happen on both paths, and a failed write is + // exactly when a leak is worst. try { persistDurably(() => { deps.store.updateSession(record('ended')); diff --git a/apps/cli/src/commands/export.ts b/apps/cli/src/commands/export.ts index 97164a33..62d9bade 100644 --- a/apps/cli/src/commands/export.ts +++ b/apps/cli/src/commands/export.ts @@ -12,6 +12,7 @@ import { import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine, stripTerminalControls } from '../render/sanitize.js'; export interface ExportCommandArgs { /** The in-file `id` of the workflow/agent to export (resolved across both project catalogs). */ @@ -59,11 +60,18 @@ export function exportCommand(args: ExportCommandArgs, deps: ExportCommandDeps): // Emit the cwd-relative path in both modes — the same contract `import` uses, so a script consuming either // command's `--json` gets the same path shape (and no absolute filesystem tree leaks into the output). if (deps.global.json) { + // `stringifyJsonLine`, not a bare `JSON.stringify` (CR-03). `targetDisplay` derives from the user-typed + // `--out`, so unlike `import --json`'s payload — whose `slug` is kebab-schema-validated and cannot carry + // one — it really can hold a C1 control or a bidi override, which `JSON.stringify` leaves raw. deps.io.writeOut( - `${JSON.stringify({ id: parsed.slug, kind: parsed.kind, path: targetDisplay })}\n`, + `${stringifyJsonLine({ id: parsed.slug, kind: parsed.kind, path: targetDisplay })}\n`, ); } else { - deps.io.writeOut(`Exported ${parsed.kind} '${parsed.slug}' to ${targetDisplay}\n`); + // The human line needs the terminal floor for the same reason: every dynamic string written to a terminal + // passes `stripTerminalControls` (security-review.md), and this one is user-supplied. + deps.io.writeOut( + `Exported ${parsed.kind} '${parsed.slug}' to ${stripTerminalControls(targetDisplay)}\n`, + ); } return EXIT_CODES.success; } diff --git a/apps/cli/src/commands/models-pricing.ts b/apps/cli/src/commands/models-pricing.ts index d861ba3b..f9fddde8 100644 --- a/apps/cli/src/commands/models-pricing.ts +++ b/apps/cli/src/commands/models-pricing.ts @@ -217,9 +217,10 @@ export function modelsPricingCommand( ? '' : `\n Overrides the catalog price for this model: input $${microcentsToUsd(shipped.inputPerMtokMicrocents)}/Mtok, output $${microcentsToUsd(shipped.outputPerMtokMicrocents)}/Mtok. Yours wins. Run \`relavium models pricing ${stripTerminalControls(args.model)} --clear\` to go back to the catalog's.`; // Strip any terminal-control byte from the (user-typed) model id before echo — parity with `renderModelList`'s - // FIX 2. `ModelListingSchema` only requires min(1), so an id can carry a control byte; the JSON path is safe on - // its own (JSON.stringify escapes them). The provider is a validated (kebab) ProviderId, and the prices are - // numbers — both already safe. + // FIX 2. `ModelListingSchema` only requires min(1), so an id can carry a control byte; the JSON path is safe + // because it goes through `stringifyJsonLine` — NOT because `JSON.stringify` escapes them, which it does not + // (see security-review.md). The provider is a validated (kebab) ProviderId, and the prices are numbers — + // both already safe. deps.io.writeOut( `Set user pricing for ${stripTerminalControls(args.model)} (${args.provider}): input $${args.inputUsdPerMtok}/Mtok, output $${args.outputUsdPerMtok}/Mtok${cachedNote}. It applies to your next run/chat and survives \`models refresh\`.${divergence}\n`, ); diff --git a/apps/cli/src/commands/models.ts b/apps/cli/src/commands/models.ts index e0b6356e..0ecaa7bc 100644 --- a/apps/cli/src/commands/models.ts +++ b/apps/cli/src/commands/models.ts @@ -247,7 +247,7 @@ function toModelJson(m: ModelCatalogListing, providerSlug: (uuid: string) => str return { // The SLUG (e.g. `anthropic`), not the internal `llm_providers` UUID the catalog row carries — matching the // `models refresh` report + the documented `{ provider }` contract. `--json` is unchanged otherwise - // (JSON.stringify escapes any control byte on its own, so the slug is not terminal-sanitized here). + // (the record goes out through `writeRecordLines` → `stringifyJsonLine`, which is what escapes the C1/bidi bytes `JSON.stringify` leaves raw — see security-review.md). provider: providerSlug(m.providerId), modelId: m.modelId, displayName: m.displayName, diff --git a/apps/cli/src/render/json-line-surfaces.test.ts b/apps/cli/src/render/json-line-surfaces.test.ts index 3d184a65..4ab3e869 100644 --- a/apps/cli/src/render/json-line-surfaces.test.ts +++ b/apps/cli/src/render/json-line-surfaces.test.ts @@ -1,50 +1,18 @@ /** - * `CR-03` — every `--json` surface goes through the safe serializer, not just the two `#W15-10` reached. + * `CR-03` — the behavioural half of the machine-output floor. * - * `#W15-10` built `stringifyJsonLine` and wired the workflow renderer and the record writer to it. Four - * sibling paths kept a bare `JSON.stringify`, which escapes `ESC` but leaves `U+009B` (the 8-bit CSI), `DEL` - * and the whole Trojan-Source bidi family RAW — and all four carry model-, tool- or artifact-derived content - * straight to a terminal. A propagation gap: the fix landed, the siblings did not get it. - * - * This is a SOURCE-level assertion rather than four end-to-end drives, deliberately. What the finding is - * about is a call site, and a call site is what a future edit reintroduces; wiring four command harnesses - * would test the harnesses. The escaping behaviour itself is covered in `sanitize.test.ts`. + * `#W15-10` built `stringifyJsonLine` and wired two call sites. `CR-03` found three more; the fold of its own + * review then found a SIXTH (`commands/export.ts`) that a source-scanning regex had missed purely because + * prettier wrapped the argument onto its own line. A guard that only checks the places someone remembered + * catches nothing new, so the CALL-SITE half is now an ESLint `no-restricted-syntax` selector in + * `eslint.config.mjs` — it fires on the shape, anywhere in `apps/cli/src`, the first time a new surface is + * written. That is the mechanism; this file keeps the reason it exists visible as executable behaviour. */ -import { readFileSync } from 'node:fs'; -import { dirname, join, resolve } from 'node:path'; -import { fileURLToPath } from 'node:url'; - import { describe, expect, it } from 'vitest'; import { stringifyJsonLine } from './sanitize.js'; -const srcRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..'); - -/** Every file that writes NDJSON to stdout under `--json`. Adding a fifth means adding it here. */ -const JSON_SURFACES = [ - 'commands/agent-run.ts', - 'commands/chat.ts', - 'commands/chat-export.ts', - 'commands/import.ts', -]; - -describe('CR-03 — the --json surfaces', () => { - it.each(JSON_SURFACES)( - '%s emits through stringifyJsonLine, never a bare JSON.stringify', - (rel) => { - const source = readFileSync(join(srcRoot, rel), 'utf8'); - // The shape that reaches a terminal: a template-literal NDJSON line written to stdout. - const bare = [...source.matchAll(/write(?:Out|Err)\(`\$\{JSON\.stringify\(/g)]; - expect( - bare, - `${rel} writes a --json line with a bare JSON.stringify. That leaves U+009B, DEL and the bidi ` + - `family raw in content the model or an imported artifact controls. Use stringifyJsonLine — the ` + - `escape is lossless, so no machine contract changes.`, - ).toEqual([]); - expect(source, `${rel} should import the safe serializer`).toContain('stringifyJsonLine'); - }, - ); - +describe('CR-03 — the --json machine-output floor', () => { it('the shared serializer neutralizes what a bare JSON.stringify leaves raw', () => { // Built from code points at RUNTIME, never typed into this file: a raw C1/bidi byte in source is a // Trojan-Source hazard in its own right, and an editor or a tool in the chain can silently rewrite it. diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index 6c30b3ec..f566628d 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -251,10 +251,21 @@ cancel path report success. The durability latch never sets, so nothing gates th **Acceptance.** A failing `writeTurn` on cancel sets `durabilityFailure`, and the next egress is refused. Break-verify by removing the wrapper. -**Closed.** One correction to the finding: the arm writes through `updateSession`, not `writeTurn` — the -terminal marks the session `ended` rather than appending a turn. The self-detach stays OUTSIDE the wrapper, in -a `finally`: `persistDurably` re-throws, and a failed write is exactly when a leaked bus listener is worst, -since every later event would re-enter a persister that cannot write. +**Closed — but the acceptance above is half unsatisfiable, and the real defect was the other half.** + +- The arm writes through `updateSession`, not `writeTurn`: the terminal marks the session `ended` rather than + appending a turn. +- **"the next egress is refused" cannot hold for a terminal event.** `session:cancelled` is emitted only by + `AgentSession.cancel()`, which sets `#status = 'cancelled'`, and every later egress entry point is already + refused by `#assertSendable()` with `not_active`. Nothing reads `durabilityFailure` again, and each + construction site builds a fresh persister with a fresh latch. The latch half is wrapped for SYMMETRY — + one arm reaching the store bare is how the next reader concludes the wrapper is optional. +- **The load-bearing half is the `finally`.** A throwing write jumped straight over the two unsubscribe lines + and left the persister attached to the bus, so every later event re-entered one that could not write. (The + user was told either way — the throw always escaped into `deliver`'s listener-error sink, so "let the cancel + path report success" was also not quite right.) Pinned by wrapping the handle's unsubscribe and asserting it + ran; the first two attempts at that assertion were vacuous — `close()` calls the same idempotent + unsubscribe, and a second `cancel()` emits no event — and both passed with the `finally` deleted. ### CR-02 — A failed turn flush leaves the turn counter incremented · Medium · ✅ CLOSED 2026-08-11 @@ -283,6 +294,13 @@ never happened — moving the increment below the flush hands back a turn the pr recording for the test: the rejection SURFACES out of `sendMessage` (ADR-0074 §2 fails the active owner loudly), so the regression awaits a rejection and then drives the cap. +**And the decision had a mirror-image hole one line further on.** A flush rejection settles through the +unclassified branch, which emitted its terminal with a hardcoded `{0,0}` — so the turn the decision insists +the provider billed consumed its cap slot AND silently dropped its real tokens from every total. That is the +opposite of the error the counter decision refuses to make, and it contradicts EA2/ADR-0055's rule to report +real usage whenever a provider engaged. The success path now captures the usage just before the flush and the +unclassified terminal reports it, falling back to zero when nothing engaged. + ### CR-03 — Three `--json` paths still bypass the safe serializer · High · ✅ CLOSED 2026-08-11 **Evidence.** Wave 1 introduced `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), which losslessly @@ -300,15 +318,28 @@ propagation gap — the fix landed, the siblings did not get it. **Acceptance.** One shared adversarial test drives a C1 + bidi payload through all three surfaces and asserts the raw code points do not survive while `JSON.parse` round-trips to the identical string. -**Closed — and it was FOUR paths, not three.** `apps/cli/src/commands/import.ts` writes the same shape and was -not in the finding; its payload carries `parsed.slug`, which comes straight out of an imported artifact, so it -is if anything the most attacker-reachable of the set. All four now go through `stringifyJsonLine`. - -The regression (`apps/cli/src/render/json-line-surfaces.test.ts`) asserts at the SOURCE, not through four -command harnesses: what the finding is about is a call site, and a call site is what a future edit -reintroduces — driving four harnesses would mostly test the harnesses. The escaping behaviour itself stays -covered in `sanitize.test.ts`. Its hostile payload is built from code points at runtime rather than typed into -the file, because a raw C1/bidi byte in source is a Trojan-Source hazard in its own right. +**Closed — and it was FIVE paths, not three.** The finding named three; `import.ts` and `export.ts` write the +same record shape and were not in it. `docs/reference/cli/commands.md` pairs the last two by that shape in one +sentence, which is how the miss was findable at all. + +Two corrections to my own reasoning while closing it, both recorded because they were stated as fact first: + +- **`import --json` is the LEAST attacker-reachable of the set, not the most.** I justified adding it by + saying `parsed.slug` "comes straight out of an imported artifact". It does — and `kebabIdSchema` rejects the + document before that value exists, so it cannot carry a C1 or bidi code point at all. The genuinely + reachable ones are the session-event streams the original finding already named, plus `export --json`, whose + `path` derives from the user-typed `--out`. Fixing `import` is cheap hygiene; the ranking was wrong. +- **A source-scanning test is the wrong mechanism, and it proved so immediately.** The first regression + matched `writeOut(\`${JSON.stringify(` and missed `export.ts` purely because prettier had wrapped the + argument onto its own line. A guard that only checks the places someone remembered catches nothing new by + construction — which is how this propagation gap reopened twice already. The call-site half is now an + ESLint `no-restricted-syntax` selector (`eslint.config.mjs`) that fires on the SHAPE anywhere in + `apps/cli/src`, so a sixth surface is caught the first time it is written rather than at review. + `json-line-surfaces.test.ts` keeps only the behavioural assertion, with its hostile payload built from code + points at runtime — a raw C1/bidi byte in source is a Trojan-Source hazard in its own right. + +`process/render-error.ts` is the one deliberate exception and is allowlisted: it pre-sanitizes with the +STRIPPING sanitizer, so the `--json` error envelope is lossy on purpose where the NDJSON stream is lossless. --- diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index d7902596..4b88a511 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -410,6 +410,16 @@ No secrets, no full prompts/responses, and no raw keys in logs — ever, **inclu *echoed back* in a provider response or error body, which must be redacted before it is logged.** Logging redaction rules live in [logging-and-observability.md](logging-and-observability.md). +**Machine output has its own arm of that floor, and it is the opposite choice.** Every `--json` record is +serialized with `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), never a bare `JSON.stringify`: +`JSON.stringify` escapes `ESC` and stops, leaving `DEL`, the whole C1 block — including `U+009B`, a working +escape-sequence introducer on real terminals — and the Trojan-Source bidi family RAW in content the model, a +tool, or an imported artifact controls. It **escapes** where the human-display path **strips**, because +`--json` is a machine contract and must reproduce its data byte for byte. The rule is enforced by an ESLint +`no-restricted-syntax` selector rather than by review, because the gap reopened twice when it was not +(`#W15-10`, then `CR-03`). `apps/cli/src/process/render-error.ts` is the single allowlisted exception: it +pre-sanitizes with the stripping sanitizer, so the `--json` ERROR envelope is deliberately lossy. + ## When a review is mandatory Any change to: key handling or the keychain bridge, IPC commands, the desktop diff --git a/eslint.config.mjs b/eslint.config.mjs index 13fd9c7c..95adf726 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -60,6 +60,36 @@ const seamSyntaxRules = /** @type {const} */ ([ }, ]); +/** + * The machine-output fence (`CR-03`) — every `--json` record leaves through `stringifyJsonLine`. + * + * `JSON.stringify` escapes `ESC` and stops there: `U+007F` (DEL), the whole C1 block including `U+009B` + * (the 8-bit CSI, a working escape sequence introducer on real terminals), and the Trojan-Source bidi family + * all survive into the emitted line. A `--json` stream carries model output, tool results and imported-artifact + * fields straight to a terminal, so a bare stringify is a terminal-injection surface. + * + * **A lint rule rather than a test, because the propagation gap kept reopening.** `#W15-10` fixed two call + * sites; `CR-03` found three more; a source-scanning regex written for `CR-03` then missed a sixth purely + * because prettier had wrapped the argument onto its own line. A guard that only checks the places someone + * remembered catches nothing new by construction. This selector fires on the SHAPE — a `JSON.stringify` call + * anywhere inside a `writeOut`/`writeErr` argument — so a new surface is caught the first time it is written. + * + * `stringifyJsonLine` ESCAPES rather than strips, deliberately: `--json` is a machine contract and must + * reproduce its data, so `JSON.parse` round-trips to the identical string. That is the opposite choice from + * the human-display floor (`stripTerminalControls`), and `process/render-error.ts` is the one deliberate + * exception — it pre-sanitizes its fields with the STRIPPING sanitizer and is therefore lossy on purpose. + */ +const JSON_LINE_MESSAGE = + 'A --json record must be serialized with stringifyJsonLine (apps/cli/src/render/sanitize.ts), not a bare ' + + 'JSON.stringify: JSON.stringify leaves DEL, the C1 block (incl. the 8-bit CSI) and the Trojan-Source bidi ' + + 'family RAW in content the model, a tool, or an imported artifact controls (CR-03). The escape is lossless, ' + + 'so no machine contract changes.'; +const jsonLineSyntaxRule = { + selector: + "CallExpression[callee.property.name=/^write(Out|Err)$/] CallExpression[callee.object.name='JSON'][callee.property.name='stringify']", + message: JSON_LINE_MESSAGE, +}; + /** * The pure-engine purity fence (CLAUDE.md rule 5). `packages/core` SHIPPING source must import no * platform-specific module and reach no Node global, so the engine runs identically in Node, the Tauri @@ -205,6 +235,15 @@ export default tseslint.config( 'no-restricted-syntax': seamSyntaxRules, }, }, + { + // The machine-output fence (CR-03). Scoped to the CLI, which is the only surface emitting NDJSON to a + // terminal today; `render-error.ts` is the documented exception (it pre-strips, and is lossy on purpose). + files: ['apps/cli/src/**/*.ts'], + ignores: ['apps/cli/src/process/render-error.ts', 'apps/cli/src/render/sanitize.ts'], + rules: { + 'no-restricted-syntax': [...seamSyntaxRules, jsonLineSyntaxRule], + }, + }, { // Plain JS (and any stray non-ignored JS): never attach type-aware rules — they // require a TS program and would otherwise crash ESLint on a file with no project. diff --git a/packages/core/src/engine/agent-session.test.ts b/packages/core/src/engine/agent-session.test.ts index 8d523fe7..163b2d54 100644 --- a/packages/core/src/engine/agent-session.test.ts +++ b/packages/core/src/engine/agent-session.test.ts @@ -384,8 +384,22 @@ describe('AgentSession (1.V) — multi-turn entry point over the shared turn cor // Turn 1 engages a provider and then fails on the flush. The rejection surfaces (ADR-0074 §2: a // durability failure fails the ACTIVE owner loudly) — and the turn still counts. await expect(s.sendMessage('first')).rejects.toThrow('disk full'); + + // The failed turn's terminal reports the REAL usage, not a hardcoded zero (`CR-02`). Consuming the cap + // slot while dropping the tokens is the mirror of the error the counter decision refuses to make: the + // provider engaged and billed either way. + const failedTurn = events.find( + (e) => e.type === 'session:turn_completed' && e.error?.code === 'internal', + ); + expect(failedTurn, `saw: ${JSON.stringify(events.map((e) => e.type))}`).toBeDefined(); + expect( + failedTurn?.type === 'session:turn_completed' ? failedTurn.tokensUsed : undefined, + ).not.toEqual({ input: 0, output: 0 }); + events.length = 0; - await s.sendMessage('second — must be over the cap'); + // Resolves — the cap refusal is a settled terminal, not a throw. Asserted so the mutation that moves the + // increment below the flush dies on the CAP assertion below rather than on a stray 'disk full' rejection. + await expect(s.sendMessage('second — must be over the cap')).resolves.toBeUndefined(); const completed = events.find((e) => e.type === 'session:turn_completed'); expect(completed?.type === 'session:turn_completed' ? completed.error?.code : undefined).toBe( diff --git a/packages/core/src/engine/agent-session.ts b/packages/core/src/engine/agent-session.ts index a04b2716..bf31a01a 100644 --- a/packages/core/src/engine/agent-session.ts +++ b/packages/core/src/engine/agent-session.ts @@ -422,6 +422,15 @@ export class AgentSession { * entries, a budget refusal, a pre-flight cancel) never burns a turn the model never took. */ #turnCount = 0; + + /** + * The last turn's real accumulated usage, captured on the success path just before the durability flush. + * + * Exists so an unclassified failure AFTER a provider engaged — in practice a `flushBudgetCommitments` + * rejection — can report what was actually billed instead of `{0,0}` (`CR-02`). Cleared at the top of each + * turn so a later failure can never inherit an earlier turn's numbers. + */ + #lastEngagedUsage: { input: number; output: number } | undefined; /** Session-wide running cost total, authoritatively stamped onto every `cost:updated`. */ #cumulativeCostMicrocents = 0; #status: SessionStatus = 'created'; @@ -612,6 +621,7 @@ export class AgentSession { // only on the NEXT turn, so the advertise-filter + approval regime stay consistent within this turn. const turnPolicy = this.#turnPolicy; try { + this.#lastEngagedUsage = undefined; // never inherit an earlier turn's numbers const result = await this.#runTurn(abort.signal, turnPolicy); // A cancel landed mid-turn — the cancel path owns the terminal session:cancelled; stay quiet, but // roll the user message back so a cancelled turn leaves no dangling user turn in the transcript @@ -654,6 +664,8 @@ export class AgentSession { // and its comment relies on "nothing is pushed after the user message on a throw" — which stopped being // true when this await landed between them: a flush rejection popped the ASSISTANT message and left the // user turn dangling, the exact shape that rollback exists to prevent. Awaiting first restores it. + // Captured BEFORE the flush, so a rejection below still knows what the provider billed (`CR-02`). + this.#lastEngagedUsage = { input: result.usage.input, output: result.usage.output }; await this.#deps.flushBudgetCommitments?.(); if (result.text.length > 0) { this.#messages.push({ role: 'assistant', content: [{ type: 'text', text: result.text }] }); @@ -762,15 +774,18 @@ export class AgentSession { } // An unexpected (non-classified) error — settle the turn LOUDLY first so the stream stays balanced (every // session:turn_started gets a terminal), then re-raise so the caller still sees the bug. - this.#emitTurnCompleted( - 'error', - { input: 0, output: 0 }, - { - code: 'internal', - message: 'the session turn failed with an unexpected error', - retryable: false, - }, - ); + // + // `#lastEngagedUsage`, not a hardcoded zero (`CR-02`). The only unclassified error that reaches here in + // practice is a `flushBudgetCommitments` rejection — a turn whose provider ALREADY engaged and billed, + // which is exactly why the same decision counts it against the cap. Reporting `{0,0}` here consumed the + // cap slot and silently dropped the turn's real tokens from every total, which is the mirror of the error + // the counter decision refuses to make, and contradicts EA2/ADR-0055's rule to report real usage whenever + // a provider engaged. Falls back to zero when nothing engaged, which stays truthful. + this.#emitTurnCompleted('error', this.#lastEngagedUsage ?? { input: 0, output: 0 }, { + code: 'internal', + message: 'the session turn failed with an unexpected error', + retryable: false, + }); throw err; } From 64300161114a1f80dd366224c6aac7418e0e0d33 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 00:50:19 +0300 Subject: [PATCH 22/26] =?UTF-8?q?docs(repo,cli):=20fold=20the=20Sonnet=20r?= =?UTF-8?q?eview=20of=20W0=20=E2=80=94=20rehome=20the=20rule,=20name=20the?= =?UTF-8?q?=20fence's=20blind=20spots?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, no Blocker and no High — the first pass this session where the implementation itself held up. Every one was about where a rule lives and what it honestly guarantees. **The machine-output rule was filed under `## Logging`.** Content accurate, home wrong: it is a terminal-injection rule, not a confidentiality one, and `security-review.md` already has `### CLI terminal-render safety` documenting its human-display sibling — the very bullet the new paragraph calls "the opposite choice". A reviewer looking for every terminal-injection defense under that header would not have found half of it. Moved beside its sibling. **The ESLint fence guarantees less than its docblock claimed.** The reviewer probed it with a scratch file: it correctly fires through a template literal, a ternary, an object literal and a `.map()` callback — esquery walks the whole subtree — but it is blind to three shapes, none of which existed in the docblock: a destructured writer (`const { writeOut } = io`, whose callee is an `Identifier` not a `MemberExpression`), indirection through a helper that stringifies and returns, and a writer under a different name. None is in the tree today, but "caught the first time it is written" was true only of the inline shape, and this gap has already reopened twice. Named as residual risk, the way the seam fence above it names its own. **The fence did not cover `.tsx`.** `apps/cli/src/**/*.ts` misses nineteen files under `render/tui/` and `home/` — and the TUI is exactly where a debug or export affordance would grow one. Widened. **`sanitize.ts`'s allowlist entry is inert** — verified by removing it: no diagnostic either way, because the file defines `stringifyJsonLine` and has no writer call of its own. Kept, but labelled defensive so the next reader does not assume it is load-bearing. `render-error.ts`'s entry IS load-bearing; removing it produces a real diagnostic, which is how the distinction was established. **And the rule had no home in the canonical contract doc.** `commands.md`'s `### The --json machine-output contract` is where a consumer learns what stdout holds, and it said nothing about the escaping. Added, with the wire contract stated as unchanged — `JSON.parse` round-trips identically, which is the whole reason the escape was chosen over a strip — and cross-linked rather than restated. The review independently re-derived and confirmed every claim from the previous fold, by reverting each fix and checking the exact failure: the `finally`'s `expected +0 to be 1`, the ESLint rule firing on the prettier-wrapped shape the regex missed, `#lastEngagedUsage` reporting `{0,0}` when captured after the flush, and `render-error.ts`'s exemption being exercised. It also walked `#lastEngagedUsage` through every path — compaction, abort, cancel, resumed first turn, a throw before `#runTurn` resolves — and proved concurrent `sendMessage` is unreachable (`#status` flips before the first `await`, so the second throws `not_active`). It can never report a previous turn's tokens. pnpm run ci EXIT=0. Refs: CR-01, CR-02, CR-03 Co-Authored-By: Claude Opus 5 --- docs/reference/cli/commands.md | 8 ++++++++ docs/standards/security-review.md | 20 ++++++++++---------- eslint.config.mjs | 16 +++++++++++++++- 3 files changed, 33 insertions(+), 11 deletions(-) diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index f5e91ee1..bd93c37f 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -50,6 +50,14 @@ who opts out of color wins over a tool/CI that opts in. (A `--json`/CI/no-TTY st ### The `--json` machine-output contract +**Every record is serialized with `stringifyJsonLine`, not a bare `JSON.stringify`.** The wire contract is +unchanged — `JSON.parse` round-trips to the identical string, which is exactly why the escape was chosen over +a strip — but what reaches a terminal differs: `JSON.stringify` escapes `ESC` and leaves `DEL`, the C1 block +(including `U+009B`, a working escape-sequence introducer) and the Trojan-Source bidi family RAW in content +the model, a tool, or an imported artifact controls. Enforced by lint rather than review; the reasoning and +the one allowlisted exception live in +[security-review.md](../../standards/security-review.md#cli-terminal-render-safety--interactive-approval). + Under `relavium run --json`, the CLI emits a stable machine contract a CI job can pipe and assert on ([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)). The contract covers a workflow **run**; `--help`, `--version`, and a bare no-command invocation are exit-`0` meta-operations that diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index 4b88a511..1b4c61b6 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -389,6 +389,16 @@ security invariants** a review must confirm are: [tool-registry.md §error taxonomy](../reference/shared-core/tool-registry.md#error-taxonomy) — not restated here. +**Machine output has its own arm of that floor, and it is the opposite choice.** Every `--json` record is +serialized with `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), never a bare `JSON.stringify`: +`JSON.stringify` escapes `ESC` and stops, leaving `DEL`, the whole C1 block — including `U+009B`, a working +escape-sequence introducer on real terminals — and the Trojan-Source bidi family RAW in content the model, a +tool, or an imported artifact controls. It **escapes** where the human-display path **strips**, because +`--json` is a machine contract and must reproduce its data byte for byte. The rule is enforced by an ESLint +`no-restricted-syntax` selector rather than by review, because the gap reopened twice when it was not +(`#W15-10`, then `CR-03`). `apps/cli/src/process/render-error.ts` is the single allowlisted exception: it +pre-sanitizes with the stripping sanitizer, so the `--json` ERROR envelope is deliberately lossy. + ## Never hand-roll crypto - We **never implement cryptography, TLS, or keychain primitives ourselves**. We use vetted @@ -410,16 +420,6 @@ No secrets, no full prompts/responses, and no raw keys in logs — ever, **inclu *echoed back* in a provider response or error body, which must be redacted before it is logged.** Logging redaction rules live in [logging-and-observability.md](logging-and-observability.md). -**Machine output has its own arm of that floor, and it is the opposite choice.** Every `--json` record is -serialized with `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), never a bare `JSON.stringify`: -`JSON.stringify` escapes `ESC` and stops, leaving `DEL`, the whole C1 block — including `U+009B`, a working -escape-sequence introducer on real terminals — and the Trojan-Source bidi family RAW in content the model, a -tool, or an imported artifact controls. It **escapes** where the human-display path **strips**, because -`--json` is a machine contract and must reproduce its data byte for byte. The rule is enforced by an ESLint -`no-restricted-syntax` selector rather than by review, because the gap reopened twice when it was not -(`#W15-10`, then `CR-03`). `apps/cli/src/process/render-error.ts` is the single allowlisted exception: it -pre-sanitizes with the stripping sanitizer, so the `--json` ERROR envelope is deliberately lossy. - ## When a review is mandatory Any change to: key handling or the keychain bridge, IPC commands, the desktop diff --git a/eslint.config.mjs b/eslint.config.mjs index 95adf726..99b97348 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -78,6 +78,14 @@ const seamSyntaxRules = /** @type {const} */ ([ * reproduce its data, so `JSON.parse` round-trips to the identical string. That is the opposite choice from * the human-display floor (`stripTerminalControls`), and `process/render-error.ts` is the one deliberate * exception — it pre-sanitizes its fields with the STRIPPING sanitizer and is therefore lossy on purpose. + * + * **Three residual blind spots, named so the guarantee is not over-trusted.** The selector is syntactic, so it + * sees a `JSON.stringify` anywhere inside a `writeOut`/`writeErr` argument — through a template literal, a + * ternary, an object literal, a callback — but NOT: (1) a destructured writer, `const { writeOut } = io`, + * whose callee is an `Identifier` rather than a `MemberExpression`; (2) indirection through a helper that + * stringifies and returns, which no syntactic rule can follow; (3) a writer under a different name. None of + * the three exists in the tree today, and each is caught by review the same way the pre-rule call sites were — + * but "caught the first time it is written" is true of the inline shape only. */ const JSON_LINE_MESSAGE = 'A --json record must be serialized with stringifyJsonLine (apps/cli/src/render/sanitize.ts), not a bare ' + @@ -238,7 +246,13 @@ export default tseslint.config( { // The machine-output fence (CR-03). Scoped to the CLI, which is the only surface emitting NDJSON to a // terminal today; `render-error.ts` is the documented exception (it pre-strips, and is lossy on purpose). - files: ['apps/cli/src/**/*.ts'], + // `.tsx` too: the TUI is where a debug/export affordance is most likely to grow one, and a `.ts`-only + // glob would leave nineteen files outside the fence. + files: ['apps/cli/src/**/*.{ts,tsx}'], + // Only `render-error.ts` is load-bearing — verified: removing it produces a real diagnostic. It is the + // documented exception (it pre-strips, so the error envelope is lossy on purpose). `sanitize.ts` is + // listed defensively: it DEFINES `stringifyJsonLine` and today has no writer call of its own, so the + // entry is currently inert rather than wrong. ignores: ['apps/cli/src/process/render-error.ts', 'apps/cli/src/render/sanitize.ts'], rules: { 'no-restricted-syntax': [...seamSyntaxRules, jsonLineSyntaxRule], From 7dd648806060e6783af5674f58237c2e939dd112 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 08:30:22 +0300 Subject: [PATCH 23/26] docs(roadmap): add CR-64 from the YAML/git-native reviews, and record why the rest is out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two design reviews — one of the YAML definition layer, one of the git-native posture — read against the tree. **Exactly one item belongs in this phase**, and the triage is recorded rather than discarded, because the same documents will be read again. **CR-64 — node-`tools` narrowing is enforced at RUN time while two ADRs say "parser-enforced".** `resolveGrant` lives in `agent-runner.ts`, the node executor, reached only after a run starts; nothing in `parser.ts`, `dag.ts` or `run-plan.ts` touches `tools` at all. ADR-0038 states the opposite in parentheses. The security boundary HOLDS — a widening node is refused at dispatch, ADR-0029(b) is not violated — but a workflow that widens passes `relavium validate`, reviews clean, and fails partway through a run after upstream nodes have spent real money. And a canonical ADR tells the next reader the parser caught it, which is how someone later removes the runtime check as redundant. Exactly the phase's second reason for existing: a claim the code does not keep. **Two of the reviews' own HIGH-priority findings are not real, and the phase doc says so with the evidence:** - "Circular `$ref` has no guard" — it cannot happen. `AgentSchema` is `.strict()` and declares neither `agents:` nor `$ref`, so an agent file cannot reference another agent. Resolution depth is exactly one hop. - "A `1.0` workflow could `$ref` a `0.9` agent" — it cannot. `schema_version` is `z.literal(SCHEMA_VERSION)`, so a file at another version fails its own parse long before cross-file skew is reachable. Checking those two took longer than adding CR-64 would have, and that is the point: a finding added on a reviewer's say-so is a finding the next person has to re-derive. **Everything else is real and belongs elsewhere** — Zod→JSON Schema for IDE validation (the reviews' strongest UX point, and tooling), `expression_type` ergonomics, the edge-condition/condition-node duplication (whose dangerous half is already CR-62), `parallel_of` DRY, "did you mean" errors, property-based schema tests, `merge_strategy` widening, session-export turn grouping (ADR-0026 scoped that deliberately), `reasoning_effort` provider validation (no false claim — ADR-0066 has the adapters withhold it), and the desktop/product items. Putting them here would dilute a phase whose scope line is "no new product surface, only the invariants an existing surface already claims". Two items the reviews raised are already covered and stay where they are: the QuickJS sandbox sits under "What must NOT change", and MCP command injection is CR-16. 46 items now; `current.md`'s count updated. pnpm run ci EXIT=0. Refs: CR-64 Co-Authored-By: Claude Opus 5 --- docs/roadmap/current.md | 2 +- ...hase-2.6.5-core-reliability-remediation.md | 57 +++++++++++++++++++ 2 files changed, 58 insertions(+), 1 deletion(-) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 97a54ef6..1ae160c8 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -458,7 +458,7 @@ gaps**: effect journal, stdio MCP consent-before-spawn, run lease, compaction tr input admission, event-log ordering. Three separate reviews landing on the same seven points is not opinion. The full, self-contained work list is -[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — **45 items** +[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — **46 items** (`CR-01`…`CR-95`) with evidence, fix, acceptance criteria and a decision/ADR/gate register, written so the work can be done from that document alone. An adversarial plan review on 2026-08-10 corrected the phase boundary, the exit rule and the execution order, and added two items (`CR-17` resume identity, `CR-63` `input_schema` diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index f566628d..a82e918d 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -217,6 +217,7 @@ to correct. | `CR-61` | **open** — needs a JSON-Schema validator dependency | new (dependency ADR) | no | — | | `CR-62` | **open** — extract dependencies, or fail loudly | — | no | — | | `CR-63` | made (verify the docs; no runtime change) | — | no | — | +| `CR-64` | made (enforce at parse / `$ref` resolution) | — | no | — | | `CR-70` | **open** — persist bounded tool pairs, or narrow the spec | — | no | — | | `CR-71` | with `CR-70` | with `CR-70`'s | no | — | | `CR-72` | **open** — implement, or reject at parse | — | no | — | @@ -781,6 +782,29 @@ run-time execution"*. Under that spec, the absence of runtime enforcement is **c an agent's `input_schema` is validated at run time; if one exists, correct it to match the spec. If none exists, close the item with that finding recorded — do not implement enforcement to satisfy a claim nobody made. +### CR-64 — Node-`tools` narrowing is enforced at RUN time, and two ADRs say "parser-enforced" · Medium + +**Evidence.** `resolveGrant` in `packages/core/src/engine/agent-runner.ts` is where a node's `tools:` is +checked against the agent's grant — the node executor, reached only after the run has started. +[ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md) states the opposite in parentheses: +*"`tools:` **narrows** the agent's grant and never widens ([ADR-0029], parser-enforced — a node listing a tool +the agent lacks fails validation)"*. Nothing in `parser.ts`, `dag.ts` or `run-plan.ts` touches `tools` at all. + +**Why it matters.** The security boundary itself HOLDS — a widening node is refused at dispatch, so no +ungranted tool ever runs; ADR-0029(b) is not violated. What is wrong is WHERE and WHEN. A workflow that widens +passes `relavium validate`, looks correct in review, and fails partway through a run — after upstream nodes +have already spent real money. And a canonical ADR tells the next reader the parser caught it, which is how +someone later "simplifies" the runtime check as redundant. + +**Fix.** Enforce it at parse for an INLINE agent, where both sides are present in one document, and at `$ref` +resolution for an external one — the point at which the agent registry has been read. The runtime check stays +as the last line of defence (a host can construct a node executor directly), but it stops being the first. + +**Acceptance.** A workflow whose node lists a tool its inline agent lacks fails `relavium validate` with a +typed error naming the node, the tool and the agent — before any run id exists. The same for a `$ref`'d agent +once resolved. ADR-0038's parenthetical is corrected in the same change, or the enforcement moves to match it; +the two must agree. + --- ## W7 — Agent product correctness @@ -1001,6 +1025,39 @@ This phase is done when **all** of the following hold: 7. **A closing register in this file states, per item, the code that closes it** — verified by reading the code, not by trusting the mark. Wave 1's completion claim was wrong twice before this discipline was adopted. +## What a later architecture review contributed — and what it did not + +Two design reviews of the YAML definition layer and of the git-native posture were read against the tree on +2026-08-11. **Exactly one item from them belongs in this phase** (`CR-64`), and that outcome is worth recording +rather than quietly discarding the rest, because the same documents will be read again. + +**Two of the reviews' own high-priority findings are NOT REAL, verified against the schema:** + +- **"Circular `$ref` has no guard."** It cannot happen. `AgentSchema` is `.strict()` and declares neither + `agents:` nor `$ref`, so an `.agent.yaml` cannot reference another agent — the resolution depth is exactly + one hop, workflow → agent. A `$ref` inside an agent file is rejected by its own parse. +- **"A workflow at `schema_version: '1.0'` could `$ref` an agent at `'0.9'`."** It cannot. The field is + `z.literal(SCHEMA_VERSION)`, not a free string, so any file carrying another version fails its own parse + before cross-file skew is reachable. + +**The rest is real but belongs elsewhere**, and putting it here would dilute a phase whose whole scope line is +"no new product surface, only the invariants an existing surface already claims": + +| finding | where it belongs | +|---|---| +| Zod → JSON Schema emit for IDE autocomplete/validation | tooling, Phase 2.6 or later — the reviews' own strongest UX point | +| `expression_type: 'js'` mandatory with no alternative | authoring ergonomics | +| Edge `condition` vs `condition` node — two mechanisms | authoring; the DANGEROUS half is already `CR-62` | +| `parallel_of` duplicating edges | authoring ergonomics; adjacent to `CR-60` | +| "did you mean" on Zod errors, property-based schema tests | DX / test tooling | +| `merge_strategy` widening (`zip`/`union`/`intersect`) | a feature, not a repair | +| Session export: linear-only, one node per turn | ADR-0026 scoped this deliberately; Phase 2.6 | +| `reasoning_effort` unvalidated against provider | no false claim — ADR-0066 has adapters withhold it; a nicety | +| Auto-commit on save, `.relavium/` discoverability, merge-conflict culture | product/desktop, Phase 3 | + +Also confirmed by those reviews and left alone: the QuickJS sandbox as a single point of failure is already +under "What must NOT change", and MCP command injection is already `CR-16`. + ## Scope note One item already in flight is NOT part of this phase and keeps its own tracking: the transaction-handle cleanup From c6a02699326f2a0dd00aead5e16730c027fc1407 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 09:05:14 +0300 Subject: [PATCH 24/26] =?UTF-8?q?fix(core,cli,tools,docs):=20fold=20the=20?= =?UTF-8?q?PR=20#82=20review=20=E2=80=94=20bind=20the=20oracle=20to=20its?= =?UTF-8?q?=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified each finding against the tree; fixed the ones that still hold. **Correctness.** `checkDurableTruth` reported foreign events and then compared against them anyway: `history` read ANOTHER run's terminal and `durableTerminalCount` counted it, so the verdict named the right cause while every downstream number described the wrong run. Every view now reads the runId-filtered log. Break-verified: reverting the filter reddens the foreign-log test at `durableTerminalCount`. `ledger B3`'s ordering assertion was vacuous — `indexOf` returns `-1` for a missing entry, which is less than everything, so a run that never settled a charge satisfied the ordering it exists to prove. Presence is asserted first. `tools/test-isolation` counted `...REPO_LOCAL_CHECKOUTS` repo-wide and required 2 — which is also what two spreads in `test.exclude` and none in `coverage.exclude` looks like, the exact half-failure assertion 4 is for. Now counted per property, exactly once each. Break-verified with that mutation. `persister.test.ts` restored its SQLITE_BUSY spies at the END of a test, so one failing assertion leaked a throwing `updateSession` into every later test in the file. Restore is suite-level now. **Docs — the contract contradicted itself.** `sse-event-schema.md` called `cost:attempt_settled.cumulativeCostMicrocents` "NOT the restore source" and `costMicrocents` "the delta a reader SUMS", while the fold rule three lines below says restore is `Math.max` over the absolutes and must never sum the deltas. The fold rule is what `checkpoint.ts` implements; both call-outs now agree with it. Roadmap: PR #81 (23 items, merged) and `#W15-1`/PR #82 (open) were recorded as one closure; separated. `45 CR items` → 46, the count the phase doc states. CR-03's Fix/Acceptance said three paths where five shipped. **Skipped, with reason.** The `run_costs` node-retry attribution identity needs a schema column + migration + snapshot regen — the caveat is recorded in `nodeSettledCost` and the run-level SUM stays exact either way; it is already a `#W15-1` follow-up. Condensing ADR-0077 conflicts with append-only, and at 214 lines it sits between ADR-0070 (204) and ADR-0073 (213), so it is not an outlier. The README architecture SVG is a curated asset, not a diagram that drifted. `settle()` stays fixed-turn: it drains to quiescence to prove an action did NOT run, which no predicate can express. Also folded: codepoint sort in `canonical()` (`localeCompare` is locale-dependent, so the instrument could manufacture a disagreement), `#flushBudgetCommitments` → `#joinMoneyDurability` (it fronts both money chains since ADR-0077; the host-supplied `AgentSessionDeps.flushBudgetCommitments` keeps its name and is correct), cognitive complexity 39 → five named helpers, the sync-throw recovery test now reuses the instance that took the throw, and two `expect:'repaired'` branches plus the derived fixture types the `as never` was hiding. `pnpm run ci` and `pnpm coverage` green — 96.97% lines / 93.34% branches. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- apps/cli/src/chat/persister.test.ts | 8 +- .../cli/src/render/json-line-surfaces.test.ts | 7 +- docs/reference/contracts/sse-event-schema.md | 6 +- docs/roadmap/current.md | 26 +- ...hase-2.6.5-core-reliability-remediation.md | 12 +- .../core/src/engine/durable-truth.test.ts | 53 +++- packages/core/src/engine/durable-truth.ts | 227 +++++++++++------- packages/core/src/engine/engine.ts | 11 +- .../src/engine/m2-e2e-harness.e2e.test.ts | 5 + .../core/src/engine/money-durability.test.ts | 24 +- tools/test-isolation/check.mjs | 36 ++- 11 files changed, 284 insertions(+), 131 deletions(-) diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index 667743e9..8936f0e4 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -48,6 +48,11 @@ describe('createSessionPersister', () => { }); afterEach(() => { client.sqlite.close(); + // Suite-level, not per-test. Several tests below spy `updateSession`/`recordSessionCost` into a throwing + // SQLITE_BUSY stub; a test-local `vi.restoreAllMocks()` at the END of the test is skipped the moment an + // assertion above it fails, and the stub then leaks into every later test in the file — turning one real + // failure into a cascade whose reported causes are all fictional. + vi.restoreAllMocks(); }); /** @@ -296,11 +301,10 @@ describe('createSessionPersister', () => { // asserted `close()` does not throw — which is true whether or not the listener leaked, because `close()` // calls the same idempotent unsubscribe. The real evidence is that a SUBSEQUENT event does not re-enter a // persister that cannot write, so `updateSession` is never called again. Deliberately WITHOUT `close()`. - expect(updateSpy.mock.calls.length).toBe(1); // the failing write happened exactly once + expect(updateSpy).toHaveBeenCalledTimes(1); // the failing write happened exactly once // THE assertion: the throw did not jump over the unsubscribe. Without the `finally` this is 0, and the // persister stays on the bus — so every later event re-enters one that cannot write. expect(unsubscribed).toBe(1); - vi.restoreAllMocks(); }); it('does not let a failed cost write advance the in-memory total (#W15-4)', async () => { diff --git a/apps/cli/src/render/json-line-surfaces.test.ts b/apps/cli/src/render/json-line-surfaces.test.ts index 4ab3e869..4321ef87 100644 --- a/apps/cli/src/render/json-line-surfaces.test.ts +++ b/apps/cli/src/render/json-line-surfaces.test.ts @@ -1,9 +1,10 @@ /** * `CR-03` — the behavioural half of the machine-output floor. * - * `#W15-10` built `stringifyJsonLine` and wired two call sites. `CR-03` found three more; the fold of its own - * review then found a SIXTH (`commands/export.ts`) that a source-scanning regex had missed purely because - * prettier wrapped the argument onto its own line. A guard that only checks the places someone remembered + * `#W15-10` built `stringifyJsonLine` and wired two call sites. `CR-03`'s finding named three more, and + * closing it routed FIVE — `commands/import.ts` and `commands/export.ts` write the same record shape and were + * not in the finding. `export.ts` was the last one found, and only because a source-scanning regex had missed + * it purely because prettier wrapped the argument onto its own line. A guard that only checks the places someone remembered * catches nothing new, so the CALL-SITE half is now an ESLint `no-restricted-syntax` selector in * `eslint.config.mjs` — it fires on the shape, anywhere in `apps/cli/src`, the first time a new surface is * written. That is the mechanism; this file keeps the reason it exists visible as executable behaviour. diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index 5857b2df..3d35c3b7 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -92,7 +92,7 @@ export type RunEvent = | `run:completed` | The run finished. | `outputs` (a record **keyed by each terminal `output` vertex's node id**, the value being that vertex's captured output — see [run-plan.md §output capture](../shared-core/run-plan.md)), `totalTokensUsed`, `totalCostMicrocents` (integer micro-cents closing total for the whole run), `durationMs` | | `run:failed` | The run failed. | `error: {code, message, retryable, nodeId?, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `nodeId` is the root-cause node; `correlationId` joins to the internal log — ADR-0036), `partialOutputs`, `cumulativeCostMicrocents?` (the run-wide running total at failure — the durable fail-cost for a **paid media job** a sibling node's failure abandoned, whose lone estimate addend is folded just before this terminal _after_ the root-cause `node:failed` snapshot, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `run:cancelled` and the `run:completed` counterpart `totalCostMicrocents`) | | `run:cancelled` | The run was cancelled. | `cumulativeCostMicrocents?` (the run-wide running total at cancellation — the durable fail-cost for a **paid media job** pending at the cancel, whose lone estimate addend is folded just before this terminal, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; the `run:completed` counterpart is `totalCostMicrocents`) | -| `cost:attempt_settled` | **One settled provider attempt's REALIZED charge, made durable** ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) — the realized twin of `budget:estimate_committed`, emitted from the same callback and joined at the same barriers ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). It exists because `cost:updated` is streamed and never persisted, so before this event a crash mid-agent-loop discarded every charge since the last node boundary — and the resumed run spent it AGAIN, against a cap understated by exactly that amount. **RUN-ONLY** (`runId` always, never `sessionId`), unlike `cost:updated` and unlike its estimate twin: the session path already records the same per-attempt increment into `session_costs` synchronously (ADR-0070 + `#W15-4`), so it needs no arm here. **Scope:** it covers the provider attempts of an AGENT TURN. A media job's realized cost (ADR-0045 §5) does **not** emit one and does not need to — it is already durable through the `node:completed` / `node:failed` / `run:*` cumulative snapshots. And no cost event of any kind makes a TOOL EFFECT idempotent; that is a separate decision about a separate failure. | `nodeId`, `model`, `attemptNumber` (**required**, 1-based within-chain — optional on `cost:updated` only because that event predates this one and has historical rows; a per-attempt ledger row that cannot say which attempt it is, is not a ledger), `inputTokens`, `outputTokens`, `costMicrocents` (**this attempt's delta**, not a cumulative — `0` is reachable for an unpriced or genuinely free model), `cumulativeCostMicrocents` (the run-wide total **after** this attempt; a display snapshot, **not** the restore source — see the fold rule below), `priced` (**required**, same reason as `attemptNumber`: without it `costMicrocents: 0` with real tokens cannot distinguish *could not price* from *genuinely free*, which is the ambiguity a ledger exists to remove) | +| `cost:attempt_settled` | **One settled provider attempt's REALIZED charge, made durable** ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) — the realized twin of `budget:estimate_committed`, emitted from the same callback and joined at the same barriers ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). It exists because `cost:updated` is streamed and never persisted, so before this event a crash mid-agent-loop discarded every charge since the last node boundary — and the resumed run spent it AGAIN, against a cap understated by exactly that amount. **RUN-ONLY** (`runId` always, never `sessionId`), unlike `cost:updated` and unlike its estimate twin: the session path already records the same per-attempt increment into `session_costs` synchronously (ADR-0070 + `#W15-4`), so it needs no arm here. **Scope:** it covers the provider attempts of an AGENT TURN. A media job's realized cost (ADR-0045 §5) does **not** emit one and does not need to — it is already durable through the `node:completed` / `node:failed` / `run:*` cumulative snapshots. And no cost event of any kind makes a TOOL EFFECT idempotent; that is a separate decision about a separate failure. | `nodeId`, `model`, `attemptNumber` (**required**, 1-based within-chain — optional on `cost:updated` only because that event predates this one and has historical rows; a per-attempt ledger row that cannot say which attempt it is, is not a ledger), `inputTokens`, `outputTokens`, `costMicrocents` (**this attempt's delta**, not a cumulative — `0` is reachable for an unpriced or genuinely free model; a delta a restoring reader must **never sum** — see the fold rule below), `cumulativeCostMicrocents` (the run-wide total **after** this attempt — a durable ABSOLUTE total, and one of the inputs a restoring reader takes `Math.max` over; the fold rule below is the canonical statement), `priced` (**required**, same reason as `attemptNumber`: without it `costMicrocents: 0` with real tokens cannot distinguish *could not price* from *genuinely free*, which is the ambiguity a ledger exists to remove) | > **Which cost event is authoritative for what — the three-way split, stated once.** `cost:updated` is the **live observation**: streamed, never persisted, last-wins, and it is what a surface renders while a run is moving. `cost:attempt_settled` is the **durable record of what was charged**: one row per settled attempt, and the only cost event a reader can trust after a restart. **Restoring** a run's realized total is `Math.max` over every durable ABSOLUTE total in the log — the node-boundary snapshots (`node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`), `budget:paused.spentMicrocents`, and `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so each is a true run-wide total at that instant and the largest is the engine's real total whatever order the rows landed in. **Do not sum the attempt deltas**, by either obvious route: adding `costMicrocents` into the same total double-counts against a node snapshot that already includes those attempts, and summing them into a *separate* accumulator to max against the snapshots **under-counts** — the two sources cover different money, since a media node writes a snapshot and emits no attempt row at all, so every attempt made after the last node boundary vanishes whenever earlier media spend is the larger figure. (`Math.max` is right here and wrong for the conservative twin below for one reason only: realized spend is monotonic, while a conservative commitment can be deliberately released.) The derived `run_costs` / `runs` rows this folds into are specified in [database-schema.md](../shared-core/database-schema.md), not here. @@ -236,8 +236,8 @@ export interface CostAttemptSettledEvent extends BaseEvent { attemptNumber: number; // 1-based WITHIN-CHAIN attempt. REQUIRED here, optional on cost:updated — see below inputTokens: number; outputTokens: number; - costMicrocents: number; // THIS attempt's realized charge — the delta a reader SUMS. 0 is reachable (unpriced/free) - cumulativeCostMicrocents: number; // the run-wide total AFTER it — a display snapshot, NOT the restore source + costMicrocents: number; // THIS attempt's realized charge. A restoring reader must NOT sum these — see the fold rule above. 0 is reachable (unpriced/free) + cumulativeCostMicrocents: number; // the run-wide total AFTER it — a durable ABSOLUTE total; restore is Math.max over the absolutes, never a sum of the deltas priced: boolean; // REQUIRED here, optional on cost:updated — see below } ``` diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 1ae160c8..ddeda7b3 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -2,7 +2,7 @@ > Status: Living > -> Last updated: 2026-08-10 +> Last updated: 2026-08-11 - **Related**: [README.md](README.md), [phases/phase-2.5-cli-consolidation.md](phases/phase-2.5-cli-consolidation.md), [phases/phase-2.5.5-hardening-and-remediation.md](phases/phase-2.5.5-hardening-and-remediation.md), [phases/phase-2-cli.md](phases/phase-2-cli.md), [deferred-tasks.md](deferred-tasks.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) @@ -90,7 +90,7 @@ flowchart TD W0["Wave 0 — One true baseline
baseline ✅ · CI truth · numbers"] W1["Wave 1 — Stop the bleeding ✅
3 CRITICALs · cost cap · ADR-0074"] LEDGER["#W15-1 — realized-cost ledger
ADR-0076 implementation"] - P265["Phase 2.6.5 — Core reliability
45 CR items · 8 P0 ADRs
absorbs the hostile-MCP class"] + P265["Phase 2.6.5 — Core reliability
46 CR items · 8 P0 ADRs
absorbs the hostile-MCP class"] W2["Wave 2 — Shut the doors
fs jail · secrets · config trust
certifies 2.5.5 EXIT 1–3"] W3["Wave 3 — Clear the ground
god-file decomposition · CLI net"] W4a["Wave 4a — The spine
2.6.A/D/H/K + 2 ADRs"] @@ -245,11 +245,17 @@ Ordered by whether the repo currently states something untrue, then by blast rad **Every item names the check that would close it, not just the defect.** -> **Status, 2026-08-10 — 24 of 24 closed.** `#W15-1`, the last one, landed 2026-08-10 behind ADR-0076 + -> ADR-0077 (see §A). Everything marked ✅ below is fixed, -> break-verified with the mutation confirmed applied, and **merged to `main` via PR #81** (2026-08-09). §E's six coverage gaps -> are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort listener is removed, which -> is the unkillable run reproduced exactly rather than an assertion standing in for it. +> **Status, 2026-08-10 — 24 of 24 closed, across TWO PRs.** Everything marked ✅ below is fixed and +> break-verified with the mutation confirmed applied; the two PRs are recorded separately because the closure +> dates and the merge states differ: +> +> - **23 items merged to `main` via PR #81** (2026-08-09). +> - **`#W15-1`, the last one, landed 2026-08-10** behind ADR-0076 + ADR-0077 (see §A) and rides **PR #82** +> (`development` → `main`), which is open at the time of writing. It is closed as work, not yet merged. +> +> §E's six coverage gaps are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort +> listener is removed, which is the unkillable run reproduced exactly rather than an assertion standing in +> for it. > > **`#W15-2` closed 2026-08-09** as [ADR-0075](../decisions/0075-fail-closed-resume-on-an-unreadable-event-log.md), > amending ADR-0074 §5: a read that feeds a REPLAY refuses when any row was skipped, a read that feeds a @@ -490,8 +496,10 @@ Two rules that changed with the plan review: inside the `ci` script; this phase edits `packages/core` heavily, and the CI coverage job does not block on `core`. -Three items are already half-closed by Wave 1 and finish right after the oracle (`CR-01`–`CR-03`); `CR-03` is a -propagation gap from `#W15-10`'s own fix — three `--json` paths never got the safe serializer. +`CR-01`–`CR-03` were half-closed by Wave 1 and finished right after the oracle — **all three are closed as of +2026-08-11** (PR #82). `CR-03` was a propagation gap from `#W15-10`'s own fix: the finding named three `--json` +paths that never got the safe serializer, and closing it found **five**, which is why the call-site half is now +an ESLint selector rather than a list. ### Wave 2 — Shut the doors diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index a82e918d..c4140c19 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -314,10 +314,14 @@ through it. Three sibling paths still use bare `JSON.stringify`: (8-bit CSI) and the bidi family raw, and these streams carry model-, tool- and persisted content. This is a propagation gap — the fix landed, the siblings did not get it. -**Fix.** Route all three through `stringifyJsonLine`. The escape is lossless, so no machine contract changes. - -**Acceptance.** One shared adversarial test drives a C1 + bidi payload through all three surfaces and asserts -the raw code points do not survive while `JSON.parse` round-trips to the identical string. +**Fix.** Route all **five** through `stringifyJsonLine` — the three above plus `commands/import.ts` and +`commands/export.ts`, which write the same record shape and were not in the finding (see the closing note). +The escape is lossless, so no machine contract changes. + +**Acceptance.** One shared adversarial test drives a C1 + bidi payload through the serializer and asserts the +raw code points do not survive while `JSON.parse` round-trips to the identical string; the CALL-SITE half — +that all five surfaces, and any sixth written later, actually go through it — is an ESLint +`no-restricted-syntax` selector, because a list only ever proves the places someone remembered. **Closed — and it was FIVE paths, not three.** The finding named three; `import.ts` and `export.ts` write the same record shape and were not in it. `docs/reference/cli/commands.md` pairs the last two by that shape in one diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts index 44916b25..68b583f0 100644 --- a/packages/core/src/engine/durable-truth.test.ts +++ b/packages/core/src/engine/durable-truth.test.ts @@ -14,10 +14,14 @@ const started: RunEvent = { executionMode: 'local', }; -const completed = (seq: number, outputs: unknown = { a: 1 }, cost = 500): RunEvent => ({ +const completed = ( + seq: number, + outputs: Record = { a: 1 }, + cost = 500, +): RunEvent => ({ type: 'run:completed', ...base(seq), - outputs: outputs as Record, + outputs, totalTokensUsed: { input: 1, output: 1 }, totalCostMicrocents: cost, durationMs: 10, @@ -188,6 +192,11 @@ describe('checkDurableTruth', () => { expect(verdict.agrees).toBe(false); expect(verdict.disagreements.some((d) => d.includes('belonging to another run'))).toBe(true); + // …and every DOWNSTREAM number describes this run, not the foreign one. Naming the right cause while + // counting the other run's terminal was the defect: `history` read `rOTHER`'s `run:completed` and this + // came back as 1, so a reader who trusted the counts over the message saw a healthy log. + expect(verdict.durableTerminalCount).toBe(0); + expect(verdict.history).toBeUndefined(); }); it('catches a live terminal carrying a different runId', async () => { @@ -310,11 +319,45 @@ describe('checkDurableTruth', () => { expect(verdict.disagreements.some((d) => d.includes('NO durable terminal'))).toBe(true); }); - /** A `run:failed` differing from the baseline in exactly one error field. */ - const failedWith = (error: Partial>): RunEvent => ({ + // The other two `expect:'repaired'` arms. Without them the mode was only ever exercised where it PASSES or + // where the restart did nothing — so a caller that asked for `'repaired'` on a run that had, in fact, + // settled would have been told it agreed, which is the mode's whole failure direction. + it('under expect:"repaired", a log that ALREADY holds a terminal is the disagreement', async () => { + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('already holds a terminal'))).toBe(true); + }); + + it('under expect:"repaired", a LIVE terminal means the run did not die — also a disagreement', async () => { + const { eventsFor } = log(); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + expect: 'repaired', + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('the live stream produced'))).toBe(true); + }); + + /** + * A `run:failed` differing from the baseline in exactly one error field. The override type is DERIVED from + * the contract rather than asserted with `as never`, so a renamed field or an invalid `code` fails to + * compile instead of silently building a fixture the oracle can never disagree about. + */ + type FailureError = Extract['error']; + const failedWith = (error: Partial): RunEvent => ({ type: 'run:failed', ...base(1), - error: { code: 'tool_failed', message: 'boom', retryable: false, ...error } as never, + error: { code: 'tool_failed', message: 'boom', retryable: false, ...error }, partialOutputs: {}, }); diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts index 9c9b7940..7f9f1d08 100644 --- a/packages/core/src/engine/durable-truth.ts +++ b/packages/core/src/engine/durable-truth.ts @@ -119,7 +119,13 @@ function canonical(value: unknown): string { return JSON.stringify(value, (_key, val: unknown): unknown => { if (typeof val === 'bigint') return `${val.toString()}n`; if (typeof val !== 'object' || val === null || Array.isArray(val)) return val; - return Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b))); + // Codepoint order, NOT `localeCompare` — the comparison must be identical on every machine that runs + // this oracle. `localeCompare` honours the host locale (and its ICU build), so two runs of the same + // payload can canonicalize to different strings under different `LC_ALL`s, which is a disagreement + // manufactured by the instrument. The order itself is arbitrary; only its stability matters. + return Object.fromEntries( + Object.entries(val).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)), + ); }); } catch (error) { // A true cycle (or any other unserializable payload) becomes a single marker rather than escaping. The @@ -243,141 +249,194 @@ export interface DurableTruthInput { } /** - * Compare the four views. Cheap enough to call at the end of every e2e run; it reads the log twice and runs - * one reconcile. + * Bind everything to THIS run before comparing anything. Without it the oracle happily reports agreement on + * another run's log — measured — and `durableTerminalCount === 1` does not catch it, because the other run + * has a terminal too. A store that ignores its id argument, or a caller that crosses two runs, both land + * here; and `CR-11`'s whole scenario is two owners over one store. */ -export async function checkDurableTruth(input: DurableTruthInput): Promise { - const disagreements: string[] = []; - const expectRepaired = input.expect === 'repaired'; - - const before = [...(await input.eventsFor(input.runId))]; - - // Bind everything to THIS run before comparing anything. Without it the oracle happily reports agreement on - // another run's log — measured — and `durableTerminalCount === 1` does not catch it, because the other run - // has a terminal too. A store that ignores its id argument, or a caller that crosses two runs, both land - // here; and `CR-11`'s whole scenario is two owners over one store. - const foreign = before.filter((event) => event.runId !== input.runId); +function checkRunBinding( + runId: string, + live: RunEvent | undefined, + before: readonly RunEvent[], +): readonly string[] { + const out: string[] = []; + const foreign = before.filter((event) => event.runId !== runId); if (foreign.length > 0) { - disagreements.push( - `eventsFor(${input.runId}) returned ${foreign.length} event(s) belonging to another run ` + + out.push( + `eventsFor(${runId}) returned ${foreign.length} event(s) belonging to another run ` + `(e.g. ${String(foreign[0]?.runId)}) — every view below would be comparing the wrong log`, ); } - if (input.live !== undefined && input.live.runId !== input.runId) { - disagreements.push(`the live terminal carries runId ${input.live.runId}, not ${input.runId}`); + if (live !== undefined && live.runId !== runId) { + out.push(`the live terminal carries runId ${live.runId}, not ${runId}`); } + return out; +} - const live = viewOf(input.live); - const history = viewOf(terminalIn(before)); - +function checkLiveVsHistory( + expectRepaired: boolean, + live: TerminalView | undefined, + history: TerminalView | undefined, +): readonly string[] { + const out: string[] = []; if (expectRepaired) { // A run that died mid-flight: nothing durable yet, and nothing live either. if (history !== undefined) { - disagreements.push( + out.push( `expected a run needing repair, but the log already holds a terminal (${describe(history)})`, ); } if (live !== undefined) { - disagreements.push( + out.push( `expected a run that died without a terminal, but the live stream produced ${describe(live)}`, ); } } else if (!sameView(live, history)) { - disagreements.push( + out.push( `live and durable history disagree: live=${describe(live)} history=${describe(history)}`, ); } + return out; +} - let reconciledCount = 0; - let afterReconcile = history; - let finalEvents = before; - if (input.reconcile !== undefined) { - const reconciled = await input.reconcile(); - // Per-RUN, not the whole array. `reconcile()` repairs every interrupted run in the store and returns all - // of their events, so attributing the raw count to this run produced a factually wrong message and a - // false failure the moment a second run existed — which is, again, `CR-11`'s exact shape. - reconciledCount = reconciled.filter((event) => event.runId === input.runId).length; - finalEvents = [...(await input.eventsFor(input.runId))]; - afterReconcile = viewOf(terminalIn(finalEvents)); - - if (expectRepaired) { - if (afterReconcile === undefined) { - disagreements.push( - 'a restart left the run with NO durable terminal — a crashed run must reconcile to one', - ); - } - if (reconciledCount === 0) { - disagreements.push('reconcile() produced no event for this run, so nothing repaired it'); - } - } else { - if (!sameView(history, afterReconcile)) { - disagreements.push( - `a restart CHANGED the terminal: before=${describe(history)} after=${describe(afterReconcile)}`, - ); - } - if (history !== undefined && reconciledCount > 0) { - disagreements.push( - `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + - `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, never one that landed`, - ); - } +function checkReconcileOutcome( + expectRepaired: boolean, + history: TerminalView | undefined, + afterReconcile: TerminalView | undefined, + reconciledCount: number, +): readonly string[] { + const out: string[] = []; + if (expectRepaired) { + if (afterReconcile === undefined) { + out.push( + 'a restart left the run with NO durable terminal — a crashed run must reconcile to one', + ); + } + if (reconciledCount === 0) { + out.push('reconcile() produced no event for this run, so nothing repaired it'); } + return out; } + if (!sameView(history, afterReconcile)) { + out.push( + `a restart CHANGED the terminal: before=${describe(history)} after=${describe(afterReconcile)}`, + ); + } + if (history !== undefined && reconciledCount > 0) { + out.push( + `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + + `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, never one that landed`, + ); + } + return out; +} - const terminalCount = finalEvents.filter((event) => TERMINALS.has(event.type)).length; +/** + * Exactly-one-terminal (ADR-0036) plus the durable ORDER property (`CR-10`). Every view above assumes the + * latter and none of them checked it: a log committed out of order, or with an event after its terminal, used + * to verdict `agrees`. + * + * **What is checkable here, and what is NOT** — the distinction cost a wrong assertion, so it is written + * down. The durable log's sequence numbers are a strictly increasing SUBSEQUENCE of the run's, never + * `0..n-1`: streamed events (`agent:token`, `cost:updated`, …) take numbers and are deliberately never + * persisted. A real completed run reads `[0,1,2,3,5,10,11,12,13,14]`, and asserting `0..n-1` failed it. + * + * So "no persisted event is missing from the middle" — CR-10's actual property — is NOT expressible from + * the log alone: a streamed event's absence is indistinguishable from a lost one. CR-10's acceptance needs + * a store harness that knows which events it was ASKED to persist. What the log can prove on its own is + * that it starts at `run:started` (always seq 0, always durable) and only ever moves forward. + */ +function checkLogShape(ours: readonly RunEvent[], terminalCount: number): readonly string[] { + const out: string[] = []; if (terminalCount !== 1) { - disagreements.push( + out.push( `the durable log holds ${terminalCount} terminal(s); exactly one closes a run (ADR-0036)`, ); } - // The durable ORDER property (`CR-10`). Every view above assumes it and none of them checked it: a log - // committed out of order, or with an event after its terminal, used to verdict `agrees`. - // - // **What is checkable here, and what is NOT** — the distinction cost a wrong assertion, so it is written - // down. The durable log's sequence numbers are a strictly increasing SUBSEQUENCE of the run's, never - // `0..n-1`: streamed events (`agent:token`, `cost:updated`, …) take numbers and are deliberately never - // persisted. A real completed run reads `[0,1,2,3,5,10,11,12,13,14]`, and asserting `0..n-1` failed it. - // - // So "no persisted event is missing from the middle" — CR-10's actual property — is NOT expressible from - // the log alone: a streamed event's absence is indistinguishable from a lost one. CR-10's acceptance needs - // a store harness that knows which events it was ASKED to persist. What the log can prove on its own is - // that it starts at `run:started` (always seq 0, always durable) and only ever moves forward. - const ours = finalEvents.filter((event) => event.runId === input.runId); const seqs = ours.map((event) => event.sequenceNumber); - const outOfOrder = seqs.some((seq, index) => index > 0 && seq <= (seqs[index - 1] ?? -1)); - if (outOfOrder) { - disagreements.push( + if (seqs.some((seq, index) => index > 0 && seq <= (seqs[index - 1] ?? -1))) { + out.push( `the durable log is not ordered: sequenceNumbers ${JSON.stringify(seqs)} — a higher seq committed ` + `before a lower one, so the checkpoint fold can read a state its causal predecessor never reached`, ); } if (seqs.length > 0 && seqs[0] !== 0) { - disagreements.push( + out.push( `the durable log starts at sequenceNumber ${String(seqs[0])}, not 0 — \`run:started\` is always ` + `persisted and always first, so the head of the log is missing`, ); } const last = ours.at(-1); if (last !== undefined && terminalCount === 1 && !TERMINALS.has(last.type)) { - disagreements.push( + out.push( `the durable log continues past its terminal (last event is '${last.type}') — exactly one terminal ` + `CLOSES a run, so nothing may follow it`, ); } + return out; +} - const localFold = reconstructCheckpointState(finalEvents)?.runStatus; - const loaded = await input.loadCheckpoint?.(input.runId); - const checkpointStatus = input.loadCheckpoint === undefined ? localFold : loaded?.runStatus; +function checkCheckpoint( + viaPort: boolean, + checkpointStatus: RunStatus | undefined, + afterReconcile: TerminalView | undefined, +): readonly string[] { const expectedStatus = afterReconcile === undefined ? undefined : STATUS_FOR[afterReconcile.type]; - if (expectedStatus !== undefined && checkpointStatus !== expectedStatus) { + if (expectedStatus === undefined || checkpointStatus === expectedStatus) return []; + return [ + `the checkpoint ${viaPort ? 'port' : 'fold'} says ` + + `'${String(checkpointStatus)}' while the durable terminal says '${expectedStatus}' — a resume seeds ` + + `itself from it, so it would do the wrong work`, + ]; +} + +/** + * Compare the four views. Cheap enough to call at the end of every e2e run; it reads the log twice and runs + * one reconcile. + */ +export async function checkDurableTruth(input: DurableTruthInput): Promise { + const disagreements: string[] = []; + const expectRepaired = input.expect === 'repaired'; + + const before = [...(await input.eventsFor(input.runId))]; + disagreements.push(...checkRunBinding(input.runId, input.live, before)); + + // EVERY view below reads the filtered log, not the raw one. Reporting the foreign events above and then + // comparing against them anyway was the bug in the first version of this binding: a store that ignored its + // id argument had `history` read ANOTHER run's terminal, and `durableTerminalCount` counted it — so the + // verdict named the right cause while every downstream number described the wrong run. + let ours = before.filter((event) => event.runId === input.runId); + + const live = viewOf(input.live); + const history = viewOf(terminalIn(ours)); + disagreements.push(...checkLiveVsHistory(expectRepaired, live, history)); + + let reconciledCount = 0; + let afterReconcile = history; + if (input.reconcile !== undefined) { + const reconciled = await input.reconcile(); + // Per-RUN, not the whole array. `reconcile()` repairs every interrupted run in the store and returns all + // of their events, so attributing the raw count to this run produced a factually wrong message and a + // false failure the moment a second run existed — which is, again, `CR-11`'s exact shape. + reconciledCount = reconciled.filter((event) => event.runId === input.runId).length; + ours = [...(await input.eventsFor(input.runId))].filter((event) => event.runId === input.runId); + afterReconcile = viewOf(terminalIn(ours)); disagreements.push( - `the checkpoint ${input.loadCheckpoint === undefined ? 'fold' : 'port'} says ` + - `'${String(checkpointStatus)}' while the durable terminal says '${expectedStatus}' — a resume seeds ` + - `itself from it, so it would do the wrong work`, + ...checkReconcileOutcome(expectRepaired, history, afterReconcile, reconciledCount), ); } + const terminalCount = ours.filter((event) => TERMINALS.has(event.type)).length; + disagreements.push(...checkLogShape(ours, terminalCount)); + + const viaPort = input.loadCheckpoint !== undefined; + const loaded = await input.loadCheckpoint?.(input.runId); + const checkpointStatus = viaPort + ? loaded?.runStatus + : reconstructCheckpointState(ours)?.runStatus; + disagreements.push(...checkCheckpoint(viaPort, checkpointStatus, afterReconcile)); + return { agrees: disagreements.length === 0, runId: input.runId, diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index fe157c50..d0246e6c 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -1171,7 +1171,7 @@ class RunExecution { // the next pre-egress check; this covers the node boundary, which a crash could otherwise land inside with // a possibly-billed call recorded nowhere. Awaited (not fire-and-forget) so a failed write fails the node // loudly; the conservative amount keeps consuming capacity either way. - await this.#flushBudgetCommitments(vertex.id); + await this.#joinMoneyDurability(vertex.id); const willRetry = outcome.kind === 'failed' && !this.#settled && @@ -2143,7 +2143,12 @@ class RunExecution { } /** - * Await the budget governor's conservative-commitment barrier at a node boundary (ADR-0074 §2). + * Await BOTH money chains at a node boundary — the conservative commitments (ADR-0074 §2) and the realized + * ledger (ADR-0077 §4). Named for the join rather than for the commitments it once awaited alone: since + * ADR-0077 this is the single barrier fronting `MoneyDurability`, and `#flushBudgetCommitments` read as + * though the ledger were still someone else's to await. (The host-supplied + * `AgentSessionDeps.flushBudgetCommitments` keeps that name and is correct — the SESSION path has no + * realized ledger to join, because its per-attempt increment is already written synchronously.) * * **The await is the substance.** `#emitDurable` resolves only after `persistEvent` has settled (its `await * settled` at the end), so waiting here means a commitment made inside the attempt is on disk — or has already @@ -2156,7 +2161,7 @@ class RunExecution { * write. Kept here so the two surfaces cannot diverge in what a durability failure means: never a released * reservation, always a loud failure. */ - async #flushBudgetCommitments(nodeId: string): Promise { + async #joinMoneyDurability(nodeId: string): Promise { // **Barrier B3 (ADR-0077)** — and it is now the SINGLE join for both money chains. The old // `if (governor === undefined) return;` is gone: it was one of §5's three barrier holes, because a run // without a budget has no conservative commitments but does have a realized ledger, and returning early diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 51ca5e11..0541249d 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -1272,6 +1272,11 @@ describe('M2 — end-to-end Node harness (1.U)', () => { ledger.release(); const events: RunEvent[] = []; for await (const event of handle.events) events.push(event); + // PRESENCE first, then order. A bare `indexOf(a) < indexOf(b)` passes vacuously when `a` is missing — + // `-1` is less than everything — so a run that never settled a charge at all would satisfy the ordering + // it is supposed to prove. + expect(ledger.persistOrder).toContain('cost:attempt_settled:work'); + expect(ledger.persistOrder).toContain('node:completed:work'); expect(ledger.persistOrder.indexOf('cost:attempt_settled:work')).toBeLessThan( ledger.persistOrder.indexOf('node:completed:work'), ); diff --git a/packages/core/src/engine/money-durability.test.ts b/packages/core/src/engine/money-durability.test.ts index b03d27eb..525ea4e2 100644 --- a/packages/core/src/engine/money-durability.test.ts +++ b/packages/core/src/engine/money-durability.test.ts @@ -82,24 +82,26 @@ describe('MoneyDurability', () => { // A `better-sqlite3`-backed store throws synchronously. A bare `emit(draft)` inside the `.then` would // escape before `.catch`/`.finally` attach: the pending count leaks at >= 1 forever and the chain is left // permanently rejected, so every later join throws with no way to clear it. + const seen: string[] = []; + let boom = true; const money = new MoneyDurability({ - emit: () => { - throw new Error('sync boom'); + emit: (d) => { + if (boom) throw new Error('sync boom'); + seen.push(d.nodeId); }, }); money.record(draft('a')); await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); - // The chain is still usable: a later successful write joins cleanly rather than re-throwing forever. - const seen: string[] = []; - const healthy = new MoneyDurability({ - emit: (d) => { - seen.push(d.nodeId); - }, - }); - healthy.record(draft('b')); - await expect(healthy.join()).resolves.toBeUndefined(); + // The recovery half, on the SAME instance — and it has to be. What a synchronous throw could brick is + // THIS chain's tail and THIS pending count; a second, healthy instance shares neither, so asserting on + // one proved only that a fresh object works. Here the later write must reach the sink and the join must + // resolve, which is false if `#inFlight` was left permanently rejected or `#pending` leaked at >= 1. + boom = false; + money.record(draft('b')); + await expect(money.join()).resolves.toBeUndefined(); expect(seen).toEqual(['b']); + expect(money.durabilityBroken).toBe(true); // sticky, as designed — usable is not the same as healthy }); it('joins the CONSERVATIVE chain too — one join, not two (ADR-0077 §4)', async () => { diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs index a5194d48..4f3c7ee3 100644 --- a/tools/test-isolation/check.mjs +++ b/tools/test-isolation/check.mjs @@ -125,16 +125,38 @@ if (patterns.length === 0) { // --- 2. The list must reach BOTH excludes (assertion 3) -------------------------------------------- -const spreadCount = (configText.match(/\.\.\.REPO_LOCAL_CHECKOUTS/g) ?? []).length; -if (spreadCount < 2) { +// A GLOBAL count of `...REPO_LOCAL_CHECKOUTS` is the wrong test, and passes the exact defect it is for: two +// spreads inside `test.exclude` and none inside `coverage.exclude` reads as 2 and goes green, while a foreign +// tree's sources sit in the coverage denominator. So each property's own array literal is inspected. The +// arrays here hold no nested brackets, which is what makes the bounded `[^\]]*` match exact rather than +// approximate; a future nested value would end the match early and fail loudly here rather than silently pass. +const coverageAt = configText.indexOf('coverage: {'); +if (coverageAt === -1) { fail( - `REPO_LOCAL_CHECKOUTS is spread ${spreadCount} time(s) in ${relative(repoRoot, CONFIG_PATH)}; both ` + - `\`test.exclude\` and \`coverage.exclude\` need it.\n` + - ` \`test.exclude\` stops a foreign tree's TESTS from running; \`coverage.exclude\` stops its SOURCES ` + - `from entering the coverage denominator.\n \`vitest list\` cannot see the second one, which is why it ` + - `is checked here as text.`, + `cannot find the \`coverage: {\` block in ${relative(repoRoot, CONFIG_PATH)}, so \`test.exclude\` and ` + + `\`coverage.exclude\` cannot be told apart.`, ); } +const excludeArrays = [...configText.matchAll(/\bexclude:\s*\[([^\]]*)\]/g)]; +const spreadsIn = (body) => (body.match(/\.\.\.REPO_LOCAL_CHECKOUTS\b/g) ?? []).length; +// `test.exclude` is the LAST one before the coverage block, `coverage.exclude` the FIRST one after it. +const testExclude = excludeArrays.filter((m) => m.index < coverageAt).at(-1); +const coverageExclude = excludeArrays.find((m) => m.index > coverageAt); +for (const [name, match, why] of [ + ['test.exclude', testExclude, "stops a foreign tree's TESTS from running"], + ['coverage.exclude', coverageExclude, 'stops its SOURCES from entering the coverage denominator'], +]) { + const count = match === undefined ? 0 : spreadsIn(match[1]); + if (count !== 1) { + fail( + `\`${name}\` in ${relative(repoRoot, CONFIG_PATH)} spreads \`...REPO_LOCAL_CHECKOUTS\` ${count} ` + + `time(s); it needs exactly 1.\n ${name} ${why}. Both halves are required and they fail ` + + `differently.\n \`vitest list\` observes \`test.exclude\` only, which is why this is checked here ` + + `as text — and why the count is checked PER PROPERTY: a repo-wide total of 2 is also what two ` + + `spreads in one array and none in the other looks like.`, + ); + } +} // --- 3. Plant one fixture per entry, collect, assert ------------------------------------------------ From 29d502bbe5ebba474bc85eb02d2a188e43e783f1 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 09:40:52 +0300 Subject: [PATCH 25/26] style(core,tools): fold two Sonar smells from the review fold itself MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both are in code the previous commit introduced. `canonical`'s key comparator was a nested ternary; it is now a named `byCodepoint` function carrying the reason it is not `localeCompare` — the comparison must be identical on every machine, or the instrument manufactures the disagreement it exists to detect. `.filter(…).at(-1)` → `.findLast(…)` when picking `test.exclude` (the last exclude array before the coverage block). Break-verified again after the change: dropping `...REPO_LOCAL_CHECKOUTS` from `coverage.exclude` still exits 1. `pnpm run ci` and `pnpm coverage` green. Co-Authored-By: Claude Opus 5 --- packages/core/src/engine/durable-truth.ts | 20 +++++++++++++------- tools/test-isolation/check.mjs | 2 +- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts index 7f9f1d08..d38bbbe6 100644 --- a/packages/core/src/engine/durable-truth.ts +++ b/packages/core/src/engine/durable-truth.ts @@ -107,6 +107,18 @@ export interface DurableTruthVerdict { readonly disagreements: readonly string[]; } +/** + * Codepoint order, NOT `localeCompare` — the comparison must be identical on every machine that runs this + * oracle. `localeCompare` honours the host locale (and its ICU build), so two runs of the same payload can + * canonicalize to different strings under different `LC_ALL`s, which is a disagreement manufactured by the + * instrument. The order itself is arbitrary; only its stability matters. + */ +function byCodepoint([a]: readonly [string, unknown], [b]: readonly [string, unknown]): number { + if (a < b) return -1; + if (a > b) return 1; + return 0; +} + /** * Stringify a payload for comparison, key-order-insensitively and WITHOUT ever throwing out of the oracle. * @@ -119,13 +131,7 @@ function canonical(value: unknown): string { return JSON.stringify(value, (_key, val: unknown): unknown => { if (typeof val === 'bigint') return `${val.toString()}n`; if (typeof val !== 'object' || val === null || Array.isArray(val)) return val; - // Codepoint order, NOT `localeCompare` — the comparison must be identical on every machine that runs - // this oracle. `localeCompare` honours the host locale (and its ICU build), so two runs of the same - // payload can canonicalize to different strings under different `LC_ALL`s, which is a disagreement - // manufactured by the instrument. The order itself is arbitrary; only its stability matters. - return Object.fromEntries( - Object.entries(val).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)), - ); + return Object.fromEntries(Object.entries(val).sort(byCodepoint)); }); } catch (error) { // A true cycle (or any other unserializable payload) becomes a single marker rather than escaping. The diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs index 4f3c7ee3..8b75e7e0 100644 --- a/tools/test-isolation/check.mjs +++ b/tools/test-isolation/check.mjs @@ -140,7 +140,7 @@ if (coverageAt === -1) { const excludeArrays = [...configText.matchAll(/\bexclude:\s*\[([^\]]*)\]/g)]; const spreadsIn = (body) => (body.match(/\.\.\.REPO_LOCAL_CHECKOUTS\b/g) ?? []).length; // `test.exclude` is the LAST one before the coverage block, `coverage.exclude` the FIRST one after it. -const testExclude = excludeArrays.filter((m) => m.index < coverageAt).at(-1); +const testExclude = excludeArrays.findLast((m) => m.index < coverageAt); const coverageExclude = excludeArrays.find((m) => m.index > coverageAt); for (const [name, match, why] of [ ['test.exclude', testExclude, "stops a foreign tree's TESTS from running"], From 93948fda45411d89373d7e7f553cacfb992c8864 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 11 Aug 2026 10:19:22 +0300 Subject: [PATCH 26/26] fix(core,tools,docs): fold the second PR #82 review round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified each finding against the tree. **The oracle reported two different cyclic payloads as equal.** Sorting the keys means the replacer hands `JSON.stringify` a fresh object at every level, which defeats its native cycle detector — so a cyclic payload never raised the `TypeError` the catch was written for; it recursed until the stack blew and BOTH sides came back `[uncomparable: RangeError]`. `canonical` now walks the tree with an ancestor stack (popped on the way back up, so a shared reference stays a shared reference) and emits a path-local `[circular]` marker. `toJSON` is applied by hand for the same reason native `stringify` applies it: without it every `Date` reduces to `{}` and any two compare equal — a false agreement the fix would have introduced. Break-verified both ways. **A ledger row could report a total that included money it never spent.** The chained write reads the engine's live counter when it RUNS — behind the previous write's `persistEvent`, which is real I/O — so under a `fan_out` (concurrent nodes, one chain) it can report attempts that settled during the wait. `MoneyDurability.record` now takes the total captured at record time and `turnPort` snapshots it synchronously. The contract's claim that each absolute is "a true run-wide total at that instant" is only true of a captured value. **`#runAttempt`'s catch-all flattened money-durability failures.** `throwMappedChainError` has two arms whose only job is to keep `LedgerDurabilityError` / `CommitmentDurabilityError` and their owning `nodeId` intact out of the turn; the catch-all then reported `the node handler threw an unexpected error`. Preserved for one frame, then discarded — and unrecoverable, since `join()` surfaces a retained failure exactly once. Corrected in the code comment after measuring: the arm is NOT reached on the ordinary path (the abort wins and the node classifies as cancelled), only where `#emitDurable` skips the abort because a sibling already set `#failure` — ADR-0077's still-unbuilt required regression. An assertion claiming otherwise was written and removed. **The isolation guard's probes all came from the list it was checking**, so deleting an entry deleted its probe. Three known locations are now planted unconditionally; the structural assertion covers this only when a real second checkout is on disk, which CI never has. Break-verified by removing `.claude` from `REPO_LOCAL_CHECKOUTS`: the control fires with the entry to paste back. Docs: `security-review.md` claimed `--json` reproduces its data "byte for byte", which contradicts escaping — what is preserved is the parsed value. The phase doc said `#W15-1` was "merged"; it is landed on `development`, pending merge to `main`. `testing.md` referred to two files in backticks rather than links. ADR-0076 gained the dated `> Amended` note `docs/decisions/README.md` step 5 requires; the in-place §1 markers stay. Also: `m2-e2e` asserted `history.errorCode === live.errorCode`, which `undefined === undefined` satisfies — the concrete code is pinned first. New tests: the record-time stamp, and both failure sources through one `join()` (whose fixture rejects ONCE, because a permanently-rejecting conservative half would mask the retained ledger failure for the rest of the run). **Skipped.** `vi.waitFor` for the two bounded microtask spins: the condition flips on a microtask, so a wall-clock poller trades a deterministic loop for a timer that can flake under CI load. `pnpm run ci` and `pnpm coverage` green — 96.91% lines / 93.31% branches. Refs: ADR-0076, ADR-0077 Co-Authored-By: Claude Opus 5 --- ...urable-per-attempt-realized-cost-ledger.md | 7 ++ ...hase-2.6.5-core-reliability-remediation.md | 5 +- docs/standards/security-review.md | 3 +- docs/standards/testing.md | 7 +- .../core/src/engine/durable-truth.test.ts | 34 ++++++ packages/core/src/engine/durable-truth.ts | 55 ++++++--- packages/core/src/engine/engine.ts | 104 +++++++++++++----- .../src/engine/m2-e2e-harness.e2e.test.ts | 13 +++ .../core/src/engine/money-durability.test.ts | 83 ++++++++++++-- packages/core/src/engine/money-durability.ts | 36 ++++-- tools/test-isolation/check.mjs | 53 +++++++-- 11 files changed, 324 insertions(+), 76 deletions(-) diff --git a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md index f93a1d69..5f03bd8b 100644 --- a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md +++ b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md @@ -4,6 +4,13 @@ - **Date**: 2026-08-09 - **Related**: [ADR-0074](0074-durable-conservative-budget-commitments.md) (the conservative half of the same problem), [ADR-0075](0075-fail-closed-resume-on-an-unreadable-event-log.md) (how this event degrades on an older binary), [ADR-0070](0070-durable-per-model-session-cost-attribution.md) (`SUM(run_costs) == runs.total_cost_microcents`), [ADR-0028](0028-workflow-resource-governance.md) (the pre-egress cap), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) (durable run events), and [sse-event-schema.md](../reference/contracts/sse-event-schema.md) (the canonical event contract). +> **Amended 2026-08-10** by [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md): §1's +> MECHANISM only. The guarantee — a realized charge is durable before the run spends or mutates again — stands +> unchanged; the "awaited inline emit at the attempt boundary" shape does not, because the seam's attempt +> observer is synchronous and there is no `await` to place there. ADR-0077 substitutes ADR-0074 §2's +> chain-and-join and adds a third barrier before tool dispatch. Not a reversal, so not a supersession. The +> in-place markers in §1 below stay where the superseded sentences are. + ## Context ADR-0074 made the **conservative** commitment durable — money a provider may already have billed for a call that returned no trustworthy usage. It deliberately did not touch the other half. A call that *did* return trustworthy usage has no equivalent barrier, and that gap is larger than it looks. diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md index c4140c19..f3e0405e 100644 --- a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -75,8 +75,9 @@ rejected: a phase whose exit criteria cannot be evaluated until a later phase is ## Prerequisite **`#W15-1` — the durable per-attempt realized-cost ledger — lands before `W1` starts. ✅ SATISFIED -(2026-08-10).** All five steps are merged, each with an Opus and a Sonnet round folded, and the decision -acquired a correction on the way: +(2026-08-10).** All five steps are implemented and landed on `development`, each with an Opus and a Sonnet +round folded; they ride **PR #82** and are **pending merge to `main`**. The decision acquired a correction on +the way: [ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) amends ADR-0076 §1, whose stated mechanism (an inline `await` at the attempt boundary) is unimplementable — the seam's attempt observer is synchronous. The ledger uses ADR-0074 §2's chain-and-join shape instead, with a diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index 1b4c61b6..f3886de7 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -394,7 +394,8 @@ serialized with `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), never a `JSON.stringify` escapes `ESC` and stops, leaving `DEL`, the whole C1 block — including `U+009B`, a working escape-sequence introducer on real terminals — and the Trojan-Source bidi family RAW in content the model, a tool, or an imported artifact controls. It **escapes** where the human-display path **strips**, because -`--json` is a machine contract and must reproduce its data byte for byte. The rule is enforced by an ESLint +`--json` is a machine contract: escaping changes the JSON *text* but `JSON.parse` returns the identical value, +so a consumer loses nothing — while stripping would silently hand it different data. The rule is enforced by an ESLint `no-restricted-syntax` selector rather than by review, because the gap reopened twice when it was not (`#W15-10`, then `CR-03`). `apps/cli/src/process/render-error.ts` is the single allowlisted exception: it pre-sanitizes with the stripping sanitizer, so the `--json` ERROR envelope is deliberately lossy. diff --git a/docs/standards/testing.md b/docs/standards/testing.md index 0a9feef3..42b06364 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -136,10 +136,11 @@ PRs must pass: typecheck, lint ([code-style-typescript.md](code-style-typescript unit tests, and the fixture-mode conformance suite. The live conformance suite and the desktop e2e suite run nightly and on release branches. A red required check blocks merge. -**Test isolation is part of that gate** (`pnpm lint:test-isolation`, `tools/test-isolation`). A root run must +**Test isolation is part of that gate** (`pnpm lint:test-isolation`, +[tools/test-isolation/check.mjs](../../tools/test-isolation/check.mjs)). A root run must collect this repo's tests and nothing else. A second checkout of the repo inside the working tree — an agent worktree, a scratch clone — is otherwise collected in full: measured once at 234 foreign test files out of 472, with its sources in the coverage denominator. The guard fails on any collected file whose ancestor carries its -own `pnpm-workspace.yaml`, so it stays true even when the exclusion list in `vitest.config.ts` is the thing -that is wrong. It also asserts every workspace still yields tests, because the cheapest way to pass an +own `pnpm-workspace.yaml`, so it stays true even when the exclusion list in +[vitest.config.ts](../../vitest.config.ts) is the thing that is wrong. It also asserts every workspace still yields tests, because the cheapest way to pass an isolation check is an over-broad exclude that goes green by not running. diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts index 68b583f0..8e912c69 100644 --- a/packages/core/src/engine/durable-truth.test.ts +++ b/packages/core/src/engine/durable-truth.test.ts @@ -436,6 +436,40 @@ describe('checkDurableTruth', () => { expect(verdict.agrees).toBe(true); }); + it('tells two DIFFERENT circular payloads apart — not one marker for every cycle', async () => { + // Comparing a cyclic payload with ITSELF (above) passes under any implementation, including the broken + // one. Sorting the keys makes the replacer hand `JSON.stringify` a fresh object at every level, which + // defeats its native cycle detector: the walk recursed until the stack blew and BOTH sides came back as + // `[uncomparable: RangeError]`. Two structurally different cyclic outputs compared EQUAL and the oracle + // verdicted `agrees` — the false agreement it exists to catch, produced by the instrument itself. + const live: Record = { note: 'live' }; + live['self'] = live; + const durable: Record = { note: 'durable' }; + durable['self'] = durable; + + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, live), + eventsFor: () => [started, completed(1, durable)], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + expect(formatDurableTruth(verdict)).toContain('[circular]'); + }); + + it('canonicalizes a Date through toJSON — walking the tree by hand must not lose it', async () => { + // Native `JSON.stringify` applies `toJSON` before the replacer; the hand-rolled cycle-aware walk has to + // apply it too. Without that a `Date` reduces to `{}` (it has no own enumerable entries) and ANY two dates + // compare equal — a false agreement introduced by the cycle fix rather than one it removes. + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { at: new Date('2026-01-01T00:00:00.000Z') }), + eventsFor: () => [started, completed(1, { at: new Date('2026-09-09T00:00:00.000Z') })], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + }); + it('treats a SHARED reference as shared, not circular — a repeated object is not a cycle', async () => { // The first cycle guard was a flat `WeakSet` that was never un-marked on the way back up, so it flagged // every REPEATED reference. `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts index d38bbbe6..96438884 100644 --- a/packages/core/src/engine/durable-truth.ts +++ b/packages/core/src/engine/durable-truth.ts @@ -125,23 +125,52 @@ function byCodepoint([a]: readonly [string, unknown], [b]: readonly [string, unk * A bare `JSON.stringify` fails twice: `{a:1,b:2}` and `{b:2,a:1}` compare unequal, which is a false failure * the moment either side has been through a store round-trip — the whole point of this module; and a circular * or `bigint` payload throws, turning the instrument itself into the failure with no verdict at all. + * + * **Cycles are detected HERE rather than left to `JSON.stringify`, and the reason is measured.** Sorting the + * keys means the replacer returns a NEW object at every level, which defeats the native cycle detector — it + * compares against the objects it is currently serializing, and it never sees the same one twice. So a cyclic + * payload did not raise the `TypeError` the catch below was written for; it recursed until the stack blew, and + * BOTH sides came back as `[uncomparable: RangeError]`. Two structurally different cyclic outputs therefore + * compared EQUAL and the oracle verdicted `agrees` — the exact false agreement it exists to catch. + * + * The ancestor stack is popped on the way back up, which is what separates a cycle from a shared reference. + * The first attempt at this used a flat `WeakSet` that was never un-marked, so it flagged every REPEATED + * reference: `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a structurally + * identical payload built from two separate objects serialized normally — two equal outputs disagreeing, and a + * `fan_in` echoing one value into two keys is enough to hit it. Ancestry is the correct predicate; the pop is + * the whole difference. */ function canonical(value: unknown): string { + const ancestors: object[] = []; + const walk = (val: unknown): unknown => { + if (typeof val === 'bigint') return `${val.toString()}n`; + if (typeof val !== 'object' || val === null) return val; + // A path-distinguishing marker, not a whole-payload one: the REST of the object still serializes, so two + // different cyclic payloads still differ everywhere they actually differ. + if (ancestors.includes(val)) return '[circular]'; + ancestors.push(val); + try { + // `toJSON` first, or a `Date` canonicalizes to `{}` — and then any two Dates would compare equal, which + // is a false agreement introduced by the fix. Native `JSON.stringify` applies it before the replacer; + // walking the tree by hand means applying it by hand. + const toJson: unknown = (val as { toJSON?: unknown }).toJSON; + if (typeof toJson === 'function') return walk((toJson as () => unknown).call(val)); + if (Array.isArray(val)) return val.map((entry: unknown) => walk(entry)); + return Object.fromEntries( + Object.entries(val) + .sort(byCodepoint) + .map(([key, entry]: [string, unknown]) => [key, walk(entry)]), + ); + } finally { + ancestors.pop(); + } + }; try { - return JSON.stringify(value, (_key, val: unknown): unknown => { - if (typeof val === 'bigint') return `${val.toString()}n`; - if (typeof val !== 'object' || val === null || Array.isArray(val)) return val; - return Object.fromEntries(Object.entries(val).sort(byCodepoint)); - }); + // `?? '[undefined]'`: `JSON.stringify(undefined)` returns `undefined`, and the return type says string. + return JSON.stringify(walk(value)) ?? '[undefined]'; } catch (error) { - // A true cycle (or any other unserializable payload) becomes a single marker rather than escaping. The - // first version tried to be cleverer, with a `WeakSet` marking objects it had already visited — and got - // it WRONG in a way that caused the very failure this function exists to prevent: the set was never - // un-marked on the way back up, so it flagged every REPEATED reference, not every circular one. - // `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a structurally identical - // payload built from two separate objects serialized normally — so two equal outputs disagreed. A - // `fan_in` echoing one value into two keys is enough to hit it. Native `JSON.stringify` already tracks - // the ANCESTRY correctly; letting it throw and catching here is both simpler and right. + // The backstop is still here — a hostile getter, a `toJSON` that throws, anything unforeseen — but it is + // no longer the cycle path, so it no longer collapses distinct payloads onto one marker. return `[uncomparable: ${error instanceof Error ? error.name : 'unknown'}]`; } } diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index d0246e6c..f5062506 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -421,7 +421,7 @@ class RunExecution { // ledger that only existed alongside a governor would silently skip every unbudgeted run. It fronts the // join for BOTH chains, so `flushConservative` is wired to the governor once that exists. this.#money = new MoneyDurability({ - emit: async (draft) => { + emit: async (draft, cumulativeCostMicrocents) => { // **The observe half, and it has to be here rather than in `MoneyDurability`.** `#emitDurable` is // TOTAL for store faults: it absorbs a `persistEvent` rejection into `#failure` and RESOLVES. So the // ledger's own `.catch` never fires for the failure mode it exists to catch, and a barrier that only @@ -434,11 +434,14 @@ class RunExecution { ...draft, type: 'cost:attempt_settled', runId: this.runId, - // Stamped HERE because the engine owns the run-wide counter — `#nodeEmit`'s `cost:updated` arm has - // already folded this attempt into it (the turn core records strictly after that emit), so reading - // it now satisfies `refineCostAttemptSettled`'s "cumulative already includes this charge" by - // construction, the same way the governor reads its counter after `+=`. - cumulativeCostMicrocents: this.#cumulativeCostMicrocents, + // The total CAPTURED AT `record()` TIME, passed in — deliberately not a fresh read of + // `#cumulativeCostMicrocents` here. `#nodeEmit`'s `cost:updated` arm folds the charge into the + // counter and the turn core records strictly after that, so the captured value satisfies + // `refineCostAttemptSettled`'s "cumulative already includes this charge" by construction. Reading it + // HERE would not: this callback is chained behind the previous write's `persistEvent`, so under a + // `fan_out` — concurrent nodes sharing one chain — it can run after several more attempts have + // settled and report their money as this attempt's running total. + cumulativeCostMicrocents, }); if (this.#failure !== failureBefore) { // Typed, not a bare `Error` (error-handling.md). `#emitDurable` discards the store error in its own @@ -1274,12 +1277,38 @@ class RunExecution { // Unconditional, unlike `preEgress` above — which `budgetApproved` deliberately drops for an approved // re-dispatch. The ledger must not be dropped with it: an approved node is the one the user just // authorised MORE money on, so it is the last place to stop recording what that money was. - money: this.#money.turnPort(), + money: this.#money.turnPort(() => this.#cumulativeCostMicrocents), }; // After the executor completes, an `output` node with `save_to` writes its produced media to the // host (1.AF/D16). A write failure FAILS the node (→ run:failed) — save_to is a real deliverable. return await this.#applySaveTo(vertex, await this.#executor.execute(ctx)); - } catch { + } catch (error) { + // A money-durability failure is NOT an anonymous handler throw. Barriers B1 and B2 (ADR-0077) both sit + // INSIDE the turn, and `throwMappedChainError` has two arms whose only job is to keep the class and its + // owning `nodeId` intact on the way out — under a `fan_out` the broken write may be a sibling's, and + // the message that names the remedy differs between the estimate and realized halves. Then + // `turnOutcomeForError` re-throws anything it cannot classify and the generic arm below reported `the + // node handler threw an unexpected error`: preserved for exactly one frame, then discarded. B3 two + // lines later cannot repair it either — `join()` surfaces a retained failure exactly ONCE, and the + // in-turn barrier already consumed it. + // + // **When this arm is actually reached, stated because measuring it corrected a claim.** On the ordinary + // path it is NOT: `#emitDurable` absorbs the store fault, sets `#failure` and aborts, so the turn ends + // at `throwIfAborted` and the node classifies as cancelled before any barrier throws. It is reached + // where the abort does not fire — `#emitDurable` skips it when `#failure` is already set by a sibling — + // which is precisely ADR-0077's stated required regression, still unbuilt. The arm is here because the + // flattening is wrong whenever it does happen, not because a test drives it today. + if (isLedgerDurabilityError(error) || error instanceof CommitmentDurabilityError) { + this.#failMoneyDurability(error, vertex.id); + return { + kind: 'failed', + error: { + code: 'internal', + message: this.#failure?.error.message ?? 'a money write could not be made durable', + retryable: false, + }, + }; + } // The catch-all: any uncaught throw from a node handler maps to a single internal failure // (a tool handler classifies its own failures as tool_failed; a sandbox throw as sandbox_error). return { @@ -2170,31 +2199,46 @@ class RunExecution { try { await this.#money.join(); } catch (error) { - // Attribute it to the node whose write actually failed, not to whichever node reached this barrier - // first — under a `fan_out` both branches await the same chain link, so the first to flush may have made no - // commitment at all. Falls back to this node when the error carries no owner. - const ledger = isLedgerDurabilityError(error); - const owner = - error instanceof CommitmentDurabilityError || ledger ? (error.nodeId ?? nodeId) : nodeId; - this.#failure ??= { - nodeId: owner, - error: { - code: 'internal', - // The cause is deliberately NOT in the message: a durable-write failure can carry a filesystem path, and - // a user-facing `run:failed` message must not. It survives on the error's `cause` for a host that - // narrows on the class — which is the only carrier, since this path has no store to log it. The two - // messages are distinct because the remedies differ: an estimate that could not be recorded leaves the - // cap conservative, while a realized charge that could not be recorded leaves it UNDERSTATED. - message: ledger - ? 'a realized provider charge could not be made durable' - : 'a conservative budget commitment could not be made durable', - retryable: false, - }, - }; - this.#abort.abort(); + this.#failMoneyDurability(error, nodeId); } } + /** + * Record a money-durability failure against the node whose write actually broke, and abort. + * + * Shared by B3's own catch and by `#runAttempt`'s catch-all, and the second caller is why it is a method: + * barriers B1 and B2 live INSIDE the turn, so their throw arrives as a bare `LedgerDurabilityError` / + * `CommitmentDurabilityError` that the catch-all used to flatten. See the note at that call site for when + * that path is actually reached — it is narrower than it looks. + * + * `??=` throughout: a sibling's already-recorded root cause always wins, which is also why this is usually + * a no-op on the ordinary path (`#emitDurable` has already set `#failure` from the same fault). + */ + #failMoneyDurability(error: unknown, nodeId: string): void { + // Attribute it to the node whose write actually failed, not to whichever node reached this barrier + // first — under a `fan_out` both branches await the same chain link, so the first to flush may have made no + // commitment at all. Falls back to this node when the error carries no owner. + const ledger = isLedgerDurabilityError(error); + const owner = + error instanceof CommitmentDurabilityError || ledger ? (error.nodeId ?? nodeId) : nodeId; + this.#failure ??= { + nodeId: owner, + error: { + code: 'internal', + // The cause is deliberately NOT in the message: a durable-write failure can carry a filesystem path, and + // a user-facing `run:failed` message must not. It survives on the error's `cause` for a host that + // narrows on the class — which is the only carrier, since this path has no store to log it. The two + // messages are distinct because the remedies differ: an estimate that could not be recorded leaves the + // cap conservative, while a realized charge that could not be recorded leaves it UNDERSTATED. + message: ledger + ? 'a realized provider charge could not be made durable' + : 'a conservative budget commitment could not be made durable', + retryable: false, + }, + }; + this.#abort.abort(); + } + async #emitDurable(draft: RunEventDraft): Promise { // Persist the boundary/terminal event, then deliver (ADR-0036 persist-before-deliver, so a crash // can never re-run a completed node or lose its output). This method is **total for store faults** (the diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 0541249d..722058a1 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -1132,6 +1132,10 @@ describe('M2 — end-to-end Node harness (1.U)', () => { expect(events.at(-1)?.type).toBe('run:failed'); const verdict = await assertDurableTruth(host, store, events, () => provider); expect(verdict.checkpointStatus).toBe('failed'); + // The concrete code FIRST. `history === live` alone is satisfied by `undefined === undefined`, so it holds + // just as well for a verdict that read no error at all from either side — including one where a refactor + // stopped populating `errorCode`. Pinning the value is what makes the equality mean something. + expect(verdict.history?.errorCode).toBe('provider_auth'); expect(verdict.history?.errorCode).toBe(verdict.live?.errorCode); }); @@ -1329,6 +1333,15 @@ describe('M2 — end-to-end Node harness (1.U)', () => { // Secret-free either way: a store error can carry a filesystem path and must never reach the user. expect(terminal.error.message).not.toContain('ledger write failed'); } + // And the NODE terminal says `the run was cancelled`, NOT the barrier's diagnosis — measured, after an + // assertion here claimed otherwise. It is the same fact as the paragraph above: the abort wins, the turn + // ends at `throwIfAborted`, and B2's `LedgerDurabilityError` is never the error that classifies this node. + // `#runAttempt` DOES have an arm that keeps that class intact (it used to flatten it to `the node handler + // threw an unexpected error`), but reaching it needs the same abort-defeating fixture named above. + const nodeFailed = events.find((e) => e.type === 'node:failed'); + expect(nodeFailed?.type === 'node:failed' ? nodeFailed.error.message : undefined).toBe( + 'the run was cancelled', + ); }); it('ledger: an UNBUDGETED run records its realized spend at all (ADR-0077 §5)', async () => { diff --git a/packages/core/src/engine/money-durability.test.ts b/packages/core/src/engine/money-durability.test.ts index 525ea4e2..6ec3f4e7 100644 --- a/packages/core/src/engine/money-durability.test.ts +++ b/packages/core/src/engine/money-durability.test.ts @@ -34,8 +34,8 @@ describe('MoneyDurability', () => { }, }); - money.record(draft('a')); - money.record(draft('b')); + money.record(draft('a'), 400); + money.record(draft('b'), 400); await vi.waitFor(() => { expect(releaseFirst).toBeDefined(); }); @@ -52,7 +52,7 @@ describe('MoneyDurability', () => { const money = new MoneyDurability({ emit: () => Promise.reject(new Error('disk full')), }); - money.record(draft('a')); + money.record(draft('a'), 400); await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); expect(money.durabilityBroken).toBe(true); @@ -65,7 +65,7 @@ describe('MoneyDurability', () => { // `run:failed`. The cause rides `cause` for a host that narrows on the class. const cause = new Error('ENOENT: /Users/someone/.relavium/history.db'); const money = new MoneyDurability({ emit: () => Promise.reject(cause) }); - money.record(draft('the-node')); + money.record(draft('the-node'), 400); await money.join().then( () => expect.unreachable('join must throw'), @@ -90,7 +90,7 @@ describe('MoneyDurability', () => { seen.push(d.nodeId); }, }); - money.record(draft('a')); + money.record(draft('a'), 400); await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); // The recovery half, on the SAME instance — and it has to be. What a synchronous throw could brick is @@ -98,7 +98,7 @@ describe('MoneyDurability', () => { // one proved only that a fresh object works. Here the later write must reach the sink and the join must // resolve, which is false if `#inFlight` was left permanently rejected or `#pending` leaked at >= 1. boom = false; - money.record(draft('b')); + money.record(draft('b'), 400); await expect(money.join()).resolves.toBeUndefined(); expect(seen).toEqual(['b']); expect(money.durabilityBroken).toBe(true); // sticky, as designed — usable is not the same as healthy @@ -119,7 +119,7 @@ describe('MoneyDurability', () => { emit: () => {}, flushConservative: () => Promise.reject(new Error('commitment write failed')), }); - money.record(draft('a')); + money.record(draft('a'), 400); await expect(money.join()).rejects.toThrow('commitment write failed'); }); @@ -138,9 +138,76 @@ describe('MoneyDurability', () => { }, }); // Destructured, because the turn core passes `params.money` around by value. - const { record, join } = money.turnPort(); + const { record, join } = money.turnPort(() => 0); record(draft('a', 700)); await join(); expect(seen).toEqual([expect.objectContaining({ nodeId: 'a', costMicrocents: 700 })]); }); + + it('captures the run-wide total at RECORD time, not when the chained write runs', async () => { + // The write is chained behind the previous one's `persistEvent` — real I/O. Reading the engine's live + // counter inside `emit` therefore yields the total including every attempt that settled DURING that wait, + // which under a `fan_out` (concurrent nodes sharing one chain) is a different number. Here the counter + // advances while `a` is in flight, and `b` must still report the total as of its own settle. + const stamped: number[] = []; + let releaseFirst: (() => void) | undefined; + const money = new MoneyDurability({ + emit: async (d, cumulative) => { + stamped.push(cumulative); + if (d.nodeId === 'a') { + await new Promise((resolve) => { + releaseFirst = resolve; + }); + } + }, + }); + let counter = 100; + const { record, join } = money.turnPort(() => counter); + + record(draft('a')); // settles at 100 + await vi.waitFor(() => { + expect(releaseFirst).toBeDefined(); + }); + counter = 250; + record(draft('b')); // settles at 250 — queued behind `a`, which is still blocked + counter = 900; // a third node settles while `a`'s write is still in flight + releaseFirst?.(); + await join(); + + // Reading the counter inside `emit` would make this `[100, 900]` — `b`'s row claiming a total that + // includes money `b` never spent, from an attempt that had not settled when `b` did. + expect(stamped).toEqual([100, 250]); + }); + + it('reports the CONSERVATIVE failure first, then the retained ledger failure on the next join', async () => { + // Both halves can be broken at once, and `join()` surfaces one failure per call. The order is not + // arbitrary: the conservative flush is awaited BEFORE the retained ledger failure is thrown, so a caller + // sees the estimate error first — and the ledger error must still be there afterwards rather than being + // dropped by the throw that overtook it. + // + // The fixture rejects ONCE on purpose, because that is `BudgetGovernor.flushCommitments`'s own contract + // (a retained failure is surfaced once, then cleared) — and the coupling is load-bearing rather than + // incidental: a conservative half that rejected on EVERY call would mask the retained ledger failure for + // the rest of the run, since `join()` never reaches the throw below. + let conservativeBroken = true; + const money = new MoneyDurability({ + emit: () => Promise.reject(new Error('ledger disk full')), + flushConservative: () => { + if (!conservativeBroken) return Promise.resolve(); + conservativeBroken = false; + return Promise.reject(new Error('commitment write failed')); + }, + }); + money.record(draft('the-node'), 400); + + await expect(money.join()).rejects.toThrow('commitment write failed'); + await money.join().then( + () => expect.unreachable('the retained ledger failure must still surface'), + (error: unknown) => { + if (!isLedgerDurabilityError(error)) expect.unreachable('wrong error class'); + expect(error.nodeId).toBe('the-node'); + }, + ); + expect(money.durabilityBroken).toBe(true); + }); }); diff --git a/packages/core/src/engine/money-durability.ts b/packages/core/src/engine/money-durability.ts index 2a7f6ebd..313ad67f 100644 --- a/packages/core/src/engine/money-durability.ts +++ b/packages/core/src/engine/money-durability.ts @@ -48,11 +48,21 @@ export interface SettledAttemptDraft { export interface MoneyDurabilityOptions { /** - * Start the durable write for one settled attempt. The engine wires this to `#emitDurable`, stamping the - * run-wide cumulative it alone owns. **May throw synchronously** — a `better-sqlite3` store does — which is - * why {@link MoneyDurability.record} never calls it bare. + * Start the durable write for one settled attempt. The engine wires this to `#emitDurable`. **May throw + * synchronously** — a `better-sqlite3` store does — which is why {@link MoneyDurability.record} never calls + * it bare. + * + * `cumulativeCostMicrocents` is passed IN rather than read here, and that is the whole point of the + * parameter: the writes are chained, so this callback runs an unbounded time after the attempt it describes + * — behind the previous write's `persistEvent`, which is real I/O. Reading the engine's live counter at that + * moment yields the total including every attempt that settled DURING the wait, which under a `fan_out` + * (concurrent nodes sharing one chain) is a different number. The contract says each absolute is "a true + * run-wide total at that instant"; only a value captured at `record()` time is. */ - readonly emit: (draft: SettledAttemptDraft) => Promise | void; + readonly emit: ( + draft: SettledAttemptDraft, + cumulativeCostMicrocents: number, + ) => Promise | void; /** * Join the CONSERVATIVE chain, when a budget governor exists. Wired to `BudgetGovernor.flushCommitments`, * which throws its own retained failure. Absent on an unbudgeted run — the realized half still applies. @@ -111,7 +121,7 @@ export class MoneyDurability { * START one settled attempt's durable write. Synchronous by necessity (the caller is a chain observer that * cannot await) and never awaited here, so a rejection is captured and re-thrown at the next barrier. */ - record(draft: SettledAttemptDraft): void { + record(draft: SettledAttemptDraft, cumulativeCostMicrocents: number): void { this.#pending += 1; this.#inFlight = this.#inFlight.then(() => // `Promise.resolve().then(…)`, NOT a bare `emit(draft)`. A sink that throws SYNCHRONOUSLY — which a @@ -120,7 +130,9 @@ export class MoneyDurability { // throws with no way to clear it. The governor's estimate twin documents the same brick at length. // The inner chain therefore ALWAYS resolves, which is why there is no onRejected arm here. Promise.resolve() - .then(() => this.#options.emit(draft)) + // The captured total, not a fresh read — see `MoneyDurabilityOptions.emit`. It rides the closure so + // the write describes the instant it was recorded, not the instant the chain got round to it. + .then(() => this.#options.emit(draft, cumulativeCostMicrocents)) .catch((error: unknown) => { // Keep the FIRST failure: it is the one that broke durability, and later writes may well fail for // the same reason. @@ -156,11 +168,17 @@ export class MoneyDurability { } } - /** The narrow port handed to the turn core. Bound so a destructured `record` still works. */ - turnPort(): TurnMoneyPort { + /** + * The narrow port handed to the turn core. Bound so a destructured `record` still works. + * + * `snapshotCumulative` is read SYNCHRONOUSLY here, inside `record`, because that is the only moment the + * run-wide total is the one this attempt produced: the turn core calls `record` immediately after the + * engine folded this charge into its counter, and the chained write runs later. + */ + turnPort(snapshotCumulative: () => number): TurnMoneyPort { return { record: (draft) => { - this.record(draft); + this.record(draft, snapshotCumulative()); }, join: () => this.join(), }; diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs index 8b75e7e0..0c50ef3a 100644 --- a/tools/test-isolation/check.mjs +++ b/tools/test-isolation/check.mjs @@ -160,11 +160,31 @@ for (const [name, match, why] of [ // --- 3. Plant one fixture per entry, collect, assert ------------------------------------------------ -const fixtures = patterns.map((pattern) => ({ - pattern, - root: join(repoRoot, plantableDir(pattern), FIXTURE_DIR), - parent: join(repoRoot, plantableDir(pattern)), - parentExisted: existsSync(join(repoRoot, plantableDir(pattern))), +/** + * Locations probed ALWAYS, whatever `REPO_LOCAL_CHECKOUTS` currently says — the three that exist today. + * + * This is the answer to the hole the header describes: because the pattern-derived fixtures below come FROM + * the list, deleting `'**\/.claude\/**'` deletes its probe too, and the guard goes green while re-collecting a + * foreign tree. The primary structural assertion covers that on a developer's machine, but only if a second + * checkout is actually on disk — and CI checks out clean, so there it covers nothing at all. A planted fixture + * at a known location is a positive control that works in an empty tree. + * + * A hardcoded list is exactly what `workspacesWithTests` was rebuilt to avoid, and the difference matters: + * this one may only ever GROW stale in the safe direction. An entry removed from `REPO_LOCAL_CHECKOUTS` still + * gets probed here and fails loudly; a NEW location nobody added here is still probed by the derived fixtures. + * Neither omission can produce a silent pass. + */ +const ALWAYS_PROBED = ['.claude', '.worktrees', 'worktrees']; + +const fixtureDirs = [...new Set([...patterns.map(plantableDir), ...ALWAYS_PROBED])]; +const fixtures = fixtureDirs.map((dir) => ({ + // The pattern that should be excluding it, for the error message. An always-probed location with no entry + // gets the pattern it OUGHT to have, which is the fix to paste back into `vitest.config.ts`. + pattern: patterns.find((p) => plantableDir(p) === dir) ?? `${dir}/**`, + dir, + root: join(repoRoot, dir, FIXTURE_DIR), + parent: join(repoRoot, dir), + parentExisted: existsSync(join(repoRoot, dir)), })); /** @@ -353,13 +373,26 @@ if (foreign.size > 0) { const leaked = files.filter((f) => f.includes(FIXTURE_DIR)); if (leaked.length > 0) { - const missed = fixtures - .filter((fx) => leaked.some((f) => f.startsWith(`${plantableDir(fx.pattern)}/`))) - .map((fx) => fx.pattern); + // The two causes read very differently, so the message names which one this is. A LISTED location that + // leaked means the pattern does not do what it says; an always-probed one that leaked means the entry that + // used to cover it was removed — the failure the `ALWAYS_PROBED` control exists for, and the one the + // derived fixtures cannot see because deleting an entry deletes its probe. + const missed = fixtures.filter((fx) => leaked.some((f) => f.startsWith(`${fx.dir}/`))); + const listed = missed.filter((fx) => patterns.some((p) => plantableDir(p) === fx.dir)); + const unlisted = missed.filter((fx) => !patterns.some((p) => plantableDir(p) === fx.dir)); fail( `a repo-local checkout leaked into the root test run (${leaked.length} file(s)):\n` + leaked.map((f) => ` ${f}`).join('\n') + - `\n Not excluded despite being listed: ${missed.map((p) => JSON.stringify(p)).join(', ')}` + + (listed.length === 0 + ? '' + : `\n Listed in REPO_LOCAL_CHECKOUTS but NOT excluded: ` + + listed.map((fx) => JSON.stringify(fx.pattern)).join(', ')) + + (unlisted.length === 0 + ? '' + : `\n MISSING from REPO_LOCAL_CHECKOUTS (vitest.config.ts): ` + + unlisted.map((fx) => JSON.stringify(fx.dir)).join(', ') + + `\n These are probed unconditionally precisely because a derived probe disappears with the entry` + + `\n it came from — removing one would otherwise go green on a clean checkout.`) + `\n A root run must see this repo's tests and nothing else — a second checkout's suites are not ours` + `\n to run, and its sources are not ours to count against the coverage floor.`, ); @@ -378,5 +411,5 @@ if (silent.length > 0) { console.log( `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout ` + - `(${patterns.length} location(s) probed, ${expected.length} workspaces with tests on disk all collected).`, + `(${fixtures.length} location(s) probed, ${expected.length} workspaces with tests on disk all collected).`, );