perf(storage): reuse unchanged storage witness proofs across warm commands

[roodboi]

HACK-1123: fewer storage witness proofs per warm command. Draft. Hosted docker-e2e timings on this PR compared with next are the measurement. This PR does not update Linear.

What changes

One directory-xattr storage proof is three fresh helper containers: root, verify and root-after. A warm up or restart that is image-only, unrouted and has no hooks now runs 4 proofs instead of 6.

proof	before	after
P1 opening (command admission)	full	full
P2 prepublication	full	full
P3 effect-entry (after pending is saved)	full	full
P4 enrollment tail	full	reuses P3 when no new volume (step A)
pre-spawn guard	reuses P4	reuses P4 (now the P3 record)
P5 first finalization (after the workload)	full	full
P6 second finalization	full	removed when there is no route, after hook, or new or saved file material (step C)

Step A (src/lib/native-compose-command-storage.ts): enroll() calls verify(generation, { adjacentProof: true }) when newVolumes.length === 0. It uses the existing per-lease takeLeaseProof record. That record is keyed by the lease's authority object, the engine ID, the generationId plus inputRevision, and the exact witness reference. It is reused only when the opening material binding is byte-identical to the binding that opened and closed the recorded proof. The material binding covers the receipt dev/ino/hash, the lease token, the checkout anchor and the generation. Every host check still runs (anchors, expectation, completion and binding).

Step C (src/lib/native-compose-command.ts): retireFiles is passed only when files !== null or a saved generation carries x-hack-native-files (new helper nativeComposeSavedFilesPresent). Without either, retirement only read documents. Leaving it out lets preparedEffectOwnership leave beforeComplete undefined when there are no routes and no after hooks. finalizeEffect then takes its existing single-pass path, the same one routed run already uses. With files === null, assertFiles and assertFilesAfterCompletion behave the same, so the single pass loses no check.

Observability: each performed verification proof now emits a passive storage.proof phase-trace span (HACK_NATIVE_COMPOSE_PHASE_TRACE=1). A reused proof emits none.

Not in this PR (each needs an explicit decision, see "Remaining"):

• step B: prepublication reusing the opening proof;
• dropping the closing proof after saved exec.

Safety argument

• Reuse stays inside one command and one mutation lease. No receipt is persisted and there is no time window. Every command, including the first one after an engine restart, opens with a full 3-helper proof. A persisted cross-command receipt was rejected for four reasons:
  • the host cannot see the volume root without a helper;
  • root identity (device, inode, uid, gid) can repeat after the volume is recreated;
  • any workload container can remove the user.* xattr;
  • Docker exposes no boot ID, and the engine ID survives restarts.
• Callers claim adjacency from code structure, never from timing.
  • Step A: newVolumes.length === 0. Between P3 and P4, the warm path runs only the pure selection compare, the no-op assertColdAbsent and the enroll loop over zero volumes. On up/restart, prepareEffectBoundary does nothing after the effect-entry check, because downHooks exists only on down.
  • Step C: beforeComplete is absent only when it would have been a no-op read.
• Receipt writes force a full proof. Pending, births, completion and before hooks all change the receipt hash/inode, so the binding check catches them.
• No reuse across pending publication or a workload. P2 and P3 stay separate, and P5 stays fresh.
• Refusal classes are unchanged. P3/P4 and P5/P6 already sit in the UNCERTAIN class, so the refusal type, the stage and the retained pending state are the same.
• Failure clears reuse. takeLeaseProof deletes the record before any check.

Invalidation table

input change between recorded proof and reuse	result
receipt write (pending, storage birth, completion, hooks)	binding mismatch → full proof
different generation or inputRevision	different key → full proof
new or cold volume	step A not claimed → seed plus full proof
changed witness reference (marker, completion, artifact)	different key → full proof
changed engine ID	different key → full proof, then refused by matchesBinding
next command / new lease / engine restart	new authority object → no record → full proof
refused or failed verification	record already deleted → next verification is full
route, after hook, or new/saved file material	beforeComplete kept → second finalization proof kept

Tests (all with injected stubs or the synthetic docker; no real socket or HOME state)

• tests/native-compose-storage-witness-xattr-owner.test.ts:
  • The warm no-op up/restart test now expects the enrollment tail to reuse the effect-entry proof (7 → 6 verify helpers).
  • New: "warm enrollment proves in full when the material binding moved since the effect-entry proof". This stands in for a storage birth between P3 and P4.
  • New: "an adjacent verification never reuses a proof across a receipt publication or a later lease". It covers:
    • a positive control;
    • the completion receipt publication;
    • a new lease, as after an engine restart;
    • a different engine ID, which refuses.
  • Mutation check: when the reuse condition ignored the binding (recorded !== undefined), both new tests failed. They pass again with the condition restored.
  • The existing xattr drift tests at prepublication and effect-entry still refuse before the workload.
• tests/native-compose-storage-activation-command.test.ts (source CLI, synthetic engine):
  • cold up: root, seed, 3 proofs, workload, 1 proof (was 2 after the workload);
  • two consecutive warm ups: 3 proofs, workload, 1 proof each (was 4 / 2);
  • exec: unchanged, 1 / 1;
  • restart: 3 / 1 (was 4 / 2).
  • New negative control: a warm up with an after hook keeps all 8 post-workload proofs. Only the enrollment tail drops (6 → 5 before the workload). The baseline numbers were measured by running the same test against next sources.
• tests/native-compose-storage-fence-cost.test.ts: up 6× → 4× PROOF_FENCES and PROOF_OBSERVATIONS. A reused proof costs no engine fence. exec is unchanged.
• tests/native-compose-phase-trace.test.ts: a one-off run with no file material now has a single finalization pass.
• Hosted: tests/e2e/scenarios/native-config-compose.ts (its existing docker-e2e shard registration) runs two consecutive unchanged warm ups with the phase trace. It asserts exactly 4 storage.proof spans and 1 finalize.witnesses span each, and logs the per-up proof time and total time.

Docs: docs/reference/native-compose-storage-witness.md (reuse rules and counts) and docs/reference/native-compose-phase-trace.md (storage.proof).

Validation (local, macOS, source CLI; nothing ran against a real engine)

• bun run --cwd packages/cli typecheck: clean. bun run --cwd packages/cli check and bun run check: 0 errors and 34 warnings, the same count as next. The complexity warning in native-compose-command.ts exists on next too.
• Focused tests:
  • tests/native-compose-storage-witness-xattr-owner.test.ts: 106 pass.
  • tests/native-compose-storage-activation-command.test.ts: the proof-count case and the new after-hook case pass.
  • tests/native-compose-storage-fence-cost.test.ts: 4 pass.
  • tests/native-compose-phase-trace.test.ts: 12 pass.
• Broad run: bun test tests/native-compose- tests/native-authored tests/integration-ownership.test.ts, 131 files.
  • First pass: 2601 pass, 40 skip, 59 fail. All 59 failures were in 11 file-material, VM-file and adoption files. Those files need the native config compiler, which this fresh worktree had not built. Before the compiler was built, the same 10 files gave the identical failure set (51) on next sources and on this branch.
  • After mise exec -- bun run build:config-compiler, those 11 files pass: 293 pass, 1 skip, 0 fail. They include the saved and new file-material retirement paths that still keep the second finalization pass.
• Side effects:
  • Synthetic docker with an isolated DOCKER_CONFIG throughout. No hack command ran against a real engine.
  • No new bun-*.ips crash reports. The only ones present date from 04:10, before this work.
• Hosted docker-e2e has not been observed yet. It is required runtime evidence.

Review follow-up (polish commit)

• New regression test in tests/native-compose-file-command.test.ts: a first up with new file material and no saved file material. It checks that finalization still runs the completion step and then a second pass (finalize.owned and finalize.witnesses, read from the phase trace). It also checks that the exact file mounts are observed 4 times after the engine starts; the post-completion files.assertReady provides the last of these.
• Mutation check: if the step C condition is only nativeComposeSavedFilesPresent(previous) (without files !== null), the new test fails. The completion step and the whole second finalize pass disappear, and the mount observations drop from 4 to 2. It passes with the PR code. This fixture has no storage, so the second finalize.witnesses pass is checked as a phase, not as a carrier proof.
• Test harnesses no longer pass the real HOME to the child CLI. HOME is now a private user-home inside the fixture root, in the storage activation test and in the native-compose-command and native-compose-files helpers. HACK_HOME was already inside the fixture. tests/native-compose-renderer-config.test.ts still passes HOME to a real docker compose config client invocation. That is a different harness and is not changed here.
• Checks:
  • The 14 native-compose file and VM-file test files: 164 pass, 15 env-gated skips, 0 fail.
  • The 16 native-compose storage test files: 453 of 455 passed in one combined run. The load average was about 20 because other jobs were running on the machine. The 2 failures (one in storage-activation-command, one in storage-witness-xattr-owner) were time-sensitive cases. Both files passed alone: 8/8 and 106/106.
  • The other users of the helpers (after-command, down-command, phase-trace and warm-up-command): 46/46 pass.
  • packages/cli typecheck and check both pass. Check reports existing warnings only.
  • No new bun-*.ips crash reports.

Expected saving (estimate, not measured here)

Basis: hosted traced run 38225752202 (#298). A warm up took about 22.0 s, with 13.0 s of witness time over 6 proofs, so about 2.2 s per proof. Proof counts are exact. Wall times vary about ±20% with the engine.

• Warm unrouted up without hooks: −2 proofs ≈ −4.3 s (about 22.0 → 17.7 s).
• Routed or hooked up: step A only, ≈ −2.2 s.
• restart in the fixture shape: −2 proofs.
• native-config-compose scenario: about 11 warm ups × 4.3 s ≈ 47 s of 535.7 s traced. The two traced warm ups this PR adds cost about 35 s, so the net scenario wall time stays about the same. The per-up numbers in the log are the comparison.

Remaining (needs decisions; not in this PR)

• Step B: prepublication reusing the opening proof saves about 2.2 s more per warm up. Trade-off: a volume replaced between P1 and P2 would be caught at P3 as UNCERTAIN (after pending) instead of as a clean prepublication refusal.
• Dropping the closing proof after saved exec: saves about 2.65 s of 5.7 s per storage exec, about 58 s per compose scenario. A replacement during the exec would then show up one command later, not in this exec's result.
• Follow-ups: merge the root and verify helpers into one run (needs a new helper image and catalog release); build the carrier factory lazily for reused proofs (about 2 CLI probes each).
• Hosted docker-e2e on this PR is the runtime evidence. It is not yet observed.

🤖 Generated with Claude Code