Fix: make large memory backups reliable

[StarshipSuperjam]

A green mechanical check below shows this change conforms to the engine's rules — not that it is correct. What covers correctness is the behavioural steps in Review you can run yourself and the change's honest self-report — not a reading of the diff for defects; a green check is never a substitute for that. Your merge is the binding gate.

About those checks: only the one that runs when the change is proposed for merge can stop a risky merge — a check that ran while the change was still being written is early advice. The engine's checks are proven against deliberately broken examples they must catch — the custom ones each against their own, the standard kinds against one shared example — so a passing check can't be one that quietly did nothing; a few are openly-noted exceptions where that kind of proof doesn't apply. Either way that speaks to the check, not to whether this change is right. And a check that could not run leaves its area unverified.

Closes #822

Purpose

Make off-site Engine memory backup and recovery reliable after one GitHub blob request is no longer enough.

#822 records that the live memory corpus outgrew the single-blob Git Data request path, so backup could fail precisely when the accumulated project history became most valuable.

• Encode one deterministic, versioned gzip multipart snapshot with explicit byte, part-count, request-size, and path bounds.
• Upload and verify every object before moving the configured ref, retry the identical whole publication, and replace only the configured namespace.
• Restore legacy v1 and multipart v2 snapshots through strict pre-mutation validation, a crash-recovery journal, writer quarantine, and hard wall-clock deadlines.

Impact: Large supported memory stores can be backed up and restored without making a partial publication current or silently exposing a partially restored local store.

Scope

This PR supplies the reliable transport and recovery substrate for the first child of the unreleased three-PR memory-and-recall program.

• memory/snapshot_format.py: deterministic v2 compression, canonical multipart layout, integrity metadata, and bounded decode.
• memory/backup_vault.py: verified ref-last Git publication, whole-publication retry, namespace allow-listing, privacy checks, and deadline outcomes.
• memory/restore_vault.py: strict v1/v2 fetch and validation, transactional replacement, durable recovery journal, and idempotent crash reconciliation.
• memory/capture.py and boot.py: fail-closed writer quarantine during recovery, bounded SessionStart reconciliation, and operator-facing recovery state.
• Focused fixtures, release-gate coverage, and regenerated Engine projections for multipart limits, failures, deadlines, and recovery.

Change profile — the shape of this pull request at a glance:

• Size: 15 files changed, +3011 / −120 lines.
• Kinds of thing touched: 12 tools, 1 doc, 2 other files (not in the engine's map).
• Where: .engine/tools, .engine/docs, .engine/knowledge, .engine/provisioning.
• Shape: 26 commits on this branch — a standalone change unless a Part of #N line below says it is one slice of a larger effort.

This is a description, not a gate — it never blocks a merge. It is here so you can weigh the change by what it touches, not by its line count.

Impact: The local memory record format stays unchanged; the remote backup representation and its recovery safety become versioned and bounded.

Behaviors

The capabilities this change delivers, each with the test that exercises it.

• A supported memory store produces deterministic multipart snapshot bytes within the 512 MiB uncompressed, 128 MiB compressed, 32-part, and 8 MiB serialized-request envelope. — .engine/tools/memory/test_snapshot_format.py (Boundary and one-unit-over fixtures reject oversized input before publication.)
• Publication uploads and verifies all snapshot objects before moving the ref, retries the same whole publication, and leaves foreign shared-vault paths untouched. — .engine/tools/memory/test_backup_vault.py (Git-step, timeout, tip-race, and namespace fixtures keep the prior head authoritative on incomplete attempts.)
• Legacy v1 and multipart v2 snapshots restore byte-exactly, while malformed, excessive, corrupt, or traversing inputs refuse before local mutation. — .engine/tools/memory/test_restore_vault.py (Immutable v1 fixtures, v2 total-loss round trips, gzip-bomb bounds, and unchanged-destination assertions cover both readers.)
• SessionStart backup and recovery stay within a 10-second envelope and foreground operations within 180 seconds, including blocked transport and recovery cleanup. — .engine/tools/memory/test_backup_vault.py, .engine/tools/memory/test_restore_vault.py, .engine/tools/test_boot.py (Subprocess and signal-aware deadline fixtures prove wall-clock refusal rather than cooperative timeout only.)
• An interrupted multi-file restore quarantines ordinary writers and can reconcile forward or roll back idempotently without exposing a mixed store. — .engine/tools/memory/test_restore_vault.py, .engine/tools/memory/test_capture.py (Crash-point, repeated-recovery, dangling-marker, post-lock quarantine, prior-index, and durable-journal-retirement fixtures cover the recovery protocol.)
• Operators are told when backup is incomplete, why recovery blocks writes, that the vault must remain private, and that replaced Git objects may remain recoverable for an unknown period. — .engine/tools/memory/test_backup_vault.py, .engine/tools/memory/test_restore_vault.py, .engine/tools/test_boot.py (Consent, warning-priority, and typed-outcome assertions keep the safety consequences visible.)

Out of scope

The change stays at the transport and recovery boundary chosen for #822 and does not pre-implement the two successor PRs.

• Encryption or vault-topology redesign — The operator chose the existing consented private GitHub vault and no Engine-provided encryption for this release; the consequence is disclosed before consent.
• Local memory-record migration — Multipart is a wire-backup representation; changing the live record belongs to the successor evidence cutover.
• Remote migration tags or a reader-only compatibility release — All three child PRs land before any supported deployment, so an intermediate deployed state is outside the program contract.
• Secure purge of superseded Git objects and history — Provider retention and garbage collection are not controllable by this client; the residual is disclosed instead of falsely claiming erasure.
• Federation, part reuse, adaptive probing, or backup-cadence redesign — Those mechanisms are unnecessary to make the observed corpus reliable inside the settled support envelope.

Impact: These boundaries keep the minor capability coherent while leaving evidence conversion and mechanized recall to their already-carried successor obligations.

Risk

4 risks, ranked, each with the bound that contains it.

• A crash during multi-file restore could expose or later recover a mixed ledger, metadata, and index set (the most safety-sensitive edit). A durable phase journal, per-file identities, same-filesystem staging, writer quarantine, atomic moves, idempotent reconciliation, and crash-point tests keep the store closed until one complete state wins.
• Hostile compressed input could amplify memory, CPU, file count, or path traversal during restore. Declared and observed compressed/decompressed byte limits, canonical part and file counts, path normalization, and bounded streaming validation run before destination mutation.
• The private GitHub vault still contains sensitive memory, and superseded Git objects may persist outside the live tree. The publisher rechecks repository privacy before every push, refuses public repositories, and discloses both the lack of extra encryption and indeterminate history retention.
• A slow network or stalled subprocess could exceed an interactive hook budget. Hard SessionStart and foreground deadlines produce typed incomplete outcomes while leaving the prior remote ref and local memory authoritative.
• Guardrail disclosure. Guarded files in the final branch divergence came from reconciling the already-reviewed current main branch; this Fix: replace oversized memory-vault blobs with compressed multipart snapshots #822 implementation does not weaken their enforcement, and the Engine guard checks passed.
• Accepted residual (USE-822-001, operator decision 2026-08-27). The operator accepted a private, unencrypted GitHub backup whose replaced objects may remain in provider history; this PR makes that residual explicit before consent rather than claiming stronger erasure or encryption.

Impact: The highest-risk mutation path is fail-closed and recoverable; remaining confidentiality limits are explicit operator choices, not hidden guarantees.

Validation

The exact reconciled head passed the affected Engine inventory and focused self-test, plus the multipart boundary suite and destructive-loss restore demonstration.

• Engine CI — the full mechanical check suite — passed at a792c1491307 — candidate evidence, not merge evidence (log sha256:e27c77d6929bcb153cb520947f867a2c269f7b61c2b26d1e4343322d3315afaf)
• Engine self-tests — affected selection against the merge base, with a run record — passed at a792c1491307, scope focused (tree 23cef14b9a8c) — candidate evidence, not merge evidence (log sha256:2a2214a0f05b55b0dcd858c177b9e8d95c7574afeb7cb1750f895bd08ae27701)
• Merge proof — not yet imported: validate final import must verify the live engine-ci run for the submitted head before ready.
• Caveat: The first live engine-ci run on the reconciled head stopped at the deliberately provisional PR body contract; all non-contract GitHub checks passed, and applying this complete contract triggers the required full rerun.
• Caveat: The live Engine memory and knowledge-graph MCP helpers were not callable in this Codex session, so grounding and impact tracing used the committed Engine status, saved memory output, wiring map, and coordinator state.
• A live helper was unavailable, so this change was authored on the committed-file fallback: mcp__engine_memory.health (The exact deferred helper was not discoverable or callable in this session; the saved-file fallback may be out of date, so trust this project's .codex/config.toml and restart Codex to restore the live helper.); mcp__engine_knowledge_graph.health (The exact deferred helper was not discoverable or callable in this session; the saved-file fallback may be out of date, so trust this project's .codex/config.toml and restart Codex to restore the live helper.). That area was not verified against live state.
• Regeneration updated 3 of the engine's generated surfaces (.engine/docs/ci-assurance.md, .engine/knowledge/graph.json, .engine/provisioning/module-surfaces.json) from the final tree — generated paths only.

Impact: Mechanical evidence is bound to the final tree; the PR-body-only live failure is visible and is being replaced by a complete, machine-checked contract before submission.

Review

Cold review repeatedly challenged the transport and recovery design; every accepted finding was repaired, and the final operator-bounded pass was scoped to technical integrity and security governance.

• Coverage. standard depth. Plan review ran before any code, on the plan side: architecture, feasibility, product-intent, risk-governance read the sealed plan; the deliverable review (divergence-hunter, security-governance, spec-conformance, technical-integrity, usability) ran after. What effort those reviewers ran at is self-reported by the spawning session and by the reviewers themselves; nothing in this engine verifies it.
• Code execution. With this PR, a reviewer ran the change's code in a throwaway copy to judge it — it never touched your project.
• Initial full review: found response-encoding verification, wall-clock deadlines, strict v1 handling, boundary fixtures, and operator wording defects; each accepted issue was repaired and regression-locked.
• First repair review: found that final restore replacement was not fully transactional and that nominal deadlines were still cooperative; rollback and hard wall-clock enforcement replaced those seams.
• Second repair review: found equal-lineage index exposure, rollback-to-absence gaps, recoverability loss, and signal-state hazards; the design moved to durable journaling, quarantine, and composable recovery state.
• Third repair review: found mutation in read-only observation, non-composable signal handling, oversized startup work, and marker-cleanup gaps; reconciliation was confined to the write-capable SessionStart path and atomic recovery was tightened.
• Fourth repair review: found recovery outside deadlines, cleanup races, incomplete missing-backup identity checks, and a trimmable warning; bounded locks, identity proofs, and never-shed operator messaging closed them.
• Final scoped review: technical-integrity and security-governance reviewers found retry idempotence for the prior index, journal-unlink durability, and a post-lock quarantine race; all three were repaired and focused plus candidate verification passed without automatic review recursion.
• Target reconciliation: current main produced only generated-artifact conflicts; the generated surfaces were rebuilt and the reconciled exact head received focused and candidate verification, with no new risk class requiring another cold panel.
• Plan findings you must still weigh.
  • FEAS-1 (feasibility, serious, rejected). The plan requires reader and writer activation with no intermediate deployment, but it specifies no rollout control that prevents an upgraded writer from publishing v2 while an older Claude/Codex installation or hook can still read only v1. Merging the code together does not make activation atomic, so the release needs an enforceable upgrade/readiness gate before the first v2 publication.
  • FEAS-2 (feasibility, serious, partially-accepted). Multipart backup multiplies GitHub API requests, yet retry and adaptive limit handling are explicitly out of scope and the request-count risk has no operating remedy. A transient API failure or rate limit safely leaves the ref unchanged, but it also leaves the system without a fresh backup - the required commit point for Plan 2 - with no specified backoff, resume, or operator recovery procedure.
  • PRODUCT-2 (product-intent, nit, rejected). The no-spec selection is asserted rather than demonstrated: the packet calls issue Fix: replace oversized memory-vault blobs with compressed multipart snapshots #822 authoritative but does not say whether semantically affected settled backup or recovery product documents were checked and found absent. The criterion mapping therefore proves coverage only of the selected issue, code, tests, and operator decisions - not that all settled backup promises were considered.
  • RISK-1 (risk-governance, serious, partially-accepted). The plan publishes memory backups as unencrypted Git objects without defining the repository-access threat model, secret-exclusion policy, or retention guarantee. Gzip provides no confidentiality, and Git history can preserve sensitive material after the live backup changes, so anyone who gains repository access may gain historical private memory as well.
  • RISK-2 (risk-governance, serious, partially-accepted). Ref-last publication prevents readers from selecting an incomplete backup, but failed attempts can still leave unreferenced commits and blobs containing private data. The plan acknowledges cleanup risk without requiring a retention or orphan-cleanup procedure, leaving abandoned backup contents and storage growth ungoverned.
• The plan review in full. 7 finding(s), of which 5 render above. The other 2 were settled before the build began (2 serious) and are not restated here. The complete text stays in this workstation's plan library, which is local and never published — it is not readable from a clone of this repository.
• Obligations carried from the predecessor plan.
  • OB-RELEASE-BARRIER — carried. Do not publish or deploy an intermediate child; release only after all three PRs complete, integrate, and qualify together. Plan 1 supplies only the transport substrate.
  • OB-MINOR-RELEASE — carried. Release the completed program under the operator-selected next minor version; implementation scope does not reclassify it. This explicit operator decision binds every child.
  • OB-EVIDENCE-CUTOVER — carried. Convert canonical live memory with a private local recovery archive, replace the live vault namespace from the verified result, delete the archive at commit, and retain no live legacy archive or remote migration tag. Plan 2 owns the conversion and cutover.
  • OB-MECHANIZED-RECALL — carried. Build the shared Claude/Codex saved-memory reflex, bounded recall helpers, and content-free evaluation telemetry over the evidence-only substrate. Plan 3 owns recall behavior.
• Operator decisions at this plan's consent gates, in their own words. Recorded by the session, not independently proven.
  • approve (2026-08-27T19:13:50Z) — “Looks good. Let's run the design audit now at standard depth.”
  • findings-presented (2026-08-27T20:29:54Z) — “Agreed”
  • seal (2026-08-27T20:52:18Z) — “Seal them.”
  • bind (2026-08-27T20:55:48Z) — “$engine-start”
• The rest of the review record. A further 86 finding(s) were raised and settled without leaving anything outstanding — 19 in deliverable (14 serious, 3 blocking, 2 nit); 67 in repair (39 blocking, 28 serious). Each is recorded in full in this Build's own evidence, which travels with the handoff.
• Reviewer disagreement R4-DIV-001: Recovery is back inside both promised wall-clock limits; unfinished work stays quarantined for the next bounded retry.
• Reviewer disagreement R4-DIV-002: Startup cleanup now uses the same lock as foreground restore and cannot delete an active recovery directory.
• Reviewer disagreement R4-SEC-001: Active recovery material is lock-protected, and quarantine clears only after every prior file is proved or atomically restored.
• Reviewer disagreement R4-SPEC-001: Interrupted-restore reconciliation now honors the same 10-second or 180-second envelope as its caller.
• Reviewer disagreement R4-SPEC-002: A live file is treated as recovered only when it matches the exact prior file recorded before publication.
• Reviewer disagreement R4-TECH-001: Orphan cleanup is now serialized with restore publication and revalidates the marker before every deletion.
• Reviewer disagreement R5-TECH-001: Fixed recovery retry so it preserves the only verified prior index.
• Reviewer disagreement R5-TECH-002: Fixed journal retirement so a reported success cannot later resurrect a rollback-capable marker.
• Reviewer disagreement R5-SEC-001: Fixed the duplicate security finding for verified prior-index preservation.
• Reviewer disagreement R5-SEC-002: Fixed the writer-lock race so capture cannot enter while restore quarantine is active.
• Escalation recorded. repair round 2 of 7 proceeded past the escalation point on recorded operator guidance: Proceed with the third full repair round.
• Escalation recorded. repair round 3 of 7 proceeded past the escalation point on recorded operator guidance: Proceed with the third full repair round.
• Escalation recorded. repair round 4 of 7 proceeded past the escalation point on recorded operator guidance: Yes
• Escalation recorded. repair round 5 of 7 proceeded past the escalation point on recorded operator guidance: Operator agreed to one further scoped review after challenging the self-perpetuating full-panel criterion; run technical-integrity and security-governance only, then adjudicate findings without automatic review recursion.
• Escalation recorded. repair round 6 of 7 proceeded past the escalation point on recorded operator guidance: Operator authorized exactly one further scoped review limited to technical-integrity and security-governance, with findings adjudicated and repaired without automatic review recursion.
• Escalation recorded. repair round 7 of 7 proceeded past the escalation point on recorded operator guidance: Operator authorized exactly one further scoped recovery review and explicitly rejected automatic review recursion; target reconciliation should receive direct verification unless it creates a genuinely new risk class.
• Repair rounds. 7 round(s) ran after the deliverable review, 5 of which dispatched a review panel. A panel round is what spends the budget; the budget is 3 panel rounds, and 6 rounds of any kind is the absolute ceiling. Passing either stop needs recorded operator guidance, disclosed above. Whether the repairs are widening is judged on code and guarded surface only; regenerated and documentation churn is listed below but never compared. A file counts as guarded if it was protected at ANY point since the deliverable review, so a guard retired mid-build still reads as guarded rather than quietly becoming ordinary work.
  • round 1: counted, full re-review across 5 lenses; the fix moved 7 authored file(s), 1 derived file(s), 526 lines
    • authored: .engine/tools/memory/backup_vault.py, .engine/tools/memory/restore_vault.py, .engine/tools/memory/snapshot_format.py, .engine/tools/memory/test_backup_vault.py and 3 more
    • derived: .engine/knowledge/graph.json
  • round 2: counted, full re-review across 5 lenses; the fix moved 4 authored file(s), 1 derived file(s), 312 lines
    • authored: .engine/tools/memory/backup_vault.py, .engine/tools/memory/restore_vault.py, .engine/tools/memory/test_backup_vault.py, .engine/tools/memory/test_restore_vault.py
    • derived: .engine/knowledge/graph.json
  • round 3: counted, full re-review across 5 lenses; the fix moved 8 authored file(s), 1 derived file(s), 615 lines
    • authored: .engine/tools/boot.py, .engine/tools/memory/backup_vault.py, .engine/tools/memory/capture.py, .engine/tools/memory/ledger.py and 4 more
    • derived: .engine/knowledge/graph.json
  • round 4: counted, full re-review across 5 lenses; the fix moved 6 authored file(s), 1 derived file(s), 564 lines
    • authored: .engine/tools/boot.py, .engine/tools/memory/backup_vault.py, .engine/tools/memory/restore_vault.py, .engine/tools/memory/test_backup_vault.py and 2 more
    • derived: .engine/knowledge/graph.json
  • round 5: counted, scoped re-review across 2 lenses, lenses named deliberately; the fix moved 4 authored file(s), 1 derived file(s), 453 lines
    • authored: .engine/tools/boot.py, .engine/tools/memory/restore_vault.py, .engine/tools/memory/test_restore_vault.py, .engine/tools/test_boot.py
    • derived: .engine/knowledge/graph.json
  • round 6: uncounted, judged no re-review needed; the fix moved 4 authored file(s), 1 derived file(s), 290 lines
    • authored: .engine/tools/memory/capture.py, .engine/tools/memory/restore_vault.py, .engine/tools/memory/test_capture.py, .engine/tools/memory/test_restore_vault.py
    • derived: .engine/knowledge/graph.json
  • round 7: uncounted, judged no re-review needed; the fix moved 46 authored file(s), 5 derived file(s), 6 guarded file(s), 5924 lines
    • guarded: .engine/policies/provider-exceptions.json, .engine/tools/close.py, .engine/tools/hooks.py, .engine/tools/session_economy.py and 2 more
    • authored: .claude/skills/engine-start/SKILL.md, .claude/skills/engine-upgrade/SKILL.md, .engine/build-orchestration-obligations.json, .engine/conduct/defaults.md and 42 more
    • derived: .agents/skills/engine-start/SKILL.md, .agents/skills/engine-upgrade/SKILL.md, .engine/docs/ci-assurance.md, .engine/knowledge/graph.json and 1 more
• Reviewed vs submitted. reviewed 50d2e6666081, submitted a792c1491307 — 61 files changed, 5629 insertions(+), 585 deletions(-); no re-review was judged necessary

Spec-derived acceptance steps

The recall proposal does not govern backup; retired eADRs and stale bases are not authority.

Impact: The review history materially strengthened crash recovery, but the stop condition is now evidence-based: the scoped findings are fixed, their exact seams are regression-tested, and no further panel is being used merely because reviewers can always find more.

Demonstration

A step you can run yourself that drives the changed surface and can genuinely fail.

• Run: uv run --directory .engine --frozen -- python tools/memory/test_snapshot_format.py && uv run --directory .engine --frozen -- python tools/memory/restore_vault.py demo
• It PASSES when: The codec reports 10 passing tests, then the practice restore exits 0 with byte-identical memory, one searchable hit before and after, explicit older-backup refusal, and no mutation on unreachable backup.
• It FAILS when: Either command exits nonzero, any boundary fixture fails, restored bytes differ, search hits disappear, an older backup applies silently, or an unreachable backup changes local memory.

Exact-head run on 2026-08-27: `Ran 10 tests ... OK`; restore reported byte-identical `True`, searchable hits `before=1, after=1`, and all eight practice sections passed.

Impact: Run it to watch the change work — an unrun step is a promise, not proof.

Files of interest

The paths that most determine this change.

• .engine/tools/memory/restore_vault.py — Owns strict v1/v2 restore, pre-mutation validation, transactional replacement, journal phases, and crash reconciliation.
• .engine/tools/memory/snapshot_format.py — Defines the deterministic multipart wire format, integrity manifest, and resource envelope.
• .engine/tools/memory/backup_vault.py — Publishes verified snapshot objects ref-last while preserving the configured namespace boundary and prior head.
• .engine/tools/memory/capture.py — Prevents ordinary memory writers from crossing an active or newly appeared restore quarantine.
• .engine/tools/test_release_gate.py — Locks the release-level multipart recovery and compatibility assertions.

Impact: These files define the wire contract, publication commit point, local recovery transaction, writer exclusion, and release acceptance boundary.

AI involvement

How this change was produced and who decided what.

• Codex (gpt-5.6-sol) — Engine Project Manager, implementation orchestrator, repair adjudicator, test runner, and PR evidence author.
• Engine cold reviewer agents (gpt-5.6-sol) — Independent spec-conformance, divergence, security, technical-integrity, and usability challenge across full rounds, followed by one operator-bounded two-lens scoped review.
• Engine Build coordinator and validation suites (deterministic local tooling) — Bound commits, findings, review rounds, change classification, generated artifacts, test inventory, and PR contract to observed repository state.
• Operator decision, 2026-08-27: The completed three-PR program will be released under the next public minor version regardless of the internal implementation shape.
• Operator decision, 2026-08-27: No supported deployment occurs between child PRs; live-memory conversion will use a private local recovery archive, a clean vault namespace, a new backup, and deletion of the temporary archive after verification.
• Operator decision, 2026-08-27: Interrupted restore recovery fails closed for memory writers until reconciliation completes, with the operator told plainly what is paused and how to retry.
• Operator decision, 2026-08-27: After challenging self-perpetuating full panels, the operator authorized exactly one final scoped technical-integrity and security-governance review and no automatic review recursion.
• The operator fixed the release version, deployment boundary, migration posture, fail-closed policy, and review budget; Codex designed and implemented the mechanism, adjudicated findings, and chose tests, while deterministic Engine tooling measured and bound the evidence.

Impact: AI judgment is load-bearing in the crash-recovery protocol and finding adjudication, and is bounded by cold adversarial review, explicit operator calls, exact-head regression suites, and an operator-runnable destructive-loss demonstration.

Release-Impact: minor — backward-compatible new capability or an explicit deprecation