feat(sql-fs): fence stale sandbox writers with durable epochs

[Hazzng]

Trek Voyage c62365e4ef1c

Ticket: 131
Base ref: main
Checkpoint commit: 5d3d3bd4ed1f11d475bdd61fcc18b28ddf4aaf14
Repair rounds: 1

Change summary

Prevent stale script transactions from mutating a sandbox after its lifecycle epoch has advanced, including after deletion and recreation with the same ID. The change persists sandbox epochs in Postgres, pins an epoch when a script transaction opens, and conditionally advances that epoch inside each advisory-locked composite mutation so stale writes fail and roll back.

What changed & why

• Add an idempotent Postgres migration that gives every live sandbox a non-null bigint version and retains per-ID lifecycle epochs in a tombstone table after deletion.
• Serialize sandbox creation and deletion with the existing advisory lock, allocate epoch zero for a new ID, advance retained epochs across recreation, and persist a deletion tombstone that fences old scopes.
• Read and pin the live sandbox epoch when a script transaction acquires its sandbox context and lock, then pass that expected epoch to composite file, directory, remove, and move mutations.
• Gate composite mutations on the sandbox epoch within their locked SQL CTEs, increment the live version on an accepted mutation, and update the transaction-local sandbox epoch before dependent inode and dirent changes run.
• Track epochs observed after writes and reject opening a later lazy script scope when its previously known epoch no longer matches the live sandbox.
• Add unit coverage for epoch propagation and fenced SQL, plus real-Postgres regressions for idempotent migration state, stale writes after delete-and-recreate, retained lifecycle fencing, and the non-superuser RLS path.

Walkthrough

On migration, each sandbox receives a live version and each sandbox ID gains durable epoch storage that survives deletion. Creating or recreating a sandbox takes the per-ID advisory lock and returns the allocated epoch; deleting it records a newer tombstone epoch. When SqlFs lazily opens a script transaction, it sets the RLS sandbox context, acquires the advisory lock, and pins the current epoch. Every composite mutation receives that pinned value: its SQL re-establishes the local context and lock, verifies the expected or transaction-local epoch against the live sandbox row, advances the version, and only then performs the filesystem change. If a stale transaction attempts its first mutation after a replacement has committed, the fenced CTE produces no eligible row, the mutation rejects, and the transaction rolls back without changing the replacement sandbox.

sequenceDiagram
    participant A as Stale scope
    participant DB as Postgres
    participant B as Live writer
    A->>DB: Read epoch zero
    B->>DB: Delete and recreate sandbox
    B->>DB: Write with live epoch
    DB-->>B: Commit and advance epoch
    A->>DB: Composite write with epoch zero
    DB->>DB: Lock and validate epoch
    DB-->>A: Reject with no rows
    A->>DB: Roll back transaction

Loading

Claimed assertions

• A1-migration-and-initial-epoch — The boot-discovered Postgres migration must be idempotent and add sandboxes.version as BIGINT NOT NULL DEFAULT 0, so a never-before-used sandbox ID begins at epoch 0. It must also install the durable lifecycle state required to retain an epoch after deletion, and it must apply successfully for configured tenant databases on both an empty database and a repeat run.
• A6-lifecycle-nonreuse — Deleting and recreating a sandbox with the same ID must not restore an epoch that can be accepted by any scope opened against the deleted sandbox. A new ID may begin at 0, but deletion must retain a monotonically advanced epoch or recreation must be rejected; this plan uses retained lifecycle epoch state so a replacement starts beyond the deleted row's reachable epoch.
• A2-transactional-composite-fence — A writable script scope must capture its expected durable epoch before its first mutation while the caller still holds the writer lease. mkdir, rm, write/append, and mv composite paths must conditionally advance sandboxes.version from that expected value inside their existing advisory-locked ctx CTE, and the scope must use the returned next epoch for later mutations in the same transaction.
• A4-rls-trusted-transaction-boundary — Epoch reads and updates must preserve the defense-in-depth boundary: database work remains under runTrustedDbAsync, and fence updates execute only after transaction-local app.sandbox_id configuration and alongside the transaction-scoped advisory lock. The change must not introduce a contextless pool query or break writes performed as the non-superuser application role.
• A5-dialect-contract-and-unit-regressions — The revised dialect API, SqlFs callers, and all affected mock dialects must remain type-aligned, and existing composite/script-transaction behavior outside a fence mismatch must retain its established semantics.
• A3-zombie-commit-rejected — When writer A has pinned an epoch, loses its lease before opening its first write transaction, and writer B commits a newer append, A's conditional epoch advance must affect zero rows and cause A's script transaction/commit to fail. B's committed file contents must remain visible, and the failure path must use existing cache recovery rather than publishing a stale cache version.

Passed assertions (evidence)

Assertion	What it verifies	Status	Evidence
A1-migration-and-initial-epoch	The boot-discovered Postgres migration must be idempotent and add sandboxes.version as BIGINT NOT NULL DEFAULT 0, so a never-before-used sandbox ID begins at epoch 0. It must also install the durable lifecycle state required to retain an epoch after deletion, and it must apply successfully for configured tenant databases on both an empty database and a repeat run.	passed	/workspace/artifacts/logs/cmd_6a2b2506-7a0c-44d5-a0e7-28f87967cd9d.out
A3-zombie-commit-rejected	When writer A has pinned an epoch, loses its lease before opening its first write transaction, and writer B commits a newer append, A's conditional epoch advance must affect zero rows and cause A's script transaction/commit to fail. B's committed file contents must remain visible, and the failure path must use existing cache recovery rather than publishing a stale cache version.	passed	/workspace/artifacts/logs/cmd_17e70f9f-17d8-4b48-a9a5-a6fc1b56ab61.out
A4-rls-trusted-transaction-boundary	Epoch reads and updates must preserve the defense-in-depth boundary: database work remains under runTrustedDbAsync, and fence updates execute only after transaction-local app.sandbox_id configuration and alongside the transaction-scoped advisory lock. The change must not introduce a contextless pool query or break writes performed as the non-superuser application role.	passed	/workspace/artifacts/logs/cmd_4c4b287c-1e2b-4df0-ab8b-87384c8403e2.out
A6-lifecycle-nonreuse	Deleting and recreating a sandbox with the same ID must not restore an epoch that can be accepted by any scope opened against the deleted sandbox. A new ID may begin at 0, but deletion must retain a monotonically advanced epoch or recreation must be rejected; this plan uses retained lifecycle epoch state so a replacement starts beyond the deleted row's reachable epoch.	passed	/workspace/artifacts/logs/cmd_1bc15bf7-2b1a-41d6-be7f-77f0045ffe40.out
ADV-447743e5	(no description)	passed	sandbox://cmd/e7a250b8-59bc-408d-9531-2dc418b6d9fa
ADV-d7e431e4	(no description)	passed	/workspace/artifacts/logs/cmd_b8af0250-4eb6-42e2-b12e-ebbd9f317a18.out

Files changed

• src/sql-fs/migrations/postgres/0007_fence_sandbox_epochs.sql
• src/api/tests/integration/migrations.integration.test.ts
• src/sql-fs/types.ts
• src/sql-fs/dialects/postgres.ts
• src/sql-fs/tests/sql-fs.composite.test.ts
• src/sql-fs/tests/sql-fs.script-tx.test.ts
• src/sql-fs/sql-fs.ts
• src/sql-fs/tests/integration/fencing.integration.test.ts

Checks executed

• bash /workspace/state/setup.sh
• pnpm exec vitest run src/sql-fs/tests/integration/fencing.integration.test.ts
• pnpm exec biome check src/sql-fs/tests/integration/fencing.integration.test.ts
• pnpm exec tsc --noEmit
• pnpm exec vitest run src/sql-fs/tests/integration/fencing.integration.test.ts -t "rejects a stale first mutation after a committed live append"
• git diff --check

Known limitations

none

Live E2E smoke test (advisory)

Verdict: error (advisory — never blocks the PR)

live smoke could not run (infrastructure): docker compose up failed (exit 1): Network trek-smoke-296b8815-e9d3-435c-90ec-8a1d5d0e563e_default Creating
Network trek-smoke-296b8815-e9d3-435c-90ec-8a1d5d0e563e_default Created
Volume trek-smoke-296b8815-e9d3-435c-90ec-8a1d5d0e563e_sqlfs-pgdata Creating
Volume trek-smoke-296b8815-e9d3-435c-90ec-8a1d5d0e563e_sqlfs-pgdata Created
Container trek-smoke-296b8815-e9d3-435c-90ec-8a1d5d0e563e-redis-1 Creating
Container trek-smoke-296b8815-e9d3-435c

Unresolved non-blocking findings

none

Rollback

To revert this change: git revert the commits on voyage/c62365e4ef1c, or
delete the branch — no merge has occurred (draft PR only).

---

Draft PR created by Trek. Do not merge — human review required.

---

Summary by cubic

Fences stale sandbox writers using durable epochs in Postgres so old script transactions cannot mutate a sandbox after delete/recreate or concurrent writes. Script scopes pin the current epoch, and all composite mutations verify and advance it atomically.

• New Features

  • Store the live epoch in sandboxes.version and retain per-ID tombstones in sandbox_epochs; serialize create/delete with advisory locks and advance epochs across recreation.
  • Script scopes pin the sandbox epoch; composite writes (mkdir, rm, writeFile, mv) validate the expected epoch inside advisory-locked CTEs, increment on success, and roll back on mismatch.
  • New/updated dialect APIs: getSandboxEpoch, createSandbox returns { epoch }, and deleteSandbox returns the advanced epoch; SqlFs tracks last-known epoch to detect stale scopes.

• Migration

  • Adds idempotent migration 0007_fence_sandbox_epochs.sql (sandboxes.version BIGINT NOT NULL DEFAULT 0; sandbox_epochs tombstone table). Run migrations to adopt.

^{Written for commit 5d3d3bd. Summary will update on new commits.}