Skip to content

fix(cache): break the session-cache bootstrap trap - #1519

Merged
marcusrbrown merged 15 commits into
mainfrom
fix/session-cache-bootstrap-trap
Sep 2, 2026
Merged

fix(cache): break the session-cache bootstrap trap#1519
marcusrbrown merged 15 commits into
mainfrom
fix/session-cache-bootstrap-trap

Conversation

@marcusrbrown

@marcusrbrown marcusrbrown commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Closes #1407.

A run whose OpenCode server failed to start persisted session state that made the next run fail the same way. Nothing in the codebase could recover from it — both reporters escaped only by deleting every opencode-* cache entry by hand.

What was actually wrong

Nothing ever checkpointed the database — not just failed runs. save.ts already said so: server.close() sends proc.kill() without awaiting a checkpoint, so a valid session can sit almost entirely in opencode.db-wal. shutdown() is synchronous and nothing awaits the child's exit, so every cached database was transported with a hot write-ahead log. The comment twenty lines away in cleanup.ts claimed the opposite — that a clean shutdown checkpoints — and it was wrong.

That reframed the fix. "Only persist state that was cleanly checkpointed" would have blocked every save, because nothing was. The checkpoint had to be added, not gated on.

Declining to save cannot escape a poisoned cache. This was my own first draft's central error. Save keys are unique per run and restore keys are prefixes returning the most recent match, so a run that declines to save writes no entry and the next run restores the same bad one. Declining forever is a stable state, not a recovery — which is why the reporters needed a manual purge.

The triage's control flow was wrong too. It concluded runCleanup never executes on bootstrap failure, so the post hook was the culprit. run.ts runs cleanup in a finally, so it does execute with a null server handle — skipping prune and shutdown, then saving anyway. A fix aimed only at the post hook would have missed the primary path.

The fix

The repair happens on restore, between restoreCache and bootstrapOpenCodeServer. No timer spans those calls — the budget covers createOpencode alone — so the recovery cost moves out of the window that was failing instead of being paid inside it. A stuck repository heals itself on its first run, with its session history intact.

Wiping the database would also break the loop, and was rejected: it discards history on every transient bootstrap failure, which is the second reporter's case exactly.

The save side checkpoints inside saveCache, which both the cleanup phase and the post hook already call and which also owns the object-store sync — so every capture path inherits it structurally, rather than through a flag two callers must agree to read. A log that cannot be merged declines the save and says why in the job summary.

Alongside: -shm is a machine-local wal-index SQLite never syncs, so it is no longer captured and any copy carried in by an older cache is deleted on restore. Declaring storage corrupt now actually clears the database, which lives beside the storage directory rather than inside it. The bootstrap budget is configurable and reported, bounded at two minutes. And session-retention finally reaches pruning — it was parsed, validated, logged, and never read.

The object-store restore path also gained the corruption and version checks it had been returning before ever running — but that is a smaller gain than it sounds, and the first draft of this description overstated it. checkStorageCorruption only asks whether the storage path is a readable directory, which the mkdir two lines above it guarantees, and checkStorageVersion treats an absent .version as compatible, which it will be for an object-store restore. Both therefore pass for a corrupt opencode.db. The checkpoint attempt is the only thing on that path that actually inspects the database.

Measurements

Opening a database with a hot 185 MB write-ahead log: 2416.8 ms, against 60.2 ms once checkpointed. Decisive at the first reporter's scale.

It does not explain the second. That repo's entire cache was 21 MB, where recovery is a few hundred milliseconds — not enough to turn 1.88 s of headroom into a timeout. The two incidents have different dominant causes, recovery cost and runner variance, and both are addressed. Neither fix alone covers both reports.

node:sqlite is available unflagged on Node 24.20.0, so the checkpoint adds no dependency. Its pragma returned {busy: 0, log: 0, checkpointed: 0} while truncating a 1.1 MB log to zero, so success is judged by the log's size on disk rather than that count.

Worth reviewing

  • Repair on restore rather than only declining on save. The decline alone is a stable failure; the repair is what recovers. Both are here, but only one of them ends the loop.
  • In-place checkpoint rather than a snapshot. VACUUM INTO or the backup API would be the textbook answer for transport, and were rejected because the cache is the next run's live working set, not an archive — a snapshot only moves the "make it live" step to restore time.
  • A declined save discards that run's session work. That is deliberate and loud, but it is a real cost, and the restore-side repair is what keeps it rare.

One residual, noted rather than solved: an object-store upload failure is non-fatal while restore prefers the object store, so the Actions cache can hold newer state than the copy that wins. That predates this change and has a different fix.

Verification

Types 0, lint 0, dist/ reproducible, 6,332 tests. A 12-reviewer pass ran before this was pushed; its findings are in the branch.

The first draft's central claim was wrong. Declining to persist bad state
prevents new poisoning but cannot escape existing poisoning: save keys are
unique per run and restore keys are prefixes returning the newest match, so a
run that declines to save leaves the poisoned entry as the newest one and the
next run restores it again. The loop stays absorbing.

The repair belongs on restore. Checkpointing a hot database between restore
and bootstrap recovers the session instead of discarding it, heals a stuck
repository in one run, and lands outside the budget that was failing, since
no timer spans those two calls.

Also corrects three things the draft asserted without grounds: that the
harness could await the child's exit, which it cannot observe; that the
checkpoint belonged in the session runtime, when both save paths and the
object-store sync share one function at the cache boundary; and that whether
the runtime could checkpoint at all was unknowable before implementation.

Refs #1407
…trap budget visible

Units 1, 4, 5 and 6 of the session-cache trap fix.

The wal-index is machine-local and SQLite never syncs it, so a copy from
another runner is stale by construction. It is no longer captured on save,
and a copy carried in by an older cache is deleted on restore rather than
left to age out. Restore still tolerates its presence so entries written
before this change still work.

Declaring storage corrupt now actually clears the database. It lives beside
the storage directory rather than inside it, so 'proceed with clean state'
had been leaving the very files it meant to discard. The object-store restore
also gains the corruption and version checks the cache path already had: it
wins on restore and returns before either one ran, so the authoritative path
was the unchecked one.

Which files constitute the database is now defined once, in the runtime
package, because the object-store sync there cannot import from the Action.
The name had been written out in three places.

The server bootstrap budget is configurable and reported. It was the SDK's
5000ms default, never passed and never logged, against runs observed landing
16ms inside it. Time spent in the spawn is measured apart from the total so
the budget comparison is exact and a slow port bind is distinguishable from
slow server init.

session-retention now reaches pruning. It was parsed, validated, logged, and
then never read, so a consumer bounding their cache with it got a validated
no-op. Its default matches the hardcoded value it replaces, so an unset input
changes nothing.

Refs #1407
No run had ever checkpointed it. server.close() sends a kill and returns
without waiting, so a valid session could sit entirely in the write-ahead log
with the database file at zero bytes, and the cache carried that hot log to
the next run, which paid recovery cost opening it — measured at 2416.8ms for
a 185MB log against 60.2ms once merged.

The checkpoint runs inside saveCache, which both the cleanup phase and the
post hook already call and which also owns the object-store sync, so every
capture path inherits it without a flag they must agree to read. It runs
before anything inspects file sizes, since merging changes which files are
non-empty.

Success is judged by the log's size on disk rather than the pragma's own
count, which reported zero on a verified-successful truncation. The outcome
has three states: collapsing 'already clean' into 'failed' would decline
saves that were always safe.

The harness cannot observe whether the writer exited, so the attempt is the
liveness probe — a live writer surfaces as busy and is retried within a bound.
A log that still cannot be merged declines the save and says why in the job
summary, because a save that silently does not happen is the failure this
repository already paid for once.

Refs #1407
This is what gets a repository out of the loop rather than merely stopping it
from entering one. Save keys are unique per run and restore keys are prefixes
returning the newest match, so a run that declines to save writes no entry and
the next restore hits the same bad one. Declining forever is a stable state.

Checkpointing the restored database heals it in place. It runs between restore
and bootstrap, where no timer applies — the budget covers createOpencode alone
— so the recovery cost moves out of the window that was timing out instead of
being paid inside it.

Repairing rather than wiping is the point. Wiping would also break the loop,
but it discards session history on every transient bootstrap failure, which is
the second reporter's case exactly.

A repair is logged because it is the evidence a stuck repository healed, and
its absence later is the evidence it stayed healed. The healthy case is silent,
and a failed repair still lets bootstrap try.

Closes #1407

Refs #1407
A comment asserted a valid session could leave opencode.db at zero bytes with
its data in the write-ahead log. Three independent reviewers read it and each
filed the same bug against the checkpoint helper's zero-byte guard, claiming
it skipped the case it exists for.

The guard is right and the comment was wrong. A database in write-ahead mode
with no checkpoint leaves the main file at its header page — 4096 bytes on
Node 24.20.0, never zero — and forcing it to actually zero makes the log
unreadable, so that shape holds no session to save. The comment now states
the measured value, and a test pins the real shape so the misreading cannot
recur.

Also from review: the errno checks in the two delete paths assumed every
thrown value carries a code, so a non-errno failure was silently treated as a
missing file; both now narrow before reading it. The bootstrap timeout is
bounded, since an unbounded value would hang a run to the job timeout rather
than failing fast, which is the opposite of what that input is for. And the
restore repair gains a test that drives a real database rather than a mock,
covering the wiring the ordering tests cannot see.

Refs #1407
fro-bot
fro-bot previously approved these changes Sep 1, 2026

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The diagnosis holds up under reading. runCleanup does execute on bootstrap failure (run.ts finally), serverHandle.shutdown() genuinely does not await a checkpoint, and restore-key prefix matching genuinely makes "decline to save" a stable state rather than a recovery. Moving the repair to restore — outside the only window timeoutMs governs — is the right shape, and cache-restore.test.ts:236-260 pins that ordering so a future refactor can't silently move the cost back inside the budget.

I verified locally: tsc --noEmit clean, and 111 tests across src/services/cache/, cache-restore.test.ts, and cleanup.test.ts pass. node:sqlite resolves unflagged on the pinned Node 24.20.0 and is bundled as an external import in the shared dist/artifact-*.js chunk, so the new dependency claim checks out.

The three-state CheckpointOutcome is the load-bearing design decision here, and the reasoning in checkpoint.ts:7-22 is correct: a two-state collapse would decline every healthy save. Judging success by WAL size rather than the pragma's checkpointed count is also correct, and checkpoint.test.ts:171-196 pins the surprising pragma behavior with a control database rather than just asserting the helper's output — that test will still be right if a future Node changes the count.

Blocking issues

None.

Non-blocking concerns

1. A malformed database still traps the repository — the same loop, different cause.
checkpointDatabase reports failed when new DatabaseSync(dbPath) throws for a non-lock reason ("file is not a database", "database disk image is malformed"). At cache-restore.ts:74-76 that only logs a warning and hands the bad database straight to bootstrapOpenCodeServer. On the save side, a malformed main DB with an absent or empty WAL returns nothing-to-checkpoint (checkpoint.ts:100-103), so the save proceeds and re-persists it under a fresh key. That is exactly the self-perpetuating loop this PR set out to break, reached through corruption instead of a hot WAL.

This is not a regression — the same database was re-saved before this change — so it does not block. But the failure is already distinguishable: isRetryableError separates "busy/locked" (transient, retry) from everything else (structurally unrecoverable). Routing the non-retryable case at restore into the existing corruption path (cleanStorage + cacheStatus = 'corrupted') would be far narrower than the "wipe on any bootstrap failure" the plan rightly rejected — it fires only when SQLite itself says the file is not a usable database, not on transient bootstrap variance.

2. The new object-store integrity checks are close to tautologies for the artifact they guard.
restoreFromObjectStore calls fs.mkdir(storagePath, {recursive: true}) at restore.ts:113, then checkStorageCorruption at :119 — which returns false as soon as storagePath is a readable directory, which the mkdir two lines earlier guarantees. checkStorageVersion returns true when .version is absent (restore.ts:47-50), and the object-store sync only transports DB-family files plus session sidecars, so .version will usually not be there. Both checks therefore pass for a corrupt opencode.db, which is the thing the object store actually restored. Adding them is still an improvement over returning before any check ran, but the PR description's framing overstates the protection gained. Related to (1): the checkpoint attempt is the only thing on this path that actually inspects the database.

3. The restore-side repair has no wall-clock bound.
maxAttempts × retryDelayMs bounds only the sleeps (800 ms at defaults). Each attempt's PRAGMA wal_checkpoint(TRUNCATE) is unbounded, and this PR's own measurement puts a 185 MB WAL at ~2.4 s. A large, contended database can add roughly 12 s ahead of bootstrap on every cache-hit run. That is deliberately outside the createOpencode budget, so it cannot cause the timeout being fixed — but it is now unbounded time on the critical path of every hit. A deadline checked between attempts would cap it without changing the healthy path.

4. The declined-save cost is smaller than the description suggests, and the reason is worth documenting.
runCleanup only sets STATE_KEYS.CACHE_SAVED when saveCache returns true (cleanup.ts:184-186), so a declined save leaves post.ts to retry — by which point the main step has ended and the OpenCode child has almost certainly exited, making the checkpoint succeed. That post-hook retry is the real reason "a declined save discards that run's session work" is rare, and it deserves a sentence in the save.ts decline comment, because a future refactor that sets CACHE_SAVED on decline (to avoid duplicate work) would silently remove the safety net.

One wrinkle: post.ts:90-98 builds cacheSaveOptions without projectIdPath, so a save that only lands in the post hook omits .git/opencode from the archive. Pre-existing, and low impact since ensureProjectId regenerates deterministically from the root commit — but the fallback path is now materially more likely to be the one that runs.

5. Comment/behavior mismatch in paths.ts:58-63.
The comment states save mode "filters by existence because @actions/cache fails if any save path is missing at archive time," but buildSaveCachePaths pushes opencode.db unconditionally at :111 while only the WAL gets an fs.access guard at :114-119. @actions/cache throws only when every path resolves to nothing, so the behavior is safe either way — but the comment states a rule the code beside it does not follow, which is the kind of thing the last commit in this branch exists to fix.

Missing tests

  • No end-to-end coverage of a malformed (not merely hot-WAL) opencode.db through restore → repair-fails → bootstrap → save. This is the gap behind concern (1); a test asserting the current behavior would at least make the residual loop explicit rather than implied.
  • No test that a cleanup-declined save is retried by the post hook. post.test.ts covers cacheSaved === 'false' → save, but not framed as decline recovery, and nothing ties the two halves together. Given that this retry is what makes the decline tradeoff acceptable, a test naming that relationship would protect it from a well-intentioned refactor.
  • No coverage of the restore-side repair against a locked databasecache-restore.test.ts:219-234 mocks the failed outcome rather than producing it, so the real path-selection code is never exercised in the failure case. The e2e suite at :270-347 does this well for the success case; the same fixture with an EXCLUSIVE-locked holder (as in checkpoint.test.ts:151-169) would close it.

Coverage is otherwise unusually strong for a change of this shape — the -shm deletion is tested on both the cache and object-store paths, including the non-ENOENT warning branch, and the input clipping has all four boundary cases.

Risk assessment: MED

Blast radius is every run: the checkpoint is now unconditional on both the restore and save paths, and saveCache gained a new way to return false. The mitigating factors are real — the decline is recoverable via the post hook, the repair sits outside the bootstrap budget, the failure modes are all logged, and checkpointDatabase never throws. Regression likelihood concentrates in one place: a run whose OpenCode child outlives the ~1 s retry bound at cleanup and whose post-hook checkpoint also fails loses that run's session work, where the previous code would have transported db+wal. Rare, but silent to the user except for a job-summary block.

No security exposure. The public API changes are additive (server-bootstrap-timeout input, new checkpointDatabase export); the one semantic change is that session-retention now actually prunes, which will surprise anyone who set it low while it was inert — worth a line in the release notes.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33469019264
Cache hit
Session ses_fa4d1db6affeL56tSeVO7ak948

…n passing it on

Review follow-ups on the trap fix.

A malformed database still trapped a repository in the loop this branch
exists to break, reached through corruption instead of a hot log. The
checkpoint already distinguished a live writer from a file SQLite says is not
a database; only the first is worth retrying, and the second now routes into
the clean-slate path restore already used for corrupt storage.

That covered a malformed database carrying a log, since that is the only
shape anything opened. One with no log was never opened at all, so it reached
bootstrap, failed, and was saved again. Restore now checks that a database it
is about to hand over can actually be read — one schema-page read, a few
milliseconds and independent of size, outside the budget. Construction alone
does not surface this: opening a file of plain text succeeds and only the
read reports it is not a database.

The repair's retries bounded their sleeps but not their work, so a large log
could spend far longer than intended ahead of bootstrap. A deadline between
attempts caps that without ever interrupting one in flight.

Also records why declining to save rarely costs a run its work — the post
hook retries once the child has exited — because a refactor that marked the
cache saved on decline would remove that quietly. And corrects a comment
claiming save paths are filtered by existence when only the log is.

Refs #1407
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

All five in, plus the gap the first fix left behind.

Concern 1 turned out to have two halves. Routing a structurally-unrecoverable checkpoint failure into the existing clean-slate path closes it for a malformed database carrying a write-ahead log — but that is the only shape anything ever opened. One with no log returned nothing-to-checkpoint before any open, reached bootstrap, failed, and was saved again: the same loop without the WAL. Restore now verifies a database is readable before handing it over.

I checked the probe rather than assuming it worked, and the result changed the implementation: new DatabaseSync() against a file of plain text succeeds. Only the schema read reports file is not a database. A construction-only check would have looked like it closed the gap while leaving it open. It reads one schema page — 0.6–5.2ms, independent of database size, and outside the bootstrap budget.

Left the save side alone deliberately: forcing every healthy save to open a database when there is nothing to checkpoint is a real cost for a benefit that only lands on restore.

Concern 3 — the retries bounded sleeps but not the checkpoint work itself, so a large log could spend far longer than intended. A deadline between attempts caps it without interrupting one in flight.

Concern 2 was right and I have corrected the description. checkStorageCorruption only asks whether the storage path is a readable directory, which the mkdir two lines above guarantees, and checkStorageVersion treats an absent .version as compatible — which it always is for an object-store restore. Both pass for a corrupt database. The checkpoint, and now the probe, are the only things on that path that inspect it.

Concern 4 is recorded in the decline comment, including that a refactor marking the cache saved on decline would remove the retry safety net. Concern 5 is fixed — and it was the same defect as the commit before it, a comment stating a rule its own code does not follow.

Your three named tests are in, plus real-database coverage for both corruption shapes and a regression guard on the healthy no-log path.

Gates: types 0, lint 0, dist/ reproducible, 6,336 tests.

Comment thread src/services/cache/integrity.test.ts Fixed
fro-bot
fro-bot previously approved these changes Sep 1, 2026

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The five concerns from the dismissed review are genuinely closed, and the sixth gap the author found on their own — a malformed database with no write-ahead log, which checkpointDatabase returns nothing-to-checkpoint for before ever opening the file — was the right catch. verifyDatabaseUsable (integrity.ts:46-66) is the correct shape for it: restore-only, schema-page read rather than PRAGMA integrity_check, and a no-op for a missing or zero-byte file so it never creates the database it was asked to inspect. The comment at integrity.ts:20-25 explaining why it is not folded into checkpointDatabase is the load-bearing reasoning and it is right — doing so would redefine nothing-to-checkpoint for the save path and force every healthy save to open a file it currently never touches.

The author's claim that new DatabaseSync() succeeds against plain text and only the schema read surfaces file is not a database matches what I see: a construction-only probe would have looked closed while leaving the gap open. integrity.test.ts:47-73 also covers the other corruption shape (a truncated disk image), which a construction-only check would also have missed.

Verified locally: tsc --noEmit clean; 126 tests across src/services/cache/, cache-restore.test.ts, cleanup.test.ts, and the new cleanup-decline-retry.test.ts pass. dist/artifact-C90iooVD.js contains both wal_checkpoint and sqlite_master, so the committed bundle reflects the source. session-retention's defaults line up (DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50), so wiring it in run.ts:80 is a genuine no-op for an unset input, and cleanup.test.ts:260-315 pins both the explicit-default and never-set branches separately — which is the distinction that makes it safe to ship.

cleanup-decline-retry.test.ts is the best addition here. It runs runCleanup and runPost against one shared state store, which is the only way to assert the relationship the decline comment now documents. Neither phase's own suite could see it.

Blocking issues

None.

Non-blocking concerns

1. isRetryableError is an allowlist of retryable, so every unrecognized SQLite error routes into the destructive path.
checkpoint.ts:82-88 returns retryable: true only for messages containing locked or busy; everything else is false, and cache-restore.ts:96-100 sends retryable: false straight to cleanStorage — deleting the session history. The comment above that branch names the two errors it means (file is not a database, database disk image is malformed), but the code reaches it for any other message too. I confirmed one concretely on this runner:

new DatabaseSync('<missing dir>/opencode.db')
  -> "unable to open database file"   retryable? false

SQLITE_CANTOPEN (fd exhaustion, permissions), SQLITE_FULL (database or disk is full), SQLITE_IOERR (disk I/O error), and attempt to write a readonly database all land the same way — environmental and transient, but classified as structural corruption and answered by wiping history. That is precisely the tradeoff the plan rejected when it declined "wipe on any bootstrap failure," reached through a narrower door. verifyDatabaseUsable has the same shape at integrity.ts:57-58: any throw from the schema read becomes usable: false.

The inversion is small and matches what both doc comments already say: match the structural messages positively (not a database, malformed, encrypted, file is encrypted or is not a database) and treat everything else as retryable / leave-alone. The healthy path is unchanged and the two tested corruption shapes still route the same way.

2. cacheResult and cacheStatus disagree after a structural-corruption downgrade.
handleStructuralCorruption (cache-restore.ts:79-86) mutates the local cacheStatus to 'corrupted' but returns the original cacheResult unchanged at :133 — still {hit: true, corrupted: false, restoredPath: storagePath} for storage that has just been deleted. Nothing reads cacheResult today (only cacheStatus and serverHandle are consumed, in finalize.ts:199, execute.ts:223,380, and run.ts:135), so this is latent rather than live. But it is a public field of CacheRestorePhaseResult, and the first consumer to read it will read a stale hit for a wiped cache. Related: metrics.setCacheSource is left at 'cache'/'storage', so the job summary will report a corrupted cache sourced from a cache that was discarded.

3. Object-store corruption is cleared locally but not at the source.
When the malformed opencode.db arrived via restoreFromObjectStore, cleanStorage removes only the runner's copy — the bucket still serves the same object on the next restore. It does self-heal, because the next saveCache uploads a healthy database over it, but only for a run that reaches syncSessionsToStore (save.ts:153-170); a run that declines the save leaves the poisoned object in place and the next run pays the wipe again. The Actions-cache side does not have this problem, since restore keys advance to the newer entry. Not a regression and not worth solving here, but the asymmetry is worth a line somewhere.

4. The WAL existence guard is computed before the checkpoint that changes it.
buildSaveCachePaths runs at save.ts:123, the checkpoint at :136. After a successful TRUNCATE the log is a zero-byte file that is still in cachePaths, so every healthy save ships an empty opencode.db-wal in the archive. Harmless — restore tolerates it and hasCacheableContent gates on the main database — but the fs.access guard described at paths.ts:58-69 never actually reflects post-checkpoint state, which is the same comment-vs-code drift the last two commits in this branch exist to fix. Computing the save paths after the checkpoint would make the comment true.

Missing tests

  • No test pins the classification of a non-structural SQLite failure. checkpoint.test.ts covers busy/locked (retryable) and not a database (non-retryable), but nothing in between — and the space between them is exactly what decides whether concern (1) destroys a repository's history. A test asserting the intended classification for unable to open database file would either confirm the current behavior is deliberate or fail and prove it is not.
  • No test asserts cacheResult/cacheStatus coherence after the corruption downgrade. Every corruption test checks result?.cacheStatus only, so concern (2) is invisible to the suite.
  • No test covers the run.ts:80 assignment itself. The cleanup half of the session-retention wiring is well covered, but the line that actually ends the "parsed, validated, logged, never read" bug is a bare assignment that run.test.ts cannot see, since it mocks runCleanup. Low value on its own, but it is the one link in that chain nothing observes.

Coverage is otherwise excellent for a change of this shape. The end-to-end suite at cache-restore.test.ts:339-533 now exercises real databases for all four states the phase distinguishes — hot WAL, malformed-with-WAL, malformed-without-WAL, and locked — using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable rather than canned outcomes, and the healthy no-log regression guard at :501-532 protects the common path. The locked-database case at :446-477 closes the exact gap flagged in the previous review.

Risk assessment: MED

Unchanged from the previous review in shape, slightly larger in blast radius. Both the checkpoint and the usability probe now run on every cache hit, and the restore path gained a code path that deletes session state — the first time this codebase discards history on the strength of a SQLite error message. The mitigations are real: the probe is a single schema page (measured under 2ms), it never runs for a missing or empty database, it sits outside the createOpencode budget, and cleanStorage failing is caught and non-fatal. The residual risk concentrates in concern (1): a runner-level fault (fd exhaustion, a full disk) presenting as a non-busy SQLite error would silently wipe a repository's session history and report a corrupted cache. Recoverable — the next run rebuilds — but silent to the user beyond a job-summary line, and the run still completes successfully.

No security exposure. Public API changes are additive (server-bootstrap-timeout input, checkpointDatabase and verifyDatabaseUsable exports). Two behavioral changes deserve release notes: session-retention now actually prunes (a no-op at the default, surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33472537265
Cache hit
Session ses_fa49e3391ffefMwAv3VUt6Rn7K

CodeQL flagged the new integrity test: it measured the database with stat and
then reopened the same path to truncate it, which resolves the path twice and
is a check-then-use window. Statting the already-open handle closes it and
also makes the truncation apply to the file that was actually measured.

Refs #1407

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CONDITIONAL

The diagnosis and the shape of the fix are right, and I want to say that clearly before the blocking issue: repairing on restore rather than only declining on save is the only thing in this change that actually ends the loop, and putting it between restoreCache and bootstrapOpenCodeServer — outside the only window timeoutMs governs — is correct. verifyDatabaseUsable (integrity.ts:46-66) closes the real gap the first draft left, and the reasoning at integrity.ts:20-25 for keeping it off the save path is sound. I independently confirmed the load-bearing claim behind it: new DatabaseSync() against a file of plain text succeeds, and only the schema read raises file is not a database. A construction-only probe would have looked closed while leaving the gap open.

Verified locally: check-types clean across all four workspace packages; 126 tests pass across src/services/cache/, cache-restore.test.ts, cleanup.test.ts, and cleanup-decline-retry.test.ts. The end-to-end suite at cache-restore.test.ts:339-533 exercises real databases for all four states the phase distinguishes, using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable rather than canned outcomes. That is unusually good coverage for a change of this shape.

The one thing I cannot pass is the polarity of the error classification that now guards a destructive path.

Blocking issues

1. Any unrecognized SQLite error is classified as structural corruption, and structural corruption now deletes the repository's session history. src/services/cache/checkpoint.ts:82-88, src/services/cache/integrity.ts:57-58, src/harness/phases/cache-restore.ts:96-100.

isRetryableError is an allowlist of retryable: it returns true only for messages containing locked or busy, and everything else falls through as false. cache-restore.ts:96-100 routes retryable: false into handleStructuralCorruption, which calls cleanStorage(storagePath) — wiping the storage directory and the entire DB family. verifyDatabaseUsable has the same polarity at integrity.ts:57-58: any throw from the schema read becomes usable: false, which reaches the same wipe.

The doc comments at checkpoint.ts:24-26 and cache-restore.ts:71-73 both name the two errors this branch is meant for (file is not a database, database disk image is malformed), but the code reaches it for anything else too. I confirmed two concrete non-structural cases on this runner:

new DatabaseSync('<dir 0555>/opencode.db') + SELECT count(*) FROM sqlite_master
  -> "attempt to write a readonly database"   retryable? false  -> wipe
new DatabaseSync('<missing dir>/opencode.db')
  -> "unable to open database file"           retryable? false  -> wipe

SQLITE_CANTOPEN (fd exhaustion, permissions), SQLITE_FULL (database or disk is full), SQLITE_IOERR (disk I/O error), and the readonly case above are all environmental and transient, but each is answered by destroying session history. Note that the probe opens read-write, so a merely non-writable database — which is perfectly readable — is classified as corrupt.

This is the tradeoff the plan explicitly rejected when it declined "wipe on any bootstrap failure," reached through a narrower door. It is also the first time this codebase discards history on the strength of a SQLite error message, which is why I am treating a wrong-polarity default as blocking rather than as a nit: the fail-safe default for an unrecognized error must be leave it alone, not delete it.

The fix is small and matches what both doc comments already claim. Match the structural messages positively — not a database, malformed, file is encrypted or is not a database — and treat everything else as retryable / leave-alone. The healthy path is unchanged, and both corruption shapes currently under test (cache-restore.test.ts:422-444 and :479-499) still route exactly as they do today. The same inversion applies to verifyDatabaseUsable: only a structurally-worded failure should report usable: false.

This was raised as concern (1) in the previous review and is unaddressed in the current head; the author's comment responded to the earlier round's five items, so I read this as not-yet-seen rather than declined.

Non-blocking concerns

1. cacheResult and cacheStatus disagree after a corruption downgrade. handleStructuralCorruption (cache-restore.ts:79-86) mutates the local cacheStatus to 'corrupted' but cache-restore.ts:133 returns the original cacheResult unchanged — still {hit: true, corrupted: false, restoredPath: storagePath} for storage that was just deleted. I confirmed nothing outside tests reads cacheResult today, so this is latent, not live. But it is a public field of CacheRestorePhaseResult, and the first consumer to read it reads a stale hit for a wiped cache. Related: metrics.setCacheSource (:50) stays at 'cache'/'storage', so the job summary reports a corrupted cache sourced from a cache that was discarded.

2. The WAL existence guard is computed before the checkpoint that changes it. buildSaveCachePaths runs at save.ts:123, the checkpoint at save.ts:136. After a successful TRUNCATE the log is a zero-byte file still present in cachePaths, so every healthy save ships an empty opencode.db-wal. Harmless — restore tolerates it and hasCacheableContent gates on the main database — but the guard described at paths.ts:66-69 never reflects post-checkpoint state. Computing the save paths after the checkpoint would make the comment true. (The rest of that comment block is now accurate; the previous round's comment/behavior drift is genuinely fixed.)

3. Object-store corruption is cleared locally but not at the source. When a malformed opencode.db arrives via restoreFromObjectStore, cleanStorage removes only the runner's copy — the bucket still serves the same object next run. It self-heals for any run that reaches syncSessionsToStore (save.ts:153-170), but a run that declines the save leaves the poisoned object in place and the next run pays the wipe again. The Actions-cache side does not have this problem, since restore keys advance. Not a regression; worth a line somewhere.

4. Two behavioral changes deserve release notes. session-retention now actually prunes — a no-op at the default (DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50, and cleanup.test.ts:260-315 pins the explicit-default and never-set branches separately), but surprising for anyone who set it low while it was inert. And opencode.db-shm is no longer captured for transport.

Missing tests

  • No test pins the classification of a non-structural SQLite failure. checkpoint.test.ts covers busy/locked (retryable) and not a database (non-retryable), but nothing between them — and that gap is exactly what decides whether the blocking issue destroys a repository's history. A case asserting that unable to open database file or attempt to write a readonly database leaves storage intact would pin the intended polarity.
  • No test asserts cacheResult/cacheStatus coherence after the downgrade. Every corruption test checks result?.cacheStatus only, so non-blocking concern (1) is invisible to the suite.
  • No test covers the run.ts:80 assignment. The cleanup half of the session-retention wiring is well covered, but the line that actually ends the "parsed, validated, logged, never read" bug is a bare assignment run.test.ts cannot observe, since it mocks runCleanup. Low value alone, but it is the one unobserved link in that chain.

Coverage is otherwise excellent. The -shm deletion is tested on both the cache and object-store paths including the non-ENOENT warning branch, the input clipping has all four boundary cases (inputs.test.ts:905-957), and the healthy no-log regression guard at cache-restore.test.ts:501-532 protects the common path with an explicit expect(mocks.loggerWarning).not.toHaveBeenCalled().

Risk assessment (MED)

Blast radius is every run: both the checkpoint and the usability probe are unconditional on the restore path, and the checkpoint is unconditional on the save path. Mitigations are real — the probe is a single schema page, it never runs for a missing or empty database, it sits outside the createOpencode budget, checkpointDatabase never throws, cleanStorage failure is caught, and a declined save is retried by the post hook (now documented at save.ts:83-88 and pinned by cleanup-decline-retry.test.ts, which is the best addition in this round — it runs runCleanup and runPost against one shared state store, which neither phase's own suite could do).

Residual risk concentrates entirely in the blocking issue: a runner-level fault presenting as a non-busy SQLite error silently wipes session history and reports a corrupted cache, while the run still completes successfully. Recoverable, since the next run rebuilds — but invisible to the user beyond a job-summary line, and triggered by exactly the transient conditions this design set out not to punish.

No security exposure. Public API changes are additive: the server-bootstrap-timeout input (clamped at 120s with a warning, inputs.ts:363-373), and the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33473192307
Cache hit
Session ses_fa494e035ffeIsNNA4D1liay5F

The clean-slate path was reached by negating retryability, and retryability
was an allowlist of busy and locked. So every error outside that pair —
a full disk, an I/O fault, exhausted descriptors, a permissions problem —
answered a transient condition by deleting the repository's session history.
That is the trade this work rejected when it declined to wipe on any
bootstrap failure, reached through a narrower door.

Corruption is now matched on its own terms rather than inferred from the
absence of a retry, so an unrecognized error leaves the database alone. Both
comments already described the allowlist; the code now agrees with them.
Verified in both directions: a file of plain text still reports it is not a
database and is still cleared, while a database that merely cannot be opened
is left where it is.

The corruption downgrade also returned the original restore result, which
still described a hit with a path that had just been deleted, and reported a
source for a cache that no longer existed. Both now match the state on disk.

Save paths are built after the checkpoint rather than before, so the guard
reflects the log the checkpoint actually left behind.

Refs #1407
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

Blocking issue fixed, and you were right to treat it as blocking rather than a nit.

The polarity was mine. My brief told the lane to route non-retryable failures into the clean-slate path, and isRetryableError is an allowlist of retryable — so everything outside busy/locked fell through to deleting session history. I reproduced it before fixing: unable to open database file from a permissions fault classified as non-retryable and wiped. A full disk, an I/O fault, or exhausted descriptors would all have done the same. That is the trade this work rejected when it declined to wipe on any bootstrap failure, reached through a door I opened.

Corruption is now matched on its own terms — isStructuralCorruptionError, a positive allowlist of not a database and malformed — as a third classification rather than the negation of retryability. retryable on the failed outcome became structural, so the two axes cannot be confused again. Verified in both directions against real node:sqlite:

file of plain text       -> "file is not a database"        structural: true   -> cleared
chmod 0o000 on a real db -> "unable to open database file"  structural: false  -> left alone, run stays a hit

Both existing corruption tests still route to the wipe, and the case that would have caught this is now pinned in three places.

Non-blocking 1cacheResult is rebuilt to match, mirroring restoreAfterCorruption's own shape, and setCacheSource(null) goes with it so the summary no longer reports a source for a cache that was just deleted.

Non-blocking 2 — save paths are now built after the checkpoint. Worth recording what that turned up: TRUNCATE zeroes the log but does not unlink it until the checkpointing connection closes, and checkpointDatabase closes its own handle in finally — so post-reorder the file is genuinely absent by the time the guard runs, and the comment is true without further edits.

Non-blocking 3 and 4 noted. The object-store copy surviving a local wipe is real; it self-heals on any run that reaches the sync, and I would rather fix it where the store's freshness is decided than bolt it onto this path.

Gates: types 0, lint 0, dist/ reproducible, 6,339 tests.

fro-bot
fro-bot previously approved these changes Sep 1, 2026

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The blocking issue from my previous review is genuinely fixed, and the fix is the right shape rather than a patch over the symptom. isStructuralCorruptionError (sqlite-errors.ts:48-53) is now a positive allowlist — not a database / malformed — living in its own module rather than as the negation of isRetryableError. Renaming the failed outcome's field from retryable to structural (checkpoint.ts:37) is what makes the correction durable: the two axes are now separate concepts with separate names, so a future change cannot silently re-conflate "I could not retry this" with "this file is corrupt." That is a better outcome than the minimal inversion I proposed.

I verified the polarity in both directions rather than reading the diff:

checkpoint.ts:157   lastStructural = isStructuralCorruptionError(caughtError)   // was: !isRetryableError
integrity.ts:68     if (isStructuralCorruptionError(error)) -> usable: false; else usable: true
cache-restore.ts:109 if (repairOutcome.structural) -> handleStructuralCorruption

The destructive path is now reachable only from a positively-matched structural error. verifyDatabaseUsable returns {usable: true} for any unrecognized throw (integrity.ts:71), which is the correct fail-safe default for a probe whose caller deletes session history.

Verified locally: check-types clean across all four workspace packages; lint 0 errors; bun run build leaves dist/ byte-identical (reproducible); 265 tests pass across src/services/cache/ and src/harness/phases/. I independently confirmed the createOpencode timeout claim against the installed SDK — node_modules/@opencode-ai/sdk/dist/server.js:7 does Object.assign({..., timeout: 5000}, options) and server.d.ts:6 declares timeout?: number, so passing DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS = 5000 explicitly is byte-for-byte identical to today's unset behavior. The comment in packages/runtime/src/shared/constants.ts is accurate.

The two non-blocking items I raised are also closed, and one of them turned up something worth recording. handleStructuralCorruption now rebuilds cacheResult to match the downgraded status and calls setCacheSource(null) (cache-restore.ts:90-98). And moving buildSaveCachePaths to after the checkpoint (save.ts:137 then :146) is correct for a reason better than the one I gave: SQLite unlinks the write-ahead log when the last connection closes, and checkpointDatabase closes its own handle in finally, so post-reorder the file is genuinely absent by the time the fs.access guard runs — the comment at paths.ts:66-69 is now true without further edits.

On the two failing tests in src/main.test.ts: both are environmental in this review sandbox, not regressions. They spawn a real child process that hits Cache Service Url not found, unable to restore cache and then hangs in auto-setup — this runner has no ACTIONS_CACHE_URL/ACTIONS_RESULTS_URL. src/main.test.ts is not in the changed-file set and the timeout occurs during setup, long before any code this PR touches. Not counted against the change.

Blocking issues

None.

Non-blocking concerns

1. acquiredServer?.close() is unguarded inside the catch block, which can break bootstrapOpenCodeServer's never-throw contract. packages/runtime/src/agent/server.ts:105.

The function's signature promises Promise<Result<OpenCodeServerHandle, Error>>, and the entire try/catch exists so a bootstrap failure returns err(...) rather than rejecting. The new cleanup call is the last statement before that return err(...), so if close() throws, the function rejects instead — the one path in this function that was previously impossible.

Reachability with the real SDK is essentially nil: stop() (@opencode-ai/sdk/dist/process.js) returns early when proc.exitCode !== null || proc.signalCode !== null, and proc.kill() on an exited child returns false rather than throwing. So this is an invariant nit, not a live bug. But note the URL-mismatch branch at :84 already calls server.close() before returning, and if that throws, control reaches the catch and closes the same handle a second time — the only double-close in the function. Wrapping :105 in the same best-effort try/catch the rest of this PR uses for db?.close() (checkpoint.ts:148-152, integrity.ts:73-77) would make it consistent with the module's own convention.

2. post.ts still omits projectIdPath, and that path is now deliberately load-bearing. src/harness/post.ts:90-98 builds cacheSaveOptions without projectIdPath, so a save that only lands in the post hook ships an archive without .git/opencode.

This is pre-existing and I flagged it as low-impact last round. I am raising it once more only because this PR promotes the post-hook retry from an incidental fallback to the documented reason a declined save is acceptable (save.ts:83-88, pinned by cleanup-decline-retry.test.ts). The mitigation still holds — ensureProjectId regenerates deterministically from the root commit — so this is genuinely non-blocking, but the asymmetry between cleanup.ts:179 and post.ts is now load-bearing enough to be worth a one-line fix or an explicit comment saying it is intentional.

3. The usability probe is skipped when the checkpoint fails non-structurally. cache-restore.ts:108-126 runs verifyDatabaseUsable only in the nothing-to-checkpoint branch. A database that is both locked and corrupt therefore reaches bootstrap unprobed.

I think this is the right call and am recording it rather than objecting: probing a database that just refused to open would fail for the same environmental reason, and re-probing would only add a second chance to misclassify. Worth a sentence in the comment block so a future reader does not mistake the gap for an oversight.

4. Release notes. Unchanged from last round: session-retention now actually prunes (a no-op at the default — DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50, and cleanup.test.ts pins the explicit-default and never-set branches separately — but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.

Missing tests

  • No test pins that bootstrapOpenCodeServer still returns err when the acquired server's close() throws. The new case at server.test.ts ("closes the server on a failure that occurs after the handle was acquired") uses a plain vi.fn() spy that cannot throw, so it proves the close happens but not that the Result contract survives it. That is exactly the gap behind concern (1); a spy with mockImplementation(() => { throw ... }) asserting result.success === false would close it.
  • No test drives the restore-side probe through the object-store source. The end-to-end suite exercises real databases well, but every case reaches the probe via a cache hit. restoreFromObjectStore returns {hit: true, source: 'storage'}, which the cacheStatus === 'hit' gate at cache-restore.ts:67 does cover — but only by inspection. Since the object store is the path that wins on restore and the PR description correctly notes its own integrity checks are near-tautologies, the probe is the only real defense there, and nothing pins it.

Coverage is otherwise excellent and materially better than either previous round. The blocking case is pinned in three separate places — checkpoint.test.ts:197 (a real permissions fault yielding unable to open database file classified non-structural), integrity.test.ts:78 (a healthy database at chmod 0o000 reported usable: true), and cache-restore.test.ts:289 (storage left intact, run still a hit) — using real node:sqlite faults rather than mocked messages, which is what makes them worth having. integrity.test.ts:62-68 sizing the truncation target through its own open handle avoids a check-then-use window that a lazier test would have carried.

Risk assessment (LOW-MED)

Blast radius remains every run — the checkpoint is unconditional on both restore and save, and the usability probe is unconditional on the healthy restore path. What changed since the last round is the residual: the destructive branch is no longer reachable by default. It now requires SQLite to positively report not a database or malformed, which is a claim about the file's contents rather than about the runner's state. The transient faults I was worried about — fd exhaustion, a full disk, an I/O error, a read-only mount — all now report non-structural and leave the database alone, verified against real node:sqlite in checkpoint.test.ts and integrity.test.ts rather than asserted.

Remaining mitigations are unchanged and real: the probe reads a single schema page (measured under 2ms, size-independent), it never opens a missing or zero-byte file, both the probe and the checkpoint sit outside the createOpencode budget, checkpointDatabase never throws, the repair is bounded by an explicit deadline (deadlineMs, checkpoint.ts:128-137) rather than by sleep count alone, and cleanStorage failure is caught. A declined save is retried by the post hook, and that relationship is now pinned by a test that runs runCleanup and runPost against one shared state store.

The loop this PR set out to break is closed at both ends, which I checked as a whole rather than per-file: a structurally corrupt database is wiped on restore and declined on save, so it is neither handed to bootstrap nor re-persisted under a fresh key. The largest remaining exposure is the object-store copy surviving a local wipe (non-blocking 3 from the previous round), which the author has correctly scoped to where store freshness is decided rather than bolting a workaround onto this path.

No security exposure. Public API changes are additive: the server-bootstrap-timeout input (validated positive, clamped to 120000ms with a warning, inputs.ts:363-373), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME re-exports from @fro-bot/runtime.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33475140166
Cache hit
Session ses_fa4791e85ffeewafG3oL989S7N

Review follow-ups.

The post hook built its save without the project-id path, so an archive that
landed only there omitted .git/opencode. That was harmless while the post
hook was an incidental fallback, but this work made it the documented reason
a declined save is acceptable, so the two call sites should agree. The path
derives from a runner-level variable available in the post process exactly
as the storage and auth paths already are.

Closing the acquired server on the bootstrap-failure path was the last
statement before returning an error, so a throw there would have rejected
from a function whose whole try/catch exists to return a result instead. It
is now best-effort, matching how this change already closes databases, and a
test drives a throwing close to prove the contract holds.

Also pins the usability probe against the object store rather than only the
cache. That source wins on restore and its own integrity checks cannot see
past the directory, so the probe is its only real defense there.

Refs #1407
fro-bot
fro-bot previously approved these changes Sep 1, 2026

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The four non-blocking items from the previous round are genuinely closed, and I checked each against the code rather than the summary:

  • post.ts derives projectIdPath. post.ts:97 builds it from getGitHubWorkspace(), symmetric with cleanup.ts:172, and post.test.ts:114 pins the exact value passed to saveCache. The reasoning in the comment is right: every other value at that call site (getOpenCodeStoragePath, getOpenCodeAuthPath) is already a runner-level environment read, so GITHUB_WORKSPACE needed no STATE handoff.
  • The close is guarded and the contract is tested. server.ts:109-113 wraps acquiredServer?.close(), and server.test.ts:258 is the case the previous round's spy could not express — it throws from close and asserts the promise still resolves a failed Result. That is the difference between proving the close happens and proving the never-throw contract survives it.
  • The object-store probe is pinned end-to-end. cache-restore.test.ts:623-691 seeds a corrupt blob under the real object key and drives restoreFromObjectStoresyncSessionsFromStorecheckpointDatabaseverifyDatabaseUsable through an in-memory adapter. Since the object store wins on restore and its own two integrity checks are near-tautologies for a corrupt opencode.db, this was the one source where the probe was the sole defense and nothing observed it.
  • The skipped-probe reasoning is recorded at cache-restore.ts:112-119, in the branch it applies to.

I re-verified the error-classification polarity that blocked the earlier round, since it guards the only destructive path in this change. isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist of not a database / malformed, consumed at checkpoint.ts:157, integrity.ts:68, and cache-restore.ts:109. verifyDatabaseUsable returns {usable: true} for any unrecognized throw (integrity.ts:71) — the correct fail-safe for a probe whose caller deletes session history. The classification is pinned against real node:sqlite faults, not mocked message strings: checkpoint.test.ts:197 (a permissions fault yielding unable to open database file, non-structural), integrity.test.ts:78 (a real database at restricted permissions reported usable), and cache-restore.test.ts:503 (an EXCLUSIVE-locked holder leaving storage intact and the run still a hit).

Gates run locally on this head: check-types clean across all four workspace packages; lint 0 errors; bun run build leaves dist/ byte-identical to the committed tree; 354 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/. I also independently confirmed the load-bearing SDK claim behind the new input: node_modules/@opencode-ai/sdk/dist/server.js:4-8 does Object.assign({hostname, port, timeout: 5000}, options), so passing DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS = 5000 explicitly is identical to today's unset behavior.

Blocking issues

None.

Non-blocking concerns

1. hasCacheableContent's doc comment now describes an unreachable state. src/services/cache/save.ts:38-44.

The comment says the checkpoint "can decline (a live writer, an unmergeable log); when it does, the WAL is where the real content still is," and concludes "We must treat any non-empty DB-family file as sufficient evidence of cacheable content." But saveCache returns false at :143 on status === 'failed', so hasCacheableContent at :149 is only ever reached after checkpointed or nothing-to-checkpoint — both of which leave the write-ahead log empty or absent. The WAL can no longer be the sole holder of content at that point.

The code is correct; the comment justifies the WAL branch with a scenario the control flow above it now forecloses. This matters more than a typical stale comment because two commits in this branch exist specifically to fix comments stating rules their own code does not follow, and checkpoint.test.ts:83 cites this comment by name as its regression guard. The header-page-only case the test actually protects is still real — it just arrives through checkpointed, not through a decline.

2. The decline block stays in the job summary even when the post-hook retry succeeds. src/services/cache/save.ts:90-103.

core.summary.write() appends by default (@actions/core/lib/summary.js:71-73 selects appendFile unless overwrite is set). So a run where cleanup declines and runPost then saves successfully still shows "Fro Bot Agent Run — Cache Save Declined" with "The next run may restore an older session," which by then is false. Given that the post-hook retry is the documented reason the decline tradeoff is acceptable (save.ts:83-88, pinned by cleanup-decline-retry.test.ts:71), the common recovery path is exactly the one that leaves a misleading summary. The decline being loud is deliberate and right; it just needs a way to be corrected, or wording that does not assert an outcome the retry usually reverses.

3. A persistently non-checkpointable log is a "never advances" stable state, which the restore-side repair does not reach.

The insight driving this fix is that declining to save is a stable state rather than a recovery, because restore keys are prefixes. The repair breaks that for a poisoned entry. It does not break it for a cache that simply stops advancing: if both cleanup's and the post hook's checkpoints fail non-structurally every run, no new entry is ever written, restore keeps repairing the same old healthy entry, and session continuity silently freezes at that snapshot while every run reports success. Mitigated by (2)'s summary firing every run, so it is loud rather than silent, and I do not think it warrants a fix here — but it is the mirror image of the argument this change is built on, and worth a line wherever the decline is documented.

4. action.yaml's literal default makes one input branch unreachable in production. action.yaml:89, src/harness/config/inputs.ts:363-367.

With default: '5000' declared, core.getInput('server-bootstrap-timeout') never returns an empty string in a real Action run, so the serverBootstrapTimeoutRaw.length > 0 ? ... : DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS fallback is only exercised by tests that omit the input. Harmless and arguably good defense-in-depth, but the two defaults must now be kept in agreement by hand — a 5000 in YAML and a DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS in TypeScript, with nothing asserting they match.

5. Object-store corruption is cleared locally but not at the source. Unchanged from the previous round and correctly scoped out: cleanStorage removes the runner's copy while the bucket still serves the same object, self-healing only on a run that reaches syncSessionsToStore. The Actions-cache side does not have this problem because restore keys advance.

6. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50, but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.

Missing tests

  • Nothing asserts the action.yaml default and DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS agree. inputs.test.ts:905 pins the constant's value through the unset-input path, but the YAML literal that actually supplies it in production is unobserved. This repo already has workflow/YAML guard tests (scripts/fro-bot-workflow.test.ts), so the shape exists; a drift here would silently change every run's bootstrap budget while every test still passed.
  • No test observes saveCache's decline path reaching hasCacheableContent — because it cannot, which is concern (1). A test asserting that a failed checkpoint returns before buildSaveCachePaths is ever called would pin the ordering the comment currently mis-describes, and would fail if a future change reintroduced the decline-then-inspect path the comment imagines.

Coverage is otherwise excellent and the strongest of the three rounds. The end-to-end suite at cache-restore.test.ts:394-592 exercises real databases for all four states the phase distinguishes — hot WAL, malformed-with-WAL, malformed-without-WAL, and EXCLUSIVE-locked — using the real checkpointDatabase / cleanStorage / verifyDatabaseUsable rather than canned outcomes, with an explicit expect(mocks.loggerWarning).not.toHaveBeenCalled() guarding the silent common path at :560. cleanup-decline-retry.test.ts remains the best structural addition: running runCleanup and runPost against one shared state store is the only way to assert the relationship the decline comment documents.

Risk assessment (LOW-MED)

Blast radius is every run — the checkpoint is unconditional on both the restore and save paths, and the usability probe is unconditional on the healthy restore path. What keeps this at the low end is that the destructive branch is no longer reachable by default: it requires SQLite to positively report not a database or malformed, a claim about the file's contents rather than about the runner's state, and the transient faults that would previously have triggered it (fd exhaustion, a full disk, an I/O error, a read-only mount) are verified non-structural against real node:sqlite rather than asserted.

Remaining mitigations are real and I confirmed each: the probe reads a single schema page and never opens a missing or zero-byte file (integrity.ts:57-60), both the probe and the checkpoint sit outside the createOpencode budget (pinned by the ordering assertion at cache-restore.test.ts:360), the repair is bounded by an explicit wall-clock deadline rather than sleep count (checkpoint.ts:128-137), checkpointDatabase never throws, cleanStorage failure is swallowed, and runPost's retry is reachable on the bootstrap-failure path because SHOULD_SAVE_CACHE is set in routing.ts:67, well before runCacheRestore.

The residual concentrates in concern (3): a repository whose OpenCode child consistently outlives both checkpoint windows stops advancing its cache while every run still reports success. Recoverable and loud, but the signal it produces is the one concern (2) makes unreliable.

No security exposure. Public API changes are additive: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms with a warning at inputs.ts:368-373), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME re-exports. ActionInputs.serverBootstrapTimeoutMs is a new required field on a @fro-bot/runtime interface, but that package is private, so no external consumer can break on it.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33483168741
Cache hit
Session ses_fa4152b4bffeXKpMdqOuiqhD6z

Review follow-ups.

The cacheable-content note justified inspecting the write-ahead log with a
case its own control flow forecloses: a declined checkpoint returns before
that check is reached, so the log there is always already merged or empty.
The header-page-only session it protects is real but arrives the other way,
and a test cited the old wording as its guard.

The decline summary claimed the next run may restore an older session, but
it is appended, not replaced, so it survived the post-hook retry that
usually saves the cache moments later and left the run asserting something
untrue. It now states only what holds when it is written.

Six action inputs declare a default in YAML that a constant repeats in
TypeScript, agreeing by hand. A guard now compares them, so a drift fails
loudly instead of quietly changing every run.

Refs #1407
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

All three closed.

The stale comment was the one worth catching. It justified inspecting the write-ahead log with a case the control flow above it forecloses — a declined checkpoint returns before that check is ever reached. The header-page-only session it protects is real, but it arrives through checkpointed, not through a decline, and checkpoint.test.ts cited the old wording as its guard. That is the third comment on this branch stating a rule its own code does not follow, which is starting to look less like carelessness and more like what happens when a comment is written before the control flow around it settles.

The summary was asserting something untrue on the recovery path. Confirmed appendFile unless overwrite, so the decline block survived the post-hook retry that usually saves moments later. Took the reword rather than a resolution note from the post hook — that would need post.ts to know a decline happened, and a second thing to keep synchronized for a cosmetic fix. Worth noting the lane's first draft replaced one false claim with another, promising a follow-up entry that post.ts never writes; it caught that itself by reading post.ts rather than assuming.

The defaults guard found six pairs, not one. session-retention, s3-prefix, timeout, server-bootstrap-timeout, dedup-window, and omo-slim-preset all declare a default in YAML that a TypeScript constant repeats. The remaining inputs have no backing constant to drift against and are excluded deliberately.

Separately, and outside this PR: scripts/build-action-dist.test.ts writes to the real dist/ rather than a temp directory, so a routine bun run test silently dirties the committed tree. I reproduced it in isolation — that one file alone leaves four files changed. It is pre-existing and not in this diff, and it explains a dist/ drift earlier in this branch that I had wrongly attributed to formatting. Filing it separately.

Gates: types 0, lint 0, dist/ reproducible. packages/harness/src/conflict-resolver.test.ts fails on this machine independent of the branch (3/3 on a pristine origin/main worktree); CI is the signal that counts.

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CONDITIONAL

The three items from the previous round are genuinely closed, and I checked each rather than reading the summary. The stale hasCacheableContent comment now derives the WAL's emptiness from the control flow above it (save.ts:38-51) instead of justifying it with a decline path that save.ts:151-157 forecloses. The decline summary's wording is now conditional (save.ts:110) with the appendFile reasoning recorded inline, which is the right call over a second synchronized write from post.ts. And scripts/action-input-defaults-guard.test.ts covers six YAML/TypeScript default pairs rather than the one I flagged, parses the YAML string back to a number so '5000.0' cannot pass, and names both locations in its failure message. It runs in CI (ci.yaml:143, bun run test:scripts), so the guard is live rather than orphaned.

Gates on this head: check-types clean across all four workspace packages; bun run build leaves dist/ byte-identical to the committed tree; 481 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, src/harness/config/inputs.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/.

The error-classification polarity that blocked round four remains correct: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:157, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable defaulting to usable: true for any unrecognized throw.

What I cannot pass is a new consequence of this change on the object-store path. I found it by asking what happens to the write-ahead log object now that healthy saves stop producing a local one — the same question deleteRestoredShm already answers for -shm.

Blocking issues

1. The checkpoint orphans opencode.db-wal in the object store, and the orphan reproduces this PR's own trap — permanently — for S3-backed repositories. packages/runtime/src/object-store/content-sync.ts:71-77, src/services/cache/save.ts:147-159, src/harness/phases/cache-restore.ts:101-110.

syncSessionsToStore uploads a transportable file only when it exists locally (content-sync.ts:73-77); when it does not, it continues. It never deletes. ObjectStoreAdapter (object-store/types.ts:7-9) exposes only upload/download/list, so nothing on this path can remove a key.

Before this change the local WAL was effectively always present at save time — that is the PR's own central premise, that nothing ever checkpointed — so the opencode.db-wal object was refreshed on every upload alongside opencode.db. After this change a healthy save checkpoints, checkpointDatabase closes its handle in finally, and SQLite unlinks the log. The author's own comments state this outcome (save.ts:143-146, paths.ts:66-69). So opencode.db-wal stops being uploaded while opencode.db keeps being overwritten at the same prefix.

syncSessionsFromStore then downloads everything list returns, unfiltered (content-sync.ts:128-149), so the next restore lands a fresh opencode.db beside a write-ahead log from an older database state. SQLite's contract is explicit that these must not be combined. I reproduced the exact checkpointDatabase sequence against real node:sqlite on this runner:

state N:   WAL with 2000 inserts, snapshot the -wal        (4120032 bytes)
close  ->  SQLite unlinks -wal                              wal exists? false
state N+1: 3000 more inserts + a DELETE, close cleanly      count = 3334
restore shape: fresh opencode.db  +  stale opencode.db-wal from N

new DatabaseSync(db); prepare('PRAGMA wal_checkpoint(TRUNCATE)').get()
  -> throws "database disk image is malformed"
  -> isStructuralCorruptionError  =>  true

That routes into handleStructuralCorruption, which calls cleanStorage and deletes the repository's entire session history. The run then saves a fresh opencode.db — and because the local WAL is again absent, the orphaned object in the bucket is still not overwritten. The next run downloads the same stale log against the new database and wipes again.

This is a stable failure state, not a one-time migration cost: exactly the "declining forever is a stable state, not a recovery" argument this PR is built on, reached through the object store instead of the Actions cache. It fires on the second run after upgrade for every repository with s3-backup: true and an existing opencode.db-wal object, which the PR's own premise makes near-universal for that population. The Actions-cache path is unaffected — an entry is an atomic archive per key, so its db and wal always come from the same save.

The -shm half of this same problem was recognized: deleteRestoredShm (restore.ts:81-93) deletes the locally-restored copy of an object that upload no longer refreshes, and content-sync.ts:13-14 explicitly notes download stays tolerant "so objects uploaded before this change still work." The -wal is the same orphan shape with the opposite consequence — it is not safely ignorable, it is actively replayed.

The narrowest fix that needs no interface change: after a successful checkpoint in saveCache, recreate opencode.db-wal as a zero-byte file before buildSaveCachePaths and syncSessionsToStore run. SQLite treats a zero-length -wal as no log, both transports then carry an explicitly-empty one, and the stale object is overwritten rather than left to accumulate. It costs the paths.ts:66-69 comment its "ordinarily absent" claim, which would need updating. Alternatives: add a delete to ObjectStoreAdapter (the S3 adapter already implements a conditional delete) and remove the remote key when the local file is absent; or have syncSessionsFromStore skip a -wal older than the opencode.db object, which needs list to carry LastModified.

Worth noting alongside: the usability probe runs only in the nothing-to-checkpoint branch, so a mixed-WAL database whose checkpoint happens to succeed while leaving the file inconsistent reaches bootstrap unprobed. That is acceptable on its own — re-probing a database the checkpoint just opened adds little — but it means a silent-corruption variant of this scenario has no second line of defense either.

Non-blocking concerns

1. post.ts reports every declined save as "no cache content to save". src/harness/post.ts:113. saveCache returns false for four distinct reasons — checkpoint declined, no cacheable content, cacheId === -1, or a caught exception — and the post hook's log attributes all of them to the second. Not silent (the decline writes its own warning and job-summary block), but this is precisely the "say what is true at the point each message is written" defect the last commit on this branch exists to fix, one call site removed.

2. A persistently non-checkpointable log is a "never advances" stable state. Unchanged from the previous round and correctly scoped out. If both cleanup's and the post hook's checkpoints fail non-structurally every run, no new entry is written, restore keeps repairing the same older healthy entry, and session continuity freezes while every run reports success. Loud rather than silent, thanks to the decline summary.

3. Object-store corruption is cleared locally but not at the source. cleanStorage removes the runner's copy while the bucket still serves the same object. This is the same missing-delete gap as the blocking issue and would likely be closed by the same fix, which is an argument for the adapter-delete variant over the zero-byte one.

4. Release notes. session-retention now actually prunes (a no-op at the default — DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50 — but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.

Missing tests

  • Nothing observes what happens to a previously-uploaded opencode.db-wal object once the local file stops existing. content-sync.test.ts:74-88 asserts that one file is uploaded when the WAL and SHM are absent, which is the upload count — it says nothing about the key that was already in the store. A test seeding both keys, running syncSessionsToStore with only opencode.db on disk, and asserting the store no longer serves a stale -wal would pin the blocking issue's fix.
  • No end-to-end restore case with a stale -wal object. The object-store suite at cache-restore.test.ts:623-691 seeds only .../sessions/opencode.db (its own comment notes "nothing else, no write-ahead log"), so the mixed fresh-db/stale-wal shape is invisible to the suite. Seeding both keys from different database states through the same in-memory adapter would reproduce the wipe loop directly.
  • No test pins that a healthy object-store round trip leaves the store's key set consistent across two saves. The single-run tests cannot see divergence that only appears on the second run.

Coverage is otherwise the strongest of the four rounds. The end-to-end suite exercises real databases for all four states the restore phase distinguishes using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable, the classification is pinned against real node:sqlite faults rather than mocked message strings (checkpoint.test.ts, integrity.test.ts), cleanup-decline-retry.test.ts runs runCleanup and runPost against one shared state store, and the new defaults guard closes a drift class that every existing unit test was structurally blind to.

Risk assessment (MED)

Blast radius is every run for the checkpoint and probe themselves, which remain well-mitigated: the destructive branch requires SQLite to positively report not a database or malformed, transient runner faults are verified non-structural against real node:sqlite, the probe reads one schema page and never opens a missing or zero-byte file, both sit outside the createOpencode budget, checkpointDatabase never throws, and cleanStorage failure is swallowed.

The residual is concentrated entirely in the blocking issue, and it is worse than the failure it replaces because the destructive branch is now reachable through a condition this change itself creates rather than through genuine on-disk corruption. Scope is limited to repositories running with s3-backup: true, but within that population it is not a rare corner: it is the steady state from the second post-upgrade run onward, it destroys session history every run, and the run still reports success. Repositories without the object store are unaffected.

No security exposure. Public API changes are additive: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms with a warning at inputs.ts:368-373, and its YAML default now guarded), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. ActionInputs.serverBootstrapTimeoutMs is a new required field on a @fro-bot/runtime interface, but that package is private.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33488201787
Cache hit
Session ses_fa3dd8ffeffe5l7TF8xcb01Qex

@fro-bot fro-bot Bot mentioned this pull request Sep 1, 2026
74 tasks
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

Correction to my previous comment: the claim that scripts/build-action-dist.test.ts writes to the real dist/ is wrong, and I should not have stated it as fact.

I re-ran it in isolation against a clean tree and dist/ stayed clean. Reading the test, it drives runBuildOrchestration through fully injected step callbacks and never spawns a bundle. A search across every *.test.ts in the repo finds no test that invokes a real build — src/main.test.ts only reads dist/main.js.

What I actually observed was four dirty files in dist/ after a test run, at a moment when the committed bundle predated uncommitted source changes. I inferred causation from that sequence without isolating it, then reported it as reproduced. That is the same mistake this branch has now corrected three times in comments — asserting a mechanism from a correlation rather than a test — and I made it in a comment about that very pattern.

The underlying question of what dirtied dist/ is unresolved and may be nothing more than my own build ordering. No issue filed; there is no evidence to file one on.

A successful checkpoint did not mean what the code assumed. shutdown()
signalled the OpenCode child and returned without awaiting its exit, so a
still-live connection could append to the write-ahead log immediately
after the checkpoint reported success:

  wal before checkpoint  2068272
  checkpoint returns {busy:0, log:0, checkpointed:0}, wal -> 0
  one late INSERT from an idle live writer            wal -> 4152

That is the same root cause as the bug this branch fixes: nothing ever
checkpointed *because* shutdown never waited.

The SDK exposes no way to await the child. createOpencode returns only
{client, server: {url, close}}, and close() calls proc.kill() on a handle
it never surfaces -- no pid, no exit event. So shutdown() now polls the
server's own port until connections are refused, which happens once the
OS reclaims the socket. The pinned server registers no SIGTERM handler,
so termination is immediate rather than drained. This is a boundary, not
a proof, and both the code and its comment say so: a timeout means the
child's fate is unknown, and cleanup warns that the checkpoint which
follows may race a writer.

With the writer actually gone, the log is empty on every path that
reaches transport, so neither transport carries it. Removing it from
DB_TRANSPORTABLE_BASENAMES only covered the object store; the Actions
cache added it independently in buildSaveCachePaths, and both had to
change.

That also fixes a regression this branch introduced. Uploads skip missing
files and never delete, so once healthy saves stopped producing a local
log, opencode.db-wal stopped being refreshed while opencode.db kept being
overwritten at the same prefix. A later restore then paired a fresh
database with a log from an older generation -- reproduced twice, once
surfacing as "database disk image is malformed" (which this branch
answers by deleting session history, then re-saving without overwriting
the orphan, so it repeats every run) and once far worse: the checkpoint
succeeded and silently replayed the stale log, turning 3000 rows back
into 2000 with no error at all.

The two sources need opposite treatment. An Actions cache entry is one
atomic archive, so a legacy pair is consistent and its log may hold
committed transactions -- it is checkpointed, never discarded. The object
store has no generation marker, and the silent-replay case proves SQLite
cannot reliably tell a stale pair from a valid one, so a downloaded log
is untrusted and deleted before anything opens the database. That runs
unconditionally after the sync, because an object store holding only a
sidecar can fall through to a cache miss and leave the file on disk.

Discarding an untrusted log can lose that generation's last
transactions. That is a real cost, recorded in the code, and still
better than silently rolling a session back.

Refs #1407
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

Blocking issue fixed at bf5839263, but not the way I first proposed — my fix was wrong and an independent review caught it before it shipped.

I argued the log was provably empty on every path reaching transport, so it could simply be dropped. That invariant is false, and I verified the counterexample rather than arguing:

wal before checkpoint  2068272
checkpoint returns {busy:0, log:0, checkpointed:0}, wal -> 0    (reported SUCCESS)
one late INSERT from a still-live idle writer            wal -> 4152

shutdown() signalled the child and returned without awaiting exit, so a live connection could append after the checkpoint reported success. SQLite's checkpoint writer lock applies while the checkpoint is pending, not after it returns. This is the same root cause as the bug the PR fixes: nothing ever checkpointed because shutdown never waited. Every option layered on a non-quiescent writer inherits that race, including the zero-byte-WAL suggestion, which would additionally be racing a live connection while writing.

So the fix goes at the root. shutdown() now polls the server's port until connections are refused. Confirmed against the pinned clone that the server registers no SIGTERM or SIGINT handler, so kill() terminates immediately rather than draining, which is what makes socket release a usable proxy for process death. Also confirmed the SDK genuinely exposes nothing better — it holds proc with an internal exit listener and never surfaces it. It is a boundary, not a proof, and the code says so: a timeout means the child's fate is unknown, and cleanup warns that the checkpoint may race a writer.

With the writer actually gone, dropping the log from transport is sound. You were right that this needed both call sites — DB_TRANSPORTABLE_BASENAMES covers only the object store, and buildSaveCachePaths added it independently for the Actions cache.

On legacy handling, I took the source-specific split rather than one rule: an Actions cache entry is atomic, so a legacy pair is consistent and its log may hold committed transactions — it is checkpointed, never discarded. The object store has no generation marker, and the silent-replay case proves SQLite cannot reliably distinguish stale from valid, so a downloaded log is untrusted and deleted before anything opens the database. That deletion is unconditional after the sync, covering the sidecar-only path that falls through to a cache miss.

Skipped the adapter delete deliberately. It does not close the non-atomicity window — a crash between upload and delete recreates the same hazard — whereas reducing the transported set to one file closes it permanently. Stale objects are one fixed key per prefix, not unbounded.

Recorded in the code that discarding an untrusted log can lose that generation's last transactions. It is a real cost, and better than silent rollback.

Gates: types 0, lint 0, dist/ rebuilt. Two pre-existing failures on this machine, both confirmed against a pristine origin/main worktree: conflict-resolver.test.ts deterministically 3/3, evals/runner.test.ts flaky 1/3.

fro-bot
fro-bot previously approved these changes Sep 2, 2026

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The blocking issue from the previous round is closed, and — as in round four — the author's fix is better than the one I proposed. I suggested recreating a zero-byte opencode.db-wal so the stale object would be overwritten. That suggestion was wrong for the reason the author demonstrated rather than argued: writing a zero-byte log while a live connection can still append races the same writer that made the checkpoint untrustworthy in the first place. The counterexample they reproduced (wal 2068272 → checkpoint reports success, wal → 0 → one late INSERT → wal 4152) invalidates the invariant my fix depended on, and it is the same root cause as the bug this PR exists to fix: nothing ever checkpointed because shutdown() never waited.

So the fix went to the root instead. I verified each of the four moving parts:

  • shutdown() now waits. waitForServerQuiescence (server.ts:57-84) polls the child's own listening port until connections are refused. The doc comment is honest about what that is — server.ts:50-55 explicitly says a timeout means the child's fate is unknown, not that it is running, and that a window remains even on quiesced: true. cleanup.ts:119-132 warns on quiesced: false and proceeds rather than failing. Calling this "a boundary, not a proof" in the code is the right register.
  • The write-ahead log left transport entirely, at both call sites. DB_TRANSPORTABLE_BASENAMES (version.ts:51) is now [opencode.db] for the object store, and buildSaveCachePaths (paths.ts:119-133) pushes only the main database for the Actions cache. I checked this was needed at both — the two lists are genuinely independent, and fixing only one would have left the Actions-cache half.
  • Download stays tolerant; restore deletes. deleteDownloadedObjectStoreWal (restore.ts:124-139) removes a downloaded log before anything opens the database, and runs regardless of mainDbRestored — which matters for the sidecar-only path that falls through to a cache miss, and is pinned by cache.test.ts:299.
  • The source-specific split is correct rather than convenient. An Actions-cache entry is one atomic archive, so a legacy db+wal pair genuinely came from the same save and is checkpointed at cache-restore.ts:101, never discarded. The object store has no generation marker across two independently-overwritten keys, so its log is untrusted and deleted. Treating the two transports differently is the correct reading of the difference between them, not an inconsistency.

Declining the adapter delete is also right, and for the stated reason: a delete does not close the non-atomicity window (a crash between upload and delete recreates it), whereas reducing the transported set to one file closes it permanently.

I also want to record something that is easy to miss in the diff. Dropping the log from the Actions cache is not merely neutral — it is safer than what it replaces. In WAL mode the main database file is only mutated during a checkpoint, so archiving opencode.db alone beside a still-appending writer yields a consistent, slightly-stale database. The previous behavior copied a log that a live writer was concurrently appending to, which is the torn-read shape that produces "database disk image is malformed". The cost is bounded and documented; the hazard removed is not.

Gates on this head: check-types clean across all four workspace packages; bun run build leaves dist/ byte-identical to the committed tree; 488 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, src/harness/config/inputs.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/.

The error-classification polarity that blocked round four still holds: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:163, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable defaulting to usable: true for any unrecognized throw.

Blocking issues

None.

Non-blocking concerns

1. The untrusted-log deletion is inside the try, not a finally, and its own comment claims otherwise. src/services/cache/restore.ts:154-165.

The comment at :115-116 states the deletion "Runs unconditionally right after syncSessionsFromStore returns." It runs on the non-throwing path only. syncSessionsFromStore calls fs.mkdir per key inside its download loop (content-sync.ts:148), which is a real throw site under ENOSPC/EACCES. Object keys are listed lexicographically, so opencode.db-wal sorts before any sessions/… sidecar — a throw while creating a directory for a later key leaves the downloaded log on disk, skips the deletion, and lands in the catch at :216, which returns a miss. restoreCache then proceeds to the Actions-cache restore at :276, which can extract a database beside that orphaned log — reconstructing exactly the fresh-db/stale-wal pairing this commit exists to prevent, including its silent-replay variant.

Doubly conditional and genuinely unlikely, which is why it is not blocking. But moving the call into a finally (or hoisting it into restoreCache ahead of the cache restore) is a one-line change that makes the comment true, and this is the fourth comment on this branch to state a rule its own control flow does not follow.

2. isPortOpen sets no socket timeout, so the "bounded" wait is not enforced. packages/runtime/src/agent/server.ts:28-39.

net.connect is awaited with no setTimeout and no deadline of its own. If a connect attempt never emits connect or error, the do/while never advances and waitForServerQuiescence never resolves — and cleanup.ts:121 awaits shutdown() with no outer bound, so the job would hang to the runner's own timeout. On loopback a connect resolves or is refused immediately, so reachability is close to nil and I am recording this rather than objecting. But the function's contract is stated as bounded, and the bound currently lives entirely in an assumption about the OS rather than in the code. A socket.setTimeout(pollIntervalMs, () => finish(false)) would put it there.

Related and smaller: port = Number(parsed.port) yields 0 for a URL with no explicit port, and a connect to port 0 errors immediately — reporting quiesced: true without having checked anything. Unreachable today, since bootstrapOpenCodeServer pins an explicit port and rejects a URL mismatch at :163-171.

3. A second, now-divergent OpenCodeServerHandle still exists. src/features/agent/server.ts:5-9.

It declares shutdown: () => void, while the real handle returned by bootstrapOpenCodeServer is () => Promise<ShutdownResult>. src/features/agent/execution.ts:3 imports this local copy, and index.ts:37 re-exports the runtime one — so the two disagree in the same module tree. No type error results (a Promise-returning function is assignable to a void-returning one), which is precisely why it is worth naming: this is the drift shape that version.ts:8-20 and paths.ts:75-81 both go out of their way to prevent for the DB filename lists, left standing for the handle type this PR just changed. execution.ts only uses server.close(), so nothing is broken today.

4. Post-checkpoint writes are now dropped rather than transported. When shutdown() reports quiesced: false and the child appends after a successful checkpoint, those transactions reach neither transport. paths.ts:68-73 documents the choice and it is the right one — the alternative is the torn-pair hazard — but it is a real cost, and it is silent apart from the quiesced: false warning.

5. post.ts:123 still reports every declined save as "no cache content to save". Carried unchanged from the previous round. saveCache returns false for four distinct reasons and the post hook attributes all of them to one. Not silent — the decline writes its own warning and summary block — but it is the "say what is true at the point each message is written" defect one call site removed.

6. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50, but surprising for anyone who set it low while it was inert). Neither opencode.db-shm nor opencode.db-wal is captured for transport any more.

Missing tests

  • Nothing exercises the throw path in restoreFromObjectStore that skips the log deletion. cache.test.ts:256 and :299 cover the sidecar-only cases well, but both are on the non-throwing path. An adapter or fs stub that throws during the download loop, asserting no write-ahead log survives into the Actions-cache restore, would pin non-blocking (1) and would fail today.
  • No test asserts the quiesced: false warning in cleanup. waitForServerQuiescence itself is covered in all four states (server.test.ts:335-389), including the timeout and the unparseable-URL case, which is good. But cleanup.test.ts:47 mocks shutdown as {quiesced: true} everywhere, so the branch at cleanup.ts:122-126 — the one that tells an operator their checkpoint may have raced a writer — is never observed.
  • No test covers a legacy Actions-cache db+wal pair surviving the round trip with its committed transactions intact. This is the case the source-specific split deliberately preserves rather than discards, and it is the one half of that decision nothing pins. cache-restore.test.ts:444 checkpoints a hot-WAL database, but from a locally-seeded fixture rather than framed as a legacy restore whose data must not be lost.

Coverage is otherwise the strongest of the five rounds. restore-save-flow.test.ts:344 and content-sync.test.ts:53 pin the exclusion at both transports independently, cache-restore.test.ts:693 drives the fresh-db/stale-wal pairing end-to-end through a real object-store adapter, and the classification remains pinned against real node:sqlite faults rather than mocked message strings.

Risk assessment (LOW-MED)

Blast radius is every run: the checkpoint is unconditional on both paths, the usability probe is unconditional on the healthy restore path, and shutdown() now adds up to 5s of cleanup wait when the child does not exit promptly — off the critical path any run is timed against, and correctly distinguished from the bootstrap budget at constants.ts:29-31.

What holds this at the low end is that the destructive branch still requires SQLite to positively report not a database or malformed, and that this round's change reduces the corruption surface rather than trading one hazard for another: the fresh-db/stale-wal pairing is now unreachable by construction on the object-store path, and the torn-WAL-copy shape is unreachable on the Actions-cache path. The residual concentrates in non-blocking (1), which requires a filesystem fault during an object-store download and a subsequent Actions-cache hit to reach — and even then produces the same recoverable wipe this design already handles, with the silent-replay variant as the worse tail.

No security exposure. Public API changes are additive or internal: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms, YAML default guarded by scripts/action-input-defaults-guard.test.ts across six pairs), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. OpenCodeServerHandle.shutdown changing from () => void to () => Promise<ShutdownResult> is a breaking signature change on a @fro-bot/runtime interface, but that package is private and both in-repo implementors (server.ts, remote-client.ts) and all call sites are updated.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33594171130
Cache hit
Session ses_f9f6fa26dffeI1RCsc7BGA7g4E

Three follow-ups from review, all of which were true of the code and not
of what its comments claimed.

The object-store write-ahead log deletion sat inside the try, while its
comment said it ran unconditionally after the download. syncSessionsFromStore
calls fs.mkdir per key, a real throw site under ENOSPC/EACCES, and object
keys list lexicographically so opencode.db-wal arrives before any sessions
sidecar. A throw partway through left the log on disk, skipped the
deletion, returned a miss, and let the Actions-cache restore extract a
database beside it -- rebuilding the exact fresh-db/stale-log pairing the
previous commit removed, silent-replay variant included.

It now runs in a finally. Hoisting the call up into restoreCache was the
other option and would have been wrong: the store-disabled guard returns
before the try, so a locally-present log on a persistent self-hosted
runner has nothing to do with the object store and must not be deleted.
The finally covers every exit from the download attempt and none of the
paths where no download happened.

isPortOpen settled only on connect or error, so the bound on the
quiescence wait lived in an assumption about loopback rather than in the
code -- if neither event fired, cleanup awaited it forever. It now has a
socket timeout of one poll interval and a settled guard so teardown
resolves exactly once. A port that parses to 0 or out of range now
reports unconfirmed instead of reporting success without having checked
anything.

The duplicate OpenCodeServerHandle in features/agent still declared
shutdown as returning void after the runtime's began returning a promise.
No type error, because a promise-returning function is assignable to a
void-returning one, which is exactly why it was worth removing -- the
same drift the DB filename lists have guards against. One definition now,
imported through the existing sibling re-export, so no new cross-layer
import.

The throw-path test was checked against the old shape and fails there:
the log survived the download failure. Also pins the quiesced:false
warning branch, and a legacy Actions-cache pair surviving a round trip
with its transactions intact -- the half of the source-specific split
that preserves data rather than discarding it.

Refs #1407
@marcusrbrown

Copy link
Copy Markdown
Collaborator Author

All three closed at 4f5703765.

Item 1 rejected my own suggested shape, correctly. I had offered hoisting the deletion into restoreCache as an alternative to a finally. That would have been a bug: the store-disabled guard returns before the try, so a locally-present opencode.db-wal at that point has nothing to do with the object store — on a persistent self-hosted runner it is live session data, and hoisting would have deleted it. The finally covers every exit from the download attempt and none of the paths where no download occurred, which is what makes the comment true rather than merely re-worded.

The throw-path test was checked against the old shape and fails there — expected fs.access(walPath) to reject, but it resolved — so the fix is proven rather than asserted.

Item 2: isPortOpen now carries a socket timeout of one poll interval plus a settled guard, so teardown resolves exactly once and the bound lives in the code instead of in an assumption about loopback. A port parsing to 0 or out of range reports unconfirmed rather than reporting success without having checked anything.

Item 3: one OpenCodeServerHandle definition, imported via the existing sibling re-export inside features/agent/ — no new cross-layer import, so the layer rule was never implicated.

Also pinned the quiesced: false warning branch and a legacy Actions-cache pair surviving a round trip with its transactions intact — the half of the source-specific split that preserves data rather than discarding it, which nothing covered before.

On your note about the Actions cache: agreed, and it is the sharper framing. Dropping the log there is not neutral, it is strictly safer — archiving opencode.db alone beside a live writer yields a consistent slightly-stale database, where copying a concurrently-appended log is the torn-read shape that produces malformed in the first place.

Items 4, 5, and 6 I am leaving. The post-checkpoint write loss is real but is the correct trade against the torn-pair hazard and is already documented at the decision point; post.ts's coarse decline message is worth fixing but belongs with the broader decline-reason plumbing rather than bolted on here; the release-note items are noted for the changelog.

Gates: types 0, lint 0, dist/ rebuilt and reproducible. operator-route.test.ts joins the two known local flakes — confirmed environmental with the implementation fully stashed.

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

The three items from the previous round are closed, and each one is closed in a shape better than the fix I would have written. The untrusted-log deletion is now in a finally around the whole object-store attempt (restore.ts:224-226), and the author was right to reject my own alternative of hoisting it into restoreCache — the store-disabled guard returns before the try, so a hoisted call would have deleted a locally-persistent write-ahead log on a self-hosted runner with the object store turned off. That reasoning is now recorded at restore.ts:114-129, in the branch it constrains. restore-save-flow.test.ts:483 pins it by throwing from a later key in the download loop and asserting the wal does not survive into the Actions-cache extract — the property, not the call.

isPortOpen now carries a socket timeout and a settled guard (server.ts:36-51), and the port-range check at server.ts:93-95 turns an unreachable invariant into an enforced one. The duplicate OpenCodeServerHandle is gone; src/features/agent/server.ts:4-9 leaves a comment naming the drift instead of a second definition, which is the right residue.

Gates on this head (4f57037): check-types clean across all four workspace packages; 372 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/; bun run build leaves dist/ byte-identical to the committed tree (git status --short dist/ empty after a full rebuild).

The error-classification polarity that blocked round four still holds: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:163, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable returning usable: true for any unrecognized throw.

Blocking issues

None.

Non-blocking concerns

1. The new socket timeout maps "unsettled" to "the child is gone", inverting the safe default of the probe it bounds. packages/runtime/src/agent/server.ts:47.

socket.setTimeout(timeoutMs, () => finish(false)) — and finish(false) means "port closed", which waitForServerQuiescence:100-102 reads as quiesced: true and returns immediately. The comment two lines above names the exact case the timeout exists for ("a firewalled port, a host that silently drops SYNs"), and in that case the function now reports the child has exited rather than that it could not tell. Everything else in this change picks the opposite default: verifyDatabaseUsable returns usable: true for an unrecognized throw, isStructuralCorruptionError returns false unless SQLite positively says otherwise. Here an unknown answer resolves to the optimistic one, and the optimistic one is what suppresses the quiesced: false warning at cleanup.ts:122-126 and the "may have raced a writer" caveat around the checkpoint that follows.

finish(true) on timeout would be consistent: an unsettled connect means "still open as far as I can tell", the do/while keeps polling, and the outer 5s deadline produces the honest quiesced: false. Reachability on loopback is close to nil — a connect either completes or is refused — so this is a polarity nit rather than a live bug, but it is the one place in this design where "I don't know" resolves to "it's fine".

Smaller, same function: finish calls socket.removeAllListeners() before socket.destroy(), which drops the error handler on a socket that may still be mid-connect. Node's afterConnect early-returns for a destroyed socket, so an unhandled 'error' should not escape — but keeping the error listener attached (or destroying first) costs nothing and does not depend on that internal.

2. A failed object-store download can leave a truncated opencode.db that the new probe is gated out of seeing. packages/runtime/src/object-store/s3-adapter.ts:183, src/services/cache/restore.ts:214-219, src/harness/phases/cache-restore.ts:67.

download pipes the response body straight into createWriteStream(localPath). A mid-transfer failure (an aborted body stream, a reset connection) leaves a partial file on disk and returns err, so mainDbRestored stays false and restoreFromObjectStore returns a miss. The finally deletes the downloaded write-ahead log — correctly — but nothing removes the partial main database. restoreCache then falls through to the Actions cache; if that also misses (the normal case for a repository leaning on S3), cacheStatus is 'miss', the repair block at cache-restore.ts:67 is skipped entirely, and a truncated opencode.db reaches bootstrapOpenCodeServer unprobed.

The blast radius is bounded — the run fails, and the next run recovers either from the intact bucket object or via the Actions-cache hit that does run the probe — so this is not the permanent trap the PR set out to break. But it is the same untrusted-artifact shape deleteDownloadedObjectStoreWal already handles for the sidecar, left open for the file that actually matters: a download attempt that did not produce a usable restore should not leave its main database behind either. Deleting it when mainDbRestored === false && failed > 0 is symmetric with the log deletion and needs no interface change.

3. "Archiving opencode.db alone beside a live writer yields a consistent database" is conditional on wal_autocheckpoint. This framing (mine, from the previous round, and agreed in the author's reply) holds only while nothing checkpoints during the archive. SQLite's default wal_autocheckpoint is 1000 pages, so a still-live child that appends ~4 MB while @actions/cache or syncSessionsToStore is reading the file will mutate the main database mid-read. It requires quiesced: false plus a genuinely busy writer, and the previous behavior (copying a concurrently-appended log) was strictly worse — so this changes nothing about the decision, only about how strongly the safety claim should be stated in the residual-risk story.

4. post.ts:123 still reports every declined save as "no cache content to save". Carried unchanged and explicitly deferred by the author to the broader decline-reason plumbing. Noted, not re-argued.

5. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50), and neither opencode.db-shm nor opencode.db-wal is captured for transport any more.

Missing tests

  • Nothing exercises isPortOpen's socket-timeout branch — the branch this commit added, and the one that decides concern (1)'s polarity. All four waitForServerQuiescence cases (server.test.ts:334-389) use a real listener that either accepts or refuses, so the timeout path is never entered. A case pointing at a black-holed address (e.g. http://10.255.255.1:9/) with a short budget would pin the intended outcome; if that is too environment-dependent for CI, extracting the timeout mapping so it can be asserted directly is the cheaper alternative.
  • No test covers a mid-transfer download failure leaving a partial opencode.db. content-sync.test.ts covers a rejected key and a failed download by return value, but nothing asserts what remains on disk afterward, and cache-restore.test.ts's object-store cases all reach the probe through a hit. A test seeding a download that writes bytes and then returns err would make concern (2) visible; today it is invisible to the suite.
  • No test pins the object-store upload path's behavior when the checkpoint declined. save.ts:152-158 returns before syncSessionsToStore, so a declined save uploads nothing — worth pinning, since the whole "the bucket self-heals on the next successful save" argument in the design notes depends on it.

Coverage is otherwise the strongest of the six rounds. cache.test.ts:913 now covers the half of the source-specific split that preserves data (a legacy Actions-cache db+wal pair surviving with its committed transaction intact), cleanup.test.ts:317 pins the quiesced: false warning branch, restore-save-flow.test.ts:483 proves the finally fix against the shape that would have failed before it, and the classification remains pinned against real node:sqlite faults rather than mocked message strings.

Risk assessment (LOW-MED)

Blast radius is every run — the checkpoint is unconditional on both the restore and save paths, the usability probe is unconditional on the healthy restore path, and shutdown() can now add up to 5s of cleanup wait (off the critical path any run is timed against, and correctly distinguished from the bootstrap budget at constants.ts:21-33).

What holds this at the low end is unchanged and re-verified: the only destructive branch requires SQLite to positively report not a database or malformed — a claim about the file's contents, not the runner's state — and the transient faults that would previously have reached it are pinned non-structural against real node:sqlite. The probe reads one schema page, never opens a missing or zero-byte file, checkpointDatabase never throws, cleanStorage failure is swallowed, and a declined save is retried by the post hook with projectIdPath now symmetric between the two call sites.

The residual sits in concern (2) — a bounded, self-healing bootstrap failure rather than a loop — and in concern (1), which degrades a warning rather than a decision. Neither is reachable on a healthy run.

No security exposure. Public API changes are additive or internal: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms, YAML default guarded across six pairs by scripts/action-input-defaults-guard.test.ts), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. OpenCodeServerHandle.shutdown changing from () => void to () => Promise<ShutdownResult> is a breaking signature change on a @fro-bot/runtime interface, but that package is private and every in-repo implementor and call site is updated — now including the local duplicate that previously masked it.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 33597659542
Cache hit
Session ses_f9f40e38bffeTg5hcSs7UTSpVN

@marcusrbrown
marcusrbrown merged commit 9d795de into main Sep 2, 2026
15 checks passed
@marcusrbrown
marcusrbrown deleted the fix/session-cache-bootstrap-trap branch September 2, 2026 06:20
marcusrbrown added a commit that referenced this pull request Sep 2, 2026
* docs(solutions): record why repair must precede capture

Writes up #1407 and #1519 for docs/solutions/. The durable lesson is an
ordering rule -- repair on restore has to happen before capture on save,
or a transient failure gets encoded into durable state that the next run
then trusts.

The most useful section is what didn't work. Five plausible fixes were
tried and refuted by evidence: "only persist cleanly-checkpointed state"
(nothing had ever produced one), "declining to save breaks the loop"
(save keys are unique per run and restore keys are prefixes, so the
poisoned entry stays newest), "wipe and save empty state" (destroys
history on every transient failure), "the log is provably empty at
transport" (a late write from a live writer regrew it after the
checkpoint reported success), and "route non-retryable failures into the
clean-slate path" (an allowlist of retryable, negated, hands a full disk
to the delete branch).

Also records the regression the fix itself introduced, since it was
caused by the fix and caught only in review: once healthy saves stopped
producing a local write-ahead log, an upload that only ever sends what
exists stopped refreshing that object while the database beside it kept
being overwritten. One reproduction reported malformed; the other
silently replayed stale content and turned 3000 rows into 2000.

The safety claim about archiving the database alone is stated
conditionally rather than absolutely -- SQLite's default
wal_autocheckpoint is 1000 pages, and a live writer crossing it mutates
the main file with no explicit checkpoint, measured going from 4096 to
860160 bytes.

Overlap against the four adjacent docs scored Moderate at highest, so
this is a new doc cross-linked in both directions rather than folded into
an existing one.

Refs #1407

* docs(solutions): add the backlinks the description claimed

The PR body said the new doc was cross-linked in both directions. It
wasn't -- only forward links existed, and no existing doc referenced it.
This repo does follow a backlink convention, so the claim was checkable
and wrong.

Adds entries from the two highest-overlap neighbours, which are where a
reader is most likely to be standing when this becomes relevant: the
cache-continuity doc (a save that failed and reported success, against
one that succeeded and persisted the wrong thing) and the S3 restore-scope
doc (the same restore-is-not-save asymmetry one layer down -- capability
there, atomicity here).

Also names the remedy in the regression section. These docs are retrieved
by section rather than read end to end, so a reader landing there from a
`malformed` search could have concluded the hazard was still live; the
fix was described in two other sections but never pointed at from the one
that describes the problem.

Drops related_components. Both values were novel to the corpus and the
field is unguarded, so they would have seeded near-synonyms for no
retrieval benefit -- tags already carry sqlite and session-cache.

* docs(solutions): fix inverted deixis in both backlinks

Both backlinks were written from the new doc's vantage point and pasted
into the neighbour docs without flipping "there" and "here", so each one
described the two documents backwards.

The convention is established three lines above the first one, in the
same bulleted list: there = the linked doc, here = the doc you are
reading. So that file simultaneously claimed "here = a write that failed
and reported success" and "here = the save genuinely succeeded", about
itself. The cache-continuity doc is the failed-save-reported-success
incident; the new doc is the succeeded-save-persisted-poison one.

The S3 backlink had the same inversion: that doc is the capability half
of the restore-is-not-save asymmetry, and the new one is the atomicity
half.

Both links resolved the whole time, which is all check-md-links can
observe -- the text they resolved to was the part that was wrong. Fitting
for a doc about claims that are checkable and unchecked.

* docs(solutions): correct the count and heading case

"Five plausible fixes" undercounted the seven that follow -- the two it
dropped are the zero-byte-log recreation and the restoreCache hoist, both
of which were refused with reasons worth keeping. Carried unresolved
through two reviews, in the section the doc itself calls the useful one.

Also aligns the regression heading with the title case the rest of the
file uses.
marcusrbrown added a commit that referenced this pull request Sep 2, 2026
…it (#1521)

* fix(runtime): report an inconclusive port probe as unknown, not as exit

Two follow-ups from the #1519 review, both cases where an unknown answer
resolved to the optimistic one.

The quiescence probe's socket timeout called finish(false), and false
means "port closed", which waitForServerQuiescence reads as quiesced and
returns immediately. So a connect that neither completed nor was refused
-- the one case the timeout exists for -- reported that the OpenCode
child had exited. That suppressed the warning telling an operator their
checkpoint may have raced a live writer, which is the whole reason the
signal exists. Every other unknown in this design resolves the other way:
verifyDatabaseUsable returns usable for an unrecognized throw, and
isStructuralCorruptionError returns false unless SQLite positively says
otherwise. It now returns true, so an inconclusive attempt keeps polling
and only the real deadline produces quiesced: false.

Testing it needed the socket to be injectable, since a real one either
connects or is refused and never reaches the timeout. The probe now takes
a connect function, and a fake socket drives the branch directly rather
than depending on a black-holed address behaving consistently in CI.

Second: a mid-transfer failure in the S3 adapter's download left a
partial opencode.db on disk. mainDbRestored stays false, so restore
reports a miss and falls through to the Actions cache; if that also
misses -- the normal case for a repository leaning on S3 -- the repair
block is skipped entirely and a truncated database reaches bootstrap
unprobed.

The suggested condition for cleaning it up was mainDbRestored === false
&& failed > 0 at the restore layer. That is unsafe: syncSessionsFromStore
increments failed for any key in the batch, so an unrelated sidecar
failure satisfies it while opencode.db was never attempted -- and on a
self-hosted runner with a pre-existing local database, deleting it
destroys live session state the failed download never touched.

Fixed at the write site instead. download() unlinks localPath only when
the pipeline itself throws, meaning the stream was open and a partial
write happened. A rejection before the pipeline is reached never touches
the path, so an existing file is provably safe -- pinned by a test.

Both fixes were checked against their pre-fix shapes and fail there.

Refs #1407

* test(runtime): pin the quiescence invariant at the call site

The polarity fix was pinned inside isPortOpen but not where it is read.
waitForServerQuiescence called it without an injector, so nothing proved
that an inconclusive probe keeps the loop polling -- and inverting the
read at the call site to `if (stillOpen) return {quiesced: true}` passed
every test in this PR, reintroducing the bug one line from where it was
fixed.

The optional connect parameter now threads through, defaulted so the
production call site is unchanged, and a test drives several poll cycles
of nothing-but-timeouts and asserts the only reachable outcome is
quiesced: false at the deadline. Verified against the inverted read: it
fails there.

Also narrows the s3-adapter cleanup comment, which claimed the unlink can
only reach a file the pipeline just opened. True for a mid-transfer
failure, not for an open failure -- if createWriteStream cannot open the
destination at all, pipeline rejects before truncating and the unlink
still fires against a file this attempt never wrote to, since unlink
needs permission on the directory rather than the file. The comment now
separates what is guaranteed from what is merely rare, and the behavior
is unchanged.

And records what the polarity flip costs: an inconclusive probe now rides
out the full 5s budget instead of returning after one poll interval. One
server per run and bounded, but whoever next tunes those constants
shouldn't have to rediscover it.

* test(runtime): decouple the quiescence test from the wall clock

Two review follow-ups, both in test code.

The call-site test asserted connectCalls > 1 against a 60ms budget with a
10ms poll interval. The do/while checks the deadline only after the first
iteration, so a single stalled delay() on a contended runner exits after
one probe and fails that assertion. Widened to a 500ms budget with a 5ms
interval so a stall cannot end the loop before a second probe. The
quiesced: false assertion was never at risk either way -- a stalled loop
still cannot produce quiesced: true -- so this only removes a spurious
red, and the comment now says so.

The s3-adapter test title claimed the pre-existing file was "at a
different path", which would be trivially true and worth nothing. It is
at the same path the download targets, which is the entire point: a
pre-pipeline rejection leaves untouched the exact file the cleanup would
otherwise delete. Title now matches what the body actually pins.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Oversized session cache traps bootstrap: prune is gated on server start, cache save is not

3 participants