fix(cache): break the session-cache bootstrap trap - #1519
Conversation
The first draft's central claim was wrong. Declining to persist bad state prevents new poisoning but cannot escape existing poisoning: save keys are unique per run and restore keys are prefixes returning the newest match, so a run that declines to save leaves the poisoned entry as the newest one and the next run restores it again. The loop stays absorbing. The repair belongs on restore. Checkpointing a hot database between restore and bootstrap recovers the session instead of discarding it, heals a stuck repository in one run, and lands outside the budget that was failing, since no timer spans those two calls. Also corrects three things the draft asserted without grounds: that the harness could await the child's exit, which it cannot observe; that the checkpoint belonged in the session runtime, when both save paths and the object-store sync share one function at the cache boundary; and that whether the runtime could checkpoint at all was unknowable before implementation. Refs #1407
…trap budget visible Units 1, 4, 5 and 6 of the session-cache trap fix. The wal-index is machine-local and SQLite never syncs it, so a copy from another runner is stale by construction. It is no longer captured on save, and a copy carried in by an older cache is deleted on restore rather than left to age out. Restore still tolerates its presence so entries written before this change still work. Declaring storage corrupt now actually clears the database. It lives beside the storage directory rather than inside it, so 'proceed with clean state' had been leaving the very files it meant to discard. The object-store restore also gains the corruption and version checks the cache path already had: it wins on restore and returns before either one ran, so the authoritative path was the unchecked one. Which files constitute the database is now defined once, in the runtime package, because the object-store sync there cannot import from the Action. The name had been written out in three places. The server bootstrap budget is configurable and reported. It was the SDK's 5000ms default, never passed and never logged, against runs observed landing 16ms inside it. Time spent in the spawn is measured apart from the total so the budget comparison is exact and a slow port bind is distinguishable from slow server init. session-retention now reaches pruning. It was parsed, validated, logged, and then never read, so a consumer bounding their cache with it got a validated no-op. Its default matches the hardcoded value it replaces, so an unset input changes nothing. Refs #1407
No run had ever checkpointed it. server.close() sends a kill and returns without waiting, so a valid session could sit entirely in the write-ahead log with the database file at zero bytes, and the cache carried that hot log to the next run, which paid recovery cost opening it — measured at 2416.8ms for a 185MB log against 60.2ms once merged. The checkpoint runs inside saveCache, which both the cleanup phase and the post hook already call and which also owns the object-store sync, so every capture path inherits it without a flag they must agree to read. It runs before anything inspects file sizes, since merging changes which files are non-empty. Success is judged by the log's size on disk rather than the pragma's own count, which reported zero on a verified-successful truncation. The outcome has three states: collapsing 'already clean' into 'failed' would decline saves that were always safe. The harness cannot observe whether the writer exited, so the attempt is the liveness probe — a live writer surfaces as busy and is retried within a bound. A log that still cannot be merged declines the save and says why in the job summary, because a save that silently does not happen is the failure this repository already paid for once. Refs #1407
This is what gets a repository out of the loop rather than merely stopping it from entering one. Save keys are unique per run and restore keys are prefixes returning the newest match, so a run that declines to save writes no entry and the next restore hits the same bad one. Declining forever is a stable state. Checkpointing the restored database heals it in place. It runs between restore and bootstrap, where no timer applies — the budget covers createOpencode alone — so the recovery cost moves out of the window that was timing out instead of being paid inside it. Repairing rather than wiping is the point. Wiping would also break the loop, but it discards session history on every transient bootstrap failure, which is the second reporter's case exactly. A repair is logged because it is the evidence a stuck repository healed, and its absence later is the evidence it stayed healed. The healthy case is silent, and a failed repair still lets bootstrap try. Closes #1407 Refs #1407
A comment asserted a valid session could leave opencode.db at zero bytes with its data in the write-ahead log. Three independent reviewers read it and each filed the same bug against the checkpoint helper's zero-byte guard, claiming it skipped the case it exists for. The guard is right and the comment was wrong. A database in write-ahead mode with no checkpoint leaves the main file at its header page — 4096 bytes on Node 24.20.0, never zero — and forcing it to actually zero makes the log unreadable, so that shape holds no session to save. The comment now states the measured value, and a test pins the real shape so the misreading cannot recur. Also from review: the errno checks in the two delete paths assumed every thrown value carries a code, so a non-errno failure was silently treated as a missing file; both now narrow before reading it. The bootstrap timeout is bounded, since an unbounded value would hang a run to the job timeout rather than failing fast, which is the opposite of what that input is for. And the restore repair gains a test that drives a real database rather than a mock, covering the wiring the ordering tests cannot see. Refs #1407
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The diagnosis holds up under reading. runCleanup does execute on bootstrap failure (run.ts finally), serverHandle.shutdown() genuinely does not await a checkpoint, and restore-key prefix matching genuinely makes "decline to save" a stable state rather than a recovery. Moving the repair to restore — outside the only window timeoutMs governs — is the right shape, and cache-restore.test.ts:236-260 pins that ordering so a future refactor can't silently move the cost back inside the budget.
I verified locally: tsc --noEmit clean, and 111 tests across src/services/cache/, cache-restore.test.ts, and cleanup.test.ts pass. node:sqlite resolves unflagged on the pinned Node 24.20.0 and is bundled as an external import in the shared dist/artifact-*.js chunk, so the new dependency claim checks out.
The three-state CheckpointOutcome is the load-bearing design decision here, and the reasoning in checkpoint.ts:7-22 is correct: a two-state collapse would decline every healthy save. Judging success by WAL size rather than the pragma's checkpointed count is also correct, and checkpoint.test.ts:171-196 pins the surprising pragma behavior with a control database rather than just asserting the helper's output — that test will still be right if a future Node changes the count.
Blocking issues
None.
Non-blocking concerns
1. A malformed database still traps the repository — the same loop, different cause.
checkpointDatabase reports failed when new DatabaseSync(dbPath) throws for a non-lock reason ("file is not a database", "database disk image is malformed"). At cache-restore.ts:74-76 that only logs a warning and hands the bad database straight to bootstrapOpenCodeServer. On the save side, a malformed main DB with an absent or empty WAL returns nothing-to-checkpoint (checkpoint.ts:100-103), so the save proceeds and re-persists it under a fresh key. That is exactly the self-perpetuating loop this PR set out to break, reached through corruption instead of a hot WAL.
This is not a regression — the same database was re-saved before this change — so it does not block. But the failure is already distinguishable: isRetryableError separates "busy/locked" (transient, retry) from everything else (structurally unrecoverable). Routing the non-retryable case at restore into the existing corruption path (cleanStorage + cacheStatus = 'corrupted') would be far narrower than the "wipe on any bootstrap failure" the plan rightly rejected — it fires only when SQLite itself says the file is not a usable database, not on transient bootstrap variance.
2. The new object-store integrity checks are close to tautologies for the artifact they guard.
restoreFromObjectStore calls fs.mkdir(storagePath, {recursive: true}) at restore.ts:113, then checkStorageCorruption at :119 — which returns false as soon as storagePath is a readable directory, which the mkdir two lines earlier guarantees. checkStorageVersion returns true when .version is absent (restore.ts:47-50), and the object-store sync only transports DB-family files plus session sidecars, so .version will usually not be there. Both checks therefore pass for a corrupt opencode.db, which is the thing the object store actually restored. Adding them is still an improvement over returning before any check ran, but the PR description's framing overstates the protection gained. Related to (1): the checkpoint attempt is the only thing on this path that actually inspects the database.
3. The restore-side repair has no wall-clock bound.
maxAttempts × retryDelayMs bounds only the sleeps (800 ms at defaults). Each attempt's PRAGMA wal_checkpoint(TRUNCATE) is unbounded, and this PR's own measurement puts a 185 MB WAL at ~2.4 s. A large, contended database can add roughly 12 s ahead of bootstrap on every cache-hit run. That is deliberately outside the createOpencode budget, so it cannot cause the timeout being fixed — but it is now unbounded time on the critical path of every hit. A deadline checked between attempts would cap it without changing the healthy path.
4. The declined-save cost is smaller than the description suggests, and the reason is worth documenting.
runCleanup only sets STATE_KEYS.CACHE_SAVED when saveCache returns true (cleanup.ts:184-186), so a declined save leaves post.ts to retry — by which point the main step has ended and the OpenCode child has almost certainly exited, making the checkpoint succeed. That post-hook retry is the real reason "a declined save discards that run's session work" is rare, and it deserves a sentence in the save.ts decline comment, because a future refactor that sets CACHE_SAVED on decline (to avoid duplicate work) would silently remove the safety net.
One wrinkle: post.ts:90-98 builds cacheSaveOptions without projectIdPath, so a save that only lands in the post hook omits .git/opencode from the archive. Pre-existing, and low impact since ensureProjectId regenerates deterministically from the root commit — but the fallback path is now materially more likely to be the one that runs.
5. Comment/behavior mismatch in paths.ts:58-63.
The comment states save mode "filters by existence because @actions/cache fails if any save path is missing at archive time," but buildSaveCachePaths pushes opencode.db unconditionally at :111 while only the WAL gets an fs.access guard at :114-119. @actions/cache throws only when every path resolves to nothing, so the behavior is safe either way — but the comment states a rule the code beside it does not follow, which is the kind of thing the last commit in this branch exists to fix.
Missing tests
- No end-to-end coverage of a malformed (not merely hot-WAL)
opencode.dbthrough restore → repair-fails → bootstrap → save. This is the gap behind concern (1); a test asserting the current behavior would at least make the residual loop explicit rather than implied. - No test that a cleanup-declined save is retried by the post hook.
post.test.tscoverscacheSaved === 'false'→ save, but not framed as decline recovery, and nothing ties the two halves together. Given that this retry is what makes the decline tradeoff acceptable, a test naming that relationship would protect it from a well-intentioned refactor. - No coverage of the restore-side repair against a locked database —
cache-restore.test.ts:219-234mocks thefailedoutcome rather than producing it, so the real path-selection code is never exercised in the failure case. The e2e suite at:270-347does this well for the success case; the same fixture with anEXCLUSIVE-locked holder (as incheckpoint.test.ts:151-169) would close it.
Coverage is otherwise unusually strong for a change of this shape — the -shm deletion is tested on both the cache and object-store paths, including the non-ENOENT warning branch, and the input clipping has all four boundary cases.
Risk assessment: MED
Blast radius is every run: the checkpoint is now unconditional on both the restore and save paths, and saveCache gained a new way to return false. The mitigating factors are real — the decline is recoverable via the post hook, the repair sits outside the bootstrap budget, the failure modes are all logged, and checkpointDatabase never throws. Regression likelihood concentrates in one place: a run whose OpenCode child outlives the ~1 s retry bound at cleanup and whose post-hook checkpoint also fails loses that run's session work, where the previous code would have transported db+wal. Rare, but silent to the user except for a job-summary block.
No security exposure. The public API changes are additive (server-bootstrap-timeout input, new checkpointDatabase export); the one semantic change is that session-retention now actually prunes, which will surprise anyone who set it low while it was inert — worth a line in the release notes.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33469019264 |
| Cache | hit |
| Session | ses_fa4d1db6affeL56tSeVO7ak948 |
…n passing it on Review follow-ups on the trap fix. A malformed database still trapped a repository in the loop this branch exists to break, reached through corruption instead of a hot log. The checkpoint already distinguished a live writer from a file SQLite says is not a database; only the first is worth retrying, and the second now routes into the clean-slate path restore already used for corrupt storage. That covered a malformed database carrying a log, since that is the only shape anything opened. One with no log was never opened at all, so it reached bootstrap, failed, and was saved again. Restore now checks that a database it is about to hand over can actually be read — one schema-page read, a few milliseconds and independent of size, outside the budget. Construction alone does not surface this: opening a file of plain text succeeds and only the read reports it is not a database. The repair's retries bounded their sleeps but not their work, so a large log could spend far longer than intended ahead of bootstrap. A deadline between attempts caps that without ever interrupting one in flight. Also records why declining to save rarely costs a run its work — the post hook retries once the child has exited — because a refactor that marked the cache saved on decline would remove that quietly. And corrects a comment claiming save paths are filtered by existence when only the log is. Refs #1407
|
All five in, plus the gap the first fix left behind. Concern 1 turned out to have two halves. Routing a structurally-unrecoverable checkpoint failure into the existing clean-slate path closes it for a malformed database carrying a write-ahead log — but that is the only shape anything ever opened. One with no log returned I checked the probe rather than assuming it worked, and the result changed the implementation: Left the save side alone deliberately: forcing every healthy save to open a database when there is nothing to checkpoint is a real cost for a benefit that only lands on restore. Concern 3 — the retries bounded sleeps but not the checkpoint work itself, so a large log could spend far longer than intended. A deadline between attempts caps it without interrupting one in flight. Concern 2 was right and I have corrected the description. Concern 4 is recorded in the decline comment, including that a refactor marking the cache saved on decline would remove the retry safety net. Concern 5 is fixed — and it was the same defect as the commit before it, a comment stating a rule its own code does not follow. Your three named tests are in, plus real-database coverage for both corruption shapes and a regression guard on the healthy no-log path. Gates: types 0, lint 0, |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The five concerns from the dismissed review are genuinely closed, and the sixth gap the author found on their own — a malformed database with no write-ahead log, which checkpointDatabase returns nothing-to-checkpoint for before ever opening the file — was the right catch. verifyDatabaseUsable (integrity.ts:46-66) is the correct shape for it: restore-only, schema-page read rather than PRAGMA integrity_check, and a no-op for a missing or zero-byte file so it never creates the database it was asked to inspect. The comment at integrity.ts:20-25 explaining why it is not folded into checkpointDatabase is the load-bearing reasoning and it is right — doing so would redefine nothing-to-checkpoint for the save path and force every healthy save to open a file it currently never touches.
The author's claim that new DatabaseSync() succeeds against plain text and only the schema read surfaces file is not a database matches what I see: a construction-only probe would have looked closed while leaving the gap open. integrity.test.ts:47-73 also covers the other corruption shape (a truncated disk image), which a construction-only check would also have missed.
Verified locally: tsc --noEmit clean; 126 tests across src/services/cache/, cache-restore.test.ts, cleanup.test.ts, and the new cleanup-decline-retry.test.ts pass. dist/artifact-C90iooVD.js contains both wal_checkpoint and sqlite_master, so the committed bundle reflects the source. session-retention's defaults line up (DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50), so wiring it in run.ts:80 is a genuine no-op for an unset input, and cleanup.test.ts:260-315 pins both the explicit-default and never-set branches separately — which is the distinction that makes it safe to ship.
cleanup-decline-retry.test.ts is the best addition here. It runs runCleanup and runPost against one shared state store, which is the only way to assert the relationship the decline comment now documents. Neither phase's own suite could see it.
Blocking issues
None.
Non-blocking concerns
1. isRetryableError is an allowlist of retryable, so every unrecognized SQLite error routes into the destructive path.
checkpoint.ts:82-88 returns retryable: true only for messages containing locked or busy; everything else is false, and cache-restore.ts:96-100 sends retryable: false straight to cleanStorage — deleting the session history. The comment above that branch names the two errors it means (file is not a database, database disk image is malformed), but the code reaches it for any other message too. I confirmed one concretely on this runner:
new DatabaseSync('<missing dir>/opencode.db')
-> "unable to open database file" retryable? false
SQLITE_CANTOPEN (fd exhaustion, permissions), SQLITE_FULL (database or disk is full), SQLITE_IOERR (disk I/O error), and attempt to write a readonly database all land the same way — environmental and transient, but classified as structural corruption and answered by wiping history. That is precisely the tradeoff the plan rejected when it declined "wipe on any bootstrap failure," reached through a narrower door. verifyDatabaseUsable has the same shape at integrity.ts:57-58: any throw from the schema read becomes usable: false.
The inversion is small and matches what both doc comments already say: match the structural messages positively (not a database, malformed, encrypted, file is encrypted or is not a database) and treat everything else as retryable / leave-alone. The healthy path is unchanged and the two tested corruption shapes still route the same way.
2. cacheResult and cacheStatus disagree after a structural-corruption downgrade.
handleStructuralCorruption (cache-restore.ts:79-86) mutates the local cacheStatus to 'corrupted' but returns the original cacheResult unchanged at :133 — still {hit: true, corrupted: false, restoredPath: storagePath} for storage that has just been deleted. Nothing reads cacheResult today (only cacheStatus and serverHandle are consumed, in finalize.ts:199, execute.ts:223,380, and run.ts:135), so this is latent rather than live. But it is a public field of CacheRestorePhaseResult, and the first consumer to read it will read a stale hit for a wiped cache. Related: metrics.setCacheSource is left at 'cache'/'storage', so the job summary will report a corrupted cache sourced from a cache that was discarded.
3. Object-store corruption is cleared locally but not at the source.
When the malformed opencode.db arrived via restoreFromObjectStore, cleanStorage removes only the runner's copy — the bucket still serves the same object on the next restore. It does self-heal, because the next saveCache uploads a healthy database over it, but only for a run that reaches syncSessionsToStore (save.ts:153-170); a run that declines the save leaves the poisoned object in place and the next run pays the wipe again. The Actions-cache side does not have this problem, since restore keys advance to the newer entry. Not a regression and not worth solving here, but the asymmetry is worth a line somewhere.
4. The WAL existence guard is computed before the checkpoint that changes it.
buildSaveCachePaths runs at save.ts:123, the checkpoint at :136. After a successful TRUNCATE the log is a zero-byte file that is still in cachePaths, so every healthy save ships an empty opencode.db-wal in the archive. Harmless — restore tolerates it and hasCacheableContent gates on the main database — but the fs.access guard described at paths.ts:58-69 never actually reflects post-checkpoint state, which is the same comment-vs-code drift the last two commits in this branch exist to fix. Computing the save paths after the checkpoint would make the comment true.
Missing tests
- No test pins the classification of a non-structural SQLite failure.
checkpoint.test.tscoversbusy/locked(retryable) andnot a database(non-retryable), but nothing in between — and the space between them is exactly what decides whether concern (1) destroys a repository's history. A test asserting the intended classification forunable to open database filewould either confirm the current behavior is deliberate or fail and prove it is not. - No test asserts
cacheResult/cacheStatuscoherence after the corruption downgrade. Every corruption test checksresult?.cacheStatusonly, so concern (2) is invisible to the suite. - No test covers the
run.ts:80assignment itself. The cleanup half of thesession-retentionwiring is well covered, but the line that actually ends the "parsed, validated, logged, never read" bug is a bare assignment thatrun.test.tscannot see, since it mocksrunCleanup. Low value on its own, but it is the one link in that chain nothing observes.
Coverage is otherwise excellent for a change of this shape. The end-to-end suite at cache-restore.test.ts:339-533 now exercises real databases for all four states the phase distinguishes — hot WAL, malformed-with-WAL, malformed-without-WAL, and locked — using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable rather than canned outcomes, and the healthy no-log regression guard at :501-532 protects the common path. The locked-database case at :446-477 closes the exact gap flagged in the previous review.
Risk assessment: MED
Unchanged from the previous review in shape, slightly larger in blast radius. Both the checkpoint and the usability probe now run on every cache hit, and the restore path gained a code path that deletes session state — the first time this codebase discards history on the strength of a SQLite error message. The mitigations are real: the probe is a single schema page (measured under 2ms), it never runs for a missing or empty database, it sits outside the createOpencode budget, and cleanStorage failing is caught and non-fatal. The residual risk concentrates in concern (1): a runner-level fault (fd exhaustion, a full disk) presenting as a non-busy SQLite error would silently wipe a repository's session history and report a corrupted cache. Recoverable — the next run rebuilds — but silent to the user beyond a job-summary line, and the run still completes successfully.
No security exposure. Public API changes are additive (server-bootstrap-timeout input, checkpointDatabase and verifyDatabaseUsable exports). Two behavioral changes deserve release notes: session-retention now actually prunes (a no-op at the default, surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33472537265 |
| Cache | hit |
| Session | ses_fa49e3391ffefMwAv3VUt6Rn7K |
CodeQL flagged the new integrity test: it measured the database with stat and then reopened the same path to truncate it, which resolves the path twice and is a check-then-use window. Statting the already-open handle closes it and also makes the truncation apply to the file that was actually measured. Refs #1407
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: CONDITIONAL
The diagnosis and the shape of the fix are right, and I want to say that clearly before the blocking issue: repairing on restore rather than only declining on save is the only thing in this change that actually ends the loop, and putting it between restoreCache and bootstrapOpenCodeServer — outside the only window timeoutMs governs — is correct. verifyDatabaseUsable (integrity.ts:46-66) closes the real gap the first draft left, and the reasoning at integrity.ts:20-25 for keeping it off the save path is sound. I independently confirmed the load-bearing claim behind it: new DatabaseSync() against a file of plain text succeeds, and only the schema read raises file is not a database. A construction-only probe would have looked closed while leaving the gap open.
Verified locally: check-types clean across all four workspace packages; 126 tests pass across src/services/cache/, cache-restore.test.ts, cleanup.test.ts, and cleanup-decline-retry.test.ts. The end-to-end suite at cache-restore.test.ts:339-533 exercises real databases for all four states the phase distinguishes, using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable rather than canned outcomes. That is unusually good coverage for a change of this shape.
The one thing I cannot pass is the polarity of the error classification that now guards a destructive path.
Blocking issues
1. Any unrecognized SQLite error is classified as structural corruption, and structural corruption now deletes the repository's session history. src/services/cache/checkpoint.ts:82-88, src/services/cache/integrity.ts:57-58, src/harness/phases/cache-restore.ts:96-100.
isRetryableError is an allowlist of retryable: it returns true only for messages containing locked or busy, and everything else falls through as false. cache-restore.ts:96-100 routes retryable: false into handleStructuralCorruption, which calls cleanStorage(storagePath) — wiping the storage directory and the entire DB family. verifyDatabaseUsable has the same polarity at integrity.ts:57-58: any throw from the schema read becomes usable: false, which reaches the same wipe.
The doc comments at checkpoint.ts:24-26 and cache-restore.ts:71-73 both name the two errors this branch is meant for (file is not a database, database disk image is malformed), but the code reaches it for anything else too. I confirmed two concrete non-structural cases on this runner:
new DatabaseSync('<dir 0555>/opencode.db') + SELECT count(*) FROM sqlite_master
-> "attempt to write a readonly database" retryable? false -> wipe
new DatabaseSync('<missing dir>/opencode.db')
-> "unable to open database file" retryable? false -> wipe
SQLITE_CANTOPEN (fd exhaustion, permissions), SQLITE_FULL (database or disk is full), SQLITE_IOERR (disk I/O error), and the readonly case above are all environmental and transient, but each is answered by destroying session history. Note that the probe opens read-write, so a merely non-writable database — which is perfectly readable — is classified as corrupt.
This is the tradeoff the plan explicitly rejected when it declined "wipe on any bootstrap failure," reached through a narrower door. It is also the first time this codebase discards history on the strength of a SQLite error message, which is why I am treating a wrong-polarity default as blocking rather than as a nit: the fail-safe default for an unrecognized error must be leave it alone, not delete it.
The fix is small and matches what both doc comments already claim. Match the structural messages positively — not a database, malformed, file is encrypted or is not a database — and treat everything else as retryable / leave-alone. The healthy path is unchanged, and both corruption shapes currently under test (cache-restore.test.ts:422-444 and :479-499) still route exactly as they do today. The same inversion applies to verifyDatabaseUsable: only a structurally-worded failure should report usable: false.
This was raised as concern (1) in the previous review and is unaddressed in the current head; the author's comment responded to the earlier round's five items, so I read this as not-yet-seen rather than declined.
Non-blocking concerns
1. cacheResult and cacheStatus disagree after a corruption downgrade. handleStructuralCorruption (cache-restore.ts:79-86) mutates the local cacheStatus to 'corrupted' but cache-restore.ts:133 returns the original cacheResult unchanged — still {hit: true, corrupted: false, restoredPath: storagePath} for storage that was just deleted. I confirmed nothing outside tests reads cacheResult today, so this is latent, not live. But it is a public field of CacheRestorePhaseResult, and the first consumer to read it reads a stale hit for a wiped cache. Related: metrics.setCacheSource (:50) stays at 'cache'/'storage', so the job summary reports a corrupted cache sourced from a cache that was discarded.
2. The WAL existence guard is computed before the checkpoint that changes it. buildSaveCachePaths runs at save.ts:123, the checkpoint at save.ts:136. After a successful TRUNCATE the log is a zero-byte file still present in cachePaths, so every healthy save ships an empty opencode.db-wal. Harmless — restore tolerates it and hasCacheableContent gates on the main database — but the guard described at paths.ts:66-69 never reflects post-checkpoint state. Computing the save paths after the checkpoint would make the comment true. (The rest of that comment block is now accurate; the previous round's comment/behavior drift is genuinely fixed.)
3. Object-store corruption is cleared locally but not at the source. When a malformed opencode.db arrives via restoreFromObjectStore, cleanStorage removes only the runner's copy — the bucket still serves the same object next run. It self-heals for any run that reaches syncSessionsToStore (save.ts:153-170), but a run that declines the save leaves the poisoned object in place and the next run pays the wipe again. The Actions-cache side does not have this problem, since restore keys advance. Not a regression; worth a line somewhere.
4. Two behavioral changes deserve release notes. session-retention now actually prunes — a no-op at the default (DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50, and cleanup.test.ts:260-315 pins the explicit-default and never-set branches separately), but surprising for anyone who set it low while it was inert. And opencode.db-shm is no longer captured for transport.
Missing tests
- No test pins the classification of a non-structural SQLite failure.
checkpoint.test.tscoversbusy/locked(retryable) andnot a database(non-retryable), but nothing between them — and that gap is exactly what decides whether the blocking issue destroys a repository's history. A case asserting thatunable to open database fileorattempt to write a readonly databaseleaves storage intact would pin the intended polarity. - No test asserts
cacheResult/cacheStatuscoherence after the downgrade. Every corruption test checksresult?.cacheStatusonly, so non-blocking concern (1) is invisible to the suite. - No test covers the
run.ts:80assignment. The cleanup half of thesession-retentionwiring is well covered, but the line that actually ends the "parsed, validated, logged, never read" bug is a bare assignmentrun.test.tscannot observe, since it mocksrunCleanup. Low value alone, but it is the one unobserved link in that chain.
Coverage is otherwise excellent. The -shm deletion is tested on both the cache and object-store paths including the non-ENOENT warning branch, the input clipping has all four boundary cases (inputs.test.ts:905-957), and the healthy no-log regression guard at cache-restore.test.ts:501-532 protects the common path with an explicit expect(mocks.loggerWarning).not.toHaveBeenCalled().
Risk assessment (MED)
Blast radius is every run: both the checkpoint and the usability probe are unconditional on the restore path, and the checkpoint is unconditional on the save path. Mitigations are real — the probe is a single schema page, it never runs for a missing or empty database, it sits outside the createOpencode budget, checkpointDatabase never throws, cleanStorage failure is caught, and a declined save is retried by the post hook (now documented at save.ts:83-88 and pinned by cleanup-decline-retry.test.ts, which is the best addition in this round — it runs runCleanup and runPost against one shared state store, which neither phase's own suite could do).
Residual risk concentrates entirely in the blocking issue: a runner-level fault presenting as a non-busy SQLite error silently wipes session history and reports a corrupted cache, while the run still completes successfully. Recoverable, since the next run rebuilds — but invisible to the user beyond a job-summary line, and triggered by exactly the transient conditions this design set out not to punish.
No security exposure. Public API changes are additive: the server-bootstrap-timeout input (clamped at 120s with a warning, inputs.ts:363-373), and the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33473192307 |
| Cache | hit |
| Session | ses_fa494e035ffeIsNNA4D1liay5F |
The clean-slate path was reached by negating retryability, and retryability was an allowlist of busy and locked. So every error outside that pair — a full disk, an I/O fault, exhausted descriptors, a permissions problem — answered a transient condition by deleting the repository's session history. That is the trade this work rejected when it declined to wipe on any bootstrap failure, reached through a narrower door. Corruption is now matched on its own terms rather than inferred from the absence of a retry, so an unrecognized error leaves the database alone. Both comments already described the allowlist; the code now agrees with them. Verified in both directions: a file of plain text still reports it is not a database and is still cleared, while a database that merely cannot be opened is left where it is. The corruption downgrade also returned the original restore result, which still described a hit with a path that had just been deleted, and reported a source for a cache that no longer existed. Both now match the state on disk. Save paths are built after the checkpoint rather than before, so the guard reflects the log the checkpoint actually left behind. Refs #1407
|
Blocking issue fixed, and you were right to treat it as blocking rather than a nit. The polarity was mine. My brief told the lane to route non-retryable failures into the clean-slate path, and Corruption is now matched on its own terms — Both existing corruption tests still route to the wipe, and the case that would have caught this is now pinned in three places. Non-blocking 1 — Non-blocking 2 — save paths are now built after the checkpoint. Worth recording what that turned up: Non-blocking 3 and 4 noted. The object-store copy surviving a local wipe is real; it self-heals on any run that reaches the sync, and I would rather fix it where the store's freshness is decided than bolt it onto this path. Gates: types 0, lint 0, |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The blocking issue from my previous review is genuinely fixed, and the fix is the right shape rather than a patch over the symptom. isStructuralCorruptionError (sqlite-errors.ts:48-53) is now a positive allowlist — not a database / malformed — living in its own module rather than as the negation of isRetryableError. Renaming the failed outcome's field from retryable to structural (checkpoint.ts:37) is what makes the correction durable: the two axes are now separate concepts with separate names, so a future change cannot silently re-conflate "I could not retry this" with "this file is corrupt." That is a better outcome than the minimal inversion I proposed.
I verified the polarity in both directions rather than reading the diff:
checkpoint.ts:157 lastStructural = isStructuralCorruptionError(caughtError) // was: !isRetryableError
integrity.ts:68 if (isStructuralCorruptionError(error)) -> usable: false; else usable: true
cache-restore.ts:109 if (repairOutcome.structural) -> handleStructuralCorruption
The destructive path is now reachable only from a positively-matched structural error. verifyDatabaseUsable returns {usable: true} for any unrecognized throw (integrity.ts:71), which is the correct fail-safe default for a probe whose caller deletes session history.
Verified locally: check-types clean across all four workspace packages; lint 0 errors; bun run build leaves dist/ byte-identical (reproducible); 265 tests pass across src/services/cache/ and src/harness/phases/. I independently confirmed the createOpencode timeout claim against the installed SDK — node_modules/@opencode-ai/sdk/dist/server.js:7 does Object.assign({..., timeout: 5000}, options) and server.d.ts:6 declares timeout?: number, so passing DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS = 5000 explicitly is byte-for-byte identical to today's unset behavior. The comment in packages/runtime/src/shared/constants.ts is accurate.
The two non-blocking items I raised are also closed, and one of them turned up something worth recording. handleStructuralCorruption now rebuilds cacheResult to match the downgraded status and calls setCacheSource(null) (cache-restore.ts:90-98). And moving buildSaveCachePaths to after the checkpoint (save.ts:137 then :146) is correct for a reason better than the one I gave: SQLite unlinks the write-ahead log when the last connection closes, and checkpointDatabase closes its own handle in finally, so post-reorder the file is genuinely absent by the time the fs.access guard runs — the comment at paths.ts:66-69 is now true without further edits.
On the two failing tests in src/main.test.ts: both are environmental in this review sandbox, not regressions. They spawn a real child process that hits Cache Service Url not found, unable to restore cache and then hangs in auto-setup — this runner has no ACTIONS_CACHE_URL/ACTIONS_RESULTS_URL. src/main.test.ts is not in the changed-file set and the timeout occurs during setup, long before any code this PR touches. Not counted against the change.
Blocking issues
None.
Non-blocking concerns
1. acquiredServer?.close() is unguarded inside the catch block, which can break bootstrapOpenCodeServer's never-throw contract. packages/runtime/src/agent/server.ts:105.
The function's signature promises Promise<Result<OpenCodeServerHandle, Error>>, and the entire try/catch exists so a bootstrap failure returns err(...) rather than rejecting. The new cleanup call is the last statement before that return err(...), so if close() throws, the function rejects instead — the one path in this function that was previously impossible.
Reachability with the real SDK is essentially nil: stop() (@opencode-ai/sdk/dist/process.js) returns early when proc.exitCode !== null || proc.signalCode !== null, and proc.kill() on an exited child returns false rather than throwing. So this is an invariant nit, not a live bug. But note the URL-mismatch branch at :84 already calls server.close() before returning, and if that throws, control reaches the catch and closes the same handle a second time — the only double-close in the function. Wrapping :105 in the same best-effort try/catch the rest of this PR uses for db?.close() (checkpoint.ts:148-152, integrity.ts:73-77) would make it consistent with the module's own convention.
2. post.ts still omits projectIdPath, and that path is now deliberately load-bearing. src/harness/post.ts:90-98 builds cacheSaveOptions without projectIdPath, so a save that only lands in the post hook ships an archive without .git/opencode.
This is pre-existing and I flagged it as low-impact last round. I am raising it once more only because this PR promotes the post-hook retry from an incidental fallback to the documented reason a declined save is acceptable (save.ts:83-88, pinned by cleanup-decline-retry.test.ts). The mitigation still holds — ensureProjectId regenerates deterministically from the root commit — so this is genuinely non-blocking, but the asymmetry between cleanup.ts:179 and post.ts is now load-bearing enough to be worth a one-line fix or an explicit comment saying it is intentional.
3. The usability probe is skipped when the checkpoint fails non-structurally. cache-restore.ts:108-126 runs verifyDatabaseUsable only in the nothing-to-checkpoint branch. A database that is both locked and corrupt therefore reaches bootstrap unprobed.
I think this is the right call and am recording it rather than objecting: probing a database that just refused to open would fail for the same environmental reason, and re-probing would only add a second chance to misclassify. Worth a sentence in the comment block so a future reader does not mistake the gap for an oversight.
4. Release notes. Unchanged from last round: session-retention now actually prunes (a no-op at the default — DEFAULT_SESSION_RETENTION 50 == DEFAULT_PRUNING_CONFIG.maxSessions 50, and cleanup.test.ts pins the explicit-default and never-set branches separately — but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.
Missing tests
- No test pins that
bootstrapOpenCodeServerstill returnserrwhen the acquired server'sclose()throws. The new case atserver.test.ts("closes the server on a failure that occurs after the handle was acquired") uses a plainvi.fn()spy that cannot throw, so it proves the close happens but not that the Result contract survives it. That is exactly the gap behind concern (1); a spy withmockImplementation(() => { throw ... })assertingresult.success === falsewould close it. - No test drives the restore-side probe through the object-store source. The end-to-end suite exercises real databases well, but every case reaches the probe via a cache hit.
restoreFromObjectStorereturns{hit: true, source: 'storage'}, which thecacheStatus === 'hit'gate atcache-restore.ts:67does cover — but only by inspection. Since the object store is the path that wins on restore and the PR description correctly notes its own integrity checks are near-tautologies, the probe is the only real defense there, and nothing pins it.
Coverage is otherwise excellent and materially better than either previous round. The blocking case is pinned in three separate places — checkpoint.test.ts:197 (a real permissions fault yielding unable to open database file classified non-structural), integrity.test.ts:78 (a healthy database at chmod 0o000 reported usable: true), and cache-restore.test.ts:289 (storage left intact, run still a hit) — using real node:sqlite faults rather than mocked messages, which is what makes them worth having. integrity.test.ts:62-68 sizing the truncation target through its own open handle avoids a check-then-use window that a lazier test would have carried.
Risk assessment (LOW-MED)
Blast radius remains every run — the checkpoint is unconditional on both restore and save, and the usability probe is unconditional on the healthy restore path. What changed since the last round is the residual: the destructive branch is no longer reachable by default. It now requires SQLite to positively report not a database or malformed, which is a claim about the file's contents rather than about the runner's state. The transient faults I was worried about — fd exhaustion, a full disk, an I/O error, a read-only mount — all now report non-structural and leave the database alone, verified against real node:sqlite in checkpoint.test.ts and integrity.test.ts rather than asserted.
Remaining mitigations are unchanged and real: the probe reads a single schema page (measured under 2ms, size-independent), it never opens a missing or zero-byte file, both the probe and the checkpoint sit outside the createOpencode budget, checkpointDatabase never throws, the repair is bounded by an explicit deadline (deadlineMs, checkpoint.ts:128-137) rather than by sleep count alone, and cleanStorage failure is caught. A declined save is retried by the post hook, and that relationship is now pinned by a test that runs runCleanup and runPost against one shared state store.
The loop this PR set out to break is closed at both ends, which I checked as a whole rather than per-file: a structurally corrupt database is wiped on restore and declined on save, so it is neither handed to bootstrap nor re-persisted under a fresh key. The largest remaining exposure is the object-store copy surviving a local wipe (non-blocking 3 from the previous round), which the author has correctly scoped to where store freshness is decided rather than bolting a workaround onto this path.
No security exposure. Public API changes are additive: the server-bootstrap-timeout input (validated positive, clamped to 120000ms with a warning, inputs.ts:363-373), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME re-exports from @fro-bot/runtime.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33475140166 |
| Cache | hit |
| Session | ses_fa4791e85ffeewafG3oL989S7N |
Review follow-ups. The post hook built its save without the project-id path, so an archive that landed only there omitted .git/opencode. That was harmless while the post hook was an incidental fallback, but this work made it the documented reason a declined save is acceptable, so the two call sites should agree. The path derives from a runner-level variable available in the post process exactly as the storage and auth paths already are. Closing the acquired server on the bootstrap-failure path was the last statement before returning an error, so a throw there would have rejected from a function whose whole try/catch exists to return a result instead. It is now best-effort, matching how this change already closes databases, and a test drives a throwing close to prove the contract holds. Also pins the usability probe against the object store rather than only the cache. That source wins on restore and its own integrity checks cannot see past the directory, so the probe is its only real defense there. Refs #1407
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The four non-blocking items from the previous round are genuinely closed, and I checked each against the code rather than the summary:
post.tsderivesprojectIdPath.post.ts:97builds it fromgetGitHubWorkspace(), symmetric withcleanup.ts:172, andpost.test.ts:114pins the exact value passed tosaveCache. The reasoning in the comment is right: every other value at that call site (getOpenCodeStoragePath,getOpenCodeAuthPath) is already a runner-level environment read, soGITHUB_WORKSPACEneeded no STATE handoff.- The close is guarded and the contract is tested.
server.ts:109-113wrapsacquiredServer?.close(), andserver.test.ts:258is the case the previous round's spy could not express — it throws fromcloseand asserts the promise still resolves a failedResult. That is the difference between proving the close happens and proving the never-throw contract survives it. - The object-store probe is pinned end-to-end.
cache-restore.test.ts:623-691seeds a corrupt blob under the real object key and drivesrestoreFromObjectStore→syncSessionsFromStore→checkpointDatabase→verifyDatabaseUsablethrough an in-memory adapter. Since the object store wins on restore and its own two integrity checks are near-tautologies for a corruptopencode.db, this was the one source where the probe was the sole defense and nothing observed it. - The skipped-probe reasoning is recorded at
cache-restore.ts:112-119, in the branch it applies to.
I re-verified the error-classification polarity that blocked the earlier round, since it guards the only destructive path in this change. isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist of not a database / malformed, consumed at checkpoint.ts:157, integrity.ts:68, and cache-restore.ts:109. verifyDatabaseUsable returns {usable: true} for any unrecognized throw (integrity.ts:71) — the correct fail-safe for a probe whose caller deletes session history. The classification is pinned against real node:sqlite faults, not mocked message strings: checkpoint.test.ts:197 (a permissions fault yielding unable to open database file, non-structural), integrity.test.ts:78 (a real database at restricted permissions reported usable), and cache-restore.test.ts:503 (an EXCLUSIVE-locked holder leaving storage intact and the run still a hit).
Gates run locally on this head: check-types clean across all four workspace packages; lint 0 errors; bun run build leaves dist/ byte-identical to the committed tree; 354 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/. I also independently confirmed the load-bearing SDK claim behind the new input: node_modules/@opencode-ai/sdk/dist/server.js:4-8 does Object.assign({hostname, port, timeout: 5000}, options), so passing DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS = 5000 explicitly is identical to today's unset behavior.
Blocking issues
None.
Non-blocking concerns
1. hasCacheableContent's doc comment now describes an unreachable state. src/services/cache/save.ts:38-44.
The comment says the checkpoint "can decline (a live writer, an unmergeable log); when it does, the WAL is where the real content still is," and concludes "We must treat any non-empty DB-family file as sufficient evidence of cacheable content." But saveCache returns false at :143 on status === 'failed', so hasCacheableContent at :149 is only ever reached after checkpointed or nothing-to-checkpoint — both of which leave the write-ahead log empty or absent. The WAL can no longer be the sole holder of content at that point.
The code is correct; the comment justifies the WAL branch with a scenario the control flow above it now forecloses. This matters more than a typical stale comment because two commits in this branch exist specifically to fix comments stating rules their own code does not follow, and checkpoint.test.ts:83 cites this comment by name as its regression guard. The header-page-only case the test actually protects is still real — it just arrives through checkpointed, not through a decline.
2. The decline block stays in the job summary even when the post-hook retry succeeds. src/services/cache/save.ts:90-103.
core.summary.write() appends by default (@actions/core/lib/summary.js:71-73 selects appendFile unless overwrite is set). So a run where cleanup declines and runPost then saves successfully still shows "Fro Bot Agent Run — Cache Save Declined" with "The next run may restore an older session," which by then is false. Given that the post-hook retry is the documented reason the decline tradeoff is acceptable (save.ts:83-88, pinned by cleanup-decline-retry.test.ts:71), the common recovery path is exactly the one that leaves a misleading summary. The decline being loud is deliberate and right; it just needs a way to be corrected, or wording that does not assert an outcome the retry usually reverses.
3. A persistently non-checkpointable log is a "never advances" stable state, which the restore-side repair does not reach.
The insight driving this fix is that declining to save is a stable state rather than a recovery, because restore keys are prefixes. The repair breaks that for a poisoned entry. It does not break it for a cache that simply stops advancing: if both cleanup's and the post hook's checkpoints fail non-structurally every run, no new entry is ever written, restore keeps repairing the same old healthy entry, and session continuity silently freezes at that snapshot while every run reports success. Mitigated by (2)'s summary firing every run, so it is loud rather than silent, and I do not think it warrants a fix here — but it is the mirror image of the argument this change is built on, and worth a line wherever the decline is documented.
4. action.yaml's literal default makes one input branch unreachable in production. action.yaml:89, src/harness/config/inputs.ts:363-367.
With default: '5000' declared, core.getInput('server-bootstrap-timeout') never returns an empty string in a real Action run, so the serverBootstrapTimeoutRaw.length > 0 ? ... : DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS fallback is only exercised by tests that omit the input. Harmless and arguably good defense-in-depth, but the two defaults must now be kept in agreement by hand — a 5000 in YAML and a DEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MS in TypeScript, with nothing asserting they match.
5. Object-store corruption is cleared locally but not at the source. Unchanged from the previous round and correctly scoped out: cleanStorage removes the runner's copy while the bucket still serves the same object, self-healing only on a run that reaches syncSessionsToStore. The Actions-cache side does not have this problem because restore keys advance.
6. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50, but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.
Missing tests
- Nothing asserts the
action.yamldefault andDEFAULT_SERVER_BOOTSTRAP_TIMEOUT_MSagree.inputs.test.ts:905pins the constant's value through the unset-input path, but the YAML literal that actually supplies it in production is unobserved. This repo already has workflow/YAML guard tests (scripts/fro-bot-workflow.test.ts), so the shape exists; a drift here would silently change every run's bootstrap budget while every test still passed. - No test observes
saveCache's decline path reachinghasCacheableContent— because it cannot, which is concern (1). A test asserting that afailedcheckpoint returns beforebuildSaveCachePathsis ever called would pin the ordering the comment currently mis-describes, and would fail if a future change reintroduced the decline-then-inspect path the comment imagines.
Coverage is otherwise excellent and the strongest of the three rounds. The end-to-end suite at cache-restore.test.ts:394-592 exercises real databases for all four states the phase distinguishes — hot WAL, malformed-with-WAL, malformed-without-WAL, and EXCLUSIVE-locked — using the real checkpointDatabase / cleanStorage / verifyDatabaseUsable rather than canned outcomes, with an explicit expect(mocks.loggerWarning).not.toHaveBeenCalled() guarding the silent common path at :560. cleanup-decline-retry.test.ts remains the best structural addition: running runCleanup and runPost against one shared state store is the only way to assert the relationship the decline comment documents.
Risk assessment (LOW-MED)
Blast radius is every run — the checkpoint is unconditional on both the restore and save paths, and the usability probe is unconditional on the healthy restore path. What keeps this at the low end is that the destructive branch is no longer reachable by default: it requires SQLite to positively report not a database or malformed, a claim about the file's contents rather than about the runner's state, and the transient faults that would previously have triggered it (fd exhaustion, a full disk, an I/O error, a read-only mount) are verified non-structural against real node:sqlite rather than asserted.
Remaining mitigations are real and I confirmed each: the probe reads a single schema page and never opens a missing or zero-byte file (integrity.ts:57-60), both the probe and the checkpoint sit outside the createOpencode budget (pinned by the ordering assertion at cache-restore.test.ts:360), the repair is bounded by an explicit wall-clock deadline rather than sleep count (checkpoint.ts:128-137), checkpointDatabase never throws, cleanStorage failure is swallowed, and runPost's retry is reachable on the bootstrap-failure path because SHOULD_SAVE_CACHE is set in routing.ts:67, well before runCacheRestore.
The residual concentrates in concern (3): a repository whose OpenCode child consistently outlives both checkpoint windows stops advancing its cache while every run still reports success. Recoverable and loud, but the signal it produces is the one concern (2) makes unreliable.
No security exposure. Public API changes are additive: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms with a warning at inputs.ts:368-373), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME re-exports. ActionInputs.serverBootstrapTimeoutMs is a new required field on a @fro-bot/runtime interface, but that package is private, so no external consumer can break on it.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33483168741 |
| Cache | hit |
| Session | ses_fa4152b4bffeXKpMdqOuiqhD6z |
Review follow-ups. The cacheable-content note justified inspecting the write-ahead log with a case its own control flow forecloses: a declined checkpoint returns before that check is reached, so the log there is always already merged or empty. The header-page-only session it protects is real but arrives the other way, and a test cited the old wording as its guard. The decline summary claimed the next run may restore an older session, but it is appended, not replaced, so it survived the post-hook retry that usually saves the cache moments later and left the run asserting something untrue. It now states only what holds when it is written. Six action inputs declare a default in YAML that a constant repeats in TypeScript, agreeing by hand. A guard now compares them, so a drift fails loudly instead of quietly changing every run. Refs #1407
|
All three closed. The stale comment was the one worth catching. It justified inspecting the write-ahead log with a case the control flow above it forecloses — a declined checkpoint returns before that check is ever reached. The header-page-only session it protects is real, but it arrives through The summary was asserting something untrue on the recovery path. Confirmed The defaults guard found six pairs, not one. Separately, and outside this PR: Gates: types 0, lint 0, |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: CONDITIONAL
The three items from the previous round are genuinely closed, and I checked each rather than reading the summary. The stale hasCacheableContent comment now derives the WAL's emptiness from the control flow above it (save.ts:38-51) instead of justifying it with a decline path that save.ts:151-157 forecloses. The decline summary's wording is now conditional (save.ts:110) with the appendFile reasoning recorded inline, which is the right call over a second synchronized write from post.ts. And scripts/action-input-defaults-guard.test.ts covers six YAML/TypeScript default pairs rather than the one I flagged, parses the YAML string back to a number so '5000.0' cannot pass, and names both locations in its failure message. It runs in CI (ci.yaml:143, bun run test:scripts), so the guard is live rather than orphaned.
Gates on this head: check-types clean across all four workspace packages; bun run build leaves dist/ byte-identical to the committed tree; 481 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, src/harness/config/inputs.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/.
The error-classification polarity that blocked round four remains correct: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:157, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable defaulting to usable: true for any unrecognized throw.
What I cannot pass is a new consequence of this change on the object-store path. I found it by asking what happens to the write-ahead log object now that healthy saves stop producing a local one — the same question deleteRestoredShm already answers for -shm.
Blocking issues
1. The checkpoint orphans opencode.db-wal in the object store, and the orphan reproduces this PR's own trap — permanently — for S3-backed repositories. packages/runtime/src/object-store/content-sync.ts:71-77, src/services/cache/save.ts:147-159, src/harness/phases/cache-restore.ts:101-110.
syncSessionsToStore uploads a transportable file only when it exists locally (content-sync.ts:73-77); when it does not, it continues. It never deletes. ObjectStoreAdapter (object-store/types.ts:7-9) exposes only upload/download/list, so nothing on this path can remove a key.
Before this change the local WAL was effectively always present at save time — that is the PR's own central premise, that nothing ever checkpointed — so the opencode.db-wal object was refreshed on every upload alongside opencode.db. After this change a healthy save checkpoints, checkpointDatabase closes its handle in finally, and SQLite unlinks the log. The author's own comments state this outcome (save.ts:143-146, paths.ts:66-69). So opencode.db-wal stops being uploaded while opencode.db keeps being overwritten at the same prefix.
syncSessionsFromStore then downloads everything list returns, unfiltered (content-sync.ts:128-149), so the next restore lands a fresh opencode.db beside a write-ahead log from an older database state. SQLite's contract is explicit that these must not be combined. I reproduced the exact checkpointDatabase sequence against real node:sqlite on this runner:
state N: WAL with 2000 inserts, snapshot the -wal (4120032 bytes)
close -> SQLite unlinks -wal wal exists? false
state N+1: 3000 more inserts + a DELETE, close cleanly count = 3334
restore shape: fresh opencode.db + stale opencode.db-wal from N
new DatabaseSync(db); prepare('PRAGMA wal_checkpoint(TRUNCATE)').get()
-> throws "database disk image is malformed"
-> isStructuralCorruptionError => true
That routes into handleStructuralCorruption, which calls cleanStorage and deletes the repository's entire session history. The run then saves a fresh opencode.db — and because the local WAL is again absent, the orphaned object in the bucket is still not overwritten. The next run downloads the same stale log against the new database and wipes again.
This is a stable failure state, not a one-time migration cost: exactly the "declining forever is a stable state, not a recovery" argument this PR is built on, reached through the object store instead of the Actions cache. It fires on the second run after upgrade for every repository with s3-backup: true and an existing opencode.db-wal object, which the PR's own premise makes near-universal for that population. The Actions-cache path is unaffected — an entry is an atomic archive per key, so its db and wal always come from the same save.
The -shm half of this same problem was recognized: deleteRestoredShm (restore.ts:81-93) deletes the locally-restored copy of an object that upload no longer refreshes, and content-sync.ts:13-14 explicitly notes download stays tolerant "so objects uploaded before this change still work." The -wal is the same orphan shape with the opposite consequence — it is not safely ignorable, it is actively replayed.
The narrowest fix that needs no interface change: after a successful checkpoint in saveCache, recreate opencode.db-wal as a zero-byte file before buildSaveCachePaths and syncSessionsToStore run. SQLite treats a zero-length -wal as no log, both transports then carry an explicitly-empty one, and the stale object is overwritten rather than left to accumulate. It costs the paths.ts:66-69 comment its "ordinarily absent" claim, which would need updating. Alternatives: add a delete to ObjectStoreAdapter (the S3 adapter already implements a conditional delete) and remove the remote key when the local file is absent; or have syncSessionsFromStore skip a -wal older than the opencode.db object, which needs list to carry LastModified.
Worth noting alongside: the usability probe runs only in the nothing-to-checkpoint branch, so a mixed-WAL database whose checkpoint happens to succeed while leaving the file inconsistent reaches bootstrap unprobed. That is acceptable on its own — re-probing a database the checkpoint just opened adds little — but it means a silent-corruption variant of this scenario has no second line of defense either.
Non-blocking concerns
1. post.ts reports every declined save as "no cache content to save". src/harness/post.ts:113. saveCache returns false for four distinct reasons — checkpoint declined, no cacheable content, cacheId === -1, or a caught exception — and the post hook's log attributes all of them to the second. Not silent (the decline writes its own warning and job-summary block), but this is precisely the "say what is true at the point each message is written" defect the last commit on this branch exists to fix, one call site removed.
2. A persistently non-checkpointable log is a "never advances" stable state. Unchanged from the previous round and correctly scoped out. If both cleanup's and the post hook's checkpoints fail non-structurally every run, no new entry is written, restore keeps repairing the same older healthy entry, and session continuity freezes while every run reports success. Loud rather than silent, thanks to the decline summary.
3. Object-store corruption is cleared locally but not at the source. cleanStorage removes the runner's copy while the bucket still serves the same object. This is the same missing-delete gap as the blocking issue and would likely be closed by the same fix, which is an argument for the adapter-delete variant over the zero-byte one.
4. Release notes. session-retention now actually prunes (a no-op at the default — DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50 — but surprising for anyone who set it low while it was inert), and opencode.db-shm is no longer captured for transport.
Missing tests
- Nothing observes what happens to a previously-uploaded
opencode.db-walobject once the local file stops existing.content-sync.test.ts:74-88asserts that one file is uploaded when the WAL and SHM are absent, which is the upload count — it says nothing about the key that was already in the store. A test seeding both keys, runningsyncSessionsToStorewith onlyopencode.dbon disk, and asserting the store no longer serves a stale-walwould pin the blocking issue's fix. - No end-to-end restore case with a stale
-walobject. The object-store suite atcache-restore.test.ts:623-691seeds only.../sessions/opencode.db(its own comment notes "nothing else, no write-ahead log"), so the mixed fresh-db/stale-wal shape is invisible to the suite. Seeding both keys from different database states through the same in-memory adapter would reproduce the wipe loop directly. - No test pins that a healthy object-store round trip leaves the store's key set consistent across two saves. The single-run tests cannot see divergence that only appears on the second run.
Coverage is otherwise the strongest of the four rounds. The end-to-end suite exercises real databases for all four states the restore phase distinguishes using the real checkpointDatabase/cleanStorage/verifyDatabaseUsable, the classification is pinned against real node:sqlite faults rather than mocked message strings (checkpoint.test.ts, integrity.test.ts), cleanup-decline-retry.test.ts runs runCleanup and runPost against one shared state store, and the new defaults guard closes a drift class that every existing unit test was structurally blind to.
Risk assessment (MED)
Blast radius is every run for the checkpoint and probe themselves, which remain well-mitigated: the destructive branch requires SQLite to positively report not a database or malformed, transient runner faults are verified non-structural against real node:sqlite, the probe reads one schema page and never opens a missing or zero-byte file, both sit outside the createOpencode budget, checkpointDatabase never throws, and cleanStorage failure is swallowed.
The residual is concentrated entirely in the blocking issue, and it is worse than the failure it replaces because the destructive branch is now reachable through a condition this change itself creates rather than through genuine on-disk corruption. Scope is limited to repositories running with s3-backup: true, but within that population it is not a rare corner: it is the steady state from the second post-upgrade run onward, it destroys session history every run, and the run still reports success. Repositories without the object store are unaffected.
No security exposure. Public API changes are additive: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms with a warning at inputs.ts:368-373, and its YAML default now guarded), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. ActionInputs.serverBootstrapTimeoutMs is a new required field on a @fro-bot/runtime interface, but that package is private.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33488201787 |
| Cache | hit |
| Session | ses_fa3dd8ffeffe5l7TF8xcb01Qex |
|
Correction to my previous comment: the claim that I re-ran it in isolation against a clean tree and What I actually observed was four dirty files in The underlying question of what dirtied |
A successful checkpoint did not mean what the code assumed. shutdown()
signalled the OpenCode child and returned without awaiting its exit, so a
still-live connection could append to the write-ahead log immediately
after the checkpoint reported success:
wal before checkpoint 2068272
checkpoint returns {busy:0, log:0, checkpointed:0}, wal -> 0
one late INSERT from an idle live writer wal -> 4152
That is the same root cause as the bug this branch fixes: nothing ever
checkpointed *because* shutdown never waited.
The SDK exposes no way to await the child. createOpencode returns only
{client, server: {url, close}}, and close() calls proc.kill() on a handle
it never surfaces -- no pid, no exit event. So shutdown() now polls the
server's own port until connections are refused, which happens once the
OS reclaims the socket. The pinned server registers no SIGTERM handler,
so termination is immediate rather than drained. This is a boundary, not
a proof, and both the code and its comment say so: a timeout means the
child's fate is unknown, and cleanup warns that the checkpoint which
follows may race a writer.
With the writer actually gone, the log is empty on every path that
reaches transport, so neither transport carries it. Removing it from
DB_TRANSPORTABLE_BASENAMES only covered the object store; the Actions
cache added it independently in buildSaveCachePaths, and both had to
change.
That also fixes a regression this branch introduced. Uploads skip missing
files and never delete, so once healthy saves stopped producing a local
log, opencode.db-wal stopped being refreshed while opencode.db kept being
overwritten at the same prefix. A later restore then paired a fresh
database with a log from an older generation -- reproduced twice, once
surfacing as "database disk image is malformed" (which this branch
answers by deleting session history, then re-saving without overwriting
the orphan, so it repeats every run) and once far worse: the checkpoint
succeeded and silently replayed the stale log, turning 3000 rows back
into 2000 with no error at all.
The two sources need opposite treatment. An Actions cache entry is one
atomic archive, so a legacy pair is consistent and its log may hold
committed transactions -- it is checkpointed, never discarded. The object
store has no generation marker, and the silent-replay case proves SQLite
cannot reliably tell a stale pair from a valid one, so a downloaded log
is untrusted and deleted before anything opens the database. That runs
unconditionally after the sync, because an object store holding only a
sidecar can fall through to a cache miss and leave the file on disk.
Discarding an untrusted log can lose that generation's last
transactions. That is a real cost, recorded in the code, and still
better than silently rolling a session back.
Refs #1407
|
Blocking issue fixed at I argued the log was provably empty on every path reaching transport, so it could simply be dropped. That invariant is false, and I verified the counterexample rather than arguing:
So the fix goes at the root. With the writer actually gone, dropping the log from transport is sound. You were right that this needed both call sites — On legacy handling, I took the source-specific split rather than one rule: an Actions cache entry is atomic, so a legacy pair is consistent and its log may hold committed transactions — it is checkpointed, never discarded. The object store has no generation marker, and the silent-replay case proves SQLite cannot reliably distinguish stale from valid, so a downloaded log is untrusted and deleted before anything opens the database. That deletion is unconditional after the sync, covering the sidecar-only path that falls through to a cache miss. Skipped the adapter delete deliberately. It does not close the non-atomicity window — a crash between upload and delete recreates the same hazard — whereas reducing the transported set to one file closes it permanently. Stale objects are one fixed key per prefix, not unbounded. Recorded in the code that discarding an untrusted log can lose that generation's last transactions. It is a real cost, and better than silent rollback. Gates: types 0, lint 0, |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The blocking issue from the previous round is closed, and — as in round four — the author's fix is better than the one I proposed. I suggested recreating a zero-byte opencode.db-wal so the stale object would be overwritten. That suggestion was wrong for the reason the author demonstrated rather than argued: writing a zero-byte log while a live connection can still append races the same writer that made the checkpoint untrustworthy in the first place. The counterexample they reproduced (wal 2068272 → checkpoint reports success, wal → 0 → one late INSERT → wal 4152) invalidates the invariant my fix depended on, and it is the same root cause as the bug this PR exists to fix: nothing ever checkpointed because shutdown() never waited.
So the fix went to the root instead. I verified each of the four moving parts:
shutdown()now waits.waitForServerQuiescence(server.ts:57-84) polls the child's own listening port until connections are refused. The doc comment is honest about what that is —server.ts:50-55explicitly says a timeout means the child's fate is unknown, not that it is running, and that a window remains even onquiesced: true.cleanup.ts:119-132warns onquiesced: falseand proceeds rather than failing. Calling this "a boundary, not a proof" in the code is the right register.- The write-ahead log left transport entirely, at both call sites.
DB_TRANSPORTABLE_BASENAMES(version.ts:51) is now[opencode.db]for the object store, andbuildSaveCachePaths(paths.ts:119-133) pushes only the main database for the Actions cache. I checked this was needed at both — the two lists are genuinely independent, and fixing only one would have left the Actions-cache half. - Download stays tolerant; restore deletes.
deleteDownloadedObjectStoreWal(restore.ts:124-139) removes a downloaded log before anything opens the database, and runs regardless ofmainDbRestored— which matters for the sidecar-only path that falls through to a cache miss, and is pinned bycache.test.ts:299. - The source-specific split is correct rather than convenient. An Actions-cache entry is one atomic archive, so a legacy
db+walpair genuinely came from the same save and is checkpointed atcache-restore.ts:101, never discarded. The object store has no generation marker across two independently-overwritten keys, so its log is untrusted and deleted. Treating the two transports differently is the correct reading of the difference between them, not an inconsistency.
Declining the adapter delete is also right, and for the stated reason: a delete does not close the non-atomicity window (a crash between upload and delete recreates it), whereas reducing the transported set to one file closes it permanently.
I also want to record something that is easy to miss in the diff. Dropping the log from the Actions cache is not merely neutral — it is safer than what it replaces. In WAL mode the main database file is only mutated during a checkpoint, so archiving opencode.db alone beside a still-appending writer yields a consistent, slightly-stale database. The previous behavior copied a log that a live writer was concurrently appending to, which is the torn-read shape that produces "database disk image is malformed". The cost is bounded and documented; the hazard removed is not.
Gates on this head: check-types clean across all four workspace packages; bun run build leaves dist/ byte-identical to the committed tree; 488 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, src/harness/config/inputs.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/.
The error-classification polarity that blocked round four still holds: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:163, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable defaulting to usable: true for any unrecognized throw.
Blocking issues
None.
Non-blocking concerns
1. The untrusted-log deletion is inside the try, not a finally, and its own comment claims otherwise. src/services/cache/restore.ts:154-165.
The comment at :115-116 states the deletion "Runs unconditionally right after syncSessionsFromStore returns." It runs on the non-throwing path only. syncSessionsFromStore calls fs.mkdir per key inside its download loop (content-sync.ts:148), which is a real throw site under ENOSPC/EACCES. Object keys are listed lexicographically, so opencode.db-wal sorts before any sessions/… sidecar — a throw while creating a directory for a later key leaves the downloaded log on disk, skips the deletion, and lands in the catch at :216, which returns a miss. restoreCache then proceeds to the Actions-cache restore at :276, which can extract a database beside that orphaned log — reconstructing exactly the fresh-db/stale-wal pairing this commit exists to prevent, including its silent-replay variant.
Doubly conditional and genuinely unlikely, which is why it is not blocking. But moving the call into a finally (or hoisting it into restoreCache ahead of the cache restore) is a one-line change that makes the comment true, and this is the fourth comment on this branch to state a rule its own control flow does not follow.
2. isPortOpen sets no socket timeout, so the "bounded" wait is not enforced. packages/runtime/src/agent/server.ts:28-39.
net.connect is awaited with no setTimeout and no deadline of its own. If a connect attempt never emits connect or error, the do/while never advances and waitForServerQuiescence never resolves — and cleanup.ts:121 awaits shutdown() with no outer bound, so the job would hang to the runner's own timeout. On loopback a connect resolves or is refused immediately, so reachability is close to nil and I am recording this rather than objecting. But the function's contract is stated as bounded, and the bound currently lives entirely in an assumption about the OS rather than in the code. A socket.setTimeout(pollIntervalMs, () => finish(false)) would put it there.
Related and smaller: port = Number(parsed.port) yields 0 for a URL with no explicit port, and a connect to port 0 errors immediately — reporting quiesced: true without having checked anything. Unreachable today, since bootstrapOpenCodeServer pins an explicit port and rejects a URL mismatch at :163-171.
3. A second, now-divergent OpenCodeServerHandle still exists. src/features/agent/server.ts:5-9.
It declares shutdown: () => void, while the real handle returned by bootstrapOpenCodeServer is () => Promise<ShutdownResult>. src/features/agent/execution.ts:3 imports this local copy, and index.ts:37 re-exports the runtime one — so the two disagree in the same module tree. No type error results (a Promise-returning function is assignable to a void-returning one), which is precisely why it is worth naming: this is the drift shape that version.ts:8-20 and paths.ts:75-81 both go out of their way to prevent for the DB filename lists, left standing for the handle type this PR just changed. execution.ts only uses server.close(), so nothing is broken today.
4. Post-checkpoint writes are now dropped rather than transported. When shutdown() reports quiesced: false and the child appends after a successful checkpoint, those transactions reach neither transport. paths.ts:68-73 documents the choice and it is the right one — the alternative is the torn-pair hazard — but it is a real cost, and it is silent apart from the quiesced: false warning.
5. post.ts:123 still reports every declined save as "no cache content to save". Carried unchanged from the previous round. saveCache returns false for four distinct reasons and the post hook attributes all of them to one. Not silent — the decline writes its own warning and summary block — but it is the "say what is true at the point each message is written" defect one call site removed.
6. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50, but surprising for anyone who set it low while it was inert). Neither opencode.db-shm nor opencode.db-wal is captured for transport any more.
Missing tests
- Nothing exercises the throw path in
restoreFromObjectStorethat skips the log deletion.cache.test.ts:256and:299cover the sidecar-only cases well, but both are on the non-throwing path. An adapter orfsstub that throws during the download loop, asserting no write-ahead log survives into the Actions-cache restore, would pin non-blocking (1) and would fail today. - No test asserts the
quiesced: falsewarning in cleanup.waitForServerQuiescenceitself is covered in all four states (server.test.ts:335-389), including the timeout and the unparseable-URL case, which is good. Butcleanup.test.ts:47mocksshutdownas{quiesced: true}everywhere, so the branch atcleanup.ts:122-126— the one that tells an operator their checkpoint may have raced a writer — is never observed. - No test covers a legacy Actions-cache
db+walpair surviving the round trip with its committed transactions intact. This is the case the source-specific split deliberately preserves rather than discards, and it is the one half of that decision nothing pins.cache-restore.test.ts:444checkpoints a hot-WAL database, but from a locally-seeded fixture rather than framed as a legacy restore whose data must not be lost.
Coverage is otherwise the strongest of the five rounds. restore-save-flow.test.ts:344 and content-sync.test.ts:53 pin the exclusion at both transports independently, cache-restore.test.ts:693 drives the fresh-db/stale-wal pairing end-to-end through a real object-store adapter, and the classification remains pinned against real node:sqlite faults rather than mocked message strings.
Risk assessment (LOW-MED)
Blast radius is every run: the checkpoint is unconditional on both paths, the usability probe is unconditional on the healthy restore path, and shutdown() now adds up to 5s of cleanup wait when the child does not exit promptly — off the critical path any run is timed against, and correctly distinguished from the bootstrap budget at constants.ts:29-31.
What holds this at the low end is that the destructive branch still requires SQLite to positively report not a database or malformed, and that this round's change reduces the corruption surface rather than trading one hazard for another: the fresh-db/stale-wal pairing is now unreachable by construction on the object-store path, and the torn-WAL-copy shape is unreachable on the Actions-cache path. The residual concentrates in non-blocking (1), which requires a filesystem fault during an object-store download and a subsequent Actions-cache hit to reach — and even then produces the same recoverable wipe this design already handles, with the silent-replay variant as the worse tail.
No security exposure. Public API changes are additive or internal: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms, YAML default guarded by scripts/action-input-defaults-guard.test.ts across six pairs), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. OpenCodeServerHandle.shutdown changing from () => void to () => Promise<ShutdownResult> is a breaking signature change on a @fro-bot/runtime interface, but that package is private and both in-repo implementors (server.ts, remote-client.ts) and all call sites are updated.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33594171130 |
| Cache | hit |
| Session | ses_f9f6fa26dffeI1RCsc7BGA7g4E |
Three follow-ups from review, all of which were true of the code and not of what its comments claimed. The object-store write-ahead log deletion sat inside the try, while its comment said it ran unconditionally after the download. syncSessionsFromStore calls fs.mkdir per key, a real throw site under ENOSPC/EACCES, and object keys list lexicographically so opencode.db-wal arrives before any sessions sidecar. A throw partway through left the log on disk, skipped the deletion, returned a miss, and let the Actions-cache restore extract a database beside it -- rebuilding the exact fresh-db/stale-log pairing the previous commit removed, silent-replay variant included. It now runs in a finally. Hoisting the call up into restoreCache was the other option and would have been wrong: the store-disabled guard returns before the try, so a locally-present log on a persistent self-hosted runner has nothing to do with the object store and must not be deleted. The finally covers every exit from the download attempt and none of the paths where no download happened. isPortOpen settled only on connect or error, so the bound on the quiescence wait lived in an assumption about loopback rather than in the code -- if neither event fired, cleanup awaited it forever. It now has a socket timeout of one poll interval and a settled guard so teardown resolves exactly once. A port that parses to 0 or out of range now reports unconfirmed instead of reporting success without having checked anything. The duplicate OpenCodeServerHandle in features/agent still declared shutdown as returning void after the runtime's began returning a promise. No type error, because a promise-returning function is assignable to a void-returning one, which is exactly why it was worth removing -- the same drift the DB filename lists have guards against. One definition now, imported through the existing sibling re-export, so no new cross-layer import. The throw-path test was checked against the old shape and fails there: the log survived the download failure. Also pins the quiesced:false warning branch, and a legacy Actions-cache pair surviving a round trip with its transactions intact -- the half of the source-specific split that preserves data rather than discarding it. Refs #1407
|
All three closed at Item 1 rejected my own suggested shape, correctly. I had offered hoisting the deletion into The throw-path test was checked against the old shape and fails there — Item 2: Item 3: one Also pinned the On your note about the Actions cache: agreed, and it is the sharper framing. Dropping the log there is not neutral, it is strictly safer — archiving Items 4, 5, and 6 I am leaving. The post-checkpoint write loss is real but is the correct trade against the torn-pair hazard and is already documented at the decision point; Gates: types 0, lint 0, |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
The three items from the previous round are closed, and each one is closed in a shape better than the fix I would have written. The untrusted-log deletion is now in a finally around the whole object-store attempt (restore.ts:224-226), and the author was right to reject my own alternative of hoisting it into restoreCache — the store-disabled guard returns before the try, so a hoisted call would have deleted a locally-persistent write-ahead log on a self-hosted runner with the object store turned off. That reasoning is now recorded at restore.ts:114-129, in the branch it constrains. restore-save-flow.test.ts:483 pins it by throwing from a later key in the download loop and asserting the wal does not survive into the Actions-cache extract — the property, not the call.
isPortOpen now carries a socket timeout and a settled guard (server.ts:36-51), and the port-range check at server.ts:93-95 turns an unreachable invariant into an enforced one. The duplicate OpenCodeServerHandle is gone; src/features/agent/server.ts:4-9 leaves a comment naming the drift instead of a second definition, which is the right residue.
Gates on this head (4f57037): check-types clean across all four workspace packages; 372 tests pass across src/services/cache/, src/harness/phases/, src/harness/post.test.ts, scripts/action-input-defaults-guard.test.ts, packages/runtime/src/agent/server.test.ts, and packages/runtime/src/object-store/; bun run build leaves dist/ byte-identical to the committed tree (git status --short dist/ empty after a full rebuild).
The error-classification polarity that blocked round four still holds: isStructuralCorruptionError (sqlite-errors.ts:48-53) is a positive allowlist consumed at checkpoint.ts:163, integrity.ts:68, and cache-restore.ts:109, with verifyDatabaseUsable returning usable: true for any unrecognized throw.
Blocking issues
None.
Non-blocking concerns
1. The new socket timeout maps "unsettled" to "the child is gone", inverting the safe default of the probe it bounds. packages/runtime/src/agent/server.ts:47.
socket.setTimeout(timeoutMs, () => finish(false)) — and finish(false) means "port closed", which waitForServerQuiescence:100-102 reads as quiesced: true and returns immediately. The comment two lines above names the exact case the timeout exists for ("a firewalled port, a host that silently drops SYNs"), and in that case the function now reports the child has exited rather than that it could not tell. Everything else in this change picks the opposite default: verifyDatabaseUsable returns usable: true for an unrecognized throw, isStructuralCorruptionError returns false unless SQLite positively says otherwise. Here an unknown answer resolves to the optimistic one, and the optimistic one is what suppresses the quiesced: false warning at cleanup.ts:122-126 and the "may have raced a writer" caveat around the checkpoint that follows.
finish(true) on timeout would be consistent: an unsettled connect means "still open as far as I can tell", the do/while keeps polling, and the outer 5s deadline produces the honest quiesced: false. Reachability on loopback is close to nil — a connect either completes or is refused — so this is a polarity nit rather than a live bug, but it is the one place in this design where "I don't know" resolves to "it's fine".
Smaller, same function: finish calls socket.removeAllListeners() before socket.destroy(), which drops the error handler on a socket that may still be mid-connect. Node's afterConnect early-returns for a destroyed socket, so an unhandled 'error' should not escape — but keeping the error listener attached (or destroying first) costs nothing and does not depend on that internal.
2. A failed object-store download can leave a truncated opencode.db that the new probe is gated out of seeing. packages/runtime/src/object-store/s3-adapter.ts:183, src/services/cache/restore.ts:214-219, src/harness/phases/cache-restore.ts:67.
download pipes the response body straight into createWriteStream(localPath). A mid-transfer failure (an aborted body stream, a reset connection) leaves a partial file on disk and returns err, so mainDbRestored stays false and restoreFromObjectStore returns a miss. The finally deletes the downloaded write-ahead log — correctly — but nothing removes the partial main database. restoreCache then falls through to the Actions cache; if that also misses (the normal case for a repository leaning on S3), cacheStatus is 'miss', the repair block at cache-restore.ts:67 is skipped entirely, and a truncated opencode.db reaches bootstrapOpenCodeServer unprobed.
The blast radius is bounded — the run fails, and the next run recovers either from the intact bucket object or via the Actions-cache hit that does run the probe — so this is not the permanent trap the PR set out to break. But it is the same untrusted-artifact shape deleteDownloadedObjectStoreWal already handles for the sidecar, left open for the file that actually matters: a download attempt that did not produce a usable restore should not leave its main database behind either. Deleting it when mainDbRestored === false && failed > 0 is symmetric with the log deletion and needs no interface change.
3. "Archiving opencode.db alone beside a live writer yields a consistent database" is conditional on wal_autocheckpoint. This framing (mine, from the previous round, and agreed in the author's reply) holds only while nothing checkpoints during the archive. SQLite's default wal_autocheckpoint is 1000 pages, so a still-live child that appends ~4 MB while @actions/cache or syncSessionsToStore is reading the file will mutate the main database mid-read. It requires quiesced: false plus a genuinely busy writer, and the previous behavior (copying a concurrently-appended log) was strictly worse — so this changes nothing about the decision, only about how strongly the safety claim should be stated in the residual-risk story.
4. post.ts:123 still reports every declined save as "no cache content to save". Carried unchanged and explicitly deferred by the author to the broader decline-reason plumbing. Noted, not re-argued.
5. Release notes. session-retention now actually prunes (a no-op at the default, since DEFAULT_SESSION_RETENTION and DEFAULT_PRUNING_CONFIG.maxSessions are both 50), and neither opencode.db-shm nor opencode.db-wal is captured for transport any more.
Missing tests
- Nothing exercises
isPortOpen's socket-timeout branch — the branch this commit added, and the one that decides concern (1)'s polarity. All fourwaitForServerQuiescencecases (server.test.ts:334-389) use a real listener that either accepts or refuses, so the timeout path is never entered. A case pointing at a black-holed address (e.g.http://10.255.255.1:9/) with a short budget would pin the intended outcome; if that is too environment-dependent for CI, extracting the timeout mapping so it can be asserted directly is the cheaper alternative. - No test covers a mid-transfer download failure leaving a partial
opencode.db.content-sync.test.tscovers a rejected key and a failed download by return value, but nothing asserts what remains on disk afterward, andcache-restore.test.ts's object-store cases all reach the probe through a hit. A test seeding a download that writes bytes and then returnserrwould make concern (2) visible; today it is invisible to the suite. - No test pins the object-store upload path's behavior when the checkpoint declined.
save.ts:152-158returns beforesyncSessionsToStore, so a declined save uploads nothing — worth pinning, since the whole "the bucket self-heals on the next successful save" argument in the design notes depends on it.
Coverage is otherwise the strongest of the six rounds. cache.test.ts:913 now covers the half of the source-specific split that preserves data (a legacy Actions-cache db+wal pair surviving with its committed transaction intact), cleanup.test.ts:317 pins the quiesced: false warning branch, restore-save-flow.test.ts:483 proves the finally fix against the shape that would have failed before it, and the classification remains pinned against real node:sqlite faults rather than mocked message strings.
Risk assessment (LOW-MED)
Blast radius is every run — the checkpoint is unconditional on both the restore and save paths, the usability probe is unconditional on the healthy restore path, and shutdown() can now add up to 5s of cleanup wait (off the critical path any run is timed against, and correctly distinguished from the bootstrap budget at constants.ts:21-33).
What holds this at the low end is unchanged and re-verified: the only destructive branch requires SQLite to positively report not a database or malformed — a claim about the file's contents, not the runner's state — and the transient faults that would previously have reached it are pinned non-structural against real node:sqlite. The probe reads one schema page, never opens a missing or zero-byte file, checkpointDatabase never throws, cleanStorage failure is swallowed, and a declined save is retried by the post hook with projectIdPath now symmetric between the two call sites.
The residual sits in concern (2) — a bounded, self-healing bootstrap failure rather than a loop — and in concern (1), which degrades a warning rather than a decision. Neither is reachable on a healthy run.
No security exposure. Public API changes are additive or internal: the server-bootstrap-timeout input (positive-validated, clamped to 120000ms, YAML default guarded across six pairs by scripts/action-input-defaults-guard.test.ts), the checkpointDatabase / verifyDatabaseUsable / cleanStorage exports, and the DB_*_BASENAME / DB_TRANSPORTABLE_BASENAMES re-exports. OpenCodeServerHandle.shutdown changing from () => void to () => Promise<ShutdownResult> is a breaking signature change on a @fro-bot/runtime interface, but that package is private and every in-repo implementor and call site is updated — now including the local duplicate that previously masked it.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 33597659542 |
| Cache | hit |
| Session | ses_f9f40e38bffeTg5hcSs7UTSpVN |
* docs(solutions): record why repair must precede capture Writes up #1407 and #1519 for docs/solutions/. The durable lesson is an ordering rule -- repair on restore has to happen before capture on save, or a transient failure gets encoded into durable state that the next run then trusts. The most useful section is what didn't work. Five plausible fixes were tried and refuted by evidence: "only persist cleanly-checkpointed state" (nothing had ever produced one), "declining to save breaks the loop" (save keys are unique per run and restore keys are prefixes, so the poisoned entry stays newest), "wipe and save empty state" (destroys history on every transient failure), "the log is provably empty at transport" (a late write from a live writer regrew it after the checkpoint reported success), and "route non-retryable failures into the clean-slate path" (an allowlist of retryable, negated, hands a full disk to the delete branch). Also records the regression the fix itself introduced, since it was caused by the fix and caught only in review: once healthy saves stopped producing a local write-ahead log, an upload that only ever sends what exists stopped refreshing that object while the database beside it kept being overwritten. One reproduction reported malformed; the other silently replayed stale content and turned 3000 rows into 2000. The safety claim about archiving the database alone is stated conditionally rather than absolutely -- SQLite's default wal_autocheckpoint is 1000 pages, and a live writer crossing it mutates the main file with no explicit checkpoint, measured going from 4096 to 860160 bytes. Overlap against the four adjacent docs scored Moderate at highest, so this is a new doc cross-linked in both directions rather than folded into an existing one. Refs #1407 * docs(solutions): add the backlinks the description claimed The PR body said the new doc was cross-linked in both directions. It wasn't -- only forward links existed, and no existing doc referenced it. This repo does follow a backlink convention, so the claim was checkable and wrong. Adds entries from the two highest-overlap neighbours, which are where a reader is most likely to be standing when this becomes relevant: the cache-continuity doc (a save that failed and reported success, against one that succeeded and persisted the wrong thing) and the S3 restore-scope doc (the same restore-is-not-save asymmetry one layer down -- capability there, atomicity here). Also names the remedy in the regression section. These docs are retrieved by section rather than read end to end, so a reader landing there from a `malformed` search could have concluded the hazard was still live; the fix was described in two other sections but never pointed at from the one that describes the problem. Drops related_components. Both values were novel to the corpus and the field is unguarded, so they would have seeded near-synonyms for no retrieval benefit -- tags already carry sqlite and session-cache. * docs(solutions): fix inverted deixis in both backlinks Both backlinks were written from the new doc's vantage point and pasted into the neighbour docs without flipping "there" and "here", so each one described the two documents backwards. The convention is established three lines above the first one, in the same bulleted list: there = the linked doc, here = the doc you are reading. So that file simultaneously claimed "here = a write that failed and reported success" and "here = the save genuinely succeeded", about itself. The cache-continuity doc is the failed-save-reported-success incident; the new doc is the succeeded-save-persisted-poison one. The S3 backlink had the same inversion: that doc is the capability half of the restore-is-not-save asymmetry, and the new one is the atomicity half. Both links resolved the whole time, which is all check-md-links can observe -- the text they resolved to was the part that was wrong. Fitting for a doc about claims that are checkable and unchecked. * docs(solutions): correct the count and heading case "Five plausible fixes" undercounted the seven that follow -- the two it dropped are the zero-byte-log recreation and the restoreCache hoist, both of which were refused with reasons worth keeping. Carried unresolved through two reviews, in the section the doc itself calls the useful one. Also aligns the regression heading with the title case the rest of the file uses.
…it (#1521) * fix(runtime): report an inconclusive port probe as unknown, not as exit Two follow-ups from the #1519 review, both cases where an unknown answer resolved to the optimistic one. The quiescence probe's socket timeout called finish(false), and false means "port closed", which waitForServerQuiescence reads as quiesced and returns immediately. So a connect that neither completed nor was refused -- the one case the timeout exists for -- reported that the OpenCode child had exited. That suppressed the warning telling an operator their checkpoint may have raced a live writer, which is the whole reason the signal exists. Every other unknown in this design resolves the other way: verifyDatabaseUsable returns usable for an unrecognized throw, and isStructuralCorruptionError returns false unless SQLite positively says otherwise. It now returns true, so an inconclusive attempt keeps polling and only the real deadline produces quiesced: false. Testing it needed the socket to be injectable, since a real one either connects or is refused and never reaches the timeout. The probe now takes a connect function, and a fake socket drives the branch directly rather than depending on a black-holed address behaving consistently in CI. Second: a mid-transfer failure in the S3 adapter's download left a partial opencode.db on disk. mainDbRestored stays false, so restore reports a miss and falls through to the Actions cache; if that also misses -- the normal case for a repository leaning on S3 -- the repair block is skipped entirely and a truncated database reaches bootstrap unprobed. The suggested condition for cleaning it up was mainDbRestored === false && failed > 0 at the restore layer. That is unsafe: syncSessionsFromStore increments failed for any key in the batch, so an unrelated sidecar failure satisfies it while opencode.db was never attempted -- and on a self-hosted runner with a pre-existing local database, deleting it destroys live session state the failed download never touched. Fixed at the write site instead. download() unlinks localPath only when the pipeline itself throws, meaning the stream was open and a partial write happened. A rejection before the pipeline is reached never touches the path, so an existing file is provably safe -- pinned by a test. Both fixes were checked against their pre-fix shapes and fail there. Refs #1407 * test(runtime): pin the quiescence invariant at the call site The polarity fix was pinned inside isPortOpen but not where it is read. waitForServerQuiescence called it without an injector, so nothing proved that an inconclusive probe keeps the loop polling -- and inverting the read at the call site to `if (stillOpen) return {quiesced: true}` passed every test in this PR, reintroducing the bug one line from where it was fixed. The optional connect parameter now threads through, defaulted so the production call site is unchanged, and a test drives several poll cycles of nothing-but-timeouts and asserts the only reachable outcome is quiesced: false at the deadline. Verified against the inverted read: it fails there. Also narrows the s3-adapter cleanup comment, which claimed the unlink can only reach a file the pipeline just opened. True for a mid-transfer failure, not for an open failure -- if createWriteStream cannot open the destination at all, pipeline rejects before truncating and the unlink still fires against a file this attempt never wrote to, since unlink needs permission on the directory rather than the file. The comment now separates what is guaranteed from what is merely rare, and the behavior is unchanged. And records what the polarity flip costs: an inconclusive probe now rides out the full 5s budget instead of returning after one poll interval. One server per run and bounded, but whoever next tunes those constants shouldn't have to rediscover it. * test(runtime): decouple the quiescence test from the wall clock Two review follow-ups, both in test code. The call-site test asserted connectCalls > 1 against a 60ms budget with a 10ms poll interval. The do/while checks the deadline only after the first iteration, so a single stalled delay() on a contended runner exits after one probe and fails that assertion. Widened to a 500ms budget with a 5ms interval so a stall cannot end the loop before a second probe. The quiesced: false assertion was never at risk either way -- a stalled loop still cannot produce quiesced: true -- so this only removes a spurious red, and the comment now says so. The s3-adapter test title claimed the pre-existing file was "at a different path", which would be trivially true and worth nothing. It is at the same path the download targets, which is the entire point: a pre-pipeline rejection leaves untouched the exact file the cleanup would otherwise delete. Title now matches what the body actually pins.
Closes #1407.
A run whose OpenCode server failed to start persisted session state that made the next run fail the same way. Nothing in the codebase could recover from it — both reporters escaped only by deleting every
opencode-*cache entry by hand.What was actually wrong
Nothing ever checkpointed the database — not just failed runs.
save.tsalready said so:server.close()sendsproc.kill()without awaiting a checkpoint, so a valid session can sit almost entirely inopencode.db-wal.shutdown()is synchronous and nothing awaits the child's exit, so every cached database was transported with a hot write-ahead log. The comment twenty lines away incleanup.tsclaimed the opposite — that a clean shutdown checkpoints — and it was wrong.That reframed the fix. "Only persist state that was cleanly checkpointed" would have blocked every save, because nothing was. The checkpoint had to be added, not gated on.
Declining to save cannot escape a poisoned cache. This was my own first draft's central error. Save keys are unique per run and restore keys are prefixes returning the most recent match, so a run that declines to save writes no entry and the next run restores the same bad one. Declining forever is a stable state, not a recovery — which is why the reporters needed a manual purge.
The triage's control flow was wrong too. It concluded
runCleanupnever executes on bootstrap failure, so the post hook was the culprit.run.tsruns cleanup in afinally, so it does execute with a null server handle — skipping prune and shutdown, then saving anyway. A fix aimed only at the post hook would have missed the primary path.The fix
The repair happens on restore, between
restoreCacheandbootstrapOpenCodeServer. No timer spans those calls — the budget coverscreateOpencodealone — so the recovery cost moves out of the window that was failing instead of being paid inside it. A stuck repository heals itself on its first run, with its session history intact.Wiping the database would also break the loop, and was rejected: it discards history on every transient bootstrap failure, which is the second reporter's case exactly.
The save side checkpoints inside
saveCache, which both the cleanup phase and the post hook already call and which also owns the object-store sync — so every capture path inherits it structurally, rather than through a flag two callers must agree to read. A log that cannot be merged declines the save and says why in the job summary.Alongside:
-shmis a machine-local wal-index SQLite never syncs, so it is no longer captured and any copy carried in by an older cache is deleted on restore. Declaring storage corrupt now actually clears the database, which lives beside the storage directory rather than inside it. The bootstrap budget is configurable and reported, bounded at two minutes. Andsession-retentionfinally reaches pruning — it was parsed, validated, logged, and never read.The object-store restore path also gained the corruption and version checks it had been returning before ever running — but that is a smaller gain than it sounds, and the first draft of this description overstated it.
checkStorageCorruptiononly asks whether the storage path is a readable directory, which themkdirtwo lines above it guarantees, andcheckStorageVersiontreats an absent.versionas compatible, which it will be for an object-store restore. Both therefore pass for a corruptopencode.db. The checkpoint attempt is the only thing on that path that actually inspects the database.Measurements
Opening a database with a hot 185 MB write-ahead log: 2416.8 ms, against 60.2 ms once checkpointed. Decisive at the first reporter's scale.
It does not explain the second. That repo's entire cache was 21 MB, where recovery is a few hundred milliseconds — not enough to turn 1.88 s of headroom into a timeout. The two incidents have different dominant causes, recovery cost and runner variance, and both are addressed. Neither fix alone covers both reports.
node:sqliteis available unflagged on Node 24.20.0, so the checkpoint adds no dependency. Its pragma returned{busy: 0, log: 0, checkpointed: 0}while truncating a 1.1 MB log to zero, so success is judged by the log's size on disk rather than that count.Worth reviewing
VACUUM INTOor the backup API would be the textbook answer for transport, and were rejected because the cache is the next run's live working set, not an archive — a snapshot only moves the "make it live" step to restore time.One residual, noted rather than solved: an object-store upload failure is non-fatal while restore prefers the object store, so the Actions cache can hold newer state than the copy that wins. That predates this change and has a different fix.
Verification
Types 0, lint 0,
dist/reproducible, 6,332 tests. A 12-reviewer pass ran before this was pushed; its findings are in the branch.