Skip to content

fix(phase2): never reset the baseline after a no-op consolidation - #6

Open
Tony-ooo wants to merge 1 commit into
moritzfl:mainfrom
Tony-ooo:fix/phase2-noop-consolidation-diff-loss
Open

Tony-ooo wants to merge 1 commit into
moritzfl:mainfrom
Tony-ooo:fix/phase2-noop-consolidation-diff-loss

Conversation

@Tony-ooo

Copy link
Copy Markdown

Problem

A phase-2 run whose consolidation agent changed nothing was recorded as succeeded.

consolidateViaSubagent resolves as soon as the helper session closes and only throws on prompt/timeout/shutdown failure. The consolidation contract explicitly permits a no-op:

No-op content updates are allowed and preferred when there is no meaningful, reusable learning worth saving.

  • INCREMENTAL UPDATE mode: if nothing is worth saving, make no file changes.
    — src/templates/consolidation.md

So a helper can read phase2_workspace_diff.md, change nothing, and exit cleanly. validateConsolidationArtifactsForVersion cannot catch this: a stale MEMORY.md and a v1-headed memory_summary.md are indistinguishable from fresh ones. The run therefore proceeded to resetBaseline and markPhase2Succeeded.

resetBaseline is the damaging part. It folds the un-consolidated rollout summaries into the git baseline, so the next pass sees zero changes, takes the no_workspace_changes early return, and never invokes the consolidator again. The new memories are stranded in the workspace permanently and silently, while every status field reports success.

The contract also states the opposite requirement — "Every changes in phase2_workspace_diff.md are authoritative and must propagated and consolidated" and "You should always make sure MEMORY.md and memory_summary.md exist and are up to date" — so a helper that no-ops over a diff full of new rollouts is not making a legitimate call.

Observed on OpenCode 2.0.16 / Windows, plugin 0.9.0

Two consecutive production runs, both reported done with an empty last_error:

phase2  09:26:21Z -> 09:41:14Z  (15 min)   MEMORY.md / memory_summary.md mtime unchanged
phase2  09:59:35Z -> 10:17:03Z  (17 min)   MEMORY.md / memory_summary.md byte-identical

22 rollout_summaries/*.md and 164 KB of raw_memories.md sat un-consolidated; memory_search returned no match for any of them. The consolidator's only tool calls in the server log were three rg phase2_workspace_diff.md reads, then nothing.

Fix

Fingerprint the artifacts the consolidator owns (MEMORY.md, memory_summary.md, skills/) before and after the helper runs. If the fingerprint is unchanged while the diff carried learning material, fail the job with a new no_artifact_changes status and skip resetBaseline, so the diff survives for the next attempt and the condition shows up in memory_inspect rather than vanishing.

diffCarriesNewLearning is scoped to the two input families the contract names as consolidation triggers — files under rollout_summaries/, and extensions/<name>/resources/. raw_memories.md is deliberately excluded: rebuildRawMemories([]) always writes a "No raw memories yet." placeholder, so on a first INIT run that file alone would look like new material and turn a legitimately empty consolidation into a permanent retry loop. A new or updated stage-1 output always changes the rollout_summaries set too, since the file stem embeds source_updated_at, so nothing real is lost.

Note on retry behaviour: phase 2 never exhausts retries by design (store.ts: "retry_remaining is informational only"), so a helper that keeps no-opping will retry hourly instead of being forgotten. That is the intended trade — visible and self-healing beats silent and permanent — but it is a behaviour change worth a maintainer opinion.

Verification

tests/phase2.test.ts gains two cases: the no-op is rejected with the diff preserved, and a no-op is still accepted when the diff genuinely carries nothing. tests/workspace.test.ts covers the fingerprint and the learning-material predicate, including the first-INIT non-regression.

Driven against pristine 0.9.0 sources, a consolidated workspace, and a stubbed helper (1 = pristine 0.9.0, 2 = this patch):

check 1 2
no-op run does not report success reported succeeded no_artifact_changes
job marked failed done failed
error names the no-op no yes
workspace diff preserved no — rollouts consumed yes
control: helper writes -> succeeded, watermark advanced, baseline reset pass pass

The control case is unchanged, so the happy path is untouched.

Local test-run caveat

I could not run the upstream suite unmodified on this machine: Bun 1.2.5 on Windows fails every fs.openSync() call that passes numeric flags — including plain O_WRONLY on an existing file — with ENOENT, while the string-flag form ("w", "wx") works. src/path-guard.ts uses numeric flags, so ensureLayout throws and the suite collapses before reaching any assertion. This does not affect production: OpenCode's bundled Bun handles numeric flags correctly, and the plugin demonstrably runs. I verified with a standalone harness (one process and one memory root per scenario) plus bun run typecheck, which passes. Flagging it as a separate portability issue rather than folding it into this PR, since path-guard.ts is security-sensitive and the fix belongs in its own change.

// Bun 1.2.5, Windows, directory confirmed to exist
fs.openSync(p, fs.constants.O_WRONLY)                              // ENOENT
fs.openSync(p, fs.constants.O_WRONLY | O_CREAT | O_EXCL)          // ENOENT
fs.openSync(p, "w")                                               // ok
fs.openSync(p, "wx")                                              // ok
fs.writeFileSync(p, "x")                                          // ok

A phase-2 run whose consolidation agent changed nothing was recorded as
`succeeded`: the git baseline was reset and the watermark advanced, even
though the new rollouts were never promoted into MEMORY.md or
memory_summary.md. Because the baseline reset is what makes the workspace
diff disappear, the next pass saw zero changes, returned
`no_workspace_changes`, and never invoked the consolidator again. The
un-consolidated memories were then stranded in the workspace permanently
and silently, while every status field reported success.

The consolidation contract explicitly allows this run: "No-op content
updates are allowed and preferred when there is no meaningful, reusable
learning worth saving." So the host cannot infer success from a clean
`consolidateViaSubagent` return, and
`validateConsolidationArtifactsForVersion` cannot catch it either - a
stale MEMORY.md and a `v1`-headed summary are indistinguishable from fresh
ones.

Fingerprint the artifacts the consolidator owns before and after the helper
runs. If nothing changed while the diff carried rollout summaries or
extension resources, fail the job with the new `no_artifact_changes`
status and skip `resetBaseline`, so the diff survives for the next attempt
and the condition is visible in `memory_inspect` instead of silent.

`raw_memories.md` is excluded from the "carries learning" test on purpose:
`rebuildRawMemories([])` always writes a placeholder, so on a first INIT
run it would look like new material and turn a legitimately empty
consolidation into a permanent retry loop. A new or updated stage-1 output
always changes the rollout_summaries set too, since the file stem embeds
source_updated_at.
@moritzfl

moritzfl commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Thanks for investigating. Two unchanged runs with only three searches for the diff are suspicious and deserve investigation.

However, our docs explicitly allow no-ops, matching Codex ("A no-op is allowed: if nothing new is worth writing, the correct result is unchanged files."). I won’t merge this guard because it would also retry legitimate no-ops indefinitely.

Could you share the three tool calls’ arguments/results and the helper’s final message, finish reason, and idle outcome, with sensitive content redacted? Also, which memory_search queries returned nothing? Rollout summaries should be searchable before consolidation.

I added regression coverage in e1343fa, crediting this PR, but haven’t reproduced your incident yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants