Skip to content

fix(store): prevent FileStore persistence starvation during continuous writes - #2346

Merged
DeliciousBuding merged 2 commits into
masterfrom
fix/file-store-persist-starvation
Sep 6, 2026
Merged

fix(store): prevent FileStore persistence starvation during continuous writes#2346
DeliciousBuding merged 2 commits into
masterfrom
fix/file-store-persist-starvation

Conversation

@DeliciousBuding

@DeliciousBuding DeliciousBuding commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Anchor FileStore's existing 50 ms batching window to the first pending write instead of postponing it after every notification. Continuous writes now make repeated durable progress without waiting for a quiet period.
  • Keep asynchronous writes and coalescing, re-arm later batches, avoid idle polling, and stop the timer when the persist loop exits.
  • Preserve full-snapshot encoding, Sync/rename, the channel capacity, backend defaults, failure reporting and close flushing. No new abstraction or configuration.
  • Update the stable FileStore lifecycle description to make the scheduling change explicit. The 50 ms window is not an I/O completion SLA.

Closes #2345

Evidence and tests

The unchanged-source regression drove 40 subsequent writes at half the existing interval: after 20 intervals, memory had 41 items and the actual disk restore path had zero. There was no final Flush/Close before the recovery read. This concerns the supported file backend; it is not a per-token transcript-write or production-frequency claim. Earlier explicit-flush cost measurements (#2256 / #2333 / #2334) are unchanged and do not cover this timer liveness bug.

Four tests use the real constructor, public writes, actual snapshot files and the real recovery reader under testing/synctest virtual time:

  • Sustained progress includes both the first item and a late item, requiring multiple batches.
  • Short bursts coalesce before the first-write deadline; later writes cannot extend it; a later batch re-arms.
  • A real filesystem fault is invisible while idle but reported after a new batch; explicit Flush recovers pending data.
  • Close saves a pending batch without a timer advance or prior Flush.

Four temporary negative controls each failed behaviorally and were restored byte-for-byte: original reset-on-every-signal loop; missing re-arm; zero-delay/no coalescing; missing close flush.

Passed locally (Go 1.26.5, from edge-server):

  • Windows, CGO_ENABLED=0: go test -p 2 ./internal/store ./internal/api ./cmd/agenthub-edge -count=1 -cpu=4 -timeout=10m.
  • Linux, CGO_ENABLED=1: same package set with go test -race -p 2 ... -count=1 -cpu=4 -timeout=10m.
  • go vet -p 2 and staticcheck for the same three packages; gofmt clean.
  • verify-doc-ssot.py, verify-project-skills.py, verify-real-e2e-contract.py, OpenAPI YAML parse and git diff --check. Existing absent-skill-root checks keep their trivial-pass/skip boundary and are not real E2E evidence.

Boundaries / pending

No public API, snapshot schema, retained-data policy, SQLite/SQL/WAL change, runtime configuration, deployment, restart or production load test. No automatic persist retry policy added. Virtual-time tests demonstrate scheduling correctness, not wall-clock throughput, actual process-kill recovery or a 50 ms durability guarantee. L3 real login/model and L4 packaged Desktop are not run. All seven required checks passed on 10f40b20205633838d050b70c599bea7bf1a5a8e; workflow 34036655934 attempt 2 succeeded. The additional complete Windows Edge suite (go test ./... -short -count=1 -timeout 15m, CGO_ENABLED=0) also passed. CodeRabbit reported Review skipped, not approval.

CI retry evidence

The first Windows Edge job reported all tested packages, including store and lifecycle, as passed, then Go failed while removing lifecycle.test.exe with Access is denied. Its exact lock owner was not identified. After inspecting that log, only the failed job and dependent jobs were requested for retry; the same source passed on attempt 2, without changing code, assertions or workflow gates. This PR does not claim to fix a lifecycle executable-lock issue.

DeliciousBuding and others added 2 commits September 6, 2026 21:21
Reproduce #2345 with real snapshot recovery before final Flush/Close and virtual-time continuous writes. The existing FileStore suite passes on the unchanged production base; this focused regression fails with 41 memory items and no durable items.

Co-authored-by: Codex <codex@vectorcontrol.tech>
Keep the existing 50ms asynchronous coalescing window without resetting it for every mutation, so continuous writes cannot starve durable snapshots. Preserve idle, error and close semantics, with real-file virtual-time regressions and an aligned architecture description.

Co-authored-by: Codex <codex@vectorcontrol.tech>
@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: fae38a73-d829-4fcb-880a-0d522b45a899

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@DeliciousBuding
DeliciousBuding merged commit a3f044b into master Sep 6, 2026
79 of 81 checks passed
@DeliciousBuding
DeliciousBuding deleted the fix/file-store-persist-starvation branch September 6, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(store): prevent FileStore persistence starvation during continuous writes

1 participant