Skip to content

Corrupt durable graph store is never quarantined: graph activation retries GRAFEO-X002 forever, keeping the project runtime unadmitted #763

Description

@ScriptedAlchemy

On the real operator profile, a SIGKILL during a WAL write (systemd stop timeout, see companion issue) left tracedecay.grafeo (12.4 GB) with a permanent CRC fault. From then on every graph activation attempt failed identically:

WARN tracedecay_code_index_runtime::code_index_scheduler::registry: graph activation failed; the sealed generation stays unseated until a new generation publishes event="code_index_graph_activation_failed" error=code-index graph projection failed: code graph database is corrupt: GRAFEO-X002: Serialization error: block 18 CRC mismatch: expected 7d877cc5, got 5a475db3

24+ identical failures logged 15:45–17:13 UTC on 2026-08-30, across multiple daemon incarnations. "Until a new generation publishes" never helps: every new generation projects through the same corrupt durable store, so the same block-18 CRC error recurs. The project runtime therefore stays unadmitted indefinitely: every project-scoped invocation (Work, workflow, semantic activation) refuses Unavailable at the front door, retention degrades (no code generation is currently serving), and nothing self-heals.

Recovery required manual surgery: stop daemon, mv tracedecay.grafeo tracedecay.grafeo.wal into a quarantine dir, restart. The daemon then recreated the store and re-projected from sealed generation replay journals with no data loss — proving a safe automatic quarantine path exists.

Expected: a corruption-class projection failure (GRAFEO-X002 and kin) on the durable graph store is a typed, terminal-for-this-store state that triggers quarantine + reprojection from replay journals as bounded background work, not an infinite identical retry that permanently disables the project's invocation surfaces.

The reasonless front-door Unavailable that masked all of this is fixed on the PR branch (652e2d7 logs the refusal cause).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions