You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Observed on the real operator profile (proj_a5b3d7e3ebe14ca7, ~4800 files, durable graph ~12.4 GB on disk) running PR #707 tip builds (4f920d6f7313, then 455ed623ee30 which includes the #754 semantic-commit fix 183f18a9ac).
After the corrupt durable graph store was quarantined (see companion issue), each daemon incarnation rebuilds/replays the durable project graph (tracedecay.grafeo). During that replay the daemon's anon RSS grows far past the on-disk graph size and the kernel OOM-kills it (oom_score_adj=200 makes it the preferred victim). The unit auto-restarts, the replay restarts, and the loop repeats whenever peer processes hold enough memory.
18:35:27 UTC — anon-rss 22,658,740 kB, same incarnation pattern
This is NOT the semantic vector commit path (#754, fixed by 183f18a): the semantic runtime never left installed during these windows; no model was loaded. The growth tracks the code-graph projection/replay writer. The shape matches the #754 defect class (whole-corpus accumulation instead of O(batch)) and also resembles #729 (startup catch-up sync unbounded), but in the durable graph replay path.
Impact: on a shared box the daemon cannot finish reprojecting a large profile graph; every OOM SIGKILL also risks corrupting the store mid-WAL-write (that is precisely how the corrupt store in the companion issue was produced). RSS curves with per-30s samples (load, MemAvailable, VmRSS/VmHWM) are preserved in /fast/tmp/td-sem-verify/rss2.ndjson.
Observed on the real operator profile (proj_a5b3d7e3ebe14ca7, ~4800 files, durable graph ~12.4 GB on disk) running PR #707 tip builds (
4f920d6f7313, then455ed623ee30which includes the #754 semantic-commit fix183f18a9ac).After the corrupt durable graph store was quarantined (see companion issue), each daemon incarnation rebuilds/replays the durable project graph (
tracedecay.grafeo). During that replay the daemon's anon RSS grows far past the on-disk graph size and the kernel OOM-kills it (oom_score_adj=200makes it the preferred victim). The unit auto-restarts, the replay restarts, and the loop repeats whenever peer processes hold enough memory.Three OOM kills on 2026-08-30 alone:
tracedecay.grafeowas being repopulated (57 KB → 1.42 GB written)This is NOT the semantic vector commit path (#754, fixed by 183f18a): the semantic runtime never left
installedduring these windows; no model was loaded. The growth tracks the code-graph projection/replay writer. The shape matches the #754 defect class (whole-corpus accumulation instead of O(batch)) and also resembles #729 (startup catch-up sync unbounded), but in the durable graph replay path.Impact: on a shared box the daemon cannot finish reprojecting a large profile graph; every OOM SIGKILL also risks corrupting the store mid-WAL-write (that is precisely how the corrupt store in the companion issue was produced). RSS curves with per-30s samples (load, MemAvailable, VmRSS/VmHWM) are preserved in /fast/tmp/td-sem-verify/rss2.ndjson.