rivet_core test binary OOMs CI: up to 100 GB RSS triggers system-wide OOM-killer on lean-mem runners

[avrabe]

Summary

The rivet_core test binary repeatedly grows to 70–100 GB resident and trips the system-wide Linux OOM-killer on the self-hosted lean-mem runners. Because it's a kernel (not cgroup-local) OOM, it can kill neighboring jobs, not just the offending one. On a 125 GB host, a single test using ~100 GB is fatal to the whole box.

Evidence (kernel OOM log, pulseengine-ci-01)

Jun 19 06:28  Killed process (rivet_core-715e)  total-vm:168.9 GB  anon-rss:94.8 GB  UID:1102
Jun 22 11:11  Killed process (rivet_core-98fe)  total-vm:172.1 GB  anon-rss:99.9 GB  UID:1104
Jun 23 10:24  Killed process (rivet_core-98fe)  total-vm:112.3 GB  anon-rss:72.4 GB  UID:1103

• rivet_core-<hash> is the rivet-core crate's compiled test binary (a native cargo test/nextest binary, not Miri).
• UIDs 1102–1104 = self-hosted runners 2/3/4, which are the lean-mem class. These runners exercise the rivet-core test suite via the RAM-heavy jobs pinned there (Miri / mutants-core). The OOMs cluster on high-load days (host ldavg ~60 on 32 cores Jun 22–23).
• 3 kernel OOM kills in 10 days — every one is rivet_core. No other process OOMs.

Why this is serious

anon-rss of 95–100 GB means the test process itself is allocating ~100 GB of heap — not build/link, not cache. That's a single test (or a fan-out of parallel test threads each holding large state) accumulating enormous memory. With MemoryHigh=32 GiB (soft) and no hard cap, it sails past 32 G, thrashes swap, and the kernel eventually OOM-kills system-wide.

What infra is doing (and why it changes this job's behavior)

We're adding a hard MemoryMax guardrail (~48 G) on the lean-mem cgroup so a runaway is killed cleanly inside its own cgroup instead of taking down the host/neighbors. After that lands, this test will FAIL fast at ~48 G instead of intermittently OOMing the box — i.e., it becomes a deterministic red until the memory use is fixed. So this needs a real fix on the rivet side, not just more RAM.

Requested

1. Identify the offending test in rivet-core — something is allocating ~100 GB. Likely candidates: a property/fuzz-style test with an unbounded input, a test that materializes a huge graph/corpus, or a proptest/quickcheck case with a large generator.
2. Bound it — cap allocation, shrink the input, or mark it #[ignore]/move it behind a feature so it doesn't run in the default test suite that Miri/mutants-core invoke.
3. Cap test concurrency as a stopgap — --test-threads=N / nextest test-threads so N parallel copies of a memory-heavy test can't sum to 100 G.

Acceptance

• No rivet_core process exceeds the lean-mem MemoryMax (~48 G); zero kernel OOM kills attributable to rivet.

cc #509 (CI reliability theme). Related infra: a 14-day audit also flagged lean-mem as the contended pool (see rivet#523).

---

Filed from a host performance investigation (OOM logs + cgroup config). Suggested labels: ci, bug.

[avrabe]

Triage note (release-driver investigation): static reconnaissance of rivet-core/src/** #[cfg(test)] did not find a smoking-gun unbounded proptest collection/recursive strategy. No prop::collection::vec(_, 0..HUGE), no prop_recursive with large depth, no string_regex with huge repetition. So the ~100 GB anon-rss is more likely an algorithmic blow-up (super-linear allocation on a moderately-sized generated/constructed input) than a giant generator.

Leading candidate for a single-test spike: export.rs::render_section_graph_over_budget_skips_layout (rivet-core/src/export.rs:3107) builds a 250-node LinkGraph and calls render_section_graph, which drives etch's graph layout — documented in-test as super-linear. 250 nodes through an O(n^2)+ layout, summed across nextest in-process threads, is a plausible path to tens of GB. Unproven — needs heaptrack/massif to confirm; I deliberately did not reproduce a 100 GB allocation locally.

Two safe, culprit-agnostic stopgaps (your ask #3) while the offender is profiled:

• Cap rivet-core test concurrency in the RAM-heavy jobs (--test-threads=N / nextest test-threads) so parallel copies cannot sum to 100 GB.
• Once the infra MemoryMax~48 G cgroup cap lands, the OOM becomes a deterministic failure that names the exact test — that is the cheapest way to identify the true offender. Recommend landing the cap first, then bounding the named test.

[avrabe]

Re-diagnosis: this is almost certainly a cargo-mutants runaway mutant, not a bindable test

I went looking for the "single test allocating ~100 GB" and found no such test exists in the unmutated source:

• Zero proptest! {} blocks in rivet-core (grep -rn "proptest! {" rivet-core/src → 0). The pathological-generated-input path does not exist here.
• No large allocation in any test: no loop bound ≥1000, no vec![_; N]/.repeat(N) with large N, no big literals in #[cfg(test)].

So the ~100 GB cannot come from the real test code. It comes from the harness, and the evidence points at mutants-core:

• The OOM process is rivet_core-<hash> — the native compiled test binary. That is what cargo-mutants runs per mutant. (Miri would show as a miri interpreter process, which is why "native, not Miri" in the original report is the right call — it points at mutants, not Miri.)
• mutants-core runs 16 shards on the lean-mem pool, nightly, cargo mutants --timeout 30 --jobs 2 -- --lib (ci.yml:590). The OOM UIDs 1102–1104 = runners 2/3/4 = lean-mem; the OOM dates cluster on nightly-run days.
• Mechanism: a mutation that breaks a loop bound, a Vec::with_capacity(n) arg, a .take(n), or a recursion base-case turns a normally-terminating test into an unbounded allocator. Allocating ~100 GB takes only a few seconds — faster than the 30 s --timeout — so the timeout never fires; the kernel OOM-kills first. This is exactly why nothing in the static source looks wrong: the bug is injected by the mutant.

Implication for the fix

"Find and bound the offending test" (the original ask #1) is chasing a test that does not exist unmutated. The correct fixes are resource caps on the mutant runner, not a code change:

1. Primary (infra, already planned): the MemoryMax~48 G cgroup cap. This is exactly right — it bounds each runner slot so a runaway mutant aborts cleanly instead of OOMing the host. Land it.
2. Repo-side defense-in-depth (optional, works now): in the mutants-core step, cap per-process address space before cargo mutants (Linux enforces RLIMIT_AS, e.g. ulimit -v 50331648 ≈ 48 G) and/or drop --jobs 2 → --jobs 1 so co-located shards cannot sum as high. The job is continue-on-error + || true, so a clipped legit build cannot fail the gate.
3. Identify the actual mutant (nice-to-have): once the cgroup cap lands, the runaway mutant fails fast and cargo-mutants logs which mutation produced the unviable test — that names the function whose mutation is dangerous, with zero guesswork.

Net: relabel from "bound a rivet-core test" to "cap the mutant runner (cgroup) + identify the runaway mutation". No unmutated-source bug to fix.

(Investigated by the release-driver. I could not safely reproduce a 100 GB allocation locally — macOS does not enforce ulimit -v/-d, so capping is impossible on Darwin; this needs the Linux cgroup cap to reproduce-and-name deterministically.)

[avrabe]

Triage bot (scheduled run, 2026-06-26):

Opened draft #599 with the optional repo-side defense-in-depth from your last comment — ulimit -v 50331648 (~48 G RLIMIT_AS) before cargo mutants in the mutants-core shard. Left --jobs 2 and the existing tuning rationale alone. Single-line addition wrapped in a multi-line run:.

Kept it as draft rather than ready-for-review because:

1. Your last comment explicitly marks the repo-side change as optional and points at the infra MemoryMax cgroup cap as the real fix — landing this before/after that should be coordinated, not raced.
2. The acceptance criterion ("zero kernel OOM kills attributable to rivet") is verifiable only by the nightly mutants-core fan-out post-merge, not inside a PR.
3. This run's mandatory pre-PR step (re-read https://pulseengine.eu/blog/ for current process guidance) failed — the host returned HTTP 503 + expired TLS cert. I did not bypass verification. Flagging in case the blog outage is news.

Flip the PR to ready, or comment "do --jobs 1 instead" / "wait for infra" / "close" and I'll act on it next run.

---

Generated by Claude Code