durable-streams-rust: memory mode uses ~5x more CPU than wal at identical low load (per-append debounced meta sidecar flush)

[balegas]

Symptom

In the 2026-07-06 benchmark round (ds-bench sustained suite: 10 append/s per stream, 256 B payloads, 90 s window, server c4d-standard-16-lssd pinned to 4 CPUs, build = main @ 640509c + #4690), --durability memory sustains the same throughput and similar latency as wal but at ~5x the CPU:

streams	wal cpu%	memory cpu%	wal p99 (ms)	memory p99 (ms)
10	0.4	0.8	0.588	0.483
100	2.2	11.7	0.98	1.091
150	3.2	16.6	1.011	1.374

Memory mode is supposed to be the cheaper path (no WAL, no fsync), so this inefficiency is pure overhead at low/medium per-stream rates.

Cause

packages/durable-streams-rust/src/handlers.rs (handle_append, the sidecar branch at the end):

• wal mode: the append only does st.meta_dirty.store(true) — the ~3 s WAL checkpoint writes the .meta sidecar in batch. This was the perf(server): remove the WAL coordination ceiling + 1M-stream cliff; memory mode 4× faster; crash-sim + 3 recovery fixes #4675 fix that removed the per-append sidecar flush (previously measured at ~40% of all server CPU under saturation, via the data-dir inode rwsem).
• memory mode: no checkpoint exists, so each append falls back to st.schedule_meta_flush() (src/store.rs, StreamState::schedule_meta_flush): a per-stream tokio::spawn + 100 ms sleep + spawn_blocking running write_meta_sync = JSON serialize + File::create(.meta.tmp) + rename (+ parent-dir inode contention).

Whenever the per-stream inter-append gap is >= the 100 ms debounce (i.e. any rate <= ~10 ops/s/stream — the common case at high stream cardinality), every stream pays a timer task plus a full sidecar rewrite every 100 ms. At 150 streams x 10/s that is ~1,500 sidecar writes/s plus timer churn — the measured 16.6% CPU.

Suggested fix

Give memory mode the same batched treatment wal mode got in #4675: replace the per-stream debounce timer with a single periodic sweeper (or a memory-mode dirty set) that flushes all meta_dirty streams in one pass every ~1-3 s.

• The contract already allows it: producer/access sidecar updates are documented as a non-durable, lagging flush; the lag bound just moves from 100 ms to the sweep cadence — exactly the trade wal mode made.
• Keep the flush-on-shutdown/final paths (write_meta_sync(st, true) call sites) unchanged.
• Expected outcome: memory-mode CPU at fixed load drops to <= wal's (it does strictly less work per append).

How to verify

• ds-bench sustained suite (wal vs memory rows) — cpu_mean in results/sustained/aggregate.csv; the table above is the baseline to beat.
• Locally: drive N streams at ~10 append/s each in --durability memory and watch process CPU; profile should show write_meta_sync + timer wakeups collapse after the fix.
• packages/durable-streams-rust/bench-latency/ (in-repo harness) confirms ack latency stays ~0.2-0.5 ms.

Context

• perf(server): remove the WAL coordination ceiling + 1M-stream cliff; memory mode 4× faster; crash-sim + 3 recovery fixes #4675 (write-path overhaul; moved wal-mode sidecar writes to the checkpoint)
• fix: complete DS_UNSAFE_FAST_FSYNC rename; drop concluded contention-investigation doc #4690 (follow-up rename + docs)
• Benchmark provenance: ds-bench sustained run 2026-07-06, durable-streams:dev image from commit 0d33d95.

🤖 Generated with Claude Code

https://claude.ai/code/session_0196ak7AuMa8we6DXbzv4zUE