From 60a52c318cb8afc27392e9f9c5c039df20a8ecaa Mon Sep 17 00:00:00 2001 From: Valter Balegas Date: Mon, 13 Jul 2026 21:58:14 +0100 Subject: [PATCH] =?UTF-8?q?refactor(durable-streams-rust):=20checkpoint=20?= =?UTF-8?q?cleanup=20=E2=80=94=20syncfs=20default-on=20(Linux),=20drop=20d?= =?UTF-8?q?ead=20fsync=20fan-out,=20refresh=20WAL=5FTUNING?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - --wal-checkpoint-syncfs defaults ON for Linux (validated +51% at the cliff, neutral below; off = escape hatch). Non-Linux unchanged (no syncfs). - Remove FSYNC_FANOUT / --wal-fsync-parallel: parallel per-file checkpoint fsync regressed in every controlled test (f8 -11% NVMe, f16 -19% constrained) and the syncfs barrier supersedes it. Flag accepted as a warning no-op so old deploy scripts don't crash. Checkpoint step 2 is now two modes: per-lane syncfs (Linux default) or the serial per-file fallback. - e2e resets restore the platform default instead of hard 'false'. - WAL_TUNING.md: stream-lanes results (1M wall broken), cardinality-based lane-split guidance, updated flag set, fd-ceiling note (#4706). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk --- packages/durable-streams-rust/WAL_TUNING.md | 45 ++++++--- packages/durable-streams-rust/src/main.rs | 17 ++-- .../durable-streams-rust/src/wal/e2e_tests.rs | 4 +- .../durable-streams-rust/src/wal/shard.rs | 98 ++++++------------- 4 files changed, 69 insertions(+), 95 deletions(-) diff --git a/packages/durable-streams-rust/WAL_TUNING.md b/packages/durable-streams-rust/WAL_TUNING.md index f442ecff4e..61fea8b6fd 100644 --- a/packages/durable-streams-rust/WAL_TUNING.md +++ b/packages/durable-streams-rust/WAL_TUNING.md @@ -20,6 +20,12 @@ is flat from 10k → 100k streams (−5%) at ~26–32× the pre-fix baseline. | **all of the above stacked** (suite `wal-stacked-1m`) | **383k** | | reference: memory durability (no fsync anywhere) | 512k | +At extreme cardinality the stacked config on ONE data lane hits a second wall — +checkpoint writeback (~40× metadata amplification of small appends) saturates +the single data device (`syncfs` = 60–74 s at 1M streams): 244k @500k, 56–68k +@1M. `--stream-lanes 3` (suite `wal-streamlanes-1m`, 3 data lanes + 3 WAL +lanes) breaks it: **374k @100k, 285k @500k, 212k @1M** (`syncfs` 5.7–11 s). + The residual ~1.6× gap to memory mode is WAL machinery (staging + double-write), not fsync — future work: io_uring segment writer (`wal/segment.rs` seam), batched mark-written. @@ -46,28 +52,35 @@ barrier). Then: raw-block node pools the default emptyDir sits on the boot PD — this single mistake mismeasures (and misdeploys) WAL mode by 5–26×. -Example (6-device box): device 0 → data root, devices 1–5 → 5 WAL shards: +Split the box between data lanes and WAL lanes **by cardinality**: -``` ---data-dir /data/wal/0 --wal-shards 5 -``` +- ≤ ~100k streams: 1 data lane is enough — e.g. device 0 → data root, + devices 1–5 → 5 WAL shards: `--data-dir /data/wal/0 --wal-shards 5` +- ≥ ~500k streams: checkpoint writeback dominates; give data more lanes — e.g. + device 0 → data root (stream lane 0), devices 1–2 → stream lanes 1–2 + (mounted at `/streams/1`, `/2`), devices 3–5 → 3 WAL shards: + `--data-dir /data/wal/0 --wal-shards 3 --stream-lanes 3` + (`--stream-lanes` is a LAYOUT choice like the shard count: it must match the + on-disk layout across restarts.) ### 2. Server flags ``` ---wal-checkpoint-syncfs on # one syncfs barrier per checkpoint instead of - # O(N-touched) per-stream fdatasync (PR #4697) --wal-checkpoint-wal-bytes 1073741824 # checkpoint a shard when ITS retained WAL # exceeds 1 GiB (PR #4704) — checkpoint cost ≈ 0, # crash-replay bounded to ≤1 GiB/shard (<1 s NVMe) --wal-checkpoint-interval-ms 60000 # fallback timer so an idle shard still recycles --wal-shards # shards = fsync lanes; on a SINGLE shared # device keep 2–4 (more only fragments batches) +--stream-lanes # hash stream files across per-device dirs + # (PR #4705); layout choice, default 1 --worker-threads ``` -Leave `--wal-fsync-parallel` at its default (1): fanout parallelizes the -per-stream fsync loop that syncfs replaces, and measured as a regression. +The syncfs checkpoint barrier (PR #4697) is **default-on for Linux** — one +`syncfs` per stream lane instead of O(N-touched) per-stream `fdatasync`; +`--wal-checkpoint-syncfs off` is the escape hatch. `--wal-fsync-parallel` is +removed (regressed in every controlled test; accepted as a warning no-op). ### 3. CPU binding (+21–24%) @@ -93,13 +106,15 @@ no longer fsync-bound, it scales with cores again — don't starve it. ## Caveats / follow-ups -- Validated to 100k streams, 256 B payloads, single node. The stacked config - measured 383k @100k, 244k @500k, and **56k @1M** — a NEW, different wall - appears near 1M streams (candidates: one open fd per live stream vs the - container nofile ceiling, an ext4 directory with 1M files, stream-map/tails - working set, page-cache pressure from 1M dirty files). Profiling pass pending; - see `CARDINALITY_1M.md` for the older analysis. Below ~500k streams the - configuration above is cliff-free. +- The 1M-stream writeback wall is diagnosed and broken (`--stream-lanes`, PR + #4705: 68k → 212k @1M). The residual slope (374k @100k → 212k @1M on 3 data + lanes) is per-file writeback amplification against total data-lane + capacity — add data lanes, or see #4695 (log-structured store) for the + structural end-state. +- fd ceiling: the server holds one fd per live stream — 1,005,724 fds at 1M + streams = 96% of the default 1,048,576 limit. Not the throughput wall, but a + hard scale ceiling just above 1M: raise LimitNOFILE, or see #4706 (lazy fd + management). - `--wal-checkpoint-syncfs` and the size trigger are opt-in; flipping defaults (syncfs on for Linux) is a candidate after soak. - Larger retained WAL = longer replay: 1 GiB/shard ≈ sub-second on local NVMe, diff --git a/packages/durable-streams-rust/src/main.rs b/packages/durable-streams-rust/src/main.rs index d874aef155..b0c908ecac 100644 --- a/packages/durable-streams-rust/src/main.rs +++ b/packages/durable-streams-rust/src/main.rs @@ -278,17 +278,16 @@ fn main() { } } } - // Checkpoint fdatasync fan-out (H4). `1` = serial baseline. + // Removed knob: parallel per-file checkpoint fsync regressed in every + // controlled test; superseded by the (default-on) syncfs barrier. + // Accepted-and-ignored so old deploy scripts don't crash the server. "--wal-fsync-parallel" => { - let n: u64 = parse_val(args.next(), "--wal-fsync-parallel"); - if n == 0 { - eprintln!("--wal-fsync-parallel must be ≥ 1"); - std::process::exit(2); - } - wal::shard::set_fsync_fanout(n); + let _ = val(args.next(), "--wal-fsync-parallel"); + eprintln!("warning: --wal-fsync-parallel is removed (no-op); the syncfs checkpoint barrier supersedes it"); } - // Checkpoint durability via ONE syncfs() barrier instead of the - // O(N_touched) per-stream fdatasync loop (cardinality-cliff #1). Linux-only. + // Checkpoint durability via per-lane syncfs() barriers instead of the + // O(N_touched) per-stream fdatasync loop (cardinality-cliff #1). + // DEFAULT ON for Linux; `off` = escape hatch. No-op elsewhere. "--wal-checkpoint-syncfs" => { let v = val(args.next(), "--wal-checkpoint-syncfs"); match v.as_str() { diff --git a/packages/durable-streams-rust/src/wal/e2e_tests.rs b/packages/durable-streams-rust/src/wal/e2e_tests.rs index 80606ff1d6..511bfcf8ca 100644 --- a/packages/durable-streams-rust/src/wal/e2e_tests.rs +++ b/packages/durable-streams-rust/src/wal/e2e_tests.rs @@ -1009,7 +1009,7 @@ async fn e2e_stream_lanes_recover_acked_records() { ); } h2.crash(); - crate::wal::shard::set_checkpoint_syncfs(false); + crate::wal::shard::set_checkpoint_syncfs(cfg!(target_os = "linux")); // restore platform default crate::store::set_stream_lanes(1); let _ = std::fs::remove_dir_all(&dir); } @@ -1057,7 +1057,7 @@ async fn e2e_checkpoint_syncfs_recovers_acked_records() { "syncfs-checkpoint acked records recover byte-identical (durability-before-recycle held)" ); h2.crash(); - crate::wal::shard::set_checkpoint_syncfs(false); + crate::wal::shard::set_checkpoint_syncfs(cfg!(target_os = "linux")); // restore platform default let _ = std::fs::remove_dir_all(&dir); } diff --git a/packages/durable-streams-rust/src/wal/shard.rs b/packages/durable-streams-rust/src/wal/shard.rs index 909b1f03b8..ccb0576155 100644 --- a/packages/durable-streams-rust/src/wal/shard.rs +++ b/packages/durable-streams-rust/src/wal/shard.rs @@ -379,43 +379,29 @@ pub struct Shard { /// recyclable. const CHECKPOINT_FILE: &str = "checkpoint"; -/// Concurrency for the checkpoint's per-stream `fdatasync` phase (cardinality-cliff -/// H4). At high stream cardinality that phase dominates the checkpoint (~99% of its -/// wall time), and a serial loop pays `latency × N_touched` while the storage -/// device's queue depth (NVMe: many in flight) sits idle. Fanning the syncs across -/// this many OS threads lets the device absorb them concurrently; all still -/// complete before `persist_durable_tails`/recycle, preserving the -/// durability-before-recycle ordering. `1` (the DEFAULT) = serial, i.e. a no-op -/// change unless `--wal-fsync-parallel N` opts in. -/// -/// Default is serial because a fixed high fan-out REGRESSES on a CPU-constrained -/// server or storage that does not do concurrent fsync: measured on a 2-vCPU Linux -/// container (Docker, virtiofs) fan-out=16 was −19% vs serial — the 16 sync threads -/// per shard stole CPU from the runtime, slowing the committer until checkpoints -/// fell behind. The win requires real NVMe (deep device queue) AND spare cores; -/// validate there before raising the default. -static FSYNC_FANOUT: AtomicU64 = AtomicU64::new(1); -pub fn set_fsync_fanout(n: u64) { - FSYNC_FANOUT.store(n.max(1), Ordering::Relaxed); -} -fn fsync_fanout() -> usize { - FSYNC_FANOUT.load(Ordering::Relaxed) as usize -} +// (Removed: FSYNC_FANOUT / --wal-fsync-parallel. Parallelizing the per-stream +// fdatasync loop regressed in every controlled test — f8 −11% on NVMe, f16 −19% +// on a CPU-constrained container — because it steals device budget/CPU from the +// commit path. The syncfs barrier (default on Linux, below) replaces the loop +// wholesale; the serial per-file loop remains only as the non-Linux fallback.) -/// Checkpoint durability strategy (cardinality-cliff #1). The default per-stream +/// Checkpoint durability strategy (cardinality-cliff #1). The per-stream /// `fdatasync` loop (above) issues ONE device durability barrier per *touched* /// stream — `O(N_touched)` syscalls, measured at ~1.4 s of `fsync` per shard at /// 200k streams. Those barriers share the device's fixed `fdatasync`/s budget with /// the commit path, so the checkpoint storm steals throughput from acks (the cliff) /// AND caps wal throughput regardless of shard count (a shared-device sweep showed -/// s1≈s24). When this is on, step 2 instead issues a SINGLE `syncfs()` on the data -/// filesystem — one barrier that flushes every touched stream file (and everything -/// else dirty on the fs) at once, collapsing the per-stream `O(N_touched)` cost to -/// `O(1)` syscalls. The durability-before-recycle ordering is unchanged: the -/// `syncfs` completes before `persist_durable_tails`/recycle. Linux-only (there is -/// no `syncfs` on macOS); on other targets it falls back to the per-stream loop. -/// Default off — opt in with `--wal-checkpoint-syncfs on`. -static CHECKPOINT_SYNCFS: AtomicBool = AtomicBool::new(false); +/// s1≈s24). When this is on, step 2 instead issues ONE `syncfs()` per stream lane — +/// a filesystem-wide barrier that flushes every touched stream file (and everything +/// else dirty on that fs) at once, collapsing the per-stream `O(N_touched)` cost to +/// `O(lanes)` syscalls. The durability-before-recycle ordering is unchanged: the +/// `syncfs` completes before `persist_durable_tails`/recycle. +/// +/// **Default ON for Linux** (validated: +51% at the 100k-stream cliff, at worst +/// neutral below it — wal-syncfs/splitlane campaigns 2026-07); `--wal-checkpoint- +/// syncfs off` is the escape hatch. Non-Linux has no `syncfs` and always uses the +/// per-stream loop regardless of this flag. +static CHECKPOINT_SYNCFS: AtomicBool = AtomicBool::new(cfg!(target_os = "linux")); pub fn set_checkpoint_syncfs(on: bool) { CHECKPOINT_SYNCFS.store(on, Ordering::Relaxed); } @@ -904,49 +890,23 @@ impl Shard { let n_touched = touched.len(); let t_capture = t_start.elapsed(); - // 2. fdatasync each touched per-stream file. Fan out across a bounded - // pool of OS threads (H4): the device's queue depth absorbs the - // syncs concurrently instead of paying latency × N_touched serially - // — the checkpoint's dominant cost at high cardinality. ALL syncs - // complete here, before persist_durable_tails/recycle below, so the - // durability-before-recycle ordering is unchanged. `fanout == 1` - // (or a single file) keeps the plain serial loop. - let fanout = fsync_fanout().min(n_touched.max(1)); + // 2. Make every touched per-stream file durable, strictly BEFORE + // persist_durable_tails/recycle (the durability-before-recycle + // ordering). Two modes: + // * syncfs (Linux default): one filesystem-wide barrier per stream + // lane — O(lanes) syscalls instead of O(N_touched) fdatasyncs + // (the cardinality cliff). Lanes are independent devices in the + // intended layout, so the barriers run in parallel. + // * per-file fdatasync loop: the non-Linux (or --wal-checkpoint- + // syncfs off) fallback. Serial on purpose: parallel fan-out + // regressed in every controlled test (it steals device budget + // from the commit path). if checkpoint_syncfs() && cfg!(target_os = "linux") && n_touched > 0 { - // #1: ONE filesystem-wide barrier for ALL touched files, replacing - // the O(N_touched) per-stream fdatasync loop. `syncfs` on any fd of - // the data fs (we pass the first touched stream file) flushes every - // dirty page on it — all touched files become durable at once. Still - // strictly before persist_durable_tails/recycle, so the - // durability-before-recycle ordering is unchanged. - // One syncfs per stream lane (touched files may span lanes on - // multi-device layouts); falls back to touched[0]'s fs when no - // lane registry exists (shard-only unit tests). crate::store::syncfs_stream_lanes(&touched[0].2)?; - } else if fanout <= 1 { + } else { for (_, _, f) in &touched { crate::store::barrier_fsync(f)?; } - } else { - let next = AtomicU64::new(0); - let first_err: Mutex> = Mutex::new(None); - std::thread::scope(|scope| { - for _ in 0..fanout { - scope.spawn(|| loop { - let i = next.fetch_add(1, Ordering::Relaxed) as usize; - let Some((_, _, f)) = touched.get(i) else { break }; - if let Err(e) = crate::store::barrier_fsync(f) { - let mut slot = first_err.lock().unwrap(); - if slot.is_none() { - *slot = Some(e); - } - } - }); - } - }); - if let Some(e) = first_err.into_inner().unwrap() { - return Err(e); - } } let t_fsync = t_start.elapsed();