Skip to content

refactor(durable-streams-rust): checkpoint cleanup — syncfs default-on (Linux), drop dead fsync fan-out #4707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
balegas merged 1 commit into from
Jul 13, 2026
Merged

refactor(durable-streams-rust): checkpoint cleanup — syncfs default-on (Linux), drop dead fsync fan-out #4707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 30 additions & 15 deletions packages/durable-streams-rust/WAL_TUNING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ is flat from 10k → 100k streams (−5%) at ~26–32× the pre-fix baseline.
| **all of the above stacked** (suite `wal-stacked-1m`) | **383k** |
| reference: memory durability (no fsync anywhere) | 512k |

At extreme cardinality the stacked config on ONE data lane hits a second wall —
checkpoint writeback (~40× metadata amplification of small appends) saturates
the single data device (`syncfs` = 60–74 s at 1M streams): 244k @500k, 56–68k
@1M. `--stream-lanes 3` (suite `wal-streamlanes-1m`, 3 data lanes + 3 WAL
lanes) breaks it: **374k @100k, 285k @500k, 212k @1M** (`syncfs` 5.7–11 s).

The residual ~1.6× gap to memory mode is WAL machinery (staging + double-write),
not fsync — future work: io_uring segment writer (`wal/segment.rs` seam),
batched mark-written.
Expand All @@ -46,28 +52,35 @@ barrier). Then:
raw-block node pools the default emptyDir sits on the boot PD — this single
mistake mismeasures (and misdeploys) WAL mode by 5–26×.

Example (6-device box): device 0 → data root, devices 1–5 → 5 WAL shards:
Split the box between data lanes and WAL lanes **by cardinality**:

```
--data-dir /data/wal/0 --wal-shards 5
```
- ≤ ~100k streams: 1 data lane is enough — e.g. device 0 → data root,
devices 1–5 → 5 WAL shards: `--data-dir /data/wal/0 --wal-shards 5`
- ≥ ~500k streams: checkpoint writeback dominates; give data more lanes — e.g.
device 0 → data root (stream lane 0), devices 1–2 → stream lanes 1–2
(mounted at `<data-dir>/streams/1`, `/2`), devices 3–5 → 3 WAL shards:
`--data-dir /data/wal/0 --wal-shards 3 --stream-lanes 3`
(`--stream-lanes` is a LAYOUT choice like the shard count: it must match the
on-disk layout across restarts.)

### 2. Server flags

```
--wal-checkpoint-syncfs on # one syncfs barrier per checkpoint instead of
# O(N-touched) per-stream fdatasync (PR #4697)
--wal-checkpoint-wal-bytes 1073741824 # checkpoint a shard when ITS retained WAL
# exceeds 1 GiB (PR #4704) — checkpoint cost ≈ 0,
# crash-replay bounded to ≤1 GiB/shard (<1 s NVMe)
--wal-checkpoint-interval-ms 60000 # fallback timer so an idle shard still recycles
--wal-shards <number of WAL devices> # shards = fsync lanes; on a SINGLE shared
# device keep 2–4 (more only fragments batches)
--stream-lanes <number of data devices> # hash stream files across per-device dirs
# (PR #4705); layout choice, default 1
--worker-threads <vCPUs>
```

Leave `--wal-fsync-parallel` at its default (1): fanout parallelizes the
per-stream fsync loop that syncfs replaces, and measured as a regression.
The syncfs checkpoint barrier (PR #4697) is **default-on for Linux** — one
`syncfs` per stream lane instead of O(N-touched) per-stream `fdatasync`;
`--wal-checkpoint-syncfs off` is the escape hatch. `--wal-fsync-parallel` is
removed (regressed in every controlled test; accepted as a warning no-op).

### 3. CPU binding (+21–24%)

Expand All @@ -93,13 +106,15 @@ no longer fsync-bound, it scales with cores again — don't starve it.

## Caveats / follow-ups

- Validated to 100k streams, 256 B payloads, single node. The stacked config
measured 383k @100k, 244k @500k, and **56k @1M** — a NEW, different wall
appears near 1M streams (candidates: one open fd per live stream vs the
container nofile ceiling, an ext4 directory with 1M files, stream-map/tails
working set, page-cache pressure from 1M dirty files). Profiling pass pending;
see `CARDINALITY_1M.md` for the older analysis. Below ~500k streams the
configuration above is cliff-free.
- The 1M-stream writeback wall is diagnosed and broken (`--stream-lanes`, PR
#4705: 68k → 212k @1M). The residual slope (374k @100k → 212k @1M on 3 data
lanes) is per-file writeback amplification against total data-lane
capacity — add data lanes, or see #4695 (log-structured store) for the
structural end-state.
- fd ceiling: the server holds one fd per live stream — 1,005,724 fds at 1M
streams = 96% of the default 1,048,576 limit. Not the throughput wall, but a
hard scale ceiling just above 1M: raise LimitNOFILE, or see #4706 (lazy fd
management).
- `--wal-checkpoint-syncfs` and the size trigger are opt-in; flipping defaults
(syncfs on for Linux) is a candidate after soak.
- Larger retained WAL = longer replay: 1 GiB/shard ≈ sub-second on local NVMe,
Expand Down
17 changes: 8 additions & 9 deletions packages/durable-streams-rust/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,17 +278,16 @@ fn main() {
}
}
}
// Checkpoint fdatasync fan-out (H4). `1` = serial baseline.
// Removed knob: parallel per-file checkpoint fsync regressed in every
// controlled test; superseded by the (default-on) syncfs barrier.
// Accepted-and-ignored so old deploy scripts don't crash the server.
"--wal-fsync-parallel" => {
let n: u64 = parse_val(args.next(), "--wal-fsync-parallel");
if n == 0 {
eprintln!("--wal-fsync-parallel must be ≥ 1");
std::process::exit(2);
}
wal::shard::set_fsync_fanout(n);
let _ = val(args.next(), "--wal-fsync-parallel");
eprintln!("warning: --wal-fsync-parallel is removed (no-op); the syncfs checkpoint barrier supersedes it");
}
// Checkpoint durability via ONE syncfs() barrier instead of the
// O(N_touched) per-stream fdatasync loop (cardinality-cliff #1). Linux-only.
// Checkpoint durability via per-lane syncfs() barriers instead of the
// O(N_touched) per-stream fdatasync loop (cardinality-cliff #1).
// DEFAULT ON for Linux; `off` = escape hatch. No-op elsewhere.
"--wal-checkpoint-syncfs" => {
let v = val(args.next(), "--wal-checkpoint-syncfs");
match v.as_str() {
Expand Down
4 changes: 2 additions & 2 deletions packages/durable-streams-rust/src/wal/e2e_tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1009,7 +1009,7 @@ async fn e2e_stream_lanes_recover_acked_records() {
);
}
h2.crash();
crate::wal::shard::set_checkpoint_syncfs(false);
crate::wal::shard::set_checkpoint_syncfs(cfg!(target_os = "linux")); // restore platform default
crate::store::set_stream_lanes(1);
let _ = std::fs::remove_dir_all(&dir);
}
Expand Down Expand Up @@ -1057,7 +1057,7 @@ async fn e2e_checkpoint_syncfs_recovers_acked_records() {
"syncfs-checkpoint acked records recover byte-identical (durability-before-recycle held)"
);
h2.crash();
crate::wal::shard::set_checkpoint_syncfs(false);
crate::wal::shard::set_checkpoint_syncfs(cfg!(target_os = "linux")); // restore platform default
let _ = std::fs::remove_dir_all(&dir);
}

Expand Down
98 changes: 29 additions & 69 deletions packages/durable-streams-rust/src/wal/shard.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,43 +379,29 @@ pub struct Shard {
/// recyclable.
const CHECKPOINT_FILE: &str = "checkpoint";

/// Concurrency for the checkpoint's per-stream `fdatasync` phase (cardinality-cliff
/// H4). At high stream cardinality that phase dominates the checkpoint (~99% of its
/// wall time), and a serial loop pays `latency × N_touched` while the storage
/// device's queue depth (NVMe: many in flight) sits idle. Fanning the syncs across
/// this many OS threads lets the device absorb them concurrently; all still
/// complete before `persist_durable_tails`/recycle, preserving the
/// durability-before-recycle ordering. `1` (the DEFAULT) = serial, i.e. a no-op
/// change unless `--wal-fsync-parallel N` opts in.
///
/// Default is serial because a fixed high fan-out REGRESSES on a CPU-constrained
/// server or storage that does not do concurrent fsync: measured on a 2-vCPU Linux
/// container (Docker, virtiofs) fan-out=16 was −19% vs serial — the 16 sync threads
/// per shard stole CPU from the runtime, slowing the committer until checkpoints
/// fell behind. The win requires real NVMe (deep device queue) AND spare cores;
/// validate there before raising the default.
static FSYNC_FANOUT: AtomicU64 = AtomicU64::new(1);
pub fn set_fsync_fanout(n: u64) {
FSYNC_FANOUT.store(n.max(1), Ordering::Relaxed);
}
fn fsync_fanout() -> usize {
FSYNC_FANOUT.load(Ordering::Relaxed) as usize
}
// (Removed: FSYNC_FANOUT / --wal-fsync-parallel. Parallelizing the per-stream
// fdatasync loop regressed in every controlled test — f8 −11% on NVMe, f16 −19%
// on a CPU-constrained container — because it steals device budget/CPU from the
// commit path. The syncfs barrier (default on Linux, below) replaces the loop
// wholesale; the serial per-file loop remains only as the non-Linux fallback.)

/// Checkpoint durability strategy (cardinality-cliff #1). The default per-stream
/// Checkpoint durability strategy (cardinality-cliff #1). The per-stream
/// `fdatasync` loop (above) issues ONE device durability barrier per *touched*
/// stream — `O(N_touched)` syscalls, measured at ~1.4 s of `fsync` per shard at
/// 200k streams. Those barriers share the device's fixed `fdatasync`/s budget with
/// the commit path, so the checkpoint storm steals throughput from acks (the cliff)
/// AND caps wal throughput regardless of shard count (a shared-device sweep showed
/// s1≈s24). When this is on, step 2 instead issues a SINGLE `syncfs()` on the data
/// filesystem — one barrier that flushes every touched stream file (and everything
/// else dirty on the fs) at once, collapsing the per-stream `O(N_touched)` cost to
/// `O(1)` syscalls. The durability-before-recycle ordering is unchanged: the
/// `syncfs` completes before `persist_durable_tails`/recycle. Linux-only (there is
/// no `syncfs` on macOS); on other targets it falls back to the per-stream loop.
/// Default off — opt in with `--wal-checkpoint-syncfs on`.
static CHECKPOINT_SYNCFS: AtomicBool = AtomicBool::new(false);
/// s1≈s24). When this is on, step 2 instead issues ONE `syncfs()` per stream lane —
/// a filesystem-wide barrier that flushes every touched stream file (and everything
/// else dirty on that fs) at once, collapsing the per-stream `O(N_touched)` cost to
/// `O(lanes)` syscalls. The durability-before-recycle ordering is unchanged: the
/// `syncfs` completes before `persist_durable_tails`/recycle.
///
/// **Default ON for Linux** (validated: +51% at the 100k-stream cliff, at worst
/// neutral below it — wal-syncfs/splitlane campaigns 2026-07); `--wal-checkpoint-
/// syncfs off` is the escape hatch. Non-Linux has no `syncfs` and always uses the
/// per-stream loop regardless of this flag.
static CHECKPOINT_SYNCFS: AtomicBool = AtomicBool::new(cfg!(target_os = "linux"));
pub fn set_checkpoint_syncfs(on: bool) {
CHECKPOINT_SYNCFS.store(on, Ordering::Relaxed);
}
Expand Down Expand Up @@ -904,49 +890,23 @@ impl Shard {
let n_touched = touched.len();
let t_capture = t_start.elapsed();

// 2. fdatasync each touched per-stream file. Fan out across a bounded
// pool of OS threads (H4): the device's queue depth absorbs the
// syncs concurrently instead of paying latency × N_touched serially
// — the checkpoint's dominant cost at high cardinality. ALL syncs
// complete here, before persist_durable_tails/recycle below, so the
// durability-before-recycle ordering is unchanged. `fanout == 1`
// (or a single file) keeps the plain serial loop.
let fanout = fsync_fanout().min(n_touched.max(1));
// 2. Make every touched per-stream file durable, strictly BEFORE
// persist_durable_tails/recycle (the durability-before-recycle
// ordering). Two modes:
// * syncfs (Linux default): one filesystem-wide barrier per stream
// lane — O(lanes) syscalls instead of O(N_touched) fdatasyncs
// (the cardinality cliff). Lanes are independent devices in the
// intended layout, so the barriers run in parallel.
// * per-file fdatasync loop: the non-Linux (or --wal-checkpoint-
// syncfs off) fallback. Serial on purpose: parallel fan-out
// regressed in every controlled test (it steals device budget
// from the commit path).
if checkpoint_syncfs() && cfg!(target_os = "linux") && n_touched > 0 {
// #1: ONE filesystem-wide barrier for ALL touched files, replacing
// the O(N_touched) per-stream fdatasync loop. `syncfs` on any fd of
// the data fs (we pass the first touched stream file) flushes every
// dirty page on it — all touched files become durable at once. Still
// strictly before persist_durable_tails/recycle, so the
// durability-before-recycle ordering is unchanged.
// One syncfs per stream lane (touched files may span lanes on
// multi-device layouts); falls back to touched[0]'s fs when no
// lane registry exists (shard-only unit tests).
crate::store::syncfs_stream_lanes(&touched[0].2)?;
} else if fanout <= 1 {
} else {
for (_, _, f) in &touched {
crate::store::barrier_fsync(f)?;
}
} else {
let next = AtomicU64::new(0);
let first_err: Mutex<Option<io::Error>> = Mutex::new(None);
std::thread::scope(|scope| {
for _ in 0..fanout {
scope.spawn(|| loop {
let i = next.fetch_add(1, Ordering::Relaxed) as usize;
let Some((_, _, f)) = touched.get(i) else { break };
if let Err(e) = crate::store::barrier_fsync(f) {
let mut slot = first_err.lock().unwrap();
if slot.is_none() {
*slot = Some(e);
}
}
});
}
});
if let Some(e) = first_err.into_inner().unwrap() {
return Err(e);
}
}
let t_fsync = t_start.elapsed();

Expand Down
Loading