[Epic] P1 — Durability Correctness & Crash Safety (v0.5)

[habibtalib]

Epic — v0.5 production-hardening, phase P1: Durability Correctness & Crash Safety

Goal: a crash at any point — mid-checkpoint, mid-flush, with a slow core — never loses an acknowledged write, and recovery replays exactly the committed suffix.

Critical (filed)

• Checkpoint truncates the WAL at the min LSN of only the responding cores — a slow core loses acknowledged writes #160 — checkpoint truncates WAL at the min LSN of only responding cores → slow core loses acknowledged writes

High

• O_DIRECT replay armed to corrupt — the WAL reader skips no inter-batch alignment padding; an O_DIRECT segment with >1 flush batch replays only the first batch. Masked only because bootstrap/wal_init.rs:25 hardcodes use_direct_io=false. nodedb-wal/src/reader.rs:83-101, writer.rs:388-429.

Medium

• DWB torn-write recovery over-advances the read offset (double-adds payload.len()) → discards records after a mid-segment recovery. nodedb-wal/src/reader.rs:118-129.
• Mid-file WAL corruption is indistinguishable from a torn tail → silently truncates the committed suffix. recovery.rs:63-88.
• ENOSPC on WAL append: retained buffer + already-advanced LSN; retry/torn-tail interaction untested. writer.rs:388-423.

Test coverage

• No nodedb-wal failpoints — add crash injection between checkpoint-marker and truncate, between DWB and WAL fsync, mid-truncate_segments.
• No multi-batch O_DIRECT replay test.

Note: verify #149 (document_strict data loss on restart) is not a shared root cause with the above during planning.

Tracked in .planning/v1-production-readiness/AUDIT.md.

---

Status

All items above are implemented and covered by tests; the full suite is green
(13,227 + 323 cluster). Held open until v0.5.0 ships.

The audit that closed these found further defects on the same goal — "a crash at
any point never loses an acknowledged write, and recovery replays exactly the
committed suffix" — which were fixed alongside them:

• LSN reuse on resume: a rolled-but-empty (or fully-torn) last segment restarted
  LSNs at 1, producing duplicates and mis-truncation. Reachable on a clean
  shutdown, no crash required.
• No cross-segment LSN continuity check: a missing middle segment replayed as a
  silent hole.
• sync() returned Ok on an empty buffer after a failed fsync, so a retry
  acknowledged writes the kernel had already dropped.
• Directory entries for new/unlinked segments were never guaranteed durable
  (first segment had no fsync_directory at all; two others discarded the result).
• Double-write buffer: detaching on error silently and permanently removed
  torn-write protection; recover_record returned the oldest matching slot, which
  could resurrect an unacknowledged payload after LSN reuse.
• Encrypted WALs were never decrypted on replay — every restart with encryption
  enabled lost the committed suffix.
• A malformed TransactionRedo record was warn-skipped, leaving a hole.
• GroupCommitter (dead code) could report durable: true for records never
  written; removed.
• A failed segment roll left the writer sealed, bricking the WAL until restart.
• O_DIRECT is now the production default, with a typed fail-stop when the
  filesystem cannot support it rather than a silent downgrade to buffered I/O.

Still outstanding for this goal: replay idempotency under partial application is
unproven across the eight engines — the design relies on per-engine LSN
watermarks, but there is no test that crashes mid-replay and replays again.