perf(wal): storage WAL group commit——并发写共享一次 fsync

[NeverENG]

背景

standalone 写路径此前每条 WAL.Append 都 Write+Sync 一次，吞吐被单机 fsync 速率（本机 ~254 次/秒）死死封顶。

实测层次对比（256B value，本机）

路径	改前	改后
PUT c=50（WAL fsync 路径）	225 qps / P50 203ms	6087 qps / P50 8ms（27x）
PUT c=200	~225 qps	22231 qps / P50 8ms（~99x）
GET（纯内存，对照）	~16.8k qps	不受影响

改动

• WAL.Append 投递请求到 reqCh，唯一的 flushLoop 攒批：写完整批只 fsync 一次，再唤醒全部等待者。并发越高摊销越充分。
• flushLoop 成为文件唯一写者，顺带消除原先无锁并发 Write+Sync 的隐患。
• 持久化契约不变：Append 返回即代表记录已随批 fsync 落盘；记录格式、Replay、撕裂尾写处理均保持不变。
• 新增并发 Append 正确性测试（-race 通过）。

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance Improvements

  • Improved write performance by batching multiple WAL updates into grouped commits.
  • Maintained durable writes by synchronizing each batch before confirming completion.

• Reliability

  • Improved concurrent append handling to prevent lost or corrupted records.
  • Ensured pending updates are completed safely when the WAL is closed.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5bf945f6-0e46-4e2c-84c6-c79d53399dbe

📥 Commits

Reviewing files that changed from the base of the PR and between e584b47 and 62dccc5.

📒 Files selected for processing (2)

• storage/wal.go
• storage/wal_test.go

---

📝 Walkthrough

Walkthrough

The WAL now queues concurrent appends for background group commits, syncing each batch once before completing requests. Shutdown drains pending requests before closing the file. A concurrent test validates replay of 500 appended records.

Changes

WAL group commit

Layer / File(s)	Summary
Batching lifecycle and append path storage/wal.go	Adds request and coordination state, starts the flush loop, batches record writes, performs one Sync per batch, reports batch results, and drains requests during Close.
Concurrent append validation storage/wal_test.go	Adds a 500-goroutine append test that verifies replayed records, values, and unique keys.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant WAL
  participant flushLoop
  participant File
  Client->>WAL: Append operation
  WAL->>flushLoop: enqueue walReq
  flushLoop->>File: writeRecord for batch
  flushLoop->>File: Sync once
  flushLoop-->>WAL: batch result
  WAL-->>Client: return result

Loading

Possibly related issues

• Issue 132 — Adds the queued group-commit concurrency fix described by the issue.
• Issue 118 — Relates to WAL batch writes and durability/error semantics.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/storage-wal-group-commit

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🐯 BanGD 数据库内核评审

整体风险：🟡 中

变更总结：将 WAL 写入从每 Append 一次 fsync 改为 group commit 模型：Append 投递 walReq 到 channel，唯一的 flushLoop goroutine 攒批后统一写入+fsync。这消除了原无锁并发 Write+Sync 的竞争隐患（已经是一个 bug），同时将 N 次 fsync 摊销为 1 次，吞吐量从 ~225 qps 提升到 22k+ qps。WAL 记录格式、Replay、撕裂尾写处理均未变化。

本评审不阻塞合入；架构级建议以 Issue 形式跟踪，普通问题在下方内联列出。

架构问题（共 2 项）

• [资源] walMaxBatch 上限可能导致死锁/饥饿 storage/wal.go → 🐯 [资源] walMaxBatch 上限可能导致死锁/饥饿 · wal.go #151
• [并发] commit 中首条失败后跳过后续写入但继续发信号可能掩盖部分成功 storage/wal.go → 🐯 [并发] commit 中首条失败后跳过后续写入但继续发信号可能掩盖部分成功 · wal.go #152

普通问题（共 1 项）

⚠️ [重要 · 资源泄漏] storage/wal_test.go:146 并发测试未在 t.Errorf 路径关闭 WAL

• TestWALConcurrentAppendGroupCommit 中，若 w.Append 返回错误，t.Errorf 会标记测试失败但不终止 goroutine，之后 wg.Wait() 等待全部完成，然后执行 replayAll(t, w) 和 w.Close()。但在 replayAll 或 Close 中可能因已存在错误而 panic。更关键的是：t.Errorf 不终止 goroutine，但 goroutine 已经捕获到 Append 错误，后续 replayAll 和 Close 仍会执行——若 Append 失败的原因是 WAL 关闭（非此测试场景），goroutine 会在 <-req.done 阻塞。当前测试不会触发此问题，但写法不健壮。
• 建议：将 t.Errorf 改为 t.Fatalf 以尽早终止，或在 defer 中确保 w.Close() 被调用。更干净的写法：使用 errgroup 管理并发 goroutine 的错误。

---

_{本次评审消耗 token：共 63611 tokens（输入 59263，输出 4348，缓存命中 0，缓存写入 0）｜维度 [concurrency, memory, storage, performance, resource]｜对抗式复核 3 票/条，过滤疑似误报 0 条}