The WAL/data design holds one open fd per live stream for its lifetime (Shared.file, opened at create/recover). Measured on the wal-1m-diag run (2026-07-13): 1,005,724 fds at 1M streams — 96% of the default container hard limit (1,048,576). It was NOT the cause of the 1M throughput wall (that was checkpoint writeback on one device — fixed by --stream-lanes, #4705), but it is a hard scale ceiling just above 1M streams: the next stream open or socket accept hits EMFILE.
Options:
- Idle-close sweeper + on-demand reopen (
Shared.file: Option<Arc<File>>), bounded by an LRU-ish cap: hot streams keep fds, idle ones close. Touches the read path (resolve_range clones the Arc), append path (ap.file), and compaction swap — the Arc refcount model makes eviction safe (in-flight sendfile keeps the fd alive).
- Deployment mitigation meanwhile: raise LimitNOFILE on the node/containerd.
Related: #4695 (log-structured store) would remove per-stream files entirely; #4705 measured the residual per-file writeback slope (374k @100k → 212k @1m on 3 lanes).
The WAL/data design holds one open fd per live stream for its lifetime (
Shared.file, opened at create/recover). Measured on the wal-1m-diag run (2026-07-13): 1,005,724 fds at 1M streams — 96% of the default container hard limit (1,048,576). It was NOT the cause of the 1M throughput wall (that was checkpoint writeback on one device — fixed by--stream-lanes, #4705), but it is a hard scale ceiling just above 1M streams: the next stream open or socket accept hits EMFILE.Options:
Shared.file: Option<Arc<File>>), bounded by an LRU-ish cap: hot streams keep fds, idle ones close. Touches the read path (resolve_rangeclones the Arc), append path (ap.file), and compaction swap — the Arc refcount model makes eviction safe (in-flight sendfile keeps the fd alive).Related: #4695 (log-structured store) would remove per-stream files entirely; #4705 measured the residual per-file writeback slope (374k @100k → 212k @1m on 3 lanes).