Bug report
With the new-architecture redo pipeline from #5956, a sustained write workload eventually changes from continuous sink progress into long zero-throughput windows followed by large bursts.
A heap profile from the affected capture retained about 1.78 GiB under pkg/sink/codec/common.NewMsg in dmlWriter.writeEncodedEventsToSpool. The redo spool currently derives its memory quota as 20% of its disk quota. With the default 10 GiB disk quota, this creates an independent 2 GiB in-memory hot set, even when the changefeed memory quota is only 1 GiB. CPU profiles then show heavy GC plus EventStore/Pebble contention.
Expected behavior
The redo spool should keep a bounded hot set in memory and spill the remaining encoded events to its local disk quota, so it does not dominate the capture heap or cause periodic sink stalls.
Proposed fix
Keep the existing 20% behavior for small spool quotas, but cap redo spool memory at 256 MiB. For the default 10 GiB disk quota this uses a 2.5% memory ratio. Add focused unit coverage for default, small, and invalid quota inputs.
Related
Bug report
With the new-architecture redo pipeline from #5956, a sustained write workload eventually changes from continuous sink progress into long zero-throughput windows followed by large bursts.
A heap profile from the affected capture retained about 1.78 GiB under
pkg/sink/codec/common.NewMsgindmlWriter.writeEncodedEventsToSpool. The redo spool currently derives its memory quota as 20% of its disk quota. With the default 10 GiB disk quota, this creates an independent 2 GiB in-memory hot set, even when the changefeed memory quota is only 1 GiB. CPU profiles then show heavy GC plus EventStore/Pebble contention.Expected behavior
The redo spool should keep a bounded hot set in memory and spill the remaining encoded events to its local disk quota, so it does not dominate the capture heap or cause periodic sink stalls.
Proposed fix
Keep the existing 20% behavior for small spool quotas, but cap redo spool memory at 256 MiB. For the default 10 GiB disk quota this uses a 2.5% memory ratio. Add focused unit coverage for default, small, and invalid quota inputs.
Related