Bug report
What did you do?
Run the new-architecture TiCDC from PR #5956 with eventual redo enabled, a MySQL sink, four continuously written tables, 16 redo encoding workers, 8 flush workers, and a 10 GiB redo spool.
What did you expect to see?
Redo's local spool should absorb transient downstream variation while feeding the redo file workers efficiently. Durable redo checkpoint callbacks should stay close to the dispatcher resolved-ts, so the normal sink can make continuous progress.
What did you see instead?
The normal sink periodically stops for minutes and then emits a burst. During a stall:
- redo input rows continue increasing;
- the persisted redo checkpoint remains pinned, which gates the normal sink;
- the redo spool grows to about 1.2 GiB / 18 segments;
- the goroutine profile shows the single spool reader doing one
pread per encoded row;
- one S3 file worker is active while the other seven are idle;
- the process spends substantial CPU in GC and retains millions of queue objects/callbacks.
The redo writer currently calls Spool.TryEnqueue once per encoded row and creates one redoSpoolEntry per row, even though the shared spool and its reader support multiple messages per entry. Under sustained write load this causes millions of small serialized entries, local reads, channel nodes, and delayed post-flush callbacks.
Proposed fix
Drain encoded rows into bounded batches before enqueueing them to the redo spool. Preserve per-row post-enqueue and post-flush callbacks, and release a spool entry only after every row in that batch has been durably flushed. Keep count and byte ceilings so batches remain bounded and allow one oversized row for liveness.
This issue is a follow-up to #6035 and the scan-window/EventStore investigation around PR #5956.
Bug report
What did you do?
Run the new-architecture TiCDC from PR #5956 with eventual redo enabled, a MySQL sink, four continuously written tables, 16 redo encoding workers, 8 flush workers, and a 10 GiB redo spool.
What did you expect to see?
Redo's local spool should absorb transient downstream variation while feeding the redo file workers efficiently. Durable redo checkpoint callbacks should stay close to the dispatcher resolved-ts, so the normal sink can make continuous progress.
What did you see instead?
The normal sink periodically stops for minutes and then emits a burst. During a stall:
preadper encoded row;The redo writer currently calls
Spool.TryEnqueueonce per encoded row and creates oneredoSpoolEntryper row, even though the shared spool and its reader support multiple messages per entry. Under sustained write load this causes millions of small serialized entries, local reads, channel nodes, and delayed post-flush callbacks.Proposed fix
Drain encoded rows into bounded batches before enqueueing them to the redo spool. Preserve per-row post-enqueue and post-flush callbacks, and release a spool entry only after every row in that batch has been durably flushed. Keep count and byte ceilings so batches remain bounded and allow one oversized row for liveness.
This issue is a follow-up to #6035 and the scan-window/EventStore investigation around PR #5956.