Bug report
What did you do?
Run the new-architecture TiCDC from PR #5956 with scan window and eventual redo enabled, a MySQL sink, sustained writes, 16 redo encoders, 8 redo flush workers, 64 MiB log files, and a 1-second flush interval.
What did you expect to see?
The file assembly goroutine should continue consuming encoded redo rows while the configured flush workers upload completed files concurrently. Durable callbacks must still be released in file creation order.
What did you see instead?
The normal sink periodically pauses and bursts while redo callbacks lag. A live goroutine dump shows the single redo file assembly goroutine alternating between building a file and waiting in flushAll. flushAll submits the current file and synchronously waits for every pending upload before it returns. During slow object-storage uploads, bgWriteLogs stops consuming input even though multiple upload workers are configured, causing backpressure through the redo sink and dispatcher.
Proposed fix
Queue files for upload without waiting in bgWriteLogs. Add one ordered callback releaser that waits for uploaded files in creation order, preserving checkpoint correctness while allowing file assembly and object-storage upload to overlap.
This issue is a follow-up to #6036 and the PR #5956 scan-window investigation.
Bug report
What did you do?
Run the new-architecture TiCDC from PR #5956 with scan window and eventual redo enabled, a MySQL sink, sustained writes, 16 redo encoders, 8 redo flush workers, 64 MiB log files, and a 1-second flush interval.
What did you expect to see?
The file assembly goroutine should continue consuming encoded redo rows while the configured flush workers upload completed files concurrently. Durable callbacks must still be released in file creation order.
What did you see instead?
The normal sink periodically pauses and bursts while redo callbacks lag. A live goroutine dump shows the single redo file assembly goroutine alternating between building a file and waiting in flushAll. flushAll submits the current file and synchronously waits for every pending upload before it returns. During slow object-storage uploads, bgWriteLogs stops consuming input even though multiple upload workers are configured, causing backpressure through the redo sink and dispatcher.
Proposed fix
Queue files for upload without waiting in bgWriteLogs. Add one ordered callback releaser that waits for uploaded files in creation order, preserving checkpoint correctness while allowing file assembly and object-storage upload to overlap.
This issue is a follow-up to #6036 and the PR #5956 scan-window investigation.