Skip to content

redo: batch encoded rows before spilling to local spool #6036

Description

@asddongmen

Bug report

What did you do?

Run the new-architecture TiCDC from PR #5956 with eventual redo enabled, a MySQL sink, four continuously written tables, 16 redo encoding workers, 8 flush workers, and a 10 GiB redo spool.

What did you expect to see?

Redo's local spool should absorb transient downstream variation while feeding the redo file workers efficiently. Durable redo checkpoint callbacks should stay close to the dispatcher resolved-ts, so the normal sink can make continuous progress.

What did you see instead?

The normal sink periodically stops for minutes and then emits a burst. During a stall:

  • redo input rows continue increasing;
  • the persisted redo checkpoint remains pinned, which gates the normal sink;
  • the redo spool grows to about 1.2 GiB / 18 segments;
  • the goroutine profile shows the single spool reader doing one pread per encoded row;
  • one S3 file worker is active while the other seven are idle;
  • the process spends substantial CPU in GC and retains millions of queue objects/callbacks.

The redo writer currently calls Spool.TryEnqueue once per encoded row and creates one redoSpoolEntry per row, even though the shared spool and its reader support multiple messages per entry. Under sustained write load this causes millions of small serialized entries, local reads, channel nodes, and delayed post-flush callbacks.

Proposed fix

Drain encoded rows into bounded batches before enqueueing them to the redo spool. Preserve per-row post-enqueue and post-flush callbacks, and release a spool entry only after every row in that batch has been durably flushed. Keep count and byte ceilings so batches remain bounded and allow one oversized row for liveness.

This issue is a follow-up to #6035 and the scan-window/EventStore investigation around PR #5956.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions