Bug report
Under sustained new-architecture TiCDC traffic, EventStore write workers drain up to 128 queued eventWithCallback values per Pebble batch but do not pass the channel byte limit. Large region event groups therefore let each of the eight workers retain and copy hundreds of MiB concurrently.
In a Fix10 live heap profile, eventStore.writeEvents retained about 2.15 GiB total, including about 1.02 GiB in pebble.Batch.grow and 1.09 GiB in Pebble manual allocations. A simultaneous CPU profile spent about 32% in GC. This remained after the redo spool hot set was successfully reduced from about 1.78 GiB to about 307 MiB, and sink output still had consecutive zero-throughput windows.
Expected behavior
EventStore should bound normal write batches by both event count and raw bytes, release log-puller callbacks in smaller increments, and still make progress for a single oversized event.
Proposed fix
Pass a 16 MiB raw-byte limit to the existing UnlimitedChannel.GetMultipleNoGroup call used by EventStore writers. Keep the existing 128-event capacity. The channel intentionally accepts the event that crosses the byte threshold, so one oversized event cannot block the queue. With four DBs and two workers per DB, ordinary concurrently held raw input is bounded to roughly 128 MiB plus at most one event per worker.
Related
Bug report
Under sustained new-architecture TiCDC traffic, EventStore write workers drain up to 128 queued
eventWithCallbackvalues per Pebble batch but do not pass the channel byte limit. Large region event groups therefore let each of the eight workers retain and copy hundreds of MiB concurrently.In a Fix10 live heap profile,
eventStore.writeEventsretained about 2.15 GiB total, including about 1.02 GiB inpebble.Batch.growand 1.09 GiB in Pebble manual allocations. A simultaneous CPU profile spent about 32% in GC. This remained after the redo spool hot set was successfully reduced from about 1.78 GiB to about 307 MiB, and sink output still had consecutive zero-throughput windows.Expected behavior
EventStore should bound normal write batches by both event count and raw bytes, release log-puller callbacks in smaller increments, and still make progress for a single oversized event.
Proposed fix
Pass a 16 MiB raw-byte limit to the existing
UnlimitedChannel.GetMultipleNoGroupcall used by EventStore writers. Keep the existing 128-event capacity. The channel intentionally accepts the event that crosses the byte threshold, so one oversized event cannot block the queue. With four DBs and two workers per DB, ordinary concurrently held raw input is bounded to roughly 128 MiB plus at most one event per worker.Related