Skip to content

Fix race between throttler changelog reads and changelog table drop - #1771

Open
ajm188 wants to merge 1 commit into
github:masterfrom
ajm188:fix/ghc-drop-race-1622
Open

ajm188 wants to merge 1 commit into
github:masterfrom
ajm188:fix/ghc-drop-race-1622

Conversation

@ajm188

@ajm188 ajm188 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #1622

The bug

Throttler.collectReplicationLag and collectControlReplicasLag read the changelog table (_ghc) from background goroutines gated only on finishedMigrating, which is set very late — in Throttler.Teardown(), called from Migrator.teardown() after finalCleanup() has already dropped _ghc.

That lets a read which was already in flight when cleanup began (or, for reads issued against a read replica via collectReplicationLag, a read racing the replication of the DROP TABLE) fail with:

Error 1146 (42S02): Table '<db>._ghc' doesn't exist

The migration itself completes fine either way — finalCleanup retries the drop and moves on — but the spurious error is confusing and shows up in logs right at the end of an otherwise-successful run. See #1622 for a full log and a walkthrough of the race, including a HeartbeatLag spike (0.06s → 1.07s) at the exact moment of the error, consistent with the drop replicating to the read replica while a stale SELECT was in flight.

The fix

  • Both ticker loops (collectReplicationLag, collectControlReplicasLag) now also check CleanupImminentFlag (set at the very start of finalCleanup) before starting a new changelog-table read, so no new reads start once cleanup begins.
  • A sync.WaitGroup (pendingChangelogReads) tracks reads that are already in flight.
  • finalCleanup calls Throttler.WaitForPendingChangelogReads() right after setting CleanupImminentFlag, so any in-flight read finishes before the changelog table is dropped below it.

Testing

  • Added regression tests in throttler_test.go:
    • TestWaitForPendingChangelogReadsReturnsImmediatelyWhenIdle
    • TestWaitForPendingChangelogReadsBlocksUntilInFlightReadCompletes
    • TestCollectReplicationLagStopsWhenCleanupImminent

Fixes github#1622

Throttler.collectReplicationLag and collectControlReplicasLag read the
changelog table (_ghc) from background goroutines gated only on
finishedMigrating, which is set very late (in Throttler.Teardown, called
from Migrator.teardown() after finalCleanup() has already dropped _ghc).
This let a read that was already in flight when cleanup began (or, for
reads against a read replica, a read racing the replication of the DROP
TABLE) fail with:

  Error 1146 (42S02): Table '<db>._ghc' doesn't exist

The migration itself completes fine either way (the drop is retried in
finalCleanup), but the spurious error is confusing and noisy in logs.

Fix:
- Both ticker loops now also check CleanupImminentFlag (set at the very
  start of finalCleanup) before starting a new changelog-table read, so
  no new reads start once cleanup begins.
- A sync.WaitGroup (pendingChangelogReads) tracks reads that are already
  in flight.
- finalCleanup calls Throttler.WaitForPendingChangelogReads() right after
  setting CleanupImminentFlag, so any in-flight read finishes before the
  changelog table is dropped.

Added regression tests for the new WaitForPendingChangelogReads gate and
for collectReplicationLag stopping once CleanupImminentFlag is set.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0b506-4700-700a-b541-fc3a71112a36
Co-authored-by: Andrew Mason <andrewmason@honeycomb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migration fails with "ghc table does not exist"

2 participants