Conversation
Fixes github#1622 Throttler.collectReplicationLag and collectControlReplicasLag read the changelog table (_ghc) from background goroutines gated only on finishedMigrating, which is set very late (in Throttler.Teardown, called from Migrator.teardown() after finalCleanup() has already dropped _ghc). This let a read that was already in flight when cleanup began (or, for reads against a read replica, a read racing the replication of the DROP TABLE) fail with: Error 1146 (42S02): Table '<db>._ghc' doesn't exist The migration itself completes fine either way (the drop is retried in finalCleanup), but the spurious error is confusing and noisy in logs. Fix: - Both ticker loops now also check CleanupImminentFlag (set at the very start of finalCleanup) before starting a new changelog-table read, so no new reads start once cleanup begins. - A sync.WaitGroup (pendingChangelogReads) tracks reads that are already in flight. - finalCleanup calls Throttler.WaitForPendingChangelogReads() right after setting CleanupImminentFlag, so any in-flight read finishes before the changelog table is dropped. Added regression tests for the new WaitForPendingChangelogReads gate and for collectReplicationLag stopping once CleanupImminentFlag is set. Amp-Thread-ID: https://ampcode.com/threads/T-01a0b506-4700-700a-b541-fc3a71112a36 Co-authored-by: Andrew Mason <andrewmason@honeycomb.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1622
The bug
Throttler.collectReplicationLagandcollectControlReplicasLagread the changelog table (_ghc) from background goroutines gated only onfinishedMigrating, which is set very late — inThrottler.Teardown(), called fromMigrator.teardown()afterfinalCleanup()has already dropped_ghc.That lets a read which was already in flight when cleanup began (or, for reads issued against a read replica via
collectReplicationLag, a read racing the replication of theDROP TABLE) fail with:The migration itself completes fine either way —
finalCleanupretries the drop and moves on — but the spurious error is confusing and shows up in logs right at the end of an otherwise-successful run. See #1622 for a full log and a walkthrough of the race, including aHeartbeatLagspike (0.06s → 1.07s) at the exact moment of the error, consistent with the drop replicating to the read replica while a staleSELECTwas in flight.The fix
collectReplicationLag,collectControlReplicasLag) now also checkCleanupImminentFlag(set at the very start offinalCleanup) before starting a new changelog-table read, so no new reads start once cleanup begins.sync.WaitGroup(pendingChangelogReads) tracks reads that are already in flight.finalCleanupcallsThrottler.WaitForPendingChangelogReads()right after settingCleanupImminentFlag, so any in-flight read finishes before the changelog table is dropped below it.Testing
throttler_test.go:TestWaitForPendingChangelogReadsReturnsImmediatelyWhenIdleTestWaitForPendingChangelogReadsBlocksUntilInFlightReadCompletesTestCollectReplicationLagStopsWhenCleanupImminent