test(execution): make TestFanOut_ContextCancelled_BlockedSemaphore deterministic - #1424
Merged
Merged
Conversation
…terministic The test intermittently reddened the Unit Tests CI job (observed 2/2 on PR #1277, a cancelled->canceled rename that does not touch fan-out logic). Root cause is a select-both-ready race in the test harness, not a production defect: after cancelling the context the test released the blocking goroutine immediately, freeing the single semaphore slot. That made both cases of the launch loop's `select { case sem <- {}: case <-ctx.Done(): }` ready, and Go's uniform-random pick would occasionally launch "second" (recording a nil error) instead of the ctx.Done branch, failing require.Error on the second item. Fix it deterministically by giving the loop's ctx.Done branch an optional, context-scoped synchronization hook (inert in production: a single ctx.Value lookup on the already-canceled path, no caller sets it). The test now waits for the queued item to be recorded as canceled before releasing the blocker, so the freed slot can never race the loop into running "second". No time.Sleep is used for synchronization; the 10s select failsafe is only a deadline guard that fails fast with a clear message on a genuine regression (never reached on a healthy tree). Verified: -race -count=50 green; 18000 runs at GOMAXPROCS=2 (the config that reproduced the flake pre-fix) green; a mutation to the pre-fix unconditional `sem <- {}` acquire is caught in 10s. golangci-lint v2.10.1 and gocyclo clean.
Member
Author
|
@coderabbitai review |
Member
Author
|
Merged to main (all CI green after the tflint-503 outage cleared). makes TestFanOut_ContextCancelled_BlockedSemaphore deterministic (a genuine test-harness select-both-ready race; production fanout.go change is inert - a ctx-value test seam on the already-cancelled path); 18000-run + mutation verified. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes
TestFanOut_ContextCancelled_BlockedSemaphore(ininternal/execution/) deterministic. The test intermittently reddened the Unit Tests CI job across the PR queue (observed 2/2 on #1277, acancelled->canceledrename that does not touch fan-out logic), blocking unrelated PRs.Flake vs real regression
Genuine flake in the test harness, not a production defect. Reproduced locally on
origin/mainatGOMAXPROCS=2(~1 failing run per several thousand on an otherwise-idle machine; loaded Linux CI hits it far more often). The failure is always the same:fanout_test.go:An error is expected but got nilon theseconditem.Root cause
After cancelling the context the test released the blocking goroutine immediately, freeing the single semaphore slot. That made both cases of the launch loop's
select { case sem <- struct{}{}: case <-ctx.Done(): }ready at once, and Go's uniform-random pick would occasionally launchsecond(which returns anilerror) instead of taking thectx.Done()branch that recordsctx.Canceled. The production code infanout.gois correct; only the test's synchronization was racy.Fix
ctx.Done()branch. It is inert in production: a singlectx.Valuelookup on the already-canceled path, and no production caller ever sets it.second.time.Sleepis used for synchronization (primary sync is a channel). The10sselectfailsafe is only a deadline guard: it is never reached on a healthy tree and fails fast with a clear message on a genuine regression.Verification
go build ./...-> 0go vet ./...-> 0go test ./internal/execution/ -run TestFanOut -race -count=50-> 0go test ./internal/execution/ -count=1-> 0GOMAXPROCS=2(the config that reproduced the flake pre-fix) -> 0 failuressem <- struct{}{}acquire is caught in 10s with the failsafe's clear message (proving the test still guards the cancellation-while-blocked invariant, not trivially passing)gocyclo -over 10on touched production file -> cleangolangci-lint run ./...at pinned CI version v2.10.1 ->0 issuesNote: labeled
type/chorebecause this repo has notype/testlabel; a test-only determinism fix is non-user-visible maintenance.