Skip to content

test(execution): make TestFanOut_ContextCancelled_BlockedSemaphore deterministic - #1424

Merged
cristim merged 1 commit into
mainfrom
test/fanout-cancel-determinism
Jul 17, 2026
Merged

cristim merged 1 commit into
mainfrom
test/fanout-cancel-determinism

Conversation

@cristim

@cristim cristim commented Jul 16, 2026

Copy link
Copy Markdown
Member

What

Makes TestFanOut_ContextCancelled_BlockedSemaphore (in internal/execution/) deterministic. The test intermittently reddened the Unit Tests CI job across the PR queue (observed 2/2 on #1277, a cancelled->canceled rename that does not touch fan-out logic), blocking unrelated PRs.

Flake vs real regression

Genuine flake in the test harness, not a production defect. Reproduced locally on origin/main at GOMAXPROCS=2 (~1 failing run per several thousand on an otherwise-idle machine; loaded Linux CI hits it far more often). The failure is always the same: fanout_test.go: An error is expected but got nil on the second item.

Root cause

After cancelling the context the test released the blocking goroutine immediately, freeing the single semaphore slot. That made both cases of the launch loop's select { case sem <- struct{}{}: case <-ctx.Done(): } ready at once, and Go's uniform-random pick would occasionally launch second (which returns a nil error) instead of taking the ctx.Done() branch that records ctx.Canceled. The production code in fanout.go is correct; only the test's synchronization was racy.

Fix

  • Add an optional, context-scoped synchronization hook to the loop's ctx.Done() branch. It is inert in production: a single ctx.Value lookup on the already-canceled path, and no production caller ever sets it.
  • The test now waits for the queued item to be recorded as canceled before releasing the blocker, so the freed slot can never race the loop into running second.
  • No time.Sleep is used for synchronization (primary sync is a channel). The 10s select failsafe is only a deadline guard: it is never reached on a healthy tree and fails fast with a clear message on a genuine regression.

Verification

  • go build ./... -> 0
  • go vet ./... -> 0
  • go test ./internal/execution/ -run TestFanOut -race -count=50 -> 0
  • go test ./internal/execution/ -count=1 -> 0
  • 18000 runs at GOMAXPROCS=2 (the config that reproduced the flake pre-fix) -> 0 failures
  • Mutation check: reverting production to the pre-fix unconditional sem <- struct{}{} acquire is caught in 10s with the failsafe's clear message (proving the test still guards the cancellation-while-blocked invariant, not trivially passing)
  • gocyclo -over 10 on touched production file -> clean
  • golangci-lint run ./... at pinned CI version v2.10.1 -> 0 issues

Note: labeled type/chore because this repo has no type/test label; a test-only determinism fix is non-user-visible maintenance.

…terministic

The test intermittently reddened the Unit Tests CI job (observed 2/2 on
PR #1277, a cancelled->canceled rename that does not touch fan-out logic).
Root cause is a select-both-ready race in the test harness, not a
production defect: after cancelling the context the test released the
blocking goroutine immediately, freeing the single semaphore slot. That
made both cases of the launch loop's `select { case sem <- {}: case
<-ctx.Done(): }` ready, and Go's uniform-random pick would occasionally
launch "second" (recording a nil error) instead of the ctx.Done branch,
failing require.Error on the second item.

Fix it deterministically by giving the loop's ctx.Done branch an optional,
context-scoped synchronization hook (inert in production: a single
ctx.Value lookup on the already-canceled path, no caller sets it). The
test now waits for the queued item to be recorded as canceled before
releasing the blocker, so the freed slot can never race the loop into
running "second". No time.Sleep is used for synchronization; the 10s
select failsafe is only a deadline guard that fails fast with a clear
message on a genuine regression (never reached on a healthy tree).

Verified: -race -count=50 green; 18000 runs at GOMAXPROCS=2 (the config
that reproduced the flake pre-fix) green; a mutation to the pre-fix
unconditional `sem <- {}` acquire is caught in 10s. golangci-lint v2.10.1
and gocyclo clean.
@cristim cristim added triaged Item has been triaged priority/p2 Backlog-worthy severity/low Minor harm type/chore Maintenance / non-user-visible labels Jul 16, 2026
@cristim

cristim commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@cristim
cristim merged commit 8843520 into main Jul 17, 2026
18 of 20 checks passed
@cristim
cristim deleted the test/fanout-cancel-determinism branch July 17, 2026 05:37
@cristim

cristim commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

Merged to main (all CI green after the tflint-503 outage cleared). makes TestFanOut_ContextCancelled_BlockedSemaphore deterministic (a genuine test-harness select-both-ready race; production fanout.go change is inert - a ctx-value test seam on the already-cancelled path); 18000-run + mutation verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority/p2 Backlog-worthy severity/low Minor harm triaged Item has been triaged type/chore Maintenance / non-user-visible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant