Skip to content

test(scheduledauth): make rebuild-rate-limit tests deterministic via injectable clock - #1399

Merged
cristim merged 1 commit into
mainfrom
fix/authflake-injectable-clock
Jul 16, 2026
Merged

cristim merged 1 commit into
mainfrom
fix/authflake-injectable-clock

Conversation

@cristim

@cristim cristim commented Jul 16, 2026

Copy link
Copy Markdown
Member

Root cause

TestVerifyRetry_RebuildRateLimited and TestVerifyRetry_RotationRecoversAfterInterval asserted exact JWKS-fetch counts (hits==3 and hits==4). Those counts depend on goroutine scheduling inside go-oidc v3, not the validator itself.

go-oidc v3's keysFromRemote starts a goroutine that calls inflight.done() (closing the channel, signalling callers) before it re-acquires its mutex to write r.cachedKeys and clear r.inflight = nil. If the next Verify call arrives in that narrow window it joins the still-live (already-done) inflight without issuing an HTTP GET. This shifts all subsequent fetch ordinals and makes any hits == N assertion non-deterministic (typically 2 instead of 3, or 3 instead of 4), causing a rare but real CI failure.

The production clock injection (v.now func() time.Time, set to time.Now in the constructor and overridden in tests) introduced by PR #1386 was already correct. No changes to validator.go.

Fix

TestVerifyRetry_RebuildRateLimited: replace hits==3 with a v.verifier pointer comparison read under v.verMu.RLock():

  • After token 1: pointer must have changed (rebuild fired -- lastRebuild was zero so rate-limit permitted it).
  • After token 2: pointer must be identical to post-token-1 (rate limiter at frozen clock t0 suppressed the rebuild).
    This is 100% deterministic; it directly asserts what the rate-limiter controls.

TestVerifyRetry_RotationRecoversAfterInterval: replace the n>=4 fetch-count gate (which assumed exactly 3 prior HTTP GETs) with an atomic.Bool phase flag set between the two Validate calls. Every fetch during the second call now reliably receives keyB, regardless of whether go-oidc's own kid-miss refresh or the validator's rebuild retry fires first. The test asserts the behavioral outcome (second Validate returns nil) rather than the internal recovery path, which is the correct invariant.

DoS semantics unchanged

  • Bad-signature tokens still cause at most one verifier rebuild per minRebuildInterval.
  • After the interval elapses, rotation recovery is unblocked.
  • The injectable clock (v.now) still drives both checks deterministically in tests and in production uses time.Now.

Verification

go build ./...                                                              build=0
go vet ./...                                                                vet=0
go test ./internal/server/scheduledauth/ -race -count=20                   race20=0
go test ./internal/server/scheduledauth/ -run TestVerifyRetry -count=50    run50=0
gocyclo -over 10 internal/server/scheduledauth/validator.go                gocyclo=0 (empty output)
golangci-lint run --new-from-rev=origin/main ./...                         lintnew=0

…pointer assertions

TestVerifyRetry_RebuildRateLimited and TestVerifyRetry_RotationRecoversAfterInterval
intermittently failed in CI because they asserted exact JWKS-fetch counts (hits==3
and hits==4). Those counts depend on goroutine scheduling inside go-oidc v3's
keysFromRemote: it calls inflight.done() before re-acquiring its mutex to update
r.cachedKeys and clear r.inflight. A Verify call that arrives in that narrow window
joins the still-live (already-done) inflight without issuing an HTTP GET, shifting
all subsequent fetch ordinals and causing the exact-count assertions to flake.

Fix:
- TestVerifyRetry_RebuildRateLimited: replace hits==3 with a v.verifier pointer
  comparison under verMu. After the first bad token the pointer must change
  (rebuild fired); after the second bad token within minRebuildInterval the
  pointer must be identical (rate limiter suppressed the rebuild). This assertion
  is 100% deterministic regardless of how many HTTP GETs go-oidc actually issued.
- TestVerifyRetry_RotationRecoversAfterInterval: replace the n>=4 fetch-count
  gate (which assumed exactly 3 prior fetches) with an atomic.Bool phase flag
  set between the two Validate calls. Every fetch during the second call now
  reliably receives keyB, whether go-oidc's own kid-miss refresh or the
  validator's rebuild retry fires first. Assert the behavioral outcome (second
  Validate returns nil) rather than the internal recovery path.

The production clock injection (v.now func() time.Time) introduced by PR #1386
was already correct. No changes to validator.go.

Verified: go test -race -count=20 and -run TestVerifyRetry -count=50 both exit 0.
@cristim cristim added triaged Item has been triaged priority/p2 Backlog-worthy severity/low Minor harm type/bug Defect labels Jul 16, 2026
@cristim

cristim commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b9358fce-90dc-4451-a746-b3c5814f5db8

📥 Commits

Reviewing files that changed from the base of the PR and between 73166c2 and f0ee4d0.

📒 Files selected for processing (1)
  • internal/server/scheduledauth/validator_test.go
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/authflake-injectable-clock

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@cristim
cristim merged commit e8da1c4 into main Jul 16, 2026
19 checks passed
@cristim
cristim deleted the fix/authflake-injectable-clock branch July 16, 2026 20:31
@cristim

cristim commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

Merged to main (CLEAN + all CI green). Removes the intermittent TestVerifyRetry Unit Tests flake queue-wide: root cause was a go-oidc goroutine race in the test's fetch-count assertion (production rebuild-rate-limit clock was already injectable/correct). Replaced fetch-count asserts with a verifier-pointer comparison + atomic phase flag; verified -race -count=20 and -count=50.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority/p2 Backlog-worthy severity/low Minor harm triaged Item has been triaged type/bug Defect

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant