Skip to content

test(cli): cmd tests fail under -shuffle=on via shared flag state #2208

Closed

Description

@cristim

Summary

The cmd package tests pass in declaration order but fail under go test -shuffle=on -count=2 ./cmd/. Several flag-validation tests (TestValidateFlags*, TestFilterFlagValidation and others) fail with errors such as coverage-lookback-days must be >= 1. That points to package-level flag/config state left behind by one test and read by another.

Evidence

  • go test -race -count=1 ./cmd/ (unshuffled) passes on origin/main (817ab4b).
  • go test -shuffle=on -count=2 ./cmd/ fails the flag-validation tests. The same shuffle seed fails on a clean origin/main copy, so the failure is pre-existing and not caused by any open PR.
  • Found during the independent review of test(cli): make engine-version error-handling tests hermetic #2207 (hermetic engine-version tests). None of the failing tests touch the engine-version code.

Expected behaviour

Every test sets up the flag/config state it depends on (or restores it via t.Cleanup), so the package passes in any order.

Suggested investigation

  • Re-run with -shuffle=on -count=2 -v and record the printed seed. Then bisect which earlier test leaves the global config (e.g. the coverage lookback default) at zero.
  • Fix the leaking test, or make the validation tests build their config from a fresh default instead of reading package globals.

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. cristim commented on Oct 16, 2026

    @cristim
    MemberAuthor

    Claiming: working on this [claude-cli-impl]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions