From d37c940f917fb3033e98071f160a2c1745100f66 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:19:55 +0100 Subject: [PATCH 1/7] chore: capture the race lane's package timeout at the edge of the default The macOS race step ran internal/surface/cli into go test's implicit 10-minute per-package timeout and ejected PR #717 from the merge queue with no test hung; the job ceilings around the step are as tight. Refs: iss-2609260319483365 Assisted-by: Claude:claude-opus-5-5 --- ...-runs-internal-surface-cli-at-the-edge-of-go.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 .abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md diff --git a/.abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md b/.abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md new file mode 100644 index 000000000..ba84c8f5c --- /dev/null +++ b/.abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md @@ -0,0 +1,14 @@ +--- +schema_version: 1 +id: "iss-2609260319483365" +slug: "the-race-lane-runs-internal-surface-cli-at-the-edge-of-go" +severity: "minor" +category: "bug" +source: "agent-finding" +found_during: "autonomous run A resumed 2026-09-25" +origin: researcher-authored +production_mode: hand-written +found_at: ".github/workflows/ci.yml" +--- + +The race lane runs internal/surface/cli at the edge of go test's default 10-minute per-package timeout on the macOS runner, so a slow runner ejects an unrelated pull request from the merge queue. PR #717 was ejected by merge_group run 36212607621 (job check (macos-latest), step Test (race, internal)): panic: test timed out after 10m0s and FAIL github.com/intentdriven/abcd/internal/surface/cli 600.120s, with no test hung; the package's total wall time crossed the default. The same package under -race on macOS took 417.9s (run 36211489533), 472.0s (36211390371) and 567.9s (36209973364), and 324-355s on ubuntu. ci.yml's race step, release.yml's verify race step and the Makefile preflight all run go test -race ./internal/... with no -timeout, so the ceiling is the implicit default rather than a budget sized from measurement. The job ceilings around the step are as tight: the macOS check job took 19.0, 21.4, 25.9 and (failing) 28.1 minutes against timeout-minutes: 30, so lifting the package timeout alone moves the failure to the job cap, where it arrives with no goroutine dump; and release.yml's verify job, ubuntu only with an uncached toolchain, took 13.2 minutes of its 15 on 2026-09-24 (run 35963282477, cli race 302s), since when the ubuntu race step has grown from 9.8 to about 12 minutes, so the next release's verify job is at risk of the same cancellation. From f0994f70d7eb0aedec21d92f0b8d55a16a65a88c Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:22:49 +0100 Subject: [PATCH 2/7] fix(ci): give the race lane a declared -timeout that fits its jobs The race lane ran internal/surface/cli under go test's implicit 10m per-package timeout. On the macOS runner that package measured 418s, 472s and 568s passing, and merge_group run 36212607621 crossed 600s with no test hung, ejecting an unrelated pull request from the queue. The lane now runs `go test -race -timeout 20m ./internal/...` in all three places that run it: ci.yml's check job, release.yml's verify job (and the scaffold template that renders it byte-for-byte), and make preflight, so local and CI judge the lane against one budget. 20m is about twice the slowest passing run; a real hang still fails, with a goroutine dump, instead of being masked. Lifting the package timeout alone would move the failure to the job ceiling, which cancels without a dump. The macOS check job already took 19.0, 21.4 and 25.9 minutes of its 30 on passing runs, spending 8.3 minutes before the race step and 9.8 inside it before cli started. So check rises to 45 minutes. release.yml's verify job (ubuntu, uncached toolchain) took 13.2 of its 15 on the last release, with cli starting 4.8 minutes into the step, and the ubuntu race step has grown about two minutes since; verify rises to 35. Managed repos rendered from the bare profile keep 15 and the default timeout: their race leg is not this package. TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds the invariant: the three commands carry one explicit -timeout, and each job's timeout-minutes covers it plus its measured headroom. Considered and not taken: shortening the package. A local -race run of the package alone took 301s, 118s of it in one test, TestHistoryCaptureAcceptsWhatTheHooksAccept, whose 9 MiB fixture is the point of the test (it must exceed the old 8 MiB cap), so trimming it means an injectable cap in production code for a test's sake. That is a CI-cost question, and the direction for CI cost awaits a ruling under iss-2609012313465609; the declared budget is needed either way. zizmor 1.26.1 (the CI pin), --persona regular: no findings. Refs: iss-2609260319483365 Refs: iss-2609012313465609 Assisted-by: Claude:claude-opus-5-5 --- .github/workflows/ci.yml | 17 ++- .github/workflows/release.yml | 7 +- AGENTS.md | 3 +- Makefile | 2 +- .../scaffold/templates/release.yml.tmpl | 9 +- internal/core/lint/racelanebudget_test.go | 117 ++++++++++++++++++ 6 files changed, 148 insertions(+), 7 deletions(-) create mode 100644 internal/core/lint/racelanebudget_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2a959729a..64faa33b5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -223,7 +223,11 @@ jobs: check: needs: changes if: ${{ !cancelled() }} - timeout-minutes: 30 + # The race step's own -timeout must fire before this does, or a hang reads + # as a cancelled job with no goroutine dump: 20m for the slowest package, + # plus about 18m the macOS leg spends before that package starts (8m of + # steps ahead of the race step, 10m of packages ahead of it inside it). + timeout-minutes: 45 strategy: fail-fast: false matrix: @@ -283,9 +287,18 @@ jobs: # Only a source change can introduce a data race, and the plain lane above # has already run the same tests over the record. + # + # -timeout is a declared budget, not go test's 10m per-package default: + # internal/surface/cli under -race on the macOS runner measured 418-568s + # passing, and a slow runner crossed 600s with no test hung, ejecting an + # unrelated pull request from the merge queue (iss-2609260319483365). 20m + # is about twice the slowest passing run, so a real hang still fails with + # a goroutine dump. make preflight and release.yml's verify job carry the + # same flag, and the job's timeout-minutes above leaves room for it to + # fire; TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds all three. - name: Test (race, internal) if: needs.changes.outputs.inert != 'true' - run: go test -race ./internal/... + run: go test -race -timeout 20m ./internal/... # Drift gate for the .abcd/development design record. Blocking: a # blocker-severity finding exits non-zero and fails the job; warn-level diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3c884d213..cb5fd2373 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -82,7 +82,10 @@ jobs: # never the default-branch tip or the re-resolvable tag name — so the gate # exercises exactly the commit whose binaries will ship. verify: - timeout-minutes: 15 + # The race step's own -timeout (20m, sized in ci.yml) must fire before this + # does, or a hang reads as a cancelled job with no goroutine dump: 20m for + # the slowest package plus about 10m this job spends before it starts. + timeout-minutes: 35 runs-on: ubuntu-latest permissions: contents: read @@ -136,7 +139,7 @@ jobs: run: go test ./... - name: Test (race, internal) - run: go test -race ./internal/... + run: go test -race -timeout 20m ./internal/... - name: Record-lint (design-record drift gate) run: go run ./cmd/record-lint diff --git a/AGENTS.md b/AGENTS.md index 848a3d205..7f8cd73db 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -316,7 +316,8 @@ irreversible; guessing downward costs nothing.** `lint-decisions`, `record-lint`, `issue-drift`, `docs-lint`, `site-render`), both tagged eval lanes (`smoke`, `evals-cold-reading`), plus `go build ./...`, - `go vet ./...`, `go test ./...`, and `go test -race ./internal/...`. The load + `go vet ./...`, `go test ./...`, and + `go test -race -timeout 20m ./internal/...`. The load check runs first (`load-check`, a warning, never a failure) and is not a gate: it exits 0 whatever it finds. The eval lanes are named separately because their files carry a build tag, so diff --git a/Makefile b/Makefile index c98b13621..c45c3aa05 100644 --- a/Makefile +++ b/Makefile @@ -308,7 +308,7 @@ preflight: load-check lint-reviews lint-issues lint-decisions record-lint issue- go build ./... go vet ./... go test ./... - go test -race ./internal/... + go test -race -timeout 20m ./internal/... @scripts/preflight-receipt.sh mint "$(PREFLIGHT_BEGAN)" # The push receipt (iss-2608290810036869, iss-2608210738378295). The pre-push hook diff --git a/internal/core/launch/scaffold/templates/release.yml.tmpl b/internal/core/launch/scaffold/templates/release.yml.tmpl index f35e5670f..cebf573a2 100644 --- a/internal/core/launch/scaffold/templates/release.yml.tmpl +++ b/internal/core/launch/scaffold/templates/release.yml.tmpl @@ -111,7 +111,14 @@ jobs: <%- end %> <%- end %> verify: +<%- if .Abcd %> + # The race step's own -timeout (20m, sized in ci.yml) must fire before this + # does, or a hang reads as a cancelled job with no goroutine dump: 20m for + # the slowest package plus about 10m this job spends before it starts. + timeout-minutes: 35 +<%- else %> timeout-minutes: 15 +<%- end %> runs-on: ubuntu-latest permissions: contents: read @@ -193,7 +200,7 @@ jobs: <%- if .Abcd %> - name: Test (race, internal) - run: go test -race ./internal/... + run: go test -race -timeout 20m ./internal/... <%- else %> # A generic module may have no internal tree, so the race leg runs over the diff --git a/internal/core/lint/racelanebudget_test.go b/internal/core/lint/racelanebudget_test.go new file mode 100644 index 000000000..f3896b5e4 --- /dev/null +++ b/internal/core/lint/racelanebudget_test.go @@ -0,0 +1,117 @@ +package lint_test + +import ( + "path/filepath" + "regexp" + "strconv" + "strings" + "testing" + "time" +) + +// The race lane's time budget is declared, not inherited (iss-2609260319483365). +// +// `go test` kills a test binary that runs past -timeout, and with no flag the +// ceiling is ten minutes per package. internal/surface/cli under -race on the +// macOS runner measured 417.9s, 472.0s and 567.9s on three passing runs, and +// a merge-group run on a slow runner crossed 600s with no test hung and ejected +// an unrelated pull request from the queue. The lane's ceiling was the default +// rather than a budget anyone had sized, so it is written down here, in three +// places that must agree: the merge gate (ci.yml), the release gate +// (release.yml's verify job) and the local pre-push gate (make preflight). +// +// The enclosing job's timeout-minutes is the other half. A package timeout that +// reaches past the job's own ceiling never fires: the runner cancels the job +// first, and a cancellation carries no goroutine dump, so a real hang would read +// as a slow runner. Each job therefore holds the package timeout plus the time +// its slowest package waits before it starts: the steps ahead of the race step, +// and the packages `go test` runs ahead of it inside the step. +func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) { + root := filepath.Join("..", "..", "..") + + recipe, ok := makeRecipe(readRepoFile(t, root, "Makefile"), "preflight") + if !ok { + t.Fatal("Makefile declares no `preflight:` recipe") + } + local := raceTimeout(t, "Makefile preflight", recipe) + + for _, c := range []struct { + file, job string + // headroom is what the job spends outside the slowest package's own + // run, measured on its slowest leg and rounded up. + headroom time.Duration + }{ + // macOS leg of run 36212607621: 8.3 minutes before the race step began, + // then 9.8 minutes inside it before internal/surface/cli started. + {".github/workflows/ci.yml", "check", 20 * time.Minute}, + // The last release (run 35963282477, ubuntu, uncached toolchain): 2.8 + // minutes before the race step, 4.8 inside it before + // internal/surface/cli started, 0.5 after it; the ubuntu race step + // has grown by about two minutes since. + {".github/workflows/release.yml", "verify", 10 * time.Minute}, + } { + where := c.file + " job " + c.job + job, ok := workflowJobBlock(readRepoFile(t, root, c.file), c.job) + if !ok { + t.Errorf("%s: no such job; the parser or the workflow changed shape", where) + continue + } + step, ok := workflowStepBlock(job, "Test (race, internal)") + if !ok { + t.Errorf("%s: no `Test (race, internal)` step", where) + continue + } + pkg := raceTimeout(t, where, step) + if pkg != local { + t.Errorf("%s runs the race lane under -timeout %s but make preflight uses %s; "+ + "local and CI must judge the lane against one budget", where, pkg, local) + } + capMin := jobTimeoutMinutes(t, where, job) + if need := pkg + c.headroom; capMin < need { + t.Errorf("%s: timeout-minutes is %s, below the package timeout %s plus %s of headroom (%s); "+ + "the runner would cancel the job before go test could report a hang", + where, capMin, pkg, c.headroom, need) + } + } +} + +// raceTimeout returns the -timeout the one `go test -race` command in text +// carries, failing the test when there is none. +func raceTimeout(t *testing.T, where, text string) time.Duration { + t.Helper() + var cmds []string + for _, l := range strings.Split(text, "\n") { + l = strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(l), "run:")) + if strings.HasPrefix(l, "go test -race") { + cmds = append(cmds, l) + } + } + if len(cmds) != 1 { + t.Fatalf("%s: want one `go test -race` command, found %d", where, len(cmds)) + } + m := regexp.MustCompile(`\s-timeout[ =](\S+)`).FindStringSubmatch(cmds[0]) + if m == nil { + t.Fatalf("%s: `%s` sets no -timeout, so the lane inherits go test's 10m default "+ + "per package, which internal/surface/cli under -race has already crossed", where, cmds[0]) + } + d, err := time.ParseDuration(m[1]) + if err != nil { + t.Fatalf("%s: -timeout %q: %v", where, m[1], err) + } + return d +} + +// jobTimeoutMinutes reads the job-level `timeout-minutes:` (four-space indent, +// directly under the job key), failing the test when the job declares none. +func jobTimeoutMinutes(t *testing.T, where, job string) time.Duration { + t.Helper() + m := regexp.MustCompile(`(?m)^ timeout-minutes: (\d+)\s*$`).FindStringSubmatch(job) + if m == nil { + t.Fatalf("%s declares no job-level timeout-minutes", where) + } + n, err := strconv.Atoi(m[1]) + if err != nil { + t.Fatalf("%s: timeout-minutes %q: %v", where, m[1], err) + } + return time.Duration(n) * time.Minute +} From b10bfd1553a31b466fada2c3b5dc69ce8f301c0c Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:23:16 +0100 Subject: [PATCH 3/7] =?UTF-8?q?chore:=20resolve=20iss-2609260319483365=20?= =?UTF-8?q?=E2=80=94=20race=20lane=20runs=20under=20a=20declared=20timeout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves: iss-2609260319483365 Assisted-by: Claude:claude-opus-5-5 --- ...ce-lane-runs-internal-surface-cli-at-the-edge-of-go.md | 8 ++++++++ 1 file changed, 8 insertions(+) rename .abcd/work/issues/{open => resolved}/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md (65%) diff --git a/.abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md b/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md similarity index 65% rename from .abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md rename to .abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md index ba84c8f5c..1eb7264ef 100644 --- a/.abcd/work/issues/open/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md +++ b/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md @@ -9,6 +9,14 @@ found_during: "autonomous run A resumed 2026-09-25" origin: researcher-authored production_mode: hand-written found_at: ".github/workflows/ci.yml" +resolution: "The race lane runs under a declared go test -race -timeout 20m in ci.yml's check job, release.yml's verify job (and its scaffold template) and make preflight; the check job's ceiling rises from 30 to 45 minutes and verify's from 15 to 35 so the package timeout fires before the job is cancelled. TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds the three commands to one explicit timeout and each job's timeout-minutes to that timeout plus its measured headroom." +impact: internal +resolved_by: + commit: "f0994f70" --- The race lane runs internal/surface/cli at the edge of go test's default 10-minute per-package timeout on the macOS runner, so a slow runner ejects an unrelated pull request from the merge queue. PR #717 was ejected by merge_group run 36212607621 (job check (macos-latest), step Test (race, internal)): panic: test timed out after 10m0s and FAIL github.com/intentdriven/abcd/internal/surface/cli 600.120s, with no test hung; the package's total wall time crossed the default. The same package under -race on macOS took 417.9s (run 36211489533), 472.0s (36211390371) and 567.9s (36209973364), and 324-355s on ubuntu. ci.yml's race step, release.yml's verify race step and the Makefile preflight all run go test -race ./internal/... with no -timeout, so the ceiling is the implicit default rather than a budget sized from measurement. The job ceilings around the step are as tight: the macOS check job took 19.0, 21.4, 25.9 and (failing) 28.1 minutes against timeout-minutes: 30, so lifting the package timeout alone moves the failure to the job cap, where it arrives with no goroutine dump; and release.yml's verify job, ubuntu only with an uncached toolchain, took 13.2 minutes of its 15 on 2026-09-24 (run 35963282477, cli race 302s), since when the ubuntu race step has grown from 9.8 to about 12 minutes, so the next release's verify job is at risk of the same cancellation. + +## Grounds + +- pursued: the macOS race step stops failing when internal/surface/cli's total -race wall time crosses 600s, while a genuinely hung test still fails with a goroutine dump at 20m; shown wrong by a merge-queue ejection from a package-timeout panic under 20m on a passing package, or by a check or verify job cancelled at its ceiling before the race step's own timeout could fire From b162ebfc4328220492444e7094eb5ea5c9c53344 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:28:52 +0100 Subject: [PATCH 4/7] fix(ci): name the preflight recipe without the phrase the load-check guard reads The race step's new comment said "make preflight", which TestCIRunsTheSkipStepBeforeEachHarness and TestHarnessJobsRunOnHostedRunners read as ci.yml running the local gate (CI runs the gates directly). The comment names the Makefile's preflight recipe instead; no step changes. Refs: iss-2609260319483365 Assisted-by: Claude:claude-opus-5-5 --- .github/workflows/ci.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 64faa33b5..6c3186fec 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -293,9 +293,10 @@ jobs: # passing, and a slow runner crossed 600s with no test hung, ejecting an # unrelated pull request from the merge queue (iss-2609260319483365). 20m # is about twice the slowest passing run, so a real hang still fails with - # a goroutine dump. make preflight and release.yml's verify job carry the - # same flag, and the job's timeout-minutes above leaves room for it to - # fire; TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds all three. + # a goroutine dump. The Makefile's preflight recipe and release.yml's + # verify job carry the same flag, and the job's timeout-minutes above + # leaves room for it to fire; TestRaceLaneBudgetIsDeclaredAndFitsItsJob + # holds all three. - name: Test (race, internal) if: needs.changes.outputs.inert != 'true' run: go test -race -timeout 20m ./internal/... From 2f7b903b15a4e1d43e3cfa62413e911378dc352b Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:45:37 +0100 Subject: [PATCH 5/7] fix(ci): hold the check job to the merge queue's response cap f0994f70 raised ci.yml's check job from 30 to 45 minutes so the race step's -timeout 20m could fire before the job was cancelled. On the event the lane was ejected on, a merge-group run, that ceiling is unreachable: the main ruleset's merge_queue rule fails the group when a required check has not concluded within check_response_timeout_minutes, which is 30 (.abcd/work/rulesets/main-protection.json). The 45 bought nothing there and made the comment, the test and the record claim a budget that is not in force. The check job goes back to 30, equal to the queue's cap, as iss-2609020215079051 deliberately set it. The race step keeps -timeout 20m: it lifts go test's 10m per-package default, which is what ejected PR #717, so a slow package runs on to the job's ceiling instead of failing at ten minutes. On the macOS leg the slowest package starts about 18 minutes in, so a genuine hang there is cancelled at 30 without a goroutine dump; the comments say so. Raising the queue's cap is an admin act on the live ruleset, not taken here and left to the technical facilitator. release.yml's verify job is not a merge-queue job and keeps 35. TestRaceLaneBudgetIsDeclaredAndFitsItsJob now pins what is true: the three -timeout values are equal; ci.yml check's timeout-minutes is at most the mirror's merge_queue check_response_timeout_minutes; release verify's timeout-minutes covers the step timeout plus its measured 10m headroom. The check job's step-plus-20m assertion is dropped, which also retires the review nit on its "about 18m" against the test's 20m. Watched red: the queue-cap assertion against the unfixed ci.yml (45 > 30) in the worktree; on a scratch copy, verify lowered to 29, the mirror's cap lowered to 25, the merge_queue rule removed, and release's -timeout changed to 25m, each failing with its own message. Refs: iss-2609260319483365 Refs: iss-2609020215079051 Assisted-by: Claude:claude-opus-5-5 --- .github/workflows/ci.yml | 25 +++-- internal/core/lint/racelanebudget_test.go | 117 ++++++++++++++++------ 2 files changed, 105 insertions(+), 37 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c3186fec..0bd2a3d1c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -223,11 +223,18 @@ jobs: check: needs: changes if: ${{ !cancelled() }} - # The race step's own -timeout must fire before this does, or a hang reads - # as a cancelled job with no goroutine dump: 20m for the slowest package, - # plus about 18m the macOS leg spends before that package starts (8m of - # steps ahead of the race step, 10m of packages ahead of it inside it). - timeout-minutes: 45 + # Equal to the merge queue's check response timeout (30 minutes, recorded in + # .abcd/work/rulesets/main-protection.json), which is the outer ceiling on a + # merge-group run: the queue fails the group when this required check has + # not concluded by then, so a job ceiling above it is unreachable there. The + # race step's -timeout lifts go test's 10m per-package default, so a slow + # package runs on to this ceiling instead of failing at ten minutes; it does + # not fit inside the ceiling on the macOS leg, which spends about 18m before + # the slowest package starts, so a hang there is cancelled without a + # goroutine dump. Raising the queue's cap is an admin act on the live + # ruleset, left to the technical facilitator; this job may follow it then. + # TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds this at or below the cap. + timeout-minutes: 30 strategy: fail-fast: false matrix: @@ -293,9 +300,11 @@ jobs: # passing, and a slow runner crossed 600s with no test hung, ejecting an # unrelated pull request from the merge queue (iss-2609260319483365). 20m # is about twice the slowest passing run, so a real hang still fails with - # a goroutine dump. The Makefile's preflight recipe and release.yml's - # verify job carry the same flag, and the job's timeout-minutes above - # leaves room for it to fire; TestRaceLaneBudgetIsDeclaredAndFitsItsJob + # a goroutine dump where the enclosing ceiling leaves room for it: in + # release.yml's verify job it does, while this job's 30 minutes, held to + # the merge queue's response cap, are the binding ceiling here (see the + # job header). The Makefile's preflight recipe and release.yml's verify + # job carry the same flag; TestRaceLaneBudgetIsDeclaredAndFitsItsJob # holds all three. - name: Test (race, internal) if: needs.changes.outputs.inert != 'true' diff --git a/internal/core/lint/racelanebudget_test.go b/internal/core/lint/racelanebudget_test.go index f3896b5e4..b2adecda8 100644 --- a/internal/core/lint/racelanebudget_test.go +++ b/internal/core/lint/racelanebudget_test.go @@ -1,6 +1,7 @@ package lint_test import ( + "encoding/json" "path/filepath" "regexp" "strconv" @@ -20,12 +21,27 @@ import ( // places that must agree: the merge gate (ci.yml), the release gate // (release.yml's verify job) and the local pre-push gate (make preflight). // -// The enclosing job's timeout-minutes is the other half. A package timeout that -// reaches past the job's own ceiling never fires: the runner cancels the job -// first, and a cancellation carries no goroutine dump, so a real hang would read -// as a slow runner. Each job therefore holds the package timeout plus the time -// its slowest package waits before it starts: the steps ahead of the race step, -// and the packages `go test` runs ahead of it inside the step. +// The enclosing ceilings are the other half, and they differ by job. +// +// ci.yml's check job is a required context, and on a merge-group run the merge +// queue fails the group when a required check has not concluded within the +// ruleset's check_response_timeout_minutes. That cap is the outer ceiling on +// the event the lane was ejected on, and a job ceiling above it is unreachable +// there: the queue gives up first, with no job failure to read. So the check +// job's timeout-minutes may not exceed the cap the ruleset mirror records. The +// step's -timeout still earns its place under that cap: it lifts the ten-minute +// per-package default, so a slow package fails on the job's clock rather than +// at ten minutes. Raising the queue's cap is an admin act on the live ruleset, +// left to the technical facilitator; this test reads the mirror, so the day the +// cap is raised the check job may follow it. +// +// release.yml's verify job is not a merge-queue job, so its own timeout-minutes +// is its ceiling. A package timeout that reaches past it never fires: the +// runner cancels the job first, and a cancellation carries no goroutine dump, +// so a real hang would read as a slow runner. verify therefore holds the +// package timeout plus the time its slowest package waits before it starts: +// the steps ahead of the race step, and the packages `go test` runs ahead of +// it inside the step. func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) { root := filepath.Join("..", "..", "..") @@ -35,44 +51,87 @@ func TestRaceLaneBudgetIsDeclaredAndFitsItsJob(t *testing.T) { } local := raceTimeout(t, "Makefile preflight", recipe) - for _, c := range []struct { - file, job string - // headroom is what the job spends outside the slowest package's own - // run, measured on its slowest leg and rounded up. - headroom time.Duration - }{ - // macOS leg of run 36212607621: 8.3 minutes before the race step began, - // then 9.8 minutes inside it before internal/surface/cli started. - {".github/workflows/ci.yml", "check", 20 * time.Minute}, - // The last release (run 35963282477, ubuntu, uncached toolchain): 2.8 - // minutes before the race step, 4.8 inside it before - // internal/surface/cli started, 0.5 after it; the ubuntu race step - // has grown by about two minutes since. - {".github/workflows/release.yml", "verify", 10 * time.Minute}, - } { - where := c.file + " job " + c.job - job, ok := workflowJobBlock(readRepoFile(t, root, c.file), c.job) + // raceJob returns a workflow job's block and the -timeout its race step + // carries, holding that timeout to the local gate's. + raceJob := func(file, job string) (string, time.Duration, bool) { + where := file + " job " + job + block, ok := workflowJobBlock(readRepoFile(t, root, file), job) if !ok { t.Errorf("%s: no such job; the parser or the workflow changed shape", where) - continue + return "", 0, false } - step, ok := workflowStepBlock(job, "Test (race, internal)") + step, ok := workflowStepBlock(block, "Test (race, internal)") if !ok { t.Errorf("%s: no `Test (race, internal)` step", where) - continue + return "", 0, false } pkg := raceTimeout(t, where, step) if pkg != local { t.Errorf("%s runs the race lane under -timeout %s but make preflight uses %s; "+ "local and CI must judge the lane against one budget", where, pkg, local) } - capMin := jobTimeoutMinutes(t, where, job) - if need := pkg + c.headroom; capMin < need { + return block, pkg, true + } + + if check, _, ok := raceJob(".github/workflows/ci.yml", "check"); ok { + const where = ".github/workflows/ci.yml job check" + capMin := jobTimeoutMinutes(t, where, check) + queue := mergeQueueResponseTimeout(t, root) + if capMin > queue { + t.Errorf("%s: timeout-minutes is %s, above the merge queue's %s check response "+ + "timeout in %s; on a merge-group run the queue fails the group first, so the "+ + "job ceiling is unreachable there and any budget sized against it is false. "+ + "Raise the live ruleset's cap (an admin act) before the job's", + where, capMin, queue, rulesetMirror) + } + } + + // The last release (run 35963282477, ubuntu, uncached toolchain): 2.8 + // minutes before the race step, 4.8 inside it before internal/surface/cli + // started, 0.5 after it; the ubuntu race step has grown by about two + // minutes since. Rounded up to 10. + const verifyHeadroom = 10 * time.Minute + if verify, pkg, ok := raceJob(".github/workflows/release.yml", "verify"); ok { + const where = ".github/workflows/release.yml job verify" + capMin := jobTimeoutMinutes(t, where, verify) + if need := pkg + verifyHeadroom; capMin < need { t.Errorf("%s: timeout-minutes is %s, below the package timeout %s plus %s of headroom (%s); "+ "the runner would cancel the job before go test could report a hang", - where, capMin, pkg, c.headroom, need) + where, capMin, pkg, verifyHeadroom, need) + } + } +} + +// rulesetMirror is the tree's record of the live branch ruleset on main. +const rulesetMirror = ".abcd/work/rulesets/main-protection.json" + +// mergeQueueResponseTimeout reads the merge_queue rule's +// check_response_timeout_minutes from the ruleset mirror, failing the test +// when the mirror declares no merge queue or no positive cap. +func mergeQueueResponseTimeout(t *testing.T, root string) time.Duration { + t.Helper() + var ruleset struct { + Rules []struct { + Type string `json:"type"` + Parameters struct { + CheckResponseTimeoutMinutes int `json:"check_response_timeout_minutes"` + } `json:"parameters"` + } `json:"rules"` + } + if err := json.Unmarshal([]byte(readRepoFile(t, root, rulesetMirror)), &ruleset); err != nil { + t.Fatalf("decoding %s: %v", rulesetMirror, err) + } + for _, r := range ruleset.Rules { + if r.Type != "merge_queue" { + continue + } + if r.Parameters.CheckResponseTimeoutMinutes <= 0 { + t.Fatalf("%s: the merge_queue rule declares no positive check_response_timeout_minutes", rulesetMirror) } + return time.Duration(r.Parameters.CheckResponseTimeoutMinutes) * time.Minute } + t.Fatalf("%s declares no merge_queue rule; the check job's ceiling is sized against it", rulesetMirror) + return 0 } // raceTimeout returns the -timeout the one `go test -race` command in text From e58ac5fb43d759ebd353ffc5773c9e23109bebd8 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 04:46:40 +0100 Subject: [PATCH 6/7] chore: correct iss-2609260319483365's resolution to the queue's ceiling The resolution said the check job's ceiling rose to 45 minutes so the package timeout fires before the job is cancelled. 2f7b903b holds the check job at the merge queue's 30-minute response cap instead, so the text says what is in force: the step timeout lifts the 10m per-package default, the queue's cap is the outer ceiling on merge-group runs, raising it is an admin act left to the technical facilitator, and only release.yml's verify job leaves room for the package timeout to fire. The grounds no longer count a merge-group cancellation at 30 as a refutation. resolved_by stays f0994f70, the commit that lifted the default; the record has not left this branch. Refs: iss-2609260319483365 Assisted-by: Claude:claude-opus-5-5 --- ...e-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md b/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md index 1eb7264ef..72876c6ef 100644 --- a/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md +++ b/.abcd/work/issues/resolved/iss-2609260319483365-the-race-lane-runs-internal-surface-cli-at-the-edge-of-go.md @@ -9,7 +9,7 @@ found_during: "autonomous run A resumed 2026-09-25" origin: researcher-authored production_mode: hand-written found_at: ".github/workflows/ci.yml" -resolution: "The race lane runs under a declared go test -race -timeout 20m in ci.yml's check job, release.yml's verify job (and its scaffold template) and make preflight; the check job's ceiling rises from 30 to 45 minutes and verify's from 15 to 35 so the package timeout fires before the job is cancelled. TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds the three commands to one explicit timeout and each job's timeout-minutes to that timeout plus its measured headroom." +resolution: "The race lane runs under a declared go test -race -timeout 20m in ci.yml's check job, release.yml's verify job (and its scaffold template) and make preflight, which lifts go test's 10m per-package default: a slow package no longer fails at ten minutes, the failure that ejected PR #717. On merge-group runs the merge queue's 30-minute check response cap (the main ruleset, mirrored in .abcd/work/rulesets/main-protection.json) is the outer ceiling, so ci.yml's check job stays at 30 minutes, equal to that cap; on the macOS leg the slowest package starts about 18 minutes in, so a genuine hang there is still cancelled at 30 without a goroutine dump. Raising the queue's cap is an admin act on the live ruleset, left to the technical facilitator. release.yml's verify job is not a merge-queue job; its ceiling rises from 15 to 35 minutes so the package timeout fires there before the job is cancelled. TestRaceLaneBudgetIsDeclaredAndFitsItsJob holds the three commands to one explicit timeout, the check job's timeout-minutes at or below the mirror's merge-queue cap, and verify's timeout-minutes to the package timeout plus its measured headroom (the check-job half corrected in 2f7b903b)." impact: internal resolved_by: commit: "f0994f70" @@ -19,4 +19,4 @@ The race lane runs internal/surface/cli at the edge of go test's default 10-minu ## Grounds -- pursued: the macOS race step stops failing when internal/surface/cli's total -race wall time crosses 600s, while a genuinely hung test still fails with a goroutine dump at 20m; shown wrong by a merge-queue ejection from a package-timeout panic under 20m on a passing package, or by a check or verify job cancelled at its ceiling before the race step's own timeout could fire +- pursued: the macOS race step stops failing when internal/surface/cli's total -race wall time crosses 600s, while a genuinely hung test still fails with a goroutine dump at 20m in release.yml's verify job; shown wrong by a merge-queue ejection from a package-timeout panic under 20m on a passing package, by a verify job cancelled at its ceiling before the race step's own timeout could fire, or by a check job set above the merge queue's response cap. A merge-group check cancelled at the 30-minute cap does not show it wrong: that ceiling is the queue's, and the record says so From fc769b33a837c4c79763c8ebcc985a50f7358544 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Sat, 26 Sep 2026 05:03:56 +0100 Subject: [PATCH 7/7] chore: recalibrate the reading windows at the merged tip Merging main (39ed6fa1) into this branch, together with the race-lane budget test this branch adds under internal/core/lint, grew the tree-bounded reading objects. Each window is re-measured by dry run over a clean clone of the merged tip 1dff258e against the rule the entries state, the smallest ten-thousand boundary leaving at least one per cent headroom. Widening measures 1,021,262, past its 1,020,000 declaration, so it moves to 1,040,000 (1.83 per cent); entailment measures 343,062, 2.02 per cent under 350,000, so its declaration stays and only its measured fields move; detection measures 1,030,298, past its 1,030,000 declaration, so it moves to 1,050,000 (1.91 per cent). Comparative is bounded by the widening run it is handed, not by the tree, and is exempt by name. Refs: iss-2609251455354719 Assisted-by: Claude:claude-opus-5-5 --- .abcd/config/reading-presets.json | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/.abcd/config/reading-presets.json b/.abcd/config/reading-presets.json index 9be0fc300..87bb306ea 100644 --- a/.abcd/config/reading-presets.json +++ b/.abcd/config/reading-presets.json @@ -60,10 +60,10 @@ "test" ], "window": { - "tokens_est": 1020000, - "measured_tokens_est": 1009052, - "measured_bytes": 3884854, - "measured_at": "e1bbef2864f42c2fa8c21f7eaf97d0747580fc68" + "tokens_est": 1040000, + "measured_tokens_est": 1021262, + "measured_bytes": 3931862, + "measured_at": "1dff258e9ea7ae29841faaefc47966913fbbd954" } }, "entailment": { @@ -133,9 +133,9 @@ ], "window": { "tokens_est": 350000, - "measured_tokens_est": 340918, - "measured_bytes": 1312538, - "measured_at": "e1bbef2864f42c2fa8c21f7eaf97d0747580fc68" + "measured_tokens_est": 343062, + "measured_bytes": 1320792, + "measured_at": "1dff258e9ea7ae29841faaefc47966913fbbd954" } }, "comparative": { @@ -216,10 +216,10 @@ "test" ], "window": { - "tokens_est": 1030000, - "measured_tokens_est": 1018088, - "measured_bytes": 3919642, - "measured_at": "e1bbef2864f42c2fa8c21f7eaf97d0747580fc68" + "tokens_est": 1050000, + "measured_tokens_est": 1030298, + "measured_bytes": 3966650, + "measured_at": "1dff258e9ea7ae29841faaefc47966913fbbd954" } } }