Repository navigation
chore(release): give tests/migrate its own race shard so the 0.10.0 release workflow can pass - #1065
Conversation
Under -race the SysML v1 migrator's suite grew past 24 minutes on the pull-request runners and past the 30-minute package timeout on CircleCI's develop builds, failing every merge since the rest shard carried it. scripts/race-shard.sh migrate names it; both CI configurations run six race shards and five coverage shards. Co-Authored-By: jason.han <hanhuijun@gmail.com>
Under -race the SysML v1 migrator suite outgrew the 30-minute package timeout on the CircleCI rest node, so the release workflow could not pass; it now runs as a shard of its own. Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
The package already ran 1370-1800+ seconds under -race on the rest node, so the 30-minute default leaves no margin even on a node of its own. Co-Authored-By: jason.han <hanhuijun@gmail.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Under -race the SysML v1 migrator's suite grew past 24 minutes on the pull-request runners and past the 30-minute package timeout on CircleCI's develop builds, failing every merge since the rest shard carried it. scripts/race-shard.sh migrate names it; both CI configurations run six race shards and five coverage shards. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: jason.han <hanhuijun@gmail.com>
| timeout=30m; [ "$shard" = migrate ] && timeout=55m | ||
| go test -v -race -pgo=off -timeout "$timeout" $pkgs |
There was a problem hiding this comment.
🟡 Migration shard retains shorter silence limit
When tests/migrate exceeds 30 minutes, the 55-minute timeout cannot prevent CircleCI from terminating the silent command. go test buffers package output, so observed 32-minute runs still fail the release workflow.
Learn more
CircleCI stops a run step when it produces no output for no_output_timeout. The repository already documents that go test buffers a package's -v output until that package finishes in the solver-absent gate at .circleci/config.yml. This shard contains only tests/migrate, so a package run can remain silent for its entire 23–32 minute duration. Raising Go's test timeout to 55 minutes therefore leaves CircleCI's independent 30-minute silence limit as the effective cap.
Example: A loaded runner completes tests/migrate in 32 minutes. CircleCI sees no output after the initial shard message, kills the step at 30 minutes, and never reaches Go's 55-minute timeout.
Recommended fix: Raise this run step's no_output_timeout above the migration shard's 55-minute Go timeout, or stream periodic output without masking the test exit status.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Not a bug: go test -v streams a single package's output as it runs, so the silence limit is never reached. Evidence from the first release build with this change (pipeline 3995 on main): the migrate node ran 33.2 minutes under no_output_timeout: 30m and passed (ok github.com/Open-MBEE/OpenSysML/tests/migrate 1923.117s). The buffering noted on the REPL gate step applies to multi-package runs; the migrate shard is one package.
What and why
Hotfix from
mainfor the 0.10.0 release. Since #1033 every CircleCIdevelopbuild has failed inGo race tests: on the 4-vCPUrestnodetests/migrateunder-racewent from 1370 s to past the 30-minute package timeout (panic: test timed out after 30m0s,FAIL github.com/Open-MBEE/OpenSysML/tests/migrate 1800.098s).mainnow carries #1033 through #1064, and every publish job in thereleaseworkflow requiresGo race tests, so a rehearsal or av0.10.0tag fails the same way; the nightly, which only publishes green CircleCI commits, has been stalled as well.tests/migratebecomes a race shard of its own (scripts/race-shard.sh migrate,make test-shard SHARD=migrate,make coverage-shard SHARD=migrate); CircleCIgo-race-testandgo-coveragego from four to five executors,pr.ymlfrom five to six race shards. Its changelog fragment is folded into the 0.10.0### Changedsection.migrateshard the Makefile's 55-minute per-package timeout (make test-shardalready uses 55 m). The package alone, with nothing beside it, takes 1423 s under-raceon an idle 8-vCPU machine, so on the 4-vCPUlargeclass a node of its own leaves no margin under 30 m.pr.ymlshard comment to two lines.No test or gate changes: the same packages run under the same flags, one more job wide.
How it was verified
circleci config validate .circleci/config.ymlpasses;scripts/race-shard.sh migrateprints onlytests/migrateandrestno longer lists it (119 packages).go test -race -pgo=off -count=1 ./tests/migratealone:ok 1422.985s(8 vCPU).python3 scripts/changelog.py check,python3 scripts/check-doc-ids.pypass; exactly one## 0.10.0section.Go race tests (migrate)/(rest)jobs, then the CircleCIrelease_rehearsalonmainafter merge, are the proof.Checklist
make testandmake lintpass locally (CI configuration change; the Go suite is unchanged)AGENTS.mdshard list, Makefile help text, via ci(test): give tests/migrate its own race shard #1060)docs/project/releasing.md)make docs-countsrun if a gate count moved (n/a)F4,K5) in the body, docs, or changelogLink to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/3db6a090aa214667b2a66d372c6e36b1
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/3db6a090aa214667b2a66d372c6e36b1?variant=devin
Requested by: @HuiJun