Speed up integration test CI by packing the matrix and trimming per-job setup

[masih]

Integration test CI is slower and pricier than it needs to be. Turns out on ubuntu-large every matrix job starts within ~1s of each other, so there's no batching: wall-clock is just prepare-cluster + slowest row, and job count only buys runner-minutes. So the trick is to cut per-job fixed cost, shorten prepare-cluster, and pack rows without making any of them long.

Before → after (main vs this branch, all jobs green):

• Wall-clock: 19m01s → 12m45s (−33%)
• Runner time: 227 → 144 runner-minutes (−37%)
• Coverage: identical. A jq dump of every (env, script) pair in the matrix is byte-for-byte the same on main and here.

What changed:

• ci-free-disk.sh skips the ~3.5 min rm -rf when / already has ≥ CI_FREE_DISK_MIN_GIB free Every call site states its own threshold: 40 for test jobs, 60 where a job builds images/seid on / (prepare-cluster, execution-specs prepare, release-upgrade, the go-lint hook). Runners boot with ~85 GiB free, so we were burning 3.5 min per job freeing space we never touched, and the check self-heals if the runner image ever shrinks.
• rpcnode image gets its own prepare-rpcnode job on ubuntu-latest, in parallel with prepare-cluster instead of on its critical path. prepare-cluster 7m20s → 4m38s.
• Sparse checkout everywhere: drops the darwin/aarch64 libwasmvm* archives (~500 MB of a 950 MB tree) that a linux/amd64 build never links. Checkout 50s → 28s per job.
• Matrix rows sharing a cluster env now run their suites back-to-back in one job: 35 → 26 jobs. Merged: Compat + RPC .io/.iox, Precompile/Endpoints + Precompiles, Autobahn Compat + Autobahn GIGA, SeiDB SS + Mint/Staking/Bank + Distribution, Gov/Oracle/Authz + Wasm, and the three Autobahn Cosmos rows. Everything else keeps its own row.
• Smaller stuff: parallel image pulls (each waited separately so a failed pull fails at the pull, not at docker tag), ::group:: per script so failures point at the right suite, npm cache, 2s boot polling, skip apt-get jq when present.

Packing policy now lives in a comment above set-matrix in the workflow (not in a row _comment that vanishes with its row): keep rows under ~8 min, suites asserting absolute chain state go first and say so in _comment, destructive rows (kill/wipe/state-sync a validator, disable wasm, upgrades) stay alone. Known couplings are recorded: TestSeiDBStateStore asserts absolute wasm code counts, TestMintModule and the Gov burn case assert absolute usei supply. Both hold today (fees aren't burned, tokenfactory doesn't burn usei).

Two things worth knowing:

• Packed rows run up to three go test invocations, so a fixed 28m -timeout per invocation no longer beats the 30m job limit. The row now has a 26m budget and each invocation gets JOB_TIMEOUT = whatever is left, so a hang anywhere in the row still ends in a goroutine dump rather than a job cancel, and adding suites to a row doesn't change that.
• 14 Integration Test (...) check contexts got renamed. If branch protection requires any of them individually rather than the Integration Test Check umbrella, the ruleset needs a tweak.

Closes PLT-1223.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 10, 2026, 11:00 AM

[cursor]

PR Summary

Medium Risk
CI-only, but packed matrix rows share chain state and renamed required checks can break branch protection; no application runtime changes.

Overview
Speeds up Docker integration test CI by cutting per-job setup, shortening prepare-cluster, and packing matrix rows so the same suites still run with fewer jobs (coverage unchanged per PR description).

ci-free-disk.sh now skips the slow rm -rf reclaim when / already has at least CI_FREE_DISK_MIN_GIB GiB free (default 40); build-heavy jobs set 60. Sparse checkout omits darwin/aarch64 wasmvm blobs (~500 MB) on linux/amd64 jobs. prepare-rpcnode builds and pushes the rpcnode image in parallel with prepare-cluster; the Makefile only requires localnode before build-seid-in-localnode-ci.

integration-test-matrix.json merges compatible suites into shared rows (35 → 26 jobs) with documented ordering for absolute-state assertions; destructive/upgrade rows stay isolated. Packed rows use ROW_BUDGET_SECONDS and a shrinking JOB_TIMEOUT per script instead of a fixed 28m per go test. Workflow tweaks include parallel GHCR pulls, npm cache, faster cluster wait polling, and ::group:: per suite script.

14 matrix job display names changed — branch protection may need updates if it keys on individual Integration Test (...) contexts rather than Integration Test Check.

^{Reviewed by Cursor Bugbot for commit de70adf. Bugbot is set up for automated code reviews on this repo. Configure here.}

[seidroid]

CI-only change that packs the integration matrix from 35 to 26 jobs and trims per-job setup; I verified the (env, script) coverage is identical to base and that every documented packing coupling (absolute usei supply in mint_test.yaml/gov_proposal_test.yaml, absolute wasm code counts in state_store_test.yaml, gov's UnbondingTime change vs. the wasm suite's code-0-only assertions, and the spec-only .io/.iox comparison) actually holds. No blockers; two suggestions plus an operational check before merge.

Findings: 0 blocking | 4 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] set-matrix still does a full checkout although it only reads integration-test-matrix.json, and it sits on integration-tests' needs chain. Adding sparse-checkout: .github/workflows there applies the same win the PR takes in every other job.
• [suggestion] Before merge, confirm the branch-protection ruleset: 14 Integration Test (...) contexts were renamed and prepare-rpcnode is a new job. If any individual context is required rather than the Integration Test Check umbrella, merges will block on contexts that no longer report.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] Rather than only documenting that JOB_TIMEOUT: 28m no longer guarantees a goroutine dump, consider lowering it. With timeout-minutes: 30, ~2 min of fixed setup, and up to three go test invocations per row, only a hang in the first invocation can still hit the Go timeout; a hang in the second or third gets a GitHub job cancel instead, which is exactly the diagnostic loss this setting exists to prevent. The packing policy directly above caps rows at ~8 min of test time, so something like JOB_TIMEOUT: 10m would restore the dump guarantee for every invocation in every row while still leaving generous slack over the 4–7 min the rows actually take.

[alexander-sei]

@masih this might make it flakey again as soon as we start adding some new tests

[devin-ai-integration]

Correction, I misread the concern above. A fixed JOB_TIMEOUT (28m or 10m) is the wrong shape for packed rows: 28m stops guaranteeing a dump once a row has more than one invocation, and 10m gets flaky as rows grow. Pushed a fix instead: the row gets a 26m budget (ROW_BUDGET_SECONDS) and each go test invocation is started with -timeout = the budget minus elapsed time, so a hang anywhere in the row hits Go's timeout before timeout-minutes cancels the job, regardless of how many suites the row contains.

[masih]

@seidroid review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.25%. Comparing base (e43f64e) to head (de70adf).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4133      +/-   ##
==========================================
- Coverage   61.25%   60.25%   -1.01%     
==========================================
  Files        2195     2078     -117     
  Lines      192497   179013   -13484     
==========================================
- Hits       117916   107866   -10050     
+ Misses      63092    60812    -2280     
+ Partials    11489    10335    -1154     

Flag	Coverage Δ
sei-db	69.80% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 117 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

CI-only change packing the integration matrix 35→26 jobs and trimming per-job setup; I re-verified that the (env, script) coverage is identical to base and that every state coupling the new row _comments claim actually holds in the code (no usei burn path, no tokenfactory creation fee, .io fixtures compared spec-only). No blockers — one log-grouping suggestion, plus a pre-existing fragility the packing slightly widens.

Findings: 0 blocking | 2 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• None at the file/PR level.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.
• 1 non-blocking pre-existing issue(s) listed below under pre-existing issues.

Pre-existing issues

• [suggestion] integration_test/mint_module/mint_test.yaml and integration_test/gov_module/gov_proposal_test.yaml assert an absolute usei total supply (5000000000333333333333). docker/localnode/scripts/step2_genesis.sh gives the cluster a 3-day token release schedule, and x/mint/keeper/hooks.go mints 333333333333 on the first epoch whose UTC date differs from LastMintDate — so any cluster that lives across a UTC midnight breaks both assertions regardless of what suites run. The new row _comments reason only about suites minting/burning and don't mention this chain-driven mint; packing widens the window between cluster boot and the assertion (Autobahn Gov now runs several minutes in rather than immediately). Asserting a delta against a supply read at row start would make these rows time-independent.

[masih]

Addressing the unrelated flakes in separate PR; that test is flaking on main as well as other open PRs.