Skip to content

ci: run the suite as four parallel shards, and drop the unit-suites step that repeated it - #163

Open
STRML wants to merge 4 commits into
mainfrom
ci/shard-the-suite
Open

STRML wants to merge 4 commits into
mainfrom
ci/shard-the-suite

Conversation

@STRML

@STRML STRML commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Summary

CI's test job ran every test one after another: about 380 seconds for run-all.sh plus 47 seconds for a second step that ran all the unit suites again, in a 9-minute run. This splits the suite into four shards that run at once.

  • tests/run-all.sh takes RUNALL_SHARD=i/n: every n-th test and unit suite, counted in file order, so n shards run the whole suite exactly once. RUNALL_LIST=1 prints what a shard would run and runs nothing. Unset, it is the whole suite as before.
  • tests.yml runs four test (i/4) shard jobs and one test job that waits on them, so the check keeps its old name.
  • The separate "unit suites" step is gone. run-all.sh already ran all 23 of those suites, so the step only repeated 47 seconds of work.
  • tests/run-parallel.sh [n] runs n shards on one machine and prints one total.

Why round-robin and not weighted: the slow tests are spread through the file (64 of 280 take over 5 seconds and account for half the time), so four shards come out at 95% of the ideal split. Measured on the real per-test timings, the longest of four holds 26% of the run time against an ideal 25%.

Status

Check Result
tests/shard.sh, 42 assertions The shards partition the suite exactly for n = 2, 3, 4, 6, 8: no entry in two shards, none missing, sizes within one. Refuses 11 malformed specs. Fails if a test is called directly instead of through the gate (bare, indented, or after &&, `
Four shards at once on a dev machine 342s and 370s against 15 to 19 minutes sequentially. Only the four known sandbox-only failures (adapter-claude needs ps, two review-cmux interactive assertions, the prune-worktrees watchdog timing).
One unsharded pass vs four shards, same code Identical: 2910 passed, 4 failed, the same four sandbox-only failures, both ways. The shards run exactly the tests the single run does, no more and none fewer.
Hosted CI on this PR All six checks green on 7bec514. Shards 1m45s, 1m27s, 1m50s, 1m33s; aggregating test 3s; wall time about 1m53s against 7 to 9 minutes for the old single job. The first sharded run (915bcfd) had one shard at 2m38s and the rest near 1m32s.
Subagent reviews (code-reviewer) 915bcfd: no blocking findings; its should-fix list is done in f1bf6b1 (!cancelled() on the aggregator, run-parallel.sh stops its shards on a signal, the gate catches any direct test_ call, tmp-cleanup.sh's child no longer inherits a shard, the doc numbers). ef2434f and 7bec514: no blocking findings; three nits left open below.

The first parallel run showed one extra failure, test_chunking_an_omp_session_cuts_the_day_then_chunks_the_live_chain. It was not parallelism: the first chunk's header line embeds the session's absolute path and is cut at the slimmer's 400-character cap, and it is 391 characters under a 9-character TMPDIR. The runner's nested temp dirs added enough path to cut it mid-JSON, and jq printed nothing. A Mac's default TMPDIR would hit it too. The assertion now checks the line's fixed prefix and passes at TMPDIR lengths 9, 28 and 62.

What the first CI run showed, and what was done about it

RUNALL_TIMES (added in f1bf6b1) makes each shard print its slowest entries. The first run had shard 0 about 65 seconds behind. The numbers named the cause: test_question_streaks_state_lives_with_the_install took 46 seconds, of which 32 were sleep 1 waits on a lock the test holds on purpose, and suite:x-bookmarks took 24 seconds on a sleep 2 that x-bookmarks.sh ran after every page, including the last one it would fetch.

  • ef2434f: AUTODREAM_QUESTION_LOCK_WAIT (default 15, unchanged; a non-number keeps it) sets the tries in acquire_lock. The two lock-held cases set it to 1 and one assertion bounds the wait. Red first (15s), then green (1s).
  • 7bec514: x-bookmarks.sh sleeps only when another page follows, so a one-page run no longer waits two seconds for nothing (the nightly run included). The suite shims sleep next to its curl shim, and a new test pins both the 2-second pacing and the absence of the trailing wait. Red first (1 and 2 sleeps), then green.

Open follow-ups

  • The header line being cut at 400 characters when a session path is long means a worker reads a header that is invalid JSON. Harmless in practice (real session paths are around 100 characters) but the header could be exempt from the cut.
  • Three review nits, not worth restarting CI for: a digit string long enough to overflow [ -gt ] in AUTODREAM_QUESTION_LOCK_WAIT would wait on a held lock forever (clamp its length); the header comment could say 0 means no wait; run_bm now reads a caller-exported X_MAX_PAGES.
  • launchd.yml still runs the whole suite unsharded on macOS (weekly and on launchd changes). Untouched.

Test plan

  • bash tests/shard.sh
  • tests/run-parallel.sh 4
  • One unsharded full pass; compare the totals with the four shards
  • Hosted CI: four test (i/4) jobs and test all green; wall time confirmed
  • Subagent reviews

STRML added 4 commits October 20, 2026 11:43
…tep that repeated it

tests/run-all.sh takes RUNALL_SHARD=i/n (every n-th test and unit suite in file order) and RUNALL_LIST=1 (print what a shard would run). tests.yml runs four shard jobs and an aggregating test job under the old name. The separate unit-suites step is removed: run-all.sh already ran all 23 of those suites, so it repeated 47s. tests/run-parallel.sh [n] runs n shards on one machine.

tests/shard.sh (40 assertions) proves the shards partition the suite for n = 2, 3, 4, 6, 8, refuses 11 malformed specs, and fails when a test invocation skips the gate or the workflow matrix and /4 disagree; its first red run found that malformed specs ran the whole suite, fixed. Measured on this machine: four shards 342s and 370s against 15 to 19 minutes sequentially; no failures beyond the four known sandbox-only ones.

Also fixes the one test the first parallel run exposed: the omp chunking assertion parsed a header line 9 characters under the slimmer's 400-character cap, so a longer TMPDIR cut it mid-JSON. It checks the fixed prefix now; passes at TMPDIR lengths 9, 28 and 62.
Review of 915bcfd: the aggregating test job uses !cancelled() instead of always(), which went red on a run a newer push had cancelled; run-parallel.sh puts each shard in its own process group and stops them on INT or TERM (a live TERM leaves no shard behind); tests/shard.sh catches a direct test_ call in any form (bare, indented, after && || ; then do) and self-tests the detector on sample lines; tmp-cleanup.sh no longer lets its child run-all inherit RUNALL_SHARD (8/8 under shards 0 to 3); the docs state what was measured.

RUNALL_TIMES=<file> records seconds per test and suite, and each CI shard prints its slowest twelve. The first CI run had shard 0 at 2m38s against about 1m32s for the others. Locally the tests balance within 10% but prune-worktrees (49s) and tmp-cleanup (11s) both land in shard 0 with a 46s test; that run's numbers say whether a weighted split is worth its upkeep.
… purpose

test_question_streaks_state_lives_with_the_install was the slowest entry in CI (46s of shard 0's 137s, from the RUNALL_TIMES output of the first sharded run). Two of its cases hold the state lock deliberately to show clear refuses while an update owns it, and acquire_lock waited its 15 one-second tries each time. AUTODREAM_QUESTION_LOCK_WAIT sets the tries (default 15 unchanged, a non-number keeps it); the two cases set it to 1 and one assertion bounds the wait. Red first (the call took 15s), then green (1s); the whole test is 47s to 19s locally.
… suite stops sleeping

bin/x-bookmarks.sh slept 2 seconds after every fetched page, including the one that ended the loop, so every run waited for nothing. It sleeps only when another page follows. tests/x-bookmarks.sh shims sleep next to its curl shim (recording to MOCK_SLEEP_LOG) so no test waits, and a new test pins the pacing: a one-page run never sleeps and a two-page run sleeps once, for 2. Red first (1 and 2 sleeps), then green. The suite was 24s of CI's shard 3 (123s) and drops from 31s to 9s locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant