Repository navigation
Conversation
…tep that repeated it tests/run-all.sh takes RUNALL_SHARD=i/n (every n-th test and unit suite in file order) and RUNALL_LIST=1 (print what a shard would run). tests.yml runs four shard jobs and an aggregating test job under the old name. The separate unit-suites step is removed: run-all.sh already ran all 23 of those suites, so it repeated 47s. tests/run-parallel.sh [n] runs n shards on one machine. tests/shard.sh (40 assertions) proves the shards partition the suite for n = 2, 3, 4, 6, 8, refuses 11 malformed specs, and fails when a test invocation skips the gate or the workflow matrix and /4 disagree; its first red run found that malformed specs ran the whole suite, fixed. Measured on this machine: four shards 342s and 370s against 15 to 19 minutes sequentially; no failures beyond the four known sandbox-only ones. Also fixes the one test the first parallel run exposed: the omp chunking assertion parsed a header line 9 characters under the slimmer's 400-character cap, so a longer TMPDIR cut it mid-JSON. It checks the fixed prefix now; passes at TMPDIR lengths 9, 28 and 62.
Review of 915bcfd: the aggregating test job uses !cancelled() instead of always(), which went red on a run a newer push had cancelled; run-parallel.sh puts each shard in its own process group and stops them on INT or TERM (a live TERM leaves no shard behind); tests/shard.sh catches a direct test_ call in any form (bare, indented, after && || ; then do) and self-tests the detector on sample lines; tmp-cleanup.sh no longer lets its child run-all inherit RUNALL_SHARD (8/8 under shards 0 to 3); the docs state what was measured. RUNALL_TIMES=<file> records seconds per test and suite, and each CI shard prints its slowest twelve. The first CI run had shard 0 at 2m38s against about 1m32s for the others. Locally the tests balance within 10% but prune-worktrees (49s) and tmp-cleanup (11s) both land in shard 0 with a 46s test; that run's numbers say whether a weighted split is worth its upkeep.
… purpose test_question_streaks_state_lives_with_the_install was the slowest entry in CI (46s of shard 0's 137s, from the RUNALL_TIMES output of the first sharded run). Two of its cases hold the state lock deliberately to show clear refuses while an update owns it, and acquire_lock waited its 15 one-second tries each time. AUTODREAM_QUESTION_LOCK_WAIT sets the tries (default 15 unchanged, a non-number keeps it); the two cases set it to 1 and one assertion bounds the wait. Red first (the call took 15s), then green (1s); the whole test is 47s to 19s locally.
… suite stops sleeping bin/x-bookmarks.sh slept 2 seconds after every fetched page, including the one that ended the loop, so every run waited for nothing. It sleeps only when another page follows. tests/x-bookmarks.sh shims sleep next to its curl shim (recording to MOCK_SLEEP_LOG) so no test waits, and a new test pins the pacing: a one-page run never sleeps and a two-page run sleeps once, for 2. Red first (1 and 2 sleeps), then green. The suite was 24s of CI's shard 3 (123s) and drops from 31s to 9s locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI's
testjob ran every test one after another: about 380 seconds forrun-all.shplus 47 seconds for a second step that ran all the unit suites again, in a 9-minute run. This splits the suite into four shards that run at once.tests/run-all.shtakesRUNALL_SHARD=i/n: every n-th test and unit suite, counted in file order, so n shards run the whole suite exactly once.RUNALL_LIST=1prints what a shard would run and runs nothing. Unset, it is the whole suite as before.tests.ymlruns fourtest (i/4)shard jobs and onetestjob that waits on them, so the check keeps its old name.run-all.shalready ran all 23 of those suites, so the step only repeated 47 seconds of work.tests/run-parallel.sh [n]runs n shards on one machine and prints one total.Why round-robin and not weighted: the slow tests are spread through the file (64 of 280 take over 5 seconds and account for half the time), so four shards come out at 95% of the ideal split. Measured on the real per-test timings, the longest of four holds 26% of the run time against an ideal 25%.
Status
tests/shard.sh, 42 assertions&&, `adapter-claudeneedsps, tworeview-cmuxinteractive assertions, theprune-worktreeswatchdog timing).7bec514. Shards 1m45s, 1m27s, 1m50s, 1m33s; aggregatingtest3s; wall time about 1m53s against 7 to 9 minutes for the old single job. The first sharded run (915bcfd) had one shard at 2m38s and the rest near 1m32s.915bcfd: no blocking findings; its should-fix list is done inf1bf6b1(!cancelled()on the aggregator,run-parallel.shstops its shards on a signal, the gate catches any directtest_call,tmp-cleanup.sh's child no longer inherits a shard, the doc numbers).ef2434fand7bec514: no blocking findings; three nits left open below.The first parallel run showed one extra failure,
test_chunking_an_omp_session_cuts_the_day_then_chunks_the_live_chain. It was not parallelism: the first chunk's header line embeds the session's absolute path and is cut at the slimmer's 400-character cap, and it is 391 characters under a 9-characterTMPDIR. The runner's nested temp dirs added enough path to cut it mid-JSON, andjqprinted nothing. A Mac's defaultTMPDIRwould hit it too. The assertion now checks the line's fixed prefix and passes atTMPDIRlengths 9, 28 and 62.What the first CI run showed, and what was done about it
RUNALL_TIMES(added inf1bf6b1) makes each shard print its slowest entries. The first run had shard 0 about 65 seconds behind. The numbers named the cause:test_question_streaks_state_lives_with_the_installtook 46 seconds, of which 32 weresleep 1waits on a lock the test holds on purpose, andsuite:x-bookmarkstook 24 seconds on asleep 2thatx-bookmarks.shran after every page, including the last one it would fetch.ef2434f:AUTODREAM_QUESTION_LOCK_WAIT(default 15, unchanged; a non-number keeps it) sets the tries inacquire_lock. The two lock-held cases set it to 1 and one assertion bounds the wait. Red first (15s), then green (1s).7bec514:x-bookmarks.shsleeps only when another page follows, so a one-page run no longer waits two seconds for nothing (the nightly run included). The suite shimssleepnext to itscurlshim, and a new test pins both the 2-second pacing and the absence of the trailing wait. Red first (1 and 2 sleeps), then green.Open follow-ups
[ -gt ]inAUTODREAM_QUESTION_LOCK_WAITwould wait on a held lock forever (clamp its length); the header comment could say0means no wait;run_bmnow reads a caller-exportedX_MAX_PAGES.launchd.ymlstill runs the whole suite unsharded on macOS (weekly and on launchd changes). Untouched.Test plan
bash tests/shard.shtests/run-parallel.sh 4test (i/4)jobs andtestall green; wall time confirmed