Make GC between test items opt-in

[davidanthoff]

Summary

execute_testrun no longer runs a full GC.gc(true) after every test item by default. Pass gc_between_testitems=true to opt in. Until now it was on whenever a run used more than one test process, matching ReTestItems. It arrived in ff943d8, an unreviewed WIP commit.

Why

Measured on JuliaWorkspaces.jl CI:

• It slows runs down. Forcing a full GC after each item made the test phase slower in 15 of 18 legs (Linux, Windows and macOS; Julia 1.11–1.13), by 1.26× at the median. Compared: GC on vs GC off.
• It doesn't save memory. Test-process RSS was the same with it on or off (median 2.84 vs 2.95 GB). What grows is live data, such as compiled code and caches, which no collection frees. Julia already collects real garbage when it needs to.
• Under memory pressure it's disastrous. On GitHub's macOS arm64 runners (7 GB, no swap), full collections took 40–75 s each and turned jobs into 5–6 h runs. With 2 test processes on Julia 1.12, a run with GC on timed out after 80 minutes at 677 of 1170 items; with GC off the same run finished in 12 minutes (diagnostic runs).

It can still help a suite whose items hold memory outside the Julia heap that only a finalizer releases, such as large C buffers, mmaps, handles or child processes. That's why it stays available as an option.

Changes

• src/testitemcontroller.jl: something(gc_between_testitems, false), with a new explanatory comment.
• src/state.jl, TestItemServer.jl and test/test_worker_lifecycle.jl: updated comments.
• New @testitem "GC between test items is off by default, even for a multi-process run". It checks both the implicit default and an explicit true, so it can't pass vacuously.
• CHANGELOG entry.

Behaviour change for JSON-RPC callers (VS Code): create_testrun never passes this option, so those runs now always run without GC between items, and they have no way to turn it on.

Testing

• test/test_worker_lifecycle.jl passes (60/60) on Julia 1.12.7.
• I also checked that the new test fails when the old default is put back.

Related

• julia-testitems/TestItemRuns.jl: memory-aware default worker count (companion PR).
• Turn off the GC between test items in CI (fixes macOS aarch64 timeouts) julia-vscode/JuliaWorkspaces.jl#346: the CI workaround this replaces.

🤖 Generated with Claude Code