From 04185905018db32e3c2d08c471faccd88c314c4e Mon Sep 17 00:00:00 2001 From: David Anthoff Date: Thu, 24 Sep 2026 13:05:15 -0700 Subject: [PATCH] Sleep GC-safe in the test server watchdog on Julia 1.12+ The watchdog thread polled with `Libc.systemsleep`, a plain `ccall`, so it stayed GC-unsafe for the whole 50 ms sleep and every stop-the-world GC in the test process waited for it to wake up: 25-50 ms of extra time-to-safepoint per GC on Windows, 50-190 ms on GitHub macOS runners, none of it counted in the GC time reported for test items. `_watchdog_loop` now sleeps through `_watchdog_sleep`, which on Julia >= 1.12 uses `@ccall gc_safe=true`: `usleep` on Unix and libuv's `uv_sleep` on Windows. `@ccall` cannot express `stdcall`, which Win32 `Sleep` needs on 32-bit Windows; `uv_sleep` is a cdecl wrapper around `Sleep` exported by the Julia runtime. Older Julia keeps `Libc.systemsleep`, behind `@static` so the new macro form is never expanded there. The explicit `GC.safepoint()` stays. Measured with a 200 MB heap and 5x GC.gc(true): time-to-safepoint per GC drops from ~23 ms mean to 0 on 1.12.7 and 1.13.0 (x64) and on 1.12.7 (x86); 1.11 is unchanged. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + testprocess/TestItemServer/src/watchdog.jl | 51 +++++++++++++++++----- 2 files changed, 43 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32d7fab..d3ef0d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- On Julia 1.12 and later, a garbage collection in a test process no longer has to wait for the hang watchdog to wake up. The watchdog paces itself with a plain OS sleep, and that `ccall` kept its thread GC-unsafe for the full 50 ms poll interval, so every stop-the-world collection stalled until the sleep ended: 25–50 ms of extra time-to-safepoint per GC on Windows and 50–190 ms on GitHub's macOS runners, none of it counted in the GC time reported for a test item. The sleep now runs in a GC-safe region (`@ccall gc_safe=true`), which brings that delay to zero. On Windows it goes through libuv's `uv_sleep` rather than Win32 `Sleep`, because `@ccall` cannot express the `stdcall` convention `Sleep` needs on 32-bit Windows. Older Julia versions keep the previous sleep. + - A test item that uses a `@testmodule` now runs with its own file path, so `@__DIR__` and `@__FILE__` inside it resolve against its own directory and its top-level frames are attributed to its own file. The test process evaluated the `@testmodule` into the same local that holds the test item's path, so the item's body was then handed the *setup's* path — an item in `test/sub/` declaring a setup defined in `test/` loaded `joinpath(@__DIR__, "data", …)` relative to `test/`, and a failure in it was reported at the setup's file. Only the first item to evaluate a given `@testmodule` on a process was affected, because later items reuse the already-evaluated module and never take that branch, which is why this presented as order- and scheduling-dependent. The working directory was always correct, so a bare relative path was unaffected. `TestItemRunner.jl`'s in-process runner never had this bug. - Test processes running with a portable `JULIA_CPU_TARGET` (a multi-versioned target such as `generic;sandybridge,...;haswell,...`, which CI sets so a cached depot survives moving between runner CPUs) now precompile their test environment from a child Julia started on that target's base variant. The variable never changes the process it is set in — Julia reads it only when spawning precompilation workers — so a test process accepted whatever native images the depot already held, compiled only the stale packages, and handed them to workers that rejected those native images: slow nested recompilation on Julia ≤ 1.12, and on Julia 1.13, whose workers run with `--compiled-modules=strict`, a failure of every `using` in the run (`Precompiled image ... not available with flags ...`). The child rejects the same images the workers would and rebuilds them under the portable target once; activation without the variable is unchanged. - A package whose `test/` folder is a `[workspace]` member (Julia 1.12+) can now run its test items. Such a test project has no manifest of its own — it resolves against the workspace's shared one — so mirroring it into a scratch environment left `TestEnv` trying to resolve a manifest from scratch, which fails as soon as the workspace contains an unregistered path-tracked package. The test process now does what `Pkg.test` does for exactly this shape and nothing else: it activates the member project in place, instantiates it and precompiles it, refusing to run rather than resolving into the user's tree when the workspace manifest is missing. Any other `test/Project.toml`, including one with a manifest of its own that `dev`s the package, is still sandboxed as `Pkg.test` would. ([#103](https://github.com/julia-testitems/TestItemControllers.jl/pull/103)) diff --git a/testprocess/TestItemServer/src/watchdog.jl b/testprocess/TestItemServer/src/watchdog.jl index 28ea9c9..7e7e100 100644 --- a/testprocess/TestItemServer/src/watchdog.jl +++ b/testprocess/TestItemServer/src/watchdog.jl @@ -6,8 +6,8 @@ # # * The watchdog must not depend on libuv. When a test item wedges the main thread no # thread services the event loop any more, so `sleep`, `Timer` and socket writes never -# complete. The watchdog therefore paces itself with `Libc.systemsleep` (a plain OS -# sleep) and writes to a file through `IOStream`, which is a blocking `write(2)` and +# complete. The watchdog therefore paces itself with a plain OS sleep (see +# `_watchdog_sleep`) and writes to a file through `IOStream`, which is a blocking `write(2)` and # needs no running loop. # * The watchdog must not share a thread with the test item. The launch reserves two # interactive threads on Julia >= 1.9 (see `src/testprocess.jl`): the main task — and @@ -229,16 +229,47 @@ function _write_dump(testitem_id::AbstractString, timeout_ms::Float64) return nothing end +# The watchdog's poll sleep. `Libc.systemsleep` is a plain `ccall`, so the thread stays +# GC-unsafe for the whole sleep and every stop-the-world collection in the process has to +# wait for it to wake up: measured at 25–50 ms per GC on Windows and 50–190 ms on macOS CI +# runners, none of it counted in the GC time reported for an item. From Julia 1.12 on, +# `@ccall gc_safe=true` sleeps in a GC-safe region instead, so a collection proceeds without +# waiting. The sleep must not call back into Julia, and neither of these does. +# +# `@ccall` always uses the C calling convention and cannot express `stdcall`, which is what +# `Libc.systemsleep` passes to Win32 `Sleep`; on 32-bit Windows calling `Sleep` as cdecl would +# corrupt the stack. So Windows calls libuv's `uv_sleep` instead — a cdecl function exported +# by the Julia runtime whose Windows implementation is exactly `Sleep(msec)`. It is a plain +# utility, not an event loop operation, so it keeps the watchdog independent of libuv's loop. +@static if VERSION >= v"1.12" + @static if Sys.iswindows() + function _watchdog_sleep(seconds::Real) + @ccall gc_safe=true uv_sleep(round(Cuint, seconds * 1e3)::Cuint)::Cvoid + return nothing + end + else + function _watchdog_sleep(seconds::Real) + @ccall gc_safe=true usleep(round(Cuint, seconds * 1e6)::Cuint)::Cint + return nothing + end + end +else + _watchdog_sleep(seconds::Real) = (Libc.systemsleep(seconds); nothing) +end + function _watchdog_loop() while !WATCHDOG_STOP[] - Libc.systemsleep(WATCHDOG_POLL_SECONDS) - - # Mandatory. This loop neither allocates nor yields, and `Libc.systemsleep` is a - # plain `ccall`, so without an explicit safepoint the thread never becomes - # collectable — and `GC.gc(true)` between test items, which stops the world, blocks - # here forever. Verified: removing this deadlocks the test process on its first - # inter-item collection, which is why the watchdog cannot use a yield-free loop - # without one. + _watchdog_sleep(WATCHDOG_POLL_SECONDS) + + # Mandatory. This loop neither allocates nor yields, so without an explicit + # safepoint the thread never becomes collectable while it is awake — and before + # Julia 1.12 `_watchdog_sleep` is a plain, GC-unsafe `ccall`, so it never does at + # all: `GC.gc(true)` between test items, which stops the world, blocks here forever. + # Verified: removing this deadlocks the test process on its first inter-item + # collection, which is why the watchdog cannot use a yield-free loop without one. + # From 1.12 on the sleep itself is GC-safe, so a collection no longer waits for the + # thread to wake; the safepoint stays because older Julia still depends on it, and + # on newer Julia it costs nothing. GC.safepoint() deadline = WATCHDOG_DEADLINE[]