recycle_on_failure and retries alternative

[christiangnrd]

While reviewing #148 I wasn't too happy with how the retries feature was reimplementing a simple version of the package so I started looking into reusing the same test machinery for the retries too.

Failing tests show up as yellow if they are to be retried.

Example output:

Running 2 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
fails      (7) │     0.05 │   failed at 2026-08-08T15:10:13.996
passes     (7) │     0.00 │   0.00 │  0.0 │       0.00 │   325.20 │
Retrying 1 failed test  (1)
fails      (8) │     0.05 │   failed at 2026-08-08T15:10:15.526
Retrying 1 failed test  (2)
fails      (8) │     0.00 │   failed at 2026-08-08T15:10:15.731

Test Summary: | Pass  Fail  Total  Time
  Overall     |    1     1      2  3.3s
    passes    |    1            1  0.0s
    fails     |          1      1  0.0s
    FAILURE

Error in testset fails:
Test Failed at REPL[18]:2
  Expression: false

It's mostly working. Only 2 things left to do:

• Figure out the failing test
• Fixup docs to sound less like Claude

[giordano]

A review from the bot:

The unification changes retry semantics in ways the docs don't reflect, and there's one correctness edge worth fixing:

The interrupted guard is now dead, and an interrupt can eat failure records. interrupted is only set in the outer catch, which jumps past the retry block — so at the if retries > 0 && !interrupted check it can never be true. Meanwhile the actual interrupt path inside a worker task calls stop_work() and returns, so nothing propagates to the outer catch: the retry block then runs with done[] == true, does filter!(r -> r.test ∉ retry_tests, results.value) up front, and spawns tasks that hit done[] && return before pushing replacement records — the interrupted run's failures silently vanish from the summary. Guarding the retry block on !done[] (instead of the dead interrupted flag) fixes both problems; alternatively, drop each old record only when its retry actually completes rather than bulk-filtering before the phase.

"Single fresh worker" is no longer guaranteed. The retry phase takes whatever comes out of worker_pool and only spawns fresh if it's nothing or dead. In the common case the pool holds nothings (workers stopped at tests_to_start == 0) so you do get a fresh worker — but with serial tests configured, the live serial worker is returned to the pool and can be handed to the retry phase, so retried tests may run on a used worker, and that worker's sibling nothings mean a second live process can exist during retries. Relatedly, the old loop always stopped the worker after a repeat failure; now that only happens with recycle_on_failure=true, so one persistently-failing retry can contaminate the next retried test in the same round. I'd force fresh-worker + recycle-on-nonpass semantics inside retry phases (it's the slow path; the cost is negligible and it's the documented contract) — or, if the reuse is intentional, rewrite the "deliberately quiesced… single fresh worker… stopped before the next retry" paragraph in advanced.md, whose example output ("passed on retry" / "failed again") is stale for the new format anyway. The "retried test runs alone" test still doesn't cover serial_position=:after, which is exactly the configuration where the invariant now breaks.

Smaller items:

• tests_to_start was initialized to length(tests) and every retry task still does atomic_sub!, driving it negative. Nothing checks == 0 afterwards today, but it silently breaks the counter's invariant for anyone who does later; skip the decrement for retry phases or document why negative is fine.
• retries > 0 && (io_ctx.nonpass_color[] = :yellow) runs even when retries will be skipped (--quickfail, and the interrupt case above), so failures print yellow with no red pass ever coming. Use the same condition as the retry block.
• The color flip is written by the coordinator while the printer task may still be draining queued messages from the previous round, so a late-printed earlier-round failure can pick up the final round's red. Cosmetic, but trivially avoided by flipping the color via a printer-channel message instead of mutating the Ref directly.
• Banner nits: the singular branch of "failed test$(tests_n > 1 ? "s" : " ")" appends a trailing space instead of nothing, and color=:white is near-invisible on light terminals — plain printstyled without a color (or :yellow, matching the scheme) would read better.
• Negative retries still isn't validated.