Let a script finish its teardown when the client goes away

[juul-charles-w]

Stacked on #215

No upstream write access, so this PR's base is main and its diff includes
#215's commit. Review the second commit only, or merge #215 first and this
will shrink to it. GitHub will show the right thing once #215 lands.

Summary

Three coupled changes to what happens after a lager python client vanishes —
cancelled CI job, killed CLI, dropped network. This is the only cleanup path
that survives the client being hard-killed, since it runs on the box and needs
nothing from the client, so it is the backstop for every executor.

Notice the client is gone. The streaming generator only checked for a dead
peer when the script next wrote something, so a quiet script kept the bench for
as long as it stayed quiet — 12.8s in the worst measured case. It now yields a
zero-length idle tick on every idle poll, and peer_is_connected reads the
socket to distinguish a closed connection from an idle one. Detection is
sub-second.

Interrupt before terminating. terminate_process went straight to SIGTERM,
whose default disposition kills the interpreter outright: no finally, no
context-manager __exit__, no atexit. For a HIL script that is the difference
between the bench being left idle and left live, because de-energising rails and
releasing the DUT live in exactly those blocks. It now escalates
SIGINT → SIGTERM → SIGKILL.

Make the next job wait. Cleanup runs after the client has gone, so the box
lock says nothing about whether the hardware is free — the client releases it on
the way out, and it lapses on its own when the client is killed. A job reaping a
bench publishes its PIDs for the duration (lager.exec.quiesce) and a starting
job waits for that to clear.

These three land together on purpose: without the gate, extending the cleanup
window would have widened the overlap it creates rather than closing it.

The cleanup budget is an idle budget

A fixed total cannot be set correctly from here — it is a guess about how long
someone else's teardown takes, and every value is wrong for someone. Ours is
~2.4s; a user with six LabJacks and three hubs could need ten times that, and a
number large enough for them means a wedged script holds a shared box that long.

wait_for_cleanup escalates only after the script has gone CLEANUP_GRACE_S
without observable progress, so a working teardown keeps its deadline pushed
out while a wedged one is still cut off promptly. CLEANUP_MAX_S caps the total
either way. This only has to exceed the longest pause within a teardown, which
is a far more stable quantity than its duration.

Why progress is context switches, not just CPU time

CPU time alone was the wrong signal, and this is the part worth scrutinising.
Hardware teardown is mostly blocking. Measured on hardware, a single Acroname
hub round trip takes ~2.2s and accrues about one 10ms clock tick over that
whole time, so sampling ticks reads a working teardown as idle:

signal	longest quiet stretch (6 runs)
CPU time only	2.44s, 3.15s, 3.36s, 3.36s, 3.66s, 4.17s
+ context switches	1.02s, 1.02s, 1.02s, 1.02s, 1.12s, 1.12s

Against the 3s budget that shipped, an end-to-end A/B on hardware put that
combination at 3 of 11 interrupted teardowns silently truncated mid-cleanup,
leaving the bench partly un-restored. With context switches added and the budget
at 5.0s (~4x the measured worst case), 11 of 11 completed.

A process wedged in a single uninterruptible syscall moves neither counter,
which is exactly the case the watchdog exists to catch — so this widens what
counts as progress without making a stuck process look busy. There is a test for
that, verified on Linux. /proc/<pid>/schedstat was measured as an alternative
and resolves no better (same underlying transaction boundaries), so this uses
status, which needs no CONFIG_SCHEDSTATS.

Test plan

• test_cleanup_watchdog.py, test_bench_quiesce.py,
  test_stream_disconnect.py and updates to test_stream_teardown.py — 42
  tests, 5 procfs-gated (they run on CI).
• Full test/unit/box test/unit/cli run: no new failures (the 5
  test_ssh_setup.py ones are a pre-existing local click version issue).
• Hardware A/B on a real box, 11 trials per arm, numbers above.
• Test box restored to released code afterwards and verified by md5.

Made with Cursor