Repository navigation
Flaky test: pipeline job control on macOS free-threaded Python (wrapper-stop-and-interrupt-busy-shell-producer) #1791
Description
Summary
tests/test_pipeline_job_control.py::test_pipeline_stops_with_cmd2_and_returns_terminal[wrapper-stop-and-interrupt-busy-shell-producer]
fails intermittently on the tests (macos-latest, 3.15t) CI job. It has failed on branches that don't
touch pipeline or job-control code, so it looks like a pre-existing race in either the test or cmd2's
job-control handling.
Failing runs
All of these failed on macos-latest, 3.15t with the same test variant:
- https://github.com/python-cmd2/cmd2/actions/runs/40803872074 (
fix/window-title-flush-v2, Flush window title updates without waiting for a prompt #1790) - https://github.com/python-cmd2/cmd2/actions/runs/40802879714 (
fix/window-title-flush, Flush window title updates without waiting for a prompt #1783) - https://github.com/python-cmd2/cmd2/actions/runs/40150857988 (
fix/window-title-flush, Flush window title updates without waiting for a prompt #1783) - https://github.com/python-cmd2/cmd2/actions/runs/39753826343 (
fix/history-range-past-end)
A related variant, [orphaned-job-control], failed once on macos-latest, 3.15 (not free-threaded):
https://github.com/python-cmd2/cmd2/actions/runs/40127777165
What fails
This variant uses finish="exit_sigint", stop_job=True, shell_child=True, launcher="sh",
producer="shell" and relay="worker". The test runs busy | bash -c 'python pager.py; :' from cmd2
inside an interactive bash, sends Ctrl-Z, and then waits for the terminal's foreground process group to
return to the outer shell (tests/test_pipeline_job_control.py:335):
send("\x1a")
wait_until(lambda: os.tcgetpgrp(master) == process.pid)This times out on the first Ctrl-Z. From the failure output of run 40803872074:
- The transcript ends with
PAGER_READY,BUSY_READYand thenPAGER_RESUMED. The pager prints
PAGER_RESUMEDfrom itsSIGCONThandler, but the test never sentfg. So the pipeline seems to have
been stopped and then continued, rather than staying stopped. SHELL_OWNS_INPUTnever appears, so the outer shell never got the terminal back.- The foreground process group is still the pipeline's group, and every process is running (
S+), not
stopped (T):
foreground process group: 2567
PID PPID PGID STAT COMMAND
2559 2089 2559 S<s bash --noprofile --norc -i
2560 2559 2560 S< /bin/bash -c .../python .../application.py; :
2561 2560 2560 S< PythonT .../application.py (cmd2)
2567 2561 2567 S<+ /bin/bash -c .../python .../pager.py; :
2568 2567 2567 S<+ PythonT .../pager.py
2570 2561 2567 S<+ PythonT .../busy.py
resizes: []
terminal size at timeout: (24, 80)
Notes for investigation
- In this variant (
relay="worker"), the test application patchessignal.pthread_killso the
job-control relay signal goes to the watcher thread that sends it, rather than to the main thread
blocked in a pipe write or wait. All the free-threaded failures are for this variant, so the race may
be in how cmd2 relaysSIGTSTP/SIGCONTand hands the terminal back when the main thread isn't the one
handling the signal. - It may be a timing problem in the test (for example Ctrl-Z arriving while the pipeline is still being
set up) or a real bug where cmd2 continues the pipeline without stopping itself and returning the
terminal to its parent shell. - It's intermittent: most macOS 3.15t runs pass, including recent runs on
main.
Activity
Wait for the simulated pager's keys in select() with a wakeup fd
Fix two races behind the flaky pipeline job-control test (python-cmd2…