Conversation
| ); | ||
| if !execution.success { | ||
| let setup_hint = if phase == "baseline" { | ||
| " A fresh parallel worktree has no existing build directory; use \ |
There was a problem hiding this comment.
Every baseline failure returns
this error
*** No errors detected
Traceback (most recent call last):
File "/tmp/bcore-mutation-workers-KSJzCG/worker-0/./build/test/functional/test_runner.py", line 962, in <module>
main()
File "/tmp/bcore-mutation-workers-KSJzCG/worker-0/./build/test/functional/test_runner.py", line 464, in main
os.makedirs(tmpdir)
File "/home/ubuntu/.pyenv/versions/3.10.14/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/test_runner_₿_🏃_20260817_212053'
Error: InvalidInput("baseline command failed in worker 0. A fresh parallel worktree has no existing build directory; use --setup-command if the test command does not configure it.")
analysis command had --setup-command, the error message is misleading
bcore-mutation analyze --sqlite mutation.db --run-id=2 --parallel 2 --file-path="src/net_processing.cpp" --setup-command 'cmake -B build && cmake --build build -j4 --target bitcoind test_bitcoin' -c 'cmake --build build -j4 --target bitcoind test_bitcoin && ./build/bin/test_bitcoin --run_test=denialofservice_tests,peerman_tests,net_tests,txdownload_tests && ./build/test/functional/test_runner.py --failfast p2p_handshake.py p2p_leak.py p2p_sendtxrcncl.py p2p_tx_download.py p2p_blocksonly.py p2p_addrv2_relay.py p2p_node_network_limited.py'
the actual error was from test_runner , got the error after the analysis had run for about 50 minutes
|
Not sure if i am testing this right But i generated mutants for verack message and then started analysis with this command Observation both workers were visibly running I could see them in the process list and the build directories growing. Then, roughly 45 minutes in,output stopped and never resumed. From the terminal I had no way to tell whether the build was still progressing, the baseline was running, a mutant was executing normally, or something had deadlocked. All four states look identical: no output. While looking at /proc from a second shell the two in-flight mutants had:
I think it makes sense to me now that i would experience more timeouts I stopped the run at ~100 minutes with zero mutant results recorded. Maybe we can add a periodic [worker N] mutant 53, 12m elapsed i am still testing using verack I’ll see how long this it takes probably more than 4 hours |
|
This time, I only had 2 mutants. |
Parallel workers get an isolated Git worktree, but their commands inherited the parent process environment, so every worker shared one system temporary directory and one functional test port range. Bitcoin Core's test_runner.py names its scratch directory after a timestamp with second granularity and creates it without exist_ok. Workers start together and do identical work, so they reach that line within the same second and all but one fail with FileExistsError. Functional test ports are derived from the test index rather than the process, so concurrent workers running the same test list would then bind the same ports. Give each worker a private temporary directory (TMPDIR, TMP, TEMP) and its own TEST_RUNNER_PORT_MIN. A runner spans three port ranges, so only three workers fit below the maximum port number; higher --parallel values now warn instead of silently sharing ports. Baseline failures also suggested --setup-command even when one was provided, hiding the real error. The hint is now conditional, and the message quotes the tail of the failing command's output.
A worker printed nothing until its command finished, so a 45 minute build, a running baseline, a working mutant and a stalled command all looked the same from the terminal. Commands now stream their output instead of buffering it, and report every 60 seconds how long they have been running along with their most recent line of output. Streaming also fixes a stall. Waiting for a buffered capture waits for the output pipes to close, not just for the shell to exit, so any background process that outlives the command kept the worker blocked with no CPU use and nothing printed until the timeout expired. The pipes are now drained with a grace period, after which the lingering processes are terminated. Commands run in their own process group, and a timeout terminates the whole group rather than only the shell. A leftover build or bitcoind node would otherwise keep holding ports and datadirs, and turn one timeout into timeouts for every mutant that followed.
|
Thanks, @naiyoma. I pushed two commits to fix some things you noticed. One of the issues is that parallel workers get an isolated Git worktree, but their commands inherited the parent process environment, so every worker shared one system temporary directory and one functional test port range. Also, changed it as commands now stream their output. |
Any reason to use 1200 seconds as timeout? |
2 participants
No description provided.