fix(pytorch_xdit): use an inactivity timeout, clean up on failure, verify artifacts

[speriaswamy-amd]

Part 9 of 12 in a stack that replaces #184. Base: #321.

Three defects in the benchmark run step, all confirmed on a live MI300X node.

Wall-clock timeout killed healthy runs

exec_cmd_list(docker_cmds, timeout=1800) is a total read cap that activity does not reset. A legitimate run was killed at exactly 1800s while still inside torch.compile warmup — rocm-smi showed all 8 GPUs at 100% and ~131GB VRAM each, i.e. computing, not hung. Left alone, the same container finished normally ~13 minutes later and produced all 5 steps and video.mp4.

Now inactivity_timeout=1800, which measures silence between output lines: a run that keeps logging is never killed, a genuine hang still fails within 30 minutes. Kept at 1800 because the silent gap between shard loading and the first compile log ranged from ~5 to 15+ minutes across nodes; 300s and 900s both produced false kills.

Timeout orphaned a GPU-holding container

When the call raised, the test failed but the container it started kept running — docker ps showed it Up and holding all 8 GPUs indefinitely. test_cleanup_stale_containers only runs at the start of the next session, so on a shared node every timeout leaks a container. The except branch now calls docker_lib.kill_docker_container before failing.

A clean log was treated as success

The run step only scanned container output for error strings; a run that logged nothing alarming but wrote no artifacts passed here and failed confusingly in the parse step. Both benchmarks now verify the expected artifacts exist and are non-empty (timing.json + a flux_*.png; a rank0_step*.json + video.mp4) on every node.

LocalPssh gains a matching inactivity path so single-node local runs behave the same as SSH runs.