Skip to content

Wait for the fast rollout before timing gather's partial return - #193

Merged
yichao-liang merged 1 commit into
masterfrom
fix-async-gather-flake
Sep 28, 2026
Merged

yichao-liang merged 1 commit into
masterfrom
fix-async-gather-flake

Conversation

@yichao-liang

@yichao-liang yichao-liang commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

tests/agent_sdk/test_async_rollouts.py::test_gather_timeout_returns_partial failed in a CI-container replay of shard 7: gather([fast, slow], timeout=2) returned with the trivial fast rollout still pending.

The test raced a forked child's start-up against a 2 s timeout.
Under pytest-cov every forked child starts its own coverage tracer, and in a shard with thousands of modules imported on a busy node that start-up alone can pass 2 s.
Measured on a quicktest node with 5,600 modules imported, launch to result took:

  • 0.06 s without coverage,
  • 0.3 s with coverage,
  • 1.0 to 2.9 s with coverage under CPU contention (the first launch took 2.86 s).

The test now waits for the fast child on its own first, then checks that a short gather timeout returns it as done and the slow child as pending without blocking.
The registry is unchanged.

Independent of the #189-#192 and #194 stack; it touches only this test file.

Test plan

  • Reproduced the start-up latency above with coverage on and the CPU contended.
  • Under the same conditions, test_async_rollouts.py passes with the fix (all 9 tests, including the overlap test next to it).
  • pylint, mypy, yapf and docformatter on the file.

🤖 Generated with Claude Code

test_gather_timeout_returns_partial raced a trivial forked child against
a 2 s gather timeout. Under pytest-cov every forked child starts its own
coverage tracer, and in a CI shard with thousands of modules imported on
a busy node that start-up alone can pass 2 s, so the fast rollout was
still pending and the test failed. Measured on a quicktest node with
5,600 modules imported: launch to result took 0.06 s without coverage,
0.3 s with it, and 1.0 to 2.9 s with coverage under CPU contention.

The test now waits for the fast child first, then checks that a short
gather timeout returns it as done and the slow child as pending without
blocking.
@yichao-liang
yichao-liang merged commit e286451 into master Sep 28, 2026
14 checks passed
@yichao-liang
yichao-liang deleted the fix-async-gather-flake branch September 28, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant