Skip to content

Make local check-structure runs as reliable as CI (swarm 0.16.1, work-system 1.29.2) - #96

Merged
gering merged 4 commits into
mainfrom
task/stabilize-local-check-structure
Nov 1, 2026
Merged

gering merged 4 commits into
mainfrom
task/stabilize-local-check-structure

Conversation

@gering

@gering gering commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Why

Local python3 scripts/check-structure.py went red while CI stayed green: per-test timeouts (F7) and test_codex_models / test_kimi_acp failures under load (F8).

What the measurements showed

Per-test wall time measured alone, with 3 full runs in parallel, and with 12 parallel runs of a single test:

  • macOS assesses every new executable on its first exec (syspolicyd, ~150 ms; a symlink to an already-run file costs ~4 ms). Parallel runs pile up on it. Tests that wrote a fresh stub per case or per Env spent most of their time waiting on it: test_kimi_acp took 67 s wall for 12 s CPU, and 419–590 s in parallel. test_agent_registry took 444 s in parallel. Linux CI has no such check. The mechanism is confirmed by research; the pile-up under load is inferred from the measurements.
  • test_codex_models gave the helper deadlines of 0.4–2 s, and a working fake peer had to start inside them. It failed in 12 of 12 parallel runs.
  • test_sessions waited a fixed 1 s per pane poll in sessions.py.
  • test_insights_handoff looked for leftover temp files in the shared $TMPDIR and saw files from parallel runs.

Changes

  • swarm tests: one shared fake per class or module (kimi, grok-compat). The codex test now separates the deadline under test from the time a working peer gets, and asserts that rejected cases were not merely timeouts. The kimi fake now waits up to 10 s, not 1 s, for the client's kill.
  • work-system: SESSIONS_POLL_MS test seam; content-addressed stub links in test_agent_registry and test_herdr_shell; a private TMPDIR for the leak check.
  • check-structure.py: a TEST_TIMEOUTS table. Only test_mandate (75 s) gets its own 300 s; the 120 s default stays. A stale entry is an error.
  • Knowledge note: deployment/ci-structure-checks.md.

Proof

  • A full run with 3 in parallel: 1360 s → 391 s. kimi 419 → 13.5 s, agent_registry 444 → 22 s, sessions 80 → 29 s, herdr_shell 67 → 31 s.
  • test_codex_models, 12 runs in parallel: 0/12 → 12/12 green.
  • Final head: 3 parallel and 3 sequential check-structure.py runs, all OK.

Review

One local /swarm:review round (Claude lenses, gpt-6.1-sol, grok-4.7; 15 of 15 voices) found 9 issues. 6 are fixed in this PR:

  • the shared fake directories now clean up after themselves (grok-compat, insights-handoff TMPDIR);
  • the stty and raw_grok stubs read their paths from env, so the stub cache now actually hits;
  • HANG_BOUND is 8 s, below the helper's 10 s default, so it still catches a helper that ignores --timeout;
  • SESSIONS_POLL_MS can only shorten the poll, never lengthen it;
  • the knowledge page's pluginVersion stamp is updated.

3 were not acted on:

  • one shared stub helper across test files: tests stay self-contained by convention;
  • the 5 s descendants deadline: 12 of 12 runs passed under 12x load, so this is an accepted residual;
  • the leak check only sees the private TMPDIR: the old check was no stronger.

check-structure.py passes on the fixed head.

Not done

  • The test_kimi_acp 1 s kill race never reproduced in 12 parallel runs. The 10 s bound comes from reading the code and costs nothing on the passing path.
  • test_herdr_launch is out of scope (its own lane).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WFDBrBNHxBX8MUG6oAC8n6

@gering
gering force-pushed the task/stabilize-local-check-structure branch 3 times, most recently from b6cd3a1 to c5ebfaa Compare October 31, 2026 11:14
@gering gering changed the title Make local check-structure runs as reliable as CI (swarm 0.16.1, work-system 1.28.3) Make local check-structure runs as reliable as CI (swarm 0.16.1, work-system 1.29.2) Oct 31, 2026
gering and others added 4 commits October 31, 2026 12:23
- test_codex_models: separate the deadline under test from the time a
  working peer gets. Only deadline cases run with a short bound; success
  and invalid-response cases get the helper's maximum and must not end
  by "timed out". 12 parallel runs: 0/12 green before, 12/12 after.
- test_kimi_acp: write the fake kimi once per class instead of once per
  helper run. macOS assesses every new executable on first exec
  (~150 ms, serialized), which was 80% of the test's wall time: 67 s ->
  13 s alone, 419-590 s under parallel load before.
- test_kimi_acp: the fake waits up to 10 s, not 1 s, for the client's
  kill before it carries on to end_turn, so a slow kill under load
  cannot pass for a client that waited.
- test_grok_compat: link one shared fake grok into each Env (21 s -> 13 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFDBrBNHxBX8MUG6oAC8n6
- sessions.py: SESSIONS_POLL_MS test seam for the pause between
  transcript looks while a reload is verified (default stays 1 s).
  test_sessions sets 100 ms: 70 s -> 33 s.
- test_agent_registry, test_herdr_shell: write each distinct stub once
  and link it into every Env. macOS assesses every new executable on its
  first exec (~150 ms, serialized across processes); ~190 fresh stubs
  were most of agent_registry's wall time (35 s -> 21 s alone, 444 s
  under parallel load before). herdr_shell: 53 s -> 30 s.
- test_insights_handoff: the temp-file leak check gets its own TMPDIR;
  in the shared one a parallel run's live files read as leftovers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFDBrBNHxBX8MUG6oAC8n6
The 120 s per-test bound is a hang guard, but on a busy machine a
long test crossed it while CI stayed green. TEST_TIMEOUTS gives each
measured long test its own bound and keeps the default tight; an
entry naming a test that no longer exists is an error, so the table
cannot rot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFDBrBNHxBX8MUG6oAC8n6
@gering
gering force-pushed the task/stabilize-local-check-structure branch from c5ebfaa to 25ae1ae Compare October 31, 2026 11:30
@gering
gering merged commit b953370 into main Nov 1, 2026
2 checks passed
@gering
gering deleted the task/stabilize-local-check-structure branch November 1, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant