perf(subagents): bound repeated full-tree discovery for missing Codex rollouts

[Juliusolsson05]

Confirmed read amplification at 5d641845 (Node v24.14.1, macOS arm64). Priority: medium/high for large Codex session archives. All evidence uses empty synthetic rollout files, with no private transcript data.

Expected and actual behavior

A child whose rollout is not present yet should have bounded, shared discovery retries. Unrelated parent records should not cause fresh full-history filesystem traversals.

findFileContaining recursively calls readdir and then stat on every entry under the sessions root. readKnownChildren caches only successful paths. A missing child therefore re-walks the entire tree on every 1,200 ms tick and every parent JSONL entry, including unrelated token-count records. Each unresolved child performs its own traversal.

Reproduction and measurements

1. Create a temporary directory named sessions, with 10 child directories containing 100 empty .jsonl files apiece (1,000 total). Ensure no filename contains missing-child.
2. Instantiate the actual CodexSubAgentTracker and call observeParentEntry with a response_item/function_call_output whose output is {"agent_id":"missing-child"}, pointing at a parent path beneath that sessions root.
3. Send zero or nine additional event_msg/token_count parent records immediately.
4. Count node:fs/promises.readdir/stat calls using wrappers that delegate to the real filesystem, await outstanding discovery, then call refresh() three more times sequentially.
5. Stop the temporary tracker and delete only the generated fixture tree.

Scenario	readdir calls	stat calls	Wall time	Process CPU time
One parent record, one missing child	11	1,010	13 ms	15 ms
Ten-record parent burst, same missing child	110	10,100	45 ms	116 ms
Three subsequent unchanged refreshes	33	3,030	not timed	not timed

Times are one warm temporary-filesystem sample under concurrent machine load, not an app benchmark. Operation counts are the primary reproducible evidence. The timer was disabled only in this counting harness so it could not contaminate counts; production's periodic retry adds work. The three sequential retries demonstrate that fixing overlapping-refresh races alone does not eliminate this waste.

Impact and proposed next steps

Main-process filesystem work and callback/parse contention grow with total archived rollout count, missing children, and parent event volume. Delayed child creation or references to unavailable rollouts keep paying this cost while the tracker retains the child id. Exact production CPU share is unmeasured.

• Prefer a shared root-scoped session-id-to-path index, or at least one coalesced directory discovery per retry window serving all unresolved children.
• Cache misses with bounded backoff and prompt invalidation when the matching rollout is created. A miss must not be permanent: freshly spawned children must eventually appear.
• Use directory entries with types to avoid a stat for every ordinary file, while preserving safe handling of symlinks and vanished entries.
• Avoid initiating child discovery for unrelated parent records; retain prompt correlation/status repaint behavior.
• Acceptance: 1,000 unrelated parent records in one retry window cause bounded discovery work, multiple missing children share the scan, and a child created after a miss appears within a documented latency bound.

Tradeoff: negative-cache lifetime exchanges discovery latency for reduced I/O. An event-backed index still needs a recovery path for missed filesystem notifications and rollover. Reuse existing path/index ownership where practical rather than creating another unbounded history cache.

Refs #103, #94. The earlier offset-tail and accumulator fixes remain useful; this is path discovery, not rereading known child transcript bodies.

Related correctness race: #802. Coalescing overlapping reads is necessary there; bounded discovery retries remain needed even after reads are serialized.

[Juliusolsson05]

Implementation is complete in dependent discovery PR #811 (base #809 / fix/subagent-refresh-coalescing). All 39 focused subagent tests, typecheck, testing contract and diff checks pass. The synthetic three-missing-child/1,000-file fixture now requires 11 directory reads and zero per-file stats; 1,000 unrelated parent records and repeated refreshes add no scans inside the retry window.

Acceptance refinement: the bounded shared scan is per tracker, serving all of its unresolved children, rather than a global archive index. Only matched active-child paths are retained. Misses retry after five seconds; discovery latency is at most that window plus the 1.2-second poll and scan duration. This avoids a new archive-sized global retention owner. New child ids do not bypass the budget. Tests cover delayed creation, continued known-child tailing, deletion/replacement, root changes and symlink cycles. Both full CI gates pass at db0f063; no review feedback is currently outstanding; #809 must merge first and this PR then retarget to main. No merge authorization has been given.

Current validation: rebased onto main f7507980 (toolkit #812 and MCP repair #818), head 64b4b105. Source changes from the earlier implementation are preserved; both required CI gates pass on this head (CI run). No merge is authorized.

2026-09-07 merge status: PR #811 now targets main after its parent merged; unchanged patches were rebased to a5539a92 and fresh CI is running. Merge is authorized once green.

Merged with explicit user authorization on 2026-09-07 as 7acc3986 after both current-head CI gates passed. No outstanding review feedback. The resolving issue is closed.