[roodboi]

Refs HACK-1419

Summary

This PR does not fix a root cause. The hosted evidence cannot tell a readiness race from a real routing defect, so it is not safe to change product code or relax the assertion. What it does is make the next failure conclusive: the log will name the URL, the phase, and the route state.

What failed

On #463, run 40057262783, job 120762873007 (docker-e2e shard 1/4), native-config-networks failed at the mixed routed ingress check:

Network command diagnostic {"step":"hack:exec","role":"primary","sequence":925,"expected":0,"exit":27,"timedOut":false,"durationMs":12872,...}
FAIL native-config-networks (1391.6s) — Fixture command must exit 0; captures retained

• What exit 27 means. It is not curl's code 27. It is the fixture's own NATIVE_NETWORK_PROBE_EXIT.timeout: the in-container peer probe (bun -e, run through hack exec <service>) hit its 4 s AbortSignal.timeout while fetching http://<target>:3000 or http://<target>-alias:3000/.
• Where it failed. The failure came 70 s after "checking mixed routed ingress". At that point the census shows all 4 primary services running, plus the proxy. So the failing exec is in markers(primary) right after the routed hack up. In that phase web is attached to the external hack-dev ingress as well as its authored outbound bridge.
• Not a service-readiness race. observe(primary) runs just before markers, and it requires every container to be healthy. The healthcheck is an in-container fetch of 127.0.0.1:3000, so every server was already listening. A server that was not listening yet would refuse the connection and exit 26 at once, not stall until timeout.
• What the log cannot show. It does not say which of the four service pairs failed, which URL (origin or alias), or whether name resolution or the HTTP exchange stalled. Hosted captures stay on the runner, and the run uploaded no artifact.

Frequency

I checked the docker-e2e shard jobs of the last 60 ci.yml runs on next and PR branches: 228 jobs, 215 green, 10 in progress, 3 failed. The 3 failures:

• This one: exit 27 at the routed ingress check. It is the only occurrence.
• Run 39916341281 (next), shard 1: a different native-config-networks failure, sibling-b hack exec with E_COMPOSE_FAILED storage-helper-start/probe-timeout. That is the storage helper budget, not ingress.
• Run 40055366376 (next), shard 4: native-compose-adoption-job-worktrees, which is unrelated.

So the exit 27 has appeared once in about 51 shard-1 runs.

Candidate causes (none confirmed)

1. Name resolution stalled. A lost UDP reply from Docker's embedded DNS (127.0.0.11 is DNAT'd in the container netns) means the resolver waits about 5 s before retrying. That is longer than the probe's 4 s budget.
2. A misroute specific to multi-homed web. Here the name answers an address that is not on a bridge the two containers share.
3. A transient connect stall on a fresh attachment.

Each of these produces exit 27 today, and they cannot be told apart from the log.

Change (test fixture only)

• nativeNetworkFixturePeerProbe resolves first, then fetches. It resolves the name (Bun.dns.lookup, IPv4 preferred) and then fetches the resolved address. Resolution and the HTTP exchange share the same 4 s per-URL budget (NATIVE_NETWORK_PEER_PROBE_BUDGET_MS), so the assertion is not loosened. Resolution failures get their own exit codes, dnsRejected 30 and dnsTimeout 31. Codes 26 and 27 now mean the HTTP phase only.
• The probe writes one closed stderr line before a failing exit: probe-evidence <origin|alias> <dns|http> <dnsMs> <elapsedMs> <addressCount>. It never includes an address or body.
• On a failed peer probe the scenario logs Network probe diagnostic {...}. The line contains:
  • the role, the service and target (fixed fixture names), the exit code, outcome and parsed evidence;
  • the route state, reduced to counts and booleans: both containers' state and health, attachment counts, shared bridges, whether the source is on the ingress, and how many addresses getent ahostsv4 <probed name> returned inside the source container, plus whether every one is the target's address on a shared bridge;
  • the outcome of one diagnostic-only rerun, which shows whether the stall is transient or persistent. The step still fails either way.
• Failures in the diagnostic path cannot hide the original failure. Any error there logs Network probe diagnostic unavailable.

How to read the next occurrence

• phase:"dns" with a passing rerun: a resolver stall. Consider a resolver-aware budget.
• resolvesToSharedTarget:false: a real routing or DNS defect. Fix it in the product, with a unit test.
• phase:"http", a shared target address, and a failing rerun: a persistent connectivity defect.

Validation

• Unit tests: bun test tests/native-networks-fixture.test.ts gives 63 pass. There are 4 new tests: probe phase and URL attribution, the evidence parser, route-state reduction, and the closed probe diagnostic. The existing probe exit-code test still passes.
• Mutation proof: each of these 8 mutations kills the tests: DNS phase mislabelled, wrong DNS rejection code, IPv4 preference dropped, budget not shared, route every changed to some, evidence regex unanchored, rerun dropped, ingress ignored.
• Gates: bun run check (includes privacy:check) passes. bun run --cwd packages/cli typecheck and bun run typecheck report 0 errors. Biome is clean on both changed files. gitleaks 8.30.1 finds no leaks over origin/next..HEAD.
• Docker: nothing ran against a local Docker socket. Hosted docker-e2e shard 1 runs native-config-networks and exercises the new probe in real containers.

Gaps

• Root cause still unknown. This PR closes only the diagnosis gap. HACK-1419's acceptance ("cause identified") needs the next occurrence, which will now be conclusive, or a deliberate reproduction on an isolated Docker host.
• The diagnostic path has not run against real Docker. It runs only when a probe fails, and its hosted coverage is limited to the success path. Its pure helpers are unit-tested.

Hosted CI at 3e88132

• Docker e2e: all four docker-e2e shards pass, as does the docker-e2e aggregate. In shard 1 (run 40060640525, job 120773164774), PASS native-config-networks (2288.2s), including the mixed routed ingress phase, with the new probe running in real containers.
• Other checks: test, secret-scan, linux-process-lifetime, runtime-images, Runtime state models, and Config compiler (ubuntu, arm, macos-latest) pass. Candidate core and Config compiler on macOS were still running when this was written.
• Runtime: the scenario took 2288 s here, against about 1290–1330 s on recent green runs. That runner was slow before the first probe ever ran: the probe-free created-before-start phase took 44 s against 15 s, and bun install took 47 s against 36 s. I attribute the difference to the runner, not the probe, but one run cannot rule out a small per-probe cost.

🤖 Generated with Claude Code

[linear-code]

HACK-1419