consult: a fast-failing model lane is recorded as SUCCESS — panels silently thin (3-way ran as 2-way for weeks, stats inflated)

[amrmelsayed]

Symptom

In the shannon workspace, porch's 3-way consult ran as 2-way (gemini absent, no output file) on at least 3 consecutive PIRs (shannon pir-3698/3699/3790 — receipt: the consult tmp dir holds 1 codex + 1 claude + 0 gemini files). Nothing failed loudly; the architect only noticed by reading the panel outputs.

Root cause (verified by direct probe)

echo 'Reply with exactly: LANE-OK' | gemini →

When using Gemini API, you must specify the GEMINI_API_KEY environment variable.

The env consult runs under lacks the key, so the gemini CLI errors immediately. The operator-side fix is trivial (set the key). The library bug is what happened next:

The bug

consult stats reports the gemini lane as 92% success, 809 calls, avg 8s over 30 days — the 8s average IS the instant auth-error exit being recorded as success. Consequences:

1. Panels silently thin — a protocol that specifies 3-way review runs 2-way indefinitely with no signal at any gate.
2. The health metric is inverted — the broken lane looks like the healthiest (fastest, 92%).

Ask

• A lane invocation that produces no output artifact (or exits with a recognizable CLI error) should be recorded as failure/skip, never success.
• Porch/consult should surface lane-degradation loudly at the consult step (e.g. 'panel ran 2/3 — gemini: GEMINI_API_KEY missing'), so a thinner-than-specified panel is a visible disposition, not a silent default.

Related but distinct: #1051 (gemini/agy sandbox behavior).