Let a scenario declare that one attempt is not enough for it

[leggetter]

Refs #47 — the second of its three open scenarios, and the one its "done when" already names a fix for.

The problem

outpost-003 publishes a coin flip on the weak model. Across three full passes:

	observations	result
frontier models	6	stable
weak model, both arms	2 x 3	split 2-1

Passing turns on which undocumented route the agent happens to guess — the operator-events API is absent from both OpenAPI definitions and from the docs (#34). The weekly schedule runs --runs 1, so one guess decides a published cell against a named vendor.

#47 asks for one of two things: multiple attempts, or hold the scenario back until #34 makes its route findable. This is the first, and it does not require withdrawing a scenario that genuinely discriminates at the bottom of the range.

The change

min_attempts in a scenario's frontmatter is a floor on --runs for that scenario alone, capped at five.

• A floor, never a ceiling. --runs 3 against min_attempts: 2 still runs three.
• Paid for only where it was already failing, because stop-on-pass means a retry costs nothing on a cell that passes first time. For this scenario that is the weak model and nothing else.
• The row says what it spent. attempts is already exported, so best-of-N is visible rather than implied.

Worth being plain about what it buys: not soundness. The published number becomes best-of-N, which is a different claim. AGENTS.md now says to use this where variance is measured rather than suspected, and records why this scenario qualifies.

Tests

Five on the frontmatter: both key spellings (min_attempts and minAttempts, as every other key here accepts), absence leaving --runs alone, and the two shapes — 0, 1.5, 30 — that would otherwise be discovered by the bill.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK