diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 3e49899..a2ffb87 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1,6 +1,6 @@ --- description: Benchmark one LLM on the llms-benchmark demo stack - drive it through a coding-agent CLI (opencode, claude or copilot) on the stored scenario, grade the observation report it produced, and propose its row of the results table -argument-hint: " " +argument-hint: " [effort, default medium]" --- Run the whole llms-benchmark protocol for one model on one CLI, end to @@ -20,9 +20,14 @@ the same way you would grade a colleague's incident report. `opencode`, `claude` or `copilot`; the **model** to benchmark, as its canonical `vendor/name` id, the OpenRouter form (`anthropic/claude-sonnet-5`, `openai/gpt-5-mini`, - `google/gemini-3.5-flash-lite`, ...). Those are the only two inputs. - Ask for whichever is missing and stop until you have both. Model and - CLI identify the row: the same model on two CLIs is two rows. + `google/gemini-3.5-flash-lite`, ...); optionally, the **effort** the + CLI runs the model at — `low`, `medium`, `high`, or any other level + the CLI's effort flag accepts — **`medium` when omitted**, the level + every row was measured at before this argument existed. Those are the + only three inputs. Ask for the CLI or the model when missing and stop + until you have both; never ask for the effort. Model, effort, CLI and + provider identify the row: the same model on two CLIs, at two efforts, + or served by two providers, is two rows. - The model id is written the same way whatever the CLI, so the two rows of one model line up. Each CLI is handed its own form of it: `opencode` takes it as `openrouter/`; `claude` takes Anthropic @@ -32,8 +37,9 @@ the same way you would grade a colleague's incident report. dashes; `claude --help` on `--model` names the accepted forms); `copilot` takes the bare name its model picker lists — `openai/gpt-5.6-luna` is `gpt-5.6-luna` (the vendor prefix dropped, - nothing else changed). A model the CLI cannot run is a preflight - failure, not a row. + nothing else changed). A model the CLI cannot run, or cannot run at + the requested effort, is a preflight failure, not a row. Below, + `` is that argument, passed verbatim to the CLI's flag. **Never ask for an API key, and never handle one.** Every credential this protocol needs — the OpenRouter provider in opencode, the Claude Code @@ -67,8 +73,18 @@ Steps: were given - or, when the listing lags OpenRouter's catalog (it did not carry `z-ai/glm-5.3-flashx` on 2026-09-19 while the model ran), a smoke run answers with a `text` event: - `opencode run --model openrouter/ --format json "reply with the single word ok" < /dev/null` - (the package is installed in step 3); + `opencode run --model openrouter/ --variant --format json "reply with the single word ok" < /dev/null` + (the package is installed in step 3) - **and the model has a + variant at the requested effort**: `opencode run` accepts any + `--variant` name, an unknown one included, records it on every + message and sends no effort at all (verified on 2026-09-25 with + `--variant bogus`). Read the model's `variants` from + `opencode models openrouter --verbose`; when `` is not + among them, launch without `--variant` and write `default` in the + Effort column - the provider's default effort is what the model + ran at (the `z-ai/glm-5.3*` models and + `deepseek/deepseek-v4.1-flash` offer `low`, `high` and `max` only, + so every row of theirs measured at "medium" ran at `default`); - `claude`: `claude --version` answers; the package is installed at **user scope** for Claude Code — `~/.claude/commands/odd-observe.md`, `~/.claude/agents/observe-run.md`, `~/.claude/skills/ --output-format json < /dev/null` + `claude -p "Reply with the single word ok" --model --effort --output-format json < /dev/null` must print a `type: result` JSON whose `modelUsage` carries the model's canonical id. Since Claude Code 2.1.270 a second key, `claude-haiku-4-5`, sits beside it on every run — a background @@ -95,7 +111,7 @@ Steps: (`~/.copilot/config.json` carries a non-empty `loggedInUsers` — the host and login, nothing else lives there); and a smoke run answers with a usage file naming the model: - `copilot -p "Reply with the single word ok" --model --allow-all-tools --usage-output-file /usage.json < /dev/null` + `copilot -p "Reply with the single word ok" --model --effort --allow-all-tools --usage-output-file /usage.json < /dev/null` must leave a `usage.json` whose `modelMetrics` has one key, the model's name. Run it from a scratch directory too — it leaves a session under `~/.copilot/session-state/`. Nothing is installed @@ -107,7 +123,7 @@ Steps: presence, never its value, and never print it. The file is gitignored; `.env.example` next to it says what goes in. -2. **Create the work branch**: `bench/--`, +2. **Create the work branch**: `bench/---`, where `` is the model id with `/` and `.` replaced by `-`. Everything the run installs, configures, and produces happens on this branch, and none of it is what ships. @@ -182,10 +198,11 @@ Steps: 4. **Select the model.** Nothing to configure: the provider is already set up (preflight), and the model and effort are passed on the command line in step 6, never persisted into a config file — `opencode`: - `--model openrouter/ --variant medium`; `claude`: - `--model --effort medium`; `copilot`: - `--model --effort medium`. The three flags name the same - effort level; that is what makes two rows of one model comparable. + `--model openrouter/ --variant `; `claude`: + `--model --effort `; `copilot`: + `--model --effort `. The three flags name the same + effort level; that is what makes two rows of one model at one effort + comparable across CLIs. 5. **Clean what the next run must not read — then recreate the demo stack, never reuse a running one.** Before every run, whatever the @@ -276,7 +293,7 @@ Steps: `opencode`: ``` - caffeinate -i opencode run --model openrouter/ --variant medium \ + caffeinate -i opencode run --model openrouter/ --variant \ --format json --auto --title "llms-benchmark " \ "" < /dev/null ``` @@ -289,7 +306,7 @@ Steps: caffeinate -i env -u CLAUDECODE -u CLAUDE_CODE_CHILD_SESSION -u CLAUDE_CODE_SESSION_ID \ -u CLAUDE_CODE_MESSAGING_SOCKET -u CLAUDE_CODE_MESSAGING_TOKEN -u CLAUDE_PID \ CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 \ - claude -p "" --model --effort medium \ + claude -p "" --model --effort \ --permission-mode bypassPermissions --output-format json \ --session-id "$SID" < /dev/null > /run.json 2> /run.err ``` @@ -298,12 +315,21 @@ Steps: ``` SID=$(uuidgen | tr 'A-Z' 'a-z') - caffeinate -i copilot -p "" --model --effort medium \ + caffeinate -i env COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200 \ + copilot -p "" --model --effort \ --allow-all --no-ask-user --additional-mcp-config @.github/mcp.json \ --session-id "$SID" --output-format json --usage-output-file /usage.json \ < /dev/null > /run.jsonl 2> /run.err ``` + `COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200` lifts prompt mode's 600 s + wait on background tasks: a root that dispatches `observe-run` with + `mode: background` and ends its turn has its subagent cancelled + 600 s later, report unwritten (`session.warning` + `background_task_wait_timeout`, `subagent.completed` with + `cancelled: true`) - the first `openai/gpt-6-luna` run of + 2026-09-24 lost its whole observation to it, ten minutes in. That + run is void, not a row. `--allow-all` is this CLI's headless auto mode (tools, paths and URLs); `--no-ask-user` removes the tool a run would otherwise use to ask a question nobody answers; `--additional-mcp-config @.github/mcp.json` @@ -933,8 +959,9 @@ Steps: resets it before launching. - **delete the run's scratch directory under the system temp dir** (`$TMPDIR/opencode/`, and `/tmp/llmbench-*`, `/tmp/oddyssey-scratch/`, - `/tmp/oddyssey-scratchpad/`, `/private/tmp/odd-scratch/` or - `$TMPDIR/oddyssey/scratch/` for a `claude` run, `$TMPDIR/oddyssey/`, + `/tmp/oddyssey-scratchpad/`, `/private/tmp/odd-scratch/`, + `/tmp/odd-observe-scratch/` or `$TMPDIR/oddyssey/scratch/` for a + `claude` run, `$TMPDIR/oddyssey/`, `/tmp/oddyssey/` or `/tmp/oddyssey-observe/` for a `copilot` run — one run of that CLI wrote under each — and **`.odd/scratch/` inside the repository**, where one `copilot` run of 2026-09-14 put its @@ -985,49 +1012,115 @@ Steps: create one naming the model and the protocol revision, then the PR that closes it. This is a step, not a fallback. - From `main`, freshly pulled, create - `docs/llms-benchmark--` and make **one** change: + `docs/llms-benchmark---` and make **one** change: the row in the results tables of `.llms-benchmark/README.md`. `## Results` holds the two tables below. **A row is identified by - model and CLI together.** The pair is not in the table yet → - append the row; already there → replace that row in place. The - same model driven through two CLIs is two rows - (`google/gemini-3.7-flash` under `opencode` and under another CLI - both appear); the oddyssey version is not part of the key — a new - run of the same model and CLI overwrites the row, whatever version - the old one carried. The table carries no history: one row per - model and CLI, always the latest run. - - **Two tables, not one.** Seventeen columns scroll the model name off + model, effort, CLI and provider together.** That key is not in the + table yet → append the row; already there → replace that row in + place. The same model driven through two CLIs, at two efforts, or + served by two providers, is two rows (`openai/gpt-6-luna` at + `medium` and at `high` both appear); the oddyssey version is not + part of the key — a new run of the same model, effort, CLI and + provider overwrites the row, whatever version the old one carried. + The table carries no history: one row per model, effort, CLI and + provider, always the latest run. + + **Two tables, not one.** Twenty-three columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of nine columns — rank, model, CLI, oddyssey - version, `confirmed / reported`, the findings by kind under a - single `Telemetry / Perf / Behavior` header written `X / X / X`, - total duration, cost, and cost per confirmed finding. It fits - without scrolling and answers the question on its own. The CLI - column names the coding-agent CLI the mission ran in — the `` - argument, `opencode`, `claude` or `copilot`, with no version: the version - belongs in the pull request, where the row's exact figures already - live. The oddyssey version sits right after it because it says - which protocol a row was taken under, which a reader needs before - any number to its right means anything; - - a **detail table** inside a `
` block — model, CLI, - oddyssey version, the three phase durations, turns, median turn - latency, input / output / cache tokens, and signals. Round the - token counts (`30.0M`, `79k`): the - exact figures live in each run's pull request, and full precision - here only costs width. - - **The rank is decided with the user, not computed.** It weighs three - axes together — findings, cost and duration — and none of them alone - survives as a rule: ranking on findings would put a 67-minute run - first, on duration would reward whichever model gives up soonest, on - cost would reward the one that barely looks. Propose a placement in - the PR and argue it on the three axes; adding or updating a model - **re-sorts the whole table**, it never just inserts a line. A row - measured under an earlier revision of the protocol is marked as such - and its placement is provisional until it is re-run. + - a **headline table**, fourteen columns, exactly these headers: + + ```text + | Rank | Model | Effort | CLI | Provider | oddyssey | Scoring | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | + | **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 64.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $N.NN | 85% | $N.NNN | 94s | + ``` + + (the two money cells are written here as patterns, never as a + dollar sign followed by a digit - see the end of step 10; in the + README they carry the figures: a dollar sign, then 0.33 and 0.030) + + It fits without scrolling and answers the question on its own. + Each cell, left to right: + - **Rank** `**#N**`, the position by Scoring, renumbered from 1 + after every re-sort; + - **Model** the canonical id in backticks; + - **Effort** the level the CLI actually applied - the `` + argument (`medium` by default), or `default` when the CLI has no + variant at that level for the model (step 1's opencode check); + - **CLI** `opencode`, `claude` or `copilot`, no version - the + version belongs in the pull request, where the row's exact + figures already live; + - **Provider** who served the model to that CLI: `OpenRouter` for + opencode, `Anthropic` for claude, `Copilot` for copilot; + - **oddyssey** the version the row was taken under (step 7), right + after them because a reader needs the protocol before any number + to its right means anything; ` ⚠` after it marks a row measured + under an earlier revision of the protocol; + - **Scoring** the row's score out of 100, one decimal (below); + - **Confirmed / reported** `X / Y` from step 8; + - **Telemetry / Perf / Behavior** the confirmed findings by kind, + `X / X / X`, summing to X; + - **Total** the run's wall clock, `NmSSs`; + - **Cost** step 7's figure, `$N.NN`; + - **Accuracy** confirmed over reported, rounded to a whole + percentage, `NN%`; + - **$/confirmed** Cost over confirmed, `$N.NNN`; + - **seconds/confirmed** Total in seconds over confirmed, rounded to + a whole second, `NNs`. + + **Bold**: Confirmed / reported on every perfect ratio (`**6 / 6**`); + in Scoring, Total, Cost, Accuracy, $/confirmed and seconds/confirmed the + best value of the table only, on every row that ties it - the + highest, the shortest, the cheapest, the highest, the cheapest, the + fastest - + re-checked after each re-sort, since a new row can take it from + another; + - a **detail table** inside a `
` block, exactly these + headers, the first five cells as in the headline row: + + ```text + | Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | + | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | + ``` + + the three phase durations, turns, median turn latency, input / + output / cache tokens and signals from step 7. Round the token + counts (`30.0M`, `79k`): the exact figures live in each run's pull + request, and full precision here only costs width. The detail + table lists the rows in the headline table's order. + + **The rank is computed: the table is sorted by Scoring, highest + first, a tie going to the cheaper run.** Adding or updating a model + recomputes nothing but its own score - every bound is fixed - and + **re-sorts the whole table**, it never just inserts a line. Scoring is + out of 100, rounded to one decimal, the weighted sum of five axes each + scored 0-100 and clamped to that range: + + - **$/confirmed**, weight 0.30: `100 * log(1.00 / c) / log(100)`, `c` + in USD - 0.01 USD scores 100, 0.10 USD 50, 1.00 USD and above 0; + - **seconds/confirmed**, weight 0.30: `100 * log(600 / s) / log(20)`, + `s` in seconds - 30 s scores 100, 600 s and above 0; + - **Total**, weight 0.20: `100 * log(3600 / T) / log(12)`, `T` the + total in seconds - 5 minutes scores 100, 60 minutes and above 0; + - **Accuracy**, weight 0.10: `100 * (a - 50) / 50`, `a` in percent - + 50 % and below scores 0, 100 % scores 100; + - **Confirmed**, weight 0.10: `100 * n / 20`, `n` the confirmed + findings - 20 and above scores 100. + + Compute it from the row's own cells as written - $/confirmed, + seconds/confirmed, Total, Accuracy and Confirmed - so any reader can + recompute it from the table. The + per-finding axes carry most of the score because they answer the + README's question - what one trustworthy finding costs in money and + in time; the total keeps a long run from winning on a low price + alone, and accuracy and volume keep a run that barely looks from + winning on its few findings. The bounds are fixed on purpose: a + score that depended on the table's best would move every other row + each time a model is added. A row measured under an earlier revision + of the protocol is marked as such and its placement is provisional + until it is re-run. Changing a weight or a bound re-scores every row + and is the maintainer's decision. Cost per confirmed finding is the column that answers the question in the README's title: cost and duration alone reward whichever model @@ -1038,7 +1131,7 @@ Steps: The PR body carries the per-finding rulings from step 8 for both runs, so the ratio is auditable and the choice between the two is too, and it names the CLI, its version and the effort flag - used. It also notes three things the table has no column for: how + used, with the effort level. It also notes three things the table has no column for: how many source files the run read **before** the drive, whether it drove any traffic of its own outside the stored scenario, and whether its report carries a replayable verification protocol. diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index b5a0b2f..d31e37a 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -4,51 +4,72 @@ A ranking, to pick the model you run the loop with and the coding-agent CLI you drive it through. Each model observes the same running stack through the same replayed -traffic, and its report is graded on evidence. One model on one CLI, one -run, one row: the same model under two CLIs is two rows, ranked against -each other like any other pair. The protocol is fixed and the only -variables are the model and the CLI. +traffic, and its report is graded on evidence. One model at one effort +on one CLI through one provider, one run, one row: the same model under +two CLIs, at two efforts or through two providers is two rows, ranked +against each other like any other pair. The protocol is fixed and the +only variables are the model, its effort, the CLI and the provider. ## Results -One row per model and CLI, always its latest run. - -| Rank | Model | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | -| **#2** | `openai/gpt-5.6-luna` | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | -| **#3** | `openai/gpt-5.6-terra` | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | -| **#4** | `openai/gpt-5.6-sol` | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | -| **#5** | `google/gemini-3.7-flash` | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | -| **#6** | `z-ai/glm-5.3` | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | -| **#7** | `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#8** | `google/gemini-3.8-flash` | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#9** | `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#10** | `anthropic/claude-opus-5` | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#11** | `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#12** | `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#13** | `anthropic/claude-sonnet-5` | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#14** | `qwen/qwen3.8-27b` | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | +One row per model, effort, CLI and provider, always its latest run. + +| Rank | Model | Effort | CLI | Provider | oddyssey | Scoring | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | **79.7** | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | +| **#2** | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 70.9 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **100%** | **$0.010** | 109s | +| **#3** | `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 68.5 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | +| **#4** | `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 67.9 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | +| **#5** | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 64.4 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | 85% | $0.212 | **44s** | +| **#6** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 64.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | 85% | $0.030 | 94s | +| **#7** | `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 63.9 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | +| **#8** | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 63.1 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | 89% | $0.082 | 70s | +| **#9** | `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 61.1 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | +| **#10** | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 60.6 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | 90% | $0.242 | 55s | +| **#11** | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 59.6 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | 89% | $0.135 | 77s | +| **#12** | `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 56.4 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | +| **#13** | `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 55.8 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | **100%** | $0.362 | 69s | +| **#14** | `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 55.4 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 53.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | **100%** | $0.191 | 103s | +| **#16** | `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 52.7 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | 94% | $0.100 | 122s | +| **#17** | `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 51.7 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | 88% | $0.014 | 276s | +| **#18** | `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 51.7 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | 89% | $0.488 | 69s | +| **#19** | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 51.4 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | 92% | $0.285 | 89s | +| **#20** | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 48.1 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | 71% | $0.199 | 99s | +| **#21** | `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 48.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | 86% | $0.634 | 75s | +| **#22** | `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | 43.2 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | **100%** | $0.561 | 139s | +| **#23** | `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | 41.2 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | **100%** | $0.147 | 220s | +| **#24** | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 39.5 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | 71% | $0.270 | 143s |
Run detail — phases, turns, tokens -| Model | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | -| `openai/gpt-5.6-luna` | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | -| `openai/gpt-5.6-sol` | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | -| `google/gemini-3.7-flash` | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | -| `z-ai/glm-5.3` | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | -| `google/gemini-3.8-flash` | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | -| `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | -| `anthropic/claude-opus-5` | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | -| `anthropic/claude-sonnet-5` | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `qwen/qwen3.8-27b` | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | +| `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | +| `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | +| `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | +| `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | +| `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | +| `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | +| `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | +| `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 3m24s | 2m02s | 18m17s | 31 | 5.3s | 3.4M | 95k | 3.4M | 4/4 | +| `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -58,16 +79,20 @@ design. **How to read the table** -- **Rank** weighs findings, cost and duration together, cost and duration the heavier since the 1.13.0 campaign. It is decided in each row's pull request, never computed: findings alone would rank a 67-minute run first, duration alone rewards whoever gives up soonest, cost alone rewards whoever barely looks. Adding a model re-sorts the whole table. +- **Rank** follows **Scoring**, highest first (a tie goes to the cheaper run); adding a model re-sorts the whole table. +- **Scoring** (0-100) weighs five axes, each scored 0-100 on fixed bounds so a row's score depends on its own figures only: **$/confirmed** 30 % (log scale, 0.01 USD = 100, 1.00 USD = 0), **seconds/confirmed** 30 % (log, 30 s = 100, 600 s = 0), **Total** 20 % (log, 5 min = 100, 60 min = 0), **Accuracy** 10 % (50 % = 0, 100 % = 100) and **Confirmed** 10 % (20 findings = 100). Cost and time per trustworthy finding carry most of it; the total duration keeps a long run from winning on a low price alone. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. +- **Accuracy** is confirmed over reported, as a percentage: how much of what the model said held up. - **$/confirmed** is what one trustworthy finding costs. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **seconds/confirmed** is how long one trustworthy finding takes: the total duration divided by the confirmed findings. +- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, Copilot for copilot. Model, effort, CLI and provider identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. -A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model and CLI, its latest run. +A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model, effort, CLI and provider, its latest run. ## How a row is produced @@ -75,11 +100,12 @@ A row measured under an earlier revision of the protocol is marked ⚠︎ and pr /launch-llms-benchmark opencode anthropic/claude-sonnet-5 /launch-llms-benchmark claude anthropic/claude-haiku-4.5 /launch-llms-benchmark copilot openai/gpt-5.6-luna +/launch-llms-benchmark copilot openai/gpt-5.6-sol high ``` -The CLI and the model id, in `vendor/name` form, are the only arguments. Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). +The CLI and the model id, in `vendor/name` form, are required; an optional third argument sets the effort (`medium` by default). Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). -The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at medium effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. +The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at the requested effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. ## The stack under observation