From 8aeb74f0df92b3c23b0bb327acecb0766e81ed40 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 00:17:50 +0200 Subject: [PATCH 01/11] feat(bench): an effort column and argument, gpt-6 luna and sol at medium and high The results tables gain an Effort column after Model (every earlier row was measured at medium), and /launch-llms-benchmark takes the effort as an optional third argument, medium by default, passed to each CLI's effort flag; model, effort and CLI identify a row. The copilot launch line lifts prompt mode's 600 s wait on background tasks, which cancelled the first gpt-6-luna observation mid-run. Four rows added: openai/gpt-6-luna and openai/gpt-6-sol on copilot, at medium and high. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 83 ++++++++++++-------- .llms-benchmark/README.md | 95 +++++++++++++---------- 2 files changed, 104 insertions(+), 74 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 3e49899..198ed31 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1,6 +1,6 @@ --- description: Benchmark one LLM on the llms-benchmark demo stack - drive it through a coding-agent CLI (opencode, claude or copilot) on the stored scenario, grade the observation report it produced, and propose its row of the results table -argument-hint: " " +argument-hint: " [effort, default medium]" --- Run the whole llms-benchmark protocol for one model on one CLI, end to @@ -20,9 +20,14 @@ the same way you would grade a colleague's incident report. `opencode`, `claude` or `copilot`; the **model** to benchmark, as its canonical `vendor/name` id, the OpenRouter form (`anthropic/claude-sonnet-5`, `openai/gpt-5-mini`, - `google/gemini-3.5-flash-lite`, ...). Those are the only two inputs. - Ask for whichever is missing and stop until you have both. Model and - CLI identify the row: the same model on two CLIs is two rows. + `google/gemini-3.5-flash-lite`, ...); optionally, the **effort** the + CLI runs the model at — `low`, `medium`, `high`, or any other level + the CLI's effort flag accepts — **`medium` when omitted**, the level + every row was measured at before this argument existed. Those are the + only three inputs. Ask for the CLI or the model when missing and stop + until you have both; never ask for the effort. Model, effort and CLI + identify the row: the same model on two CLIs, or at two efforts, is + two rows. - The model id is written the same way whatever the CLI, so the two rows of one model line up. Each CLI is handed its own form of it: `opencode` takes it as `openrouter/`; `claude` takes Anthropic @@ -32,8 +37,9 @@ the same way you would grade a colleague's incident report. dashes; `claude --help` on `--model` names the accepted forms); `copilot` takes the bare name its model picker lists — `openai/gpt-5.6-luna` is `gpt-5.6-luna` (the vendor prefix dropped, - nothing else changed). A model the CLI cannot run is a preflight - failure, not a row. + nothing else changed). A model the CLI cannot run, or cannot run at + the requested effort, is a preflight failure, not a row. Below, + `` is that argument, passed verbatim to the CLI's flag. **Never ask for an API key, and never handle one.** Every credential this protocol needs — the OpenRouter provider in opencode, the Claude Code @@ -67,7 +73,7 @@ Steps: were given - or, when the listing lags OpenRouter's catalog (it did not carry `z-ai/glm-5.3-flashx` on 2026-09-19 while the model ran), a smoke run answers with a `text` event: - `opencode run --model openrouter/ --format json "reply with the single word ok" < /dev/null` + `opencode run --model openrouter/ --variant --format json "reply with the single word ok" < /dev/null` (the package is installed in step 3); - `claude`: `claude --version` answers; the package is installed at **user scope** for Claude Code — `~/.claude/commands/odd-observe.md`, @@ -81,7 +87,7 @@ Steps: configuration (`~/.claude.json`, `mcpServers` carries `oddyssey` — the name only, never its contents); and a smoke run answers with a result naming the model: - `claude -p "Reply with the single word ok" --model --output-format json < /dev/null` + `claude -p "Reply with the single word ok" --model --effort --output-format json < /dev/null` must print a `type: result` JSON whose `modelUsage` carries the model's canonical id. Since Claude Code 2.1.270 a second key, `claude-haiku-4-5`, sits beside it on every run — a background @@ -95,7 +101,7 @@ Steps: (`~/.copilot/config.json` carries a non-empty `loggedInUsers` — the host and login, nothing else lives there); and a smoke run answers with a usage file naming the model: - `copilot -p "Reply with the single word ok" --model --allow-all-tools --usage-output-file /usage.json < /dev/null` + `copilot -p "Reply with the single word ok" --model --effort --allow-all-tools --usage-output-file /usage.json < /dev/null` must leave a `usage.json` whose `modelMetrics` has one key, the model's name. Run it from a scratch directory too — it leaves a session under `~/.copilot/session-state/`. Nothing is installed @@ -107,7 +113,7 @@ Steps: presence, never its value, and never print it. The file is gitignored; `.env.example` next to it says what goes in. -2. **Create the work branch**: `bench/--`, +2. **Create the work branch**: `bench/---`, where `` is the model id with `/` and `.` replaced by `-`. Everything the run installs, configures, and produces happens on this branch, and none of it is what ships. @@ -182,10 +188,11 @@ Steps: 4. **Select the model.** Nothing to configure: the provider is already set up (preflight), and the model and effort are passed on the command line in step 6, never persisted into a config file — `opencode`: - `--model openrouter/ --variant medium`; `claude`: - `--model --effort medium`; `copilot`: - `--model --effort medium`. The three flags name the same - effort level; that is what makes two rows of one model comparable. + `--model openrouter/ --variant `; `claude`: + `--model --effort `; `copilot`: + `--model --effort `. The three flags name the same + effort level; that is what makes two rows of one model at one effort + comparable across CLIs. 5. **Clean what the next run must not read — then recreate the demo stack, never reuse a running one.** Before every run, whatever the @@ -276,7 +283,7 @@ Steps: `opencode`: ``` - caffeinate -i opencode run --model openrouter/ --variant medium \ + caffeinate -i opencode run --model openrouter/ --variant \ --format json --auto --title "llms-benchmark " \ "" < /dev/null ``` @@ -289,7 +296,7 @@ Steps: caffeinate -i env -u CLAUDECODE -u CLAUDE_CODE_CHILD_SESSION -u CLAUDE_CODE_SESSION_ID \ -u CLAUDE_CODE_MESSAGING_SOCKET -u CLAUDE_CODE_MESSAGING_TOKEN -u CLAUDE_PID \ CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 \ - claude -p "" --model --effort medium \ + claude -p "" --model --effort \ --permission-mode bypassPermissions --output-format json \ --session-id "$SID" < /dev/null > /run.json 2> /run.err ``` @@ -298,12 +305,21 @@ Steps: ``` SID=$(uuidgen | tr 'A-Z' 'a-z') - caffeinate -i copilot -p "" --model --effort medium \ + caffeinate -i env COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200 \ + copilot -p "" --model --effort \ --allow-all --no-ask-user --additional-mcp-config @.github/mcp.json \ --session-id "$SID" --output-format json --usage-output-file /usage.json \ < /dev/null > /run.jsonl 2> /run.err ``` + `COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200` lifts prompt mode's 600 s + wait on background tasks: a root that dispatches `observe-run` with + `mode: background` and ends its turn has its subagent cancelled + 600 s later, report unwritten (`session.warning` + `background_task_wait_timeout`, `subagent.completed` with + `cancelled: true`) - the first `openai/gpt-6-luna` run of + 2026-09-24 lost its whole observation to it, ten minutes in. That + run is void, not a row. `--allow-all` is this CLI's headless auto mode (tools, paths and URLs); `--no-ask-user` removes the tool a run would otherwise use to ask a question nobody answers; `--additional-mcp-config @.github/mcp.json` @@ -985,34 +1001,37 @@ Steps: create one naming the model and the protocol revision, then the PR that closes it. This is a step, not a fallback. - From `main`, freshly pulled, create - `docs/llms-benchmark--` and make **one** change: + `docs/llms-benchmark---` and make **one** change: the row in the results tables of `.llms-benchmark/README.md`. `## Results` holds the two tables below. **A row is identified by - model and CLI together.** The pair is not in the table yet → - append the row; already there → replace that row in place. The - same model driven through two CLIs is two rows - (`google/gemini-3.7-flash` under `opencode` and under another CLI - both appear); the oddyssey version is not part of the key — a new - run of the same model and CLI overwrites the row, whatever version - the old one carried. The table carries no history: one row per - model and CLI, always the latest run. - - **Two tables, not one.** Seventeen columns scroll the model name off + model, effort and CLI together.** The triple is not in the table + yet → append the row; already there → replace that row in place. + The same model driven through two CLIs, or at two efforts, is two + rows (`google/gemini-3.7-flash` under `opencode` and under another + CLI both appear); the oddyssey version is not part of the key — a + new run of the same model, effort and CLI overwrites the row, + whatever version the old one carried. The table carries no + history: one row per model, effort and CLI, always the latest + run. + + **Two tables, not one.** Eighteen columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of nine columns — rank, model, CLI, oddyssey + - a **headline table** of ten columns — rank, model, effort, CLI, oddyssey version, `confirmed / reported`, the findings by kind under a single `Telemetry / Perf / Behavior` header written `X / X / X`, total duration, cost, and cost per confirmed finding. It fits - without scrolling and answers the question on its own. The CLI + without scrolling and answers the question on its own. The effort + column is the `` argument as passed to the CLI's flag + (`medium` by default), right after the model it qualifies. The CLI column names the coding-agent CLI the mission ran in — the `` argument, `opencode`, `claude` or `copilot`, with no version: the version belongs in the pull request, where the row's exact figures already live. The oddyssey version sits right after it because it says which protocol a row was taken under, which a reader needs before any number to its right means anything; - - a **detail table** inside a `
` block — model, CLI, + - a **detail table** inside a `
` block — model, effort, CLI, oddyssey version, the three phase durations, turns, median turn latency, input / output / cache tokens, and signals. Round the token counts (`30.0M`, `79k`): the @@ -1038,7 +1057,7 @@ Steps: The PR body carries the per-finding rulings from step 8 for both runs, so the ratio is auditable and the choice between the two is too, and it names the CLI, its version and the effort flag - used. It also notes three things the table has no column for: how + used, with the effort level. It also notes three things the table has no column for: how many source files the run read **before** the drive, whether it drove any traffic of its own outside the stored scenario, and whether its report carries a replayable verification protocol. diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index b5a0b2f..d155d4d 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -4,51 +4,60 @@ A ranking, to pick the model you run the loop with and the coding-agent CLI you drive it through. Each model observes the same running stack through the same replayed -traffic, and its report is graded on evidence. One model on one CLI, one -run, one row: the same model under two CLIs is two rows, ranked against -each other like any other pair. The protocol is fixed and the only -variables are the model and the CLI. +traffic, and its report is graded on evidence. One model at one effort +on one CLI, one run, one row: the same model under two CLIs, or at two +efforts, is two rows, ranked against each other like any other pair. The +protocol is fixed and the only variables are the model, its effort and +the CLI. ## Results -One row per model and CLI, always its latest run. - -| Rank | Model | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | -| **#2** | `openai/gpt-5.6-luna` | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | -| **#3** | `openai/gpt-5.6-terra` | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | -| **#4** | `openai/gpt-5.6-sol` | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | -| **#5** | `google/gemini-3.7-flash` | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | -| **#6** | `z-ai/glm-5.3` | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | -| **#7** | `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#8** | `google/gemini-3.8-flash` | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#9** | `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#10** | `anthropic/claude-opus-5` | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#11** | `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#12** | `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#13** | `anthropic/claude-sonnet-5` | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#14** | `qwen/qwen3.8-27b` | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | +One row per model, effort and CLI, always its latest run. + +| Rank | Model | Effort | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | +| **#2** | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | +| **#3** | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | +| **#4** | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | +| **#5** | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | $0.133 | +| **#6** | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | +| **#7** | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | $0.161 | +| **#8** | `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | +| **#9** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | +| **#10** | `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | +| **#11** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | +| **#12** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | +| **#13** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | +| **#14** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | +| **#15** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | +| **#16** | `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#17** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#18** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens -| Model | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | -| `openai/gpt-5.6-luna` | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | -| `openai/gpt-5.6-sol` | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | -| `google/gemini-3.7-flash` | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | -| `z-ai/glm-5.3` | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | -| `google/gemini-3.8-flash` | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | -| `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | -| `anthropic/claude-opus-5` | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | -| `anthropic/claude-sonnet-5` | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `qwen/qwen3.8-27b` | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| Model | Effort | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-6-sol` | high | copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | +| `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | +| `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | +| `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | +| `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | +| `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -62,12 +71,13 @@ design. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. -A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model and CLI, its latest run. +A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model, effort and CLI, its latest run. ## How a row is produced @@ -75,11 +85,12 @@ A row measured under an earlier revision of the protocol is marked ⚠︎ and pr /launch-llms-benchmark opencode anthropic/claude-sonnet-5 /launch-llms-benchmark claude anthropic/claude-haiku-4.5 /launch-llms-benchmark copilot openai/gpt-5.6-luna +/launch-llms-benchmark copilot openai/gpt-5.6-sol high ``` -The CLI and the model id, in `vendor/name` form, are the only arguments. Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). +The CLI and the model id, in `vendor/name` form, are required; an optional third argument sets the effort (`medium` by default). Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). -The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at medium effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. +The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at the requested effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. ## The stack under observation From 8e547a66a1d87341655360e16a8fd2850c0d59d6 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 16:42:28 +0200 Subject: [PATCH 02/11] feat(bench): opus 5.5 on claude, four opencode models at medium and high, effective effort per row Adds anthropic/claude-opus-5.5 (claude) and openai/gpt-6-luna-pro, openai/gpt-6-sol-pro, x-ai/grok-4.7 and z-ai/glm-5.3-prime (opencode) at medium and high, and re-sorts the table. opencode accepts any --variant name and sends no effort for an unknown one: the z-ai/glm-5.3* models and deepseek/deepseek-v4.1-flash offer no medium variant, so their rows read `default`, and the preflight now checks the model's variants. The claude teardown also clears /tmp/odd-observe-scratch/. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 17 +++++-- .llms-benchmark/README.md | 54 ++++++++++++++++------- 2 files changed, 51 insertions(+), 20 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 198ed31..a40b4df 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -74,7 +74,17 @@ Steps: did not carry `z-ai/glm-5.3-flashx` on 2026-09-19 while the model ran), a smoke run answers with a `text` event: `opencode run --model openrouter/ --variant --format json "reply with the single word ok" < /dev/null` - (the package is installed in step 3); + (the package is installed in step 3) - **and the model has a + variant at the requested effort**: `opencode run` accepts any + `--variant` name, an unknown one included, records it on every + message and sends no effort at all (verified on 2026-09-25 with + `--variant bogus`). Read the model's `variants` from + `opencode models openrouter --verbose`; when `` is not + among them, launch without `--variant` and write `default` in the + Effort column - the provider's default effort is what the model + ran at (the `z-ai/glm-5.3*` models and + `deepseek/deepseek-v4.1-flash` offer `low`, `high` and `max` only, + so every row of theirs measured at "medium" ran at `default`); - `claude`: `claude --version` answers; the package is installed at **user scope** for Claude Code — `~/.claude/commands/odd-observe.md`, `~/.claude/agents/observe-run.md`, `~/.claude/skills/ Run detail — phases, turns, tokens | Model | Effort | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | -| `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `z-ai/glm-5.3` | default | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | +| `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 2m11s | 2m02s | 21m33s | 43 | 24.1s | 15.9M | 167k | 15.0M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | high | opencode | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | +| `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 3m45s | 2m02s | 13m16s | 46 | 17.6s | 8.2M | 82k | 7.0M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 4m41s | 2m01s | 16m27s | 49 | 17.7s | 5.3M | 125k | 5.0M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 8m27s | 2m02s | 59m58s | 70 | 40.9s | 28.0M | 505k | 25.6M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -71,7 +91,7 @@ design. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. -- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. +- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. - **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). From d8ec1e06778f66a0d0aacef1eef8e17d2cc0fc54 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 16:51:24 +0200 Subject: [PATCH 03/11] docs(bench): drop the gpt-6 pro rows and glm-5.3-prime at default effort Removes openai/gpt-6-luna-pro (medium and high), openai/gpt-6-sol-pro (high) and z-ai/glm-5.3-prime (default) from both results tables on the maintainer's decision, and renumbers the ranking (24 rows). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .llms-benchmark/README.md | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 983f982..f1951db 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -27,23 +27,19 @@ One row per model, effort and CLI, always its latest run. | **#9** | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | $0.212 | | **#10** | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | $0.242 | | **#11** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | -| **#12** | `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 11 / 13 | 6 / 2 / 3 | 25m46s | $0.59 | $0.053 | -| **#13** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#14** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | -| **#15** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | -| **#16** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#17** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | -| **#18** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | -| **#19** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | -| **#20** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#21** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#22** | `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 14 / 15 | 8 / 3 / 3 | 19m03s | $7.34 | $0.525 | -| **#23** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#24** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#25** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#26** | `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 7 / 8 | 4 / 3 / 0 | 23m09s | $4.73 | $0.675 | -| **#27** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | -| **#28** | `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 10 / 11 | 6 / 3 / 1 | 70m27s | $1.32 | $0.132 | +| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | +| **#13** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | +| **#14** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | +| **#16** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | +| **#17** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | +| **#18** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | +| **#19** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | +| **#20** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | +| **#21** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | +| **#22** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#23** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#24** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens @@ -61,7 +57,6 @@ One row per model, effort and CLI, always its latest run. | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 2m11s | 2m02s | 21m33s | 43 | 24.1s | 15.9M | 167k | 15.0M | 4/4 | | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | @@ -71,13 +66,10 @@ One row per model, effort and CLI, always its latest run. | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 3m45s | 2m02s | 13m16s | 46 | 17.6s | 8.2M | 82k | 7.0M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 4m41s | 2m01s | 16m27s | 49 | 17.7s | 5.3M | 125k | 5.0M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | -| `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 8m27s | 2m02s | 59m58s | 70 | 40.9s | 28.0M | 505k | 25.6M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by From ce4313cf209f120f7f6fd93327290566643de11c Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 08:58:23 +0200 Subject: [PATCH 04/11] docs(bench): claude-fable-5.1 re-run under 1.13.0 at medium, and a high row Replaces the provisional 1.12.0 row of anthropic/claude-fable-5.1 (claude) with a run under 1.13.0 at medium, adds its high row, and renumbers the ranking (25 rows, none provisional). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .llms-benchmark/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index f1951db..96b7af7 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -36,10 +36,11 @@ One row per model, effort and CLI, always its latest run. | **#18** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | | **#19** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | | **#20** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#21** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#22** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#23** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#24** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | +| **#21** | `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | $0.488 | +| **#22** | `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | $0.634 | +| **#23** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#24** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#25** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens @@ -66,7 +67,8 @@ One row per model, effort and CLI, always its latest run. | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | +| `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | +| `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 3m24s | 2m02s | 18m17s | 31 | 5.3s | 3.4M | 95k | 3.4M | 4/4 | | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | From a78d814a47a2c1590f61cbca2d906071483d6192 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:04:52 +0200 Subject: [PATCH 05/11] feat(bench): a seconds/confirmed column, and drop the gpt-6-sol-pro medium row The headline table gains seconds/confirmed after $/confirmed - the total duration divided by the confirmed findings, the time one trustworthy finding takes - and the command's table contract says so. The openai/gpt-6-sol-pro medium row is removed on the maintainer's decision (24 rows). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 8 ++-- .llms-benchmark/README.md | 55 +++++++++++------------ 2 files changed, 32 insertions(+), 31 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index a40b4df..0444e94 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1025,14 +1025,16 @@ Steps: history: one row per model, effort and CLI, always the latest run. - **Two tables, not one.** Eighteen columns scroll the model name off + **Two tables, not one.** Nineteen columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of ten columns — rank, model, effort, CLI, oddyssey + - a **headline table** of eleven columns — rank, model, effort, CLI, oddyssey version, `confirmed / reported`, the findings by kind under a single `Telemetry / Perf / Behavior` header written `X / X / X`, - total duration, cost, and cost per confirmed finding. It fits + total duration, cost, cost per confirmed finding and seconds per + confirmed finding (the total duration divided by the confirmed + findings). It fits without scrolling and answers the question on its own. The effort column is the `` argument as passed to the CLI's flag (`medium` by default), right after the model it qualifies. The CLI diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 96b7af7..8c78e73 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -14,33 +14,32 @@ the CLI. One row per model, effort and CLI, always its latest run. -| Rank | Model | Effort | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | -| **#2** | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | -| **#3** | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | -| **#4** | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | -| **#5** | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | $0.133 | -| **#6** | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | -| **#7** | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | $0.161 | -| **#8** | `z-ai/glm-5.3` | default | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | -| **#9** | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | $0.212 | -| **#10** | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | $0.242 | -| **#11** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | -| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#13** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | -| **#14** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | -| **#15** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#16** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | -| **#17** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | -| **#18** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | -| **#19** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#20** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#21** | `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | $0.488 | -| **#22** | `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | $0.634 | -| **#23** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#24** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#25** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | +| Rank | Model | Effort | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | seconds/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | 94s | +| **#2** | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | 56s | +| **#3** | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | 51s | +| **#4** | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | 48s | +| **#5** | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | $0.133 | 64s | +| **#6** | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | 77s | +| **#7** | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | $0.161 | 78s | +| **#8** | `z-ai/glm-5.3` | default | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | 70s | +| **#9** | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | $0.212 | **44s** | +| **#10** | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | $0.242 | 55s | +| **#11** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | 139s | +| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | 109s | +| **#13** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | 99s | +| **#14** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | 151s | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | 103s | +| **#16** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | 89s | +| **#17** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | 143s | +| **#18** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | 122s | +| **#19** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | 69s | +| **#20** | `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | $0.488 | 69s | +| **#21** | `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | $0.634 | 75s | +| **#22** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | 276s | +| **#23** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | 139s | +| **#24** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | 220s |
Run detail — phases, turns, tokens @@ -64,7 +63,6 @@ One row per model, effort and CLI, always its latest run. | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | -| `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | @@ -85,6 +83,7 @@ design. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. +- **seconds/confirmed** is how long one trustworthy finding takes: the total duration divided by the confirmed findings. - **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. - **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. From 9e08424aeff8463075540695e88d1d4832521ac0 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:13:39 +0200 Subject: [PATCH 06/11] feat(bench): Provider and Accuracy columns in the results tables Provider follows CLI in both tables - OpenRouter for opencode, Anthropic for claude, GitHub Copilot for copilot - and Accuracy (confirmed over reported, as a percentage) sits before $/confirmed in the headline table. The reading guide and the command's table contract say so. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 13 ++- .llms-benchmark/README.md | 107 +++++++++++----------- 2 files changed, 63 insertions(+), 57 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 0444e94..2466979 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1025,14 +1025,16 @@ Steps: history: one row per model, effort and CLI, always the latest run. - **Two tables, not one.** Nineteen columns scroll the model name off + **Two tables, not one.** Twenty-two columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of eleven columns — rank, model, effort, CLI, oddyssey + - a **headline table** of thirteen columns — rank, model, effort, CLI, + provider, oddyssey version, `confirmed / reported`, the findings by kind under a single `Telemetry / Perf / Behavior` header written `X / X / X`, - total duration, cost, cost per confirmed finding and seconds per + total duration, cost, accuracy (confirmed over reported, as a + percentage), cost per confirmed finding and seconds per confirmed finding (the total duration divided by the confirmed findings). It fits without scrolling and answers the question on its own. The effort @@ -1041,10 +1043,13 @@ Steps: column names the coding-agent CLI the mission ran in — the `` argument, `opencode`, `claude` or `copilot`, with no version: the version belongs in the pull request, where the row's exact figures already - live. The oddyssey version sits right after it because it says + live. The provider column follows it and names who served the model: + `OpenRouter` for opencode, `Anthropic` for claude, `GitHub Copilot` + for copilot. The oddyssey version sits right after it because it says which protocol a row was taken under, which a reader needs before any number to its right means anything; - a **detail table** inside a `
` block — model, effort, CLI, + provider, oddyssey version, the three phase durations, turns, median turn latency, input / output / cache tokens, and signals. Round the token counts (`30.0M`, `79k`): the diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 8c78e73..1c0a8c3 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -14,62 +14,62 @@ the CLI. One row per model, effort and CLI, always its latest run. -| Rank | Model | Effort | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | seconds/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | 94s | -| **#2** | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | 56s | -| **#3** | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | 51s | -| **#4** | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | 48s | -| **#5** | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | $0.133 | 64s | -| **#6** | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | 77s | -| **#7** | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | $0.161 | 78s | -| **#8** | `z-ai/glm-5.3` | default | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | 70s | -| **#9** | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | $0.212 | **44s** | -| **#10** | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | $0.242 | 55s | -| **#11** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | 139s | -| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | 109s | -| **#13** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | 99s | -| **#14** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | 151s | -| **#15** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | 103s | -| **#16** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | 89s | -| **#17** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | 143s | -| **#18** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | 122s | -| **#19** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | 69s | -| **#20** | `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | $0.488 | 69s | -| **#21** | `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | $0.634 | 75s | -| **#22** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | 276s | -| **#23** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | 139s | -| **#24** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | 220s | +| Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | 85% | $0.030 | 94s | +| **#2** | `openai/gpt-5.6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | +| **#3** | `openai/gpt-5.6-terra` | medium | copilot | GitHub Copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | +| **#4** | `openai/gpt-5.6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | +| **#5** | `openai/gpt-6-sol` | high | copilot | GitHub Copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | +| **#6** | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | 89% | $0.135 | 77s | +| **#7** | `openai/gpt-6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | +| **#8** | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | 89% | $0.082 | 70s | +| **#9** | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | 85% | $0.212 | **44s** | +| **#10** | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | 90% | $0.242 | 55s | +| **#11** | `openai/gpt-6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | +| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **100%** | **$0.010** | 109s | +| **#13** | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | 71% | $0.199 | 99s | +| **#14** | `openai/gpt-6-luna` | high | copilot | GitHub Copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | **100%** | $0.191 | 103s | +| **#16** | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | 92% | $0.285 | 89s | +| **#17** | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | 71% | $0.270 | 143s | +| **#18** | `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | 94% | $0.100 | 122s | +| **#19** | `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | **100%** | $0.362 | 69s | +| **#20** | `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | 89% | $0.488 | 69s | +| **#21** | `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | 86% | $0.634 | 75s | +| **#22** | `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | 88% | $0.014 | 276s | +| **#23** | `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | **100%** | $0.561 | 139s | +| **#24** | `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | **100%** | $0.147 | 220s |
Run detail — phases, turns, tokens -| Model | Effort | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | -| `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | -| `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | -| `openai/gpt-6-sol` | high | copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | -| `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | -| `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | -| `z-ai/glm-5.3` | default | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | -| `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | -| `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | -| `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | -| `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | -| `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | -| `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | -| `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | -| `x-ai/grok-4.7` | high | opencode | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | -| `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | -| `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `anthropic/claude-fable-5.1` | medium | claude | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | -| `anthropic/claude-fable-5.1` | high | claude | 1.13.0 | 3m24s | 2m02s | 18m17s | 31 | 5.3s | 3.4M | 95k | 3.4M | 4/4 | -| `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | -| `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-5.6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | GitHub Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `openai/gpt-5.6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-6-sol` | high | copilot | GitHub Copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | +| `openai/gpt-6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | +| `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | +| `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | +| `openai/gpt-6-luna` | high | copilot | GitHub Copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | +| `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | +| `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | +| `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | +| `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 3m24s | 2m02s | 18m17s | 31 | 5.3s | 3.4M | 95k | 3.4M | 4/4 | +| `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -82,10 +82,11 @@ design. - **Rank** weighs findings, cost and duration together, cost and duration the heavier since the 1.13.0 campaign. It is decided in each row's pull request, never computed: findings alone would rank a 67-minute run first, duration alone rewards whoever gives up soonest, cost alone rewards whoever barely looks. Adding a model re-sorts the whole table. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. +- **Accuracy** is confirmed over reported, as a percentage: how much of what the model said held up. - **$/confirmed** is what one trustworthy finding costs. - **seconds/confirmed** is how long one trustworthy finding takes: the total duration divided by the confirmed findings. - **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, GitHub Copilot for copilot. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. From a90c6bb83c10d0f254025169adc92f6adea6cae5 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:15:44 +0200 Subject: [PATCH 07/11] docs(bench): the copilot provider reads Copilot Shortens the Provider label of the copilot rows from "GitHub Copilot" to "Copilot" in both tables, the reading guide and the command. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 2 +- .llms-benchmark/README.md | 30 +++++++++++------------ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 2466979..a4b80a6 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1044,7 +1044,7 @@ Steps: argument, `opencode`, `claude` or `copilot`, with no version: the version belongs in the pull request, where the row's exact figures already live. The provider column follows it and names who served the model: - `OpenRouter` for opencode, `Anthropic` for claude, `GitHub Copilot` + `OpenRouter` for opencode, `Anthropic` for claude, `Copilot` for copilot. The oddyssey version sits right after it because it says which protocol a row was taken under, which a reader needs before any number to its right means anything; diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 1c0a8c3..b478df0 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -17,19 +17,19 @@ One row per model, effort and CLI, always its latest run. | Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | 85% | $0.030 | 94s | -| **#2** | `openai/gpt-5.6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | -| **#3** | `openai/gpt-5.6-terra` | medium | copilot | GitHub Copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | -| **#4** | `openai/gpt-5.6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | -| **#5** | `openai/gpt-6-sol` | high | copilot | GitHub Copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | +| **#2** | `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | +| **#3** | `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | +| **#4** | `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | +| **#5** | `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | | **#6** | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | 89% | $0.135 | 77s | -| **#7** | `openai/gpt-6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | +| **#7** | `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | | **#8** | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | 89% | $0.082 | 70s | | **#9** | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | 85% | $0.212 | **44s** | | **#10** | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | 90% | $0.242 | 55s | -| **#11** | `openai/gpt-6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | +| **#11** | `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | | **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **100%** | **$0.010** | 109s | | **#13** | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | 71% | $0.199 | 99s | -| **#14** | `openai/gpt-6-luna` | high | copilot | GitHub Copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | +| **#14** | `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | | **#15** | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | **100%** | $0.191 | 103s | | **#16** | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | 92% | $0.285 | 89s | | **#17** | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | 71% | $0.270 | 143s | @@ -47,19 +47,19 @@ One row per model, effort and CLI, always its latest run. | Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | -| `openai/gpt-5.6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | medium | copilot | GitHub Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | -| `openai/gpt-5.6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | -| `openai/gpt-6-sol` | high | copilot | GitHub Copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | -| `openai/gpt-6-sol` | medium | copilot | GitHub Copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | +| `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | -| `openai/gpt-6-luna` | medium | copilot | GitHub Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | -| `openai/gpt-6-luna` | high | copilot | GitHub Copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | +| `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | @@ -86,7 +86,7 @@ design. - **$/confirmed** is what one trustworthy finding costs. - **seconds/confirmed** is how long one trustworthy finding takes: the total duration divided by the confirmed findings. - **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, GitHub Copilot for copilot. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, Copilot for copilot. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. From 9f914b746705dea1b247861543f72dd43a012575 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:17:57 +0200 Subject: [PATCH 08/11] docs(commands): launch-llms-benchmark states every results column and its format The table contract now gives the exact headers of both tables, an example row, what fills each headline cell and in which format (Effort as applied, Provider per CLI, Accuracy, $/confirmed, seconds/confirmed) and the bolding rule, so the next row is written without guessing. The example's money cells are patterns: the command's text never carries a dollar sign followed by a digit. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 83 ++++++++++++++++------- 1 file changed, 57 insertions(+), 26 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index a4b80a6..07f1d53 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1029,32 +1029,63 @@ Steps: the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of thirteen columns — rank, model, effort, CLI, - provider, oddyssey - version, `confirmed / reported`, the findings by kind under a - single `Telemetry / Perf / Behavior` header written `X / X / X`, - total duration, cost, accuracy (confirmed over reported, as a - percentage), cost per confirmed finding and seconds per - confirmed finding (the total duration divided by the confirmed - findings). It fits - without scrolling and answers the question on its own. The effort - column is the `` argument as passed to the CLI's flag - (`medium` by default), right after the model it qualifies. The CLI - column names the coding-agent CLI the mission ran in — the `` - argument, `opencode`, `claude` or `copilot`, with no version: the version - belongs in the pull request, where the row's exact figures already - live. The provider column follows it and names who served the model: - `OpenRouter` for opencode, `Anthropic` for claude, `Copilot` - for copilot. The oddyssey version sits right after it because it says - which protocol a row was taken under, which a reader needs before - any number to its right means anything; - - a **detail table** inside a `
` block — model, effort, CLI, - provider, - oddyssey version, the three phase durations, turns, median turn - latency, input / output / cache tokens, and signals. Round the - token counts (`30.0M`, `79k`): the - exact figures live in each run's pull request, and full precision - here only costs width. + - a **headline table**, thirteen columns, exactly these headers: + + ```text + | Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | + | **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $N.NN | 85% | $N.NNN | 94s | + ``` + + (the two money cells are written here as patterns, never as a + dollar sign followed by a digit - see the end of step 10; in the + README they carry the figures: a dollar sign, then 0.33 and 0.030) + + It fits without scrolling and answers the question on its own. + Each cell, left to right: + - **Rank** `**#N**`, renumbered from 1 after every re-sort; + - **Model** the canonical id in backticks; + - **Effort** the level the CLI actually applied - the `` + argument (`medium` by default), or `default` when the CLI has no + variant at that level for the model (step 1's opencode check); + - **CLI** `opencode`, `claude` or `copilot`, no version - the + version belongs in the pull request, where the row's exact + figures already live; + - **Provider** who served the model to that CLI: `OpenRouter` for + opencode, `Anthropic` for claude, `Copilot` for copilot; + - **oddyssey** the version the row was taken under (step 7), right + after them because a reader needs the protocol before any number + to its right means anything; ` ⚠︎` after it marks a row measured + under an earlier revision of the protocol; + - **Confirmed / reported** `X / Y` from step 8; + - **Telemetry / Perf / Behavior** the confirmed findings by kind, + `X / X / X`, summing to X; + - **Total** the run's wall clock, `NmSSs`; + - **Cost** step 7's figure, `$N.NN`; + - **Accuracy** confirmed over reported, rounded to a whole + percentage, `NN%`; + - **$/confirmed** Cost over confirmed, `$N.NNN`; + - **seconds/confirmed** Total in seconds over confirmed, rounded to + a whole second, `NNs`. + + **Bold**: Confirmed / reported on every perfect ratio (`**6 / 6**`); + in Total, Cost, Accuracy, $/confirmed and seconds/confirmed the + best value of the table only, on every row that ties it - the + shortest, the cheapest, the highest, the cheapest, the fastest - + re-checked after each re-sort, since a new row can take it from + another; + - a **detail table** inside a `
` block, exactly these + headers, the first five cells as in the headline row: + + ```text + | Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | + | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | + ``` + + the three phase durations, turns, median turn latency, input / + output / cache tokens and signals from step 7. Round the token + counts (`30.0M`, `79k`): the exact figures live in each run's pull + request, and full precision here only costs width. The detail + table lists the rows in the headline table's order. **The rank is decided with the user, not computed.** It weighs three axes together — findings, cost and duration — and none of them alone From 4e1821af4bed9edd9b08be92499a29c5bd1d0e6b Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:19:32 +0200 Subject: [PATCH 09/11] fix(commands): drop the hidden variation selector from the provisional-row mark The warning sign written in the table contract carried U+FE0E, which the package audit flags as a hidden character and fails on; the mark is now the plain sign. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 07f1d53..805a7c2 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1054,7 +1054,7 @@ Steps: opencode, `Anthropic` for claude, `Copilot` for copilot; - **oddyssey** the version the row was taken under (step 7), right after them because a reader needs the protocol before any number - to its right means anything; ` ⚠︎` after it marks a row measured + to its right means anything; ` ⚠` after it marks a row measured under an earlier revision of the protocol; - **Confirmed / reported** `X / Y` from step 8; - **Telemetry / Perf / Behavior** the confirmed findings by kind, From 9cb6fc19b5cf03aab4039b173471051d53c5d792 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:21:35 +0200 Subject: [PATCH 10/11] feat(bench): model, effort, CLI and provider identify a results row The row key gains the provider beside model, effort and CLI, in the command's arguments and table contract and in the README's intro and reading guide: the same model through two providers is two rows. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 24 +++++++++++------------ .llms-benchmark/README.md | 14 ++++++------- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 805a7c2..2c25b2a 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -25,9 +25,9 @@ the same way you would grade a colleague's incident report. the CLI's effort flag accepts — **`medium` when omitted**, the level every row was measured at before this argument existed. Those are the only three inputs. Ask for the CLI or the model when missing and stop - until you have both; never ask for the effort. Model, effort and CLI - identify the row: the same model on two CLIs, or at two efforts, is - two rows. + until you have both; never ask for the effort. Model, effort, CLI and + provider identify the row: the same model on two CLIs, at two efforts, + or served by two providers, is two rows. - The model id is written the same way whatever the CLI, so the two rows of one model line up. Each CLI is handed its own form of it: `opencode` takes it as `openrouter/`; `claude` takes Anthropic @@ -1015,15 +1015,15 @@ Steps: `docs/llms-benchmark---` and make **one** change: the row in the results tables of `.llms-benchmark/README.md`. `## Results` holds the two tables below. **A row is identified by - model, effort and CLI together.** The triple is not in the table - yet → append the row; already there → replace that row in place. - The same model driven through two CLIs, or at two efforts, is two - rows (`google/gemini-3.7-flash` under `opencode` and under another - CLI both appear); the oddyssey version is not part of the key — a - new run of the same model, effort and CLI overwrites the row, - whatever version the old one carried. The table carries no - history: one row per model, effort and CLI, always the latest - run. + model, effort, CLI and provider together.** That key is not in the + table yet → append the row; already there → replace that row in + place. The same model driven through two CLIs, at two efforts, or + served by two providers, is two rows (`openai/gpt-6-luna` at + `medium` and at `high` both appear); the oddyssey version is not + part of the key — a new run of the same model, effort, CLI and + provider overwrites the row, whatever version the old one carried. + The table carries no history: one row per model, effort, CLI and + provider, always the latest run. **Two tables, not one.** Twenty-two columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index b478df0..f683e0d 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -5,14 +5,14 @@ CLI you drive it through. Each model observes the same running stack through the same replayed traffic, and its report is graded on evidence. One model at one effort -on one CLI, one run, one row: the same model under two CLIs, or at two -efforts, is two rows, ranked against each other like any other pair. The -protocol is fixed and the only variables are the model, its effort and -the CLI. +on one CLI through one provider, one run, one row: the same model under +two CLIs, at two efforts or through two providers is two rows, ranked +against each other like any other pair. The protocol is fixed and the +only variables are the model, its effort, the CLI and the provider. ## Results -One row per model, effort and CLI, always its latest run. +One row per model, effort, CLI and provider, always its latest run. | Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -86,12 +86,12 @@ design. - **$/confirmed** is what one trustworthy finding costs. - **seconds/confirmed** is how long one trustworthy finding takes: the total duration divided by the confirmed findings. - **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, Copilot for copilot. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. **Provider** is who served the model to that CLI: OpenRouter for opencode, Anthropic for claude, Copilot for copilot. Model, effort, CLI and provider identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. -A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model, effort and CLI, its latest run. +A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model, effort, CLI and provider, its latest run. ## How a row is produced From 1ca2e7b7c28ffa580c966e9b3428a14e940727f7 Mon Sep 17 00:00:00 2001 From: using-system Date: Sat, 26 Sep 2026 09:33:30 +0200 Subject: [PATCH 11/11] feat(bench): a Scoring column ranks the results table The headline table gains Scoring (0-100) before Confirmed / reported, and the rank is now the table sorted by it. The score weighs $/confirmed 30 %, seconds/confirmed 30 %, Total 20 % (all three on a log scale between fixed bounds), Accuracy 10 % and Confirmed 10 %; the fixed bounds keep a row's score independent of the other rows. The README's reading guide describes it in one bullet, the command states the formula, the bounds and the sort. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 57 +++++++++++----- .llms-benchmark/README.md | 79 ++++++++++++----------- 2 files changed, 81 insertions(+), 55 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 2c25b2a..a2ffb87 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1025,15 +1025,15 @@ Steps: The table carries no history: one row per model, effort, CLI and provider, always the latest run. - **Two tables, not one.** Twenty-two columns scroll the model name off + **Two tables, not one.** Twenty-three columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table**, thirteen columns, exactly these headers: + - a **headline table**, fourteen columns, exactly these headers: ```text - | Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | - | **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $N.NN | 85% | $N.NNN | 94s | + | Rank | Model | Effort | CLI | Provider | oddyssey | Scoring | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | + | **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 64.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $N.NN | 85% | $N.NNN | 94s | ``` (the two money cells are written here as patterns, never as a @@ -1042,7 +1042,8 @@ Steps: It fits without scrolling and answers the question on its own. Each cell, left to right: - - **Rank** `**#N**`, renumbered from 1 after every re-sort; + - **Rank** `**#N**`, the position by Scoring, renumbered from 1 + after every re-sort; - **Model** the canonical id in backticks; - **Effort** the level the CLI actually applied - the `` argument (`medium` by default), or `default` when the CLI has no @@ -1056,6 +1057,7 @@ Steps: after them because a reader needs the protocol before any number to its right means anything; ` ⚠` after it marks a row measured under an earlier revision of the protocol; + - **Scoring** the row's score out of 100, one decimal (below); - **Confirmed / reported** `X / Y` from step 8; - **Telemetry / Perf / Behavior** the confirmed findings by kind, `X / X / X`, summing to X; @@ -1068,9 +1070,10 @@ Steps: a whole second, `NNs`. **Bold**: Confirmed / reported on every perfect ratio (`**6 / 6**`); - in Total, Cost, Accuracy, $/confirmed and seconds/confirmed the + in Scoring, Total, Cost, Accuracy, $/confirmed and seconds/confirmed the best value of the table only, on every row that ties it - the - shortest, the cheapest, the highest, the cheapest, the fastest - + highest, the shortest, the cheapest, the highest, the cheapest, the + fastest - re-checked after each re-sort, since a new row can take it from another; - a **detail table** inside a `
` block, exactly these @@ -1087,15 +1090,37 @@ Steps: request, and full precision here only costs width. The detail table lists the rows in the headline table's order. - **The rank is decided with the user, not computed.** It weighs three - axes together — findings, cost and duration — and none of them alone - survives as a rule: ranking on findings would put a 67-minute run - first, on duration would reward whichever model gives up soonest, on - cost would reward the one that barely looks. Propose a placement in - the PR and argue it on the three axes; adding or updating a model - **re-sorts the whole table**, it never just inserts a line. A row - measured under an earlier revision of the protocol is marked as such - and its placement is provisional until it is re-run. + **The rank is computed: the table is sorted by Scoring, highest + first, a tie going to the cheaper run.** Adding or updating a model + recomputes nothing but its own score - every bound is fixed - and + **re-sorts the whole table**, it never just inserts a line. Scoring is + out of 100, rounded to one decimal, the weighted sum of five axes each + scored 0-100 and clamped to that range: + + - **$/confirmed**, weight 0.30: `100 * log(1.00 / c) / log(100)`, `c` + in USD - 0.01 USD scores 100, 0.10 USD 50, 1.00 USD and above 0; + - **seconds/confirmed**, weight 0.30: `100 * log(600 / s) / log(20)`, + `s` in seconds - 30 s scores 100, 600 s and above 0; + - **Total**, weight 0.20: `100 * log(3600 / T) / log(12)`, `T` the + total in seconds - 5 minutes scores 100, 60 minutes and above 0; + - **Accuracy**, weight 0.10: `100 * (a - 50) / 50`, `a` in percent - + 50 % and below scores 0, 100 % scores 100; + - **Confirmed**, weight 0.10: `100 * n / 20`, `n` the confirmed + findings - 20 and above scores 100. + + Compute it from the row's own cells as written - $/confirmed, + seconds/confirmed, Total, Accuracy and Confirmed - so any reader can + recompute it from the table. The + per-finding axes carry most of the score because they answer the + README's question - what one trustworthy finding costs in money and + in time; the total keeps a long run from winning on a low price + alone, and accuracy and volume keep a run that barely looks from + winning on its few findings. The bounds are fixed on purpose: a + score that depended on the table's best would move every other row + each time a model is added. A row measured under an earlier revision + of the protocol is marked as such and its placement is provisional + until it is re-run. Changing a weight or a bound re-scores every row + and is the maintainer's decision. Cost per confirmed finding is the column that answers the question in the README's title: cost and duration alone reward whichever model diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index f683e0d..d31e37a 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -14,62 +14,62 @@ only variables are the model, its effort, the CLI and the provider. One row per model, effort, CLI and provider, always its latest run. -| Rank | Model | Effort | CLI | Provider | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | 85% | $0.030 | 94s | -| **#2** | `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | -| **#3** | `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | -| **#4** | `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | -| **#5** | `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | -| **#6** | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | 89% | $0.135 | 77s | -| **#7** | `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | -| **#8** | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | 89% | $0.082 | 70s | -| **#9** | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | 85% | $0.212 | **44s** | -| **#10** | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | 90% | $0.242 | 55s | -| **#11** | `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | -| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **100%** | **$0.010** | 109s | -| **#13** | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | 71% | $0.199 | 99s | -| **#14** | `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | -| **#15** | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | **100%** | $0.191 | 103s | -| **#16** | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | 92% | $0.285 | 89s | -| **#17** | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | 71% | $0.270 | 143s | -| **#18** | `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | 94% | $0.100 | 122s | -| **#19** | `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | **100%** | $0.362 | 69s | -| **#20** | `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | 89% | $0.488 | 69s | -| **#21** | `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | 86% | $0.634 | 75s | -| **#22** | `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | 88% | $0.014 | 276s | -| **#23** | `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | **100%** | $0.561 | 139s | -| **#24** | `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | **100%** | $0.147 | 220s | +| Rank | Model | Effort | CLI | Provider | oddyssey | Scoring | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | Accuracy | $/confirmed | seconds/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | **79.7** | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | 88% | $0.016 | 56s | +| **#2** | `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 70.9 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **100%** | **$0.010** | 109s | +| **#3** | `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 68.5 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | 92% | $0.117 | 48s | +| **#4** | `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 67.9 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | 88% | $0.126 | 51s | +| **#5** | `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 64.4 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | 85% | $0.212 | **44s** | +| **#6** | `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 64.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | 85% | $0.030 | 94s | +| **#7** | `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 63.9 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | **100%** | $0.016 | 139s | +| **#8** | `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 63.1 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | 89% | $0.082 | 70s | +| **#9** | `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 61.1 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | 85% | $0.133 | 64s | +| **#10** | `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 60.6 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | 90% | $0.242 | 55s | +| **#11** | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 59.6 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | 89% | $0.135 | 77s | +| **#12** | `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 56.4 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | 80% | $0.161 | 78s | +| **#13** | `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 55.8 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | **100%** | $0.362 | 69s | +| **#14** | `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 55.4 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | 67% | $0.018 | 151s | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 53.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | **100%** | $0.191 | 103s | +| **#16** | `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 52.7 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | 94% | $0.100 | 122s | +| **#17** | `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 51.7 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | 88% | $0.014 | 276s | +| **#18** | `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 51.7 | 16 / 18 | 7 / 5 / 4 | 18m19s | $7.80 | 89% | $0.488 | 69s | +| **#19** | `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 51.4 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | 92% | $0.285 | 89s | +| **#20** | `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 48.1 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | 71% | $0.199 | 99s | +| **#21** | `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 48.0 | 19 / 22 | 10 / 6 / 3 | 23m43s | $12.05 | 86% | $0.634 | 75s | +| **#22** | `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | 43.2 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | **100%** | $0.561 | 139s | +| **#23** | `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | 41.2 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | **100%** | $0.147 | 220s | +| **#24** | `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 39.5 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | 71% | $0.270 | 143s |
Run detail — phases, turns, tokens | Model | Effort | CLI | Provider | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | | `openai/gpt-5.6-luna` | medium | copilot | Copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | | `openai/gpt-5.6-sol` | medium | copilot | Copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | Copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | +| `z-ai/glm-5.3-flashx` | default | opencode | OpenRouter | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | | `openai/gpt-6-sol` | high | copilot | Copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | | `google/gemini-3.7-flash` | medium | opencode | OpenRouter | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | | `openai/gpt-6-sol` | medium | copilot | Copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | -| `z-ai/glm-5.3` | default | opencode | OpenRouter | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | -| `anthropic/claude-opus-5.5` | medium | claude | Anthropic | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | -| `anthropic/claude-opus-5.5` | high | claude | Anthropic | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | -| `openai/gpt-6-luna` | medium | copilot | Copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | default | opencode | OpenRouter | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | -| `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | +| `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | | `openai/gpt-6-luna` | high | copilot | Copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | | `google/gemini-3.8-flash` | medium | opencode | OpenRouter | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | -| `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | -| `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | OpenRouter | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | -| `anthropic/claude-opus-5` | medium | claude | Anthropic | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | Anthropic | 1.13.0 | 1m53s | 2m01s | 14m25s | 41 | 4.5s | 4.4M | 67k | 4.4M | 4/4 | +| `z-ai/glm-5.3-prime` | high | opencode | OpenRouter | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | medium | opencode | OpenRouter | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | | `anthropic/claude-fable-5.1` | high | claude | Anthropic | 1.13.0 | 3m24s | 2m02s | 18m17s | 31 | 5.3s | 3.4M | 95k | 3.4M | 4/4 | -| `z-ai/glm-5.3-flash` | default | opencode | OpenRouter | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | Anthropic | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | OpenRouter | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | high | opencode | OpenRouter | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -79,7 +79,8 @@ design. **How to read the table** -- **Rank** weighs findings, cost and duration together, cost and duration the heavier since the 1.13.0 campaign. It is decided in each row's pull request, never computed: findings alone would rank a 67-minute run first, duration alone rewards whoever gives up soonest, cost alone rewards whoever barely looks. Adding a model re-sorts the whole table. +- **Rank** follows **Scoring**, highest first (a tie goes to the cheaper run); adding a model re-sorts the whole table. +- **Scoring** (0-100) weighs five axes, each scored 0-100 on fixed bounds so a row's score depends on its own figures only: **$/confirmed** 30 % (log scale, 0.01 USD = 100, 1.00 USD = 0), **seconds/confirmed** 30 % (log, 30 s = 100, 600 s = 0), **Total** 20 % (log, 5 min = 100, 60 min = 0), **Accuracy** 10 % (50 % = 0, 100 % = 100) and **Confirmed** 10 % (20 findings = 100). Cost and time per trustworthy finding carry most of it; the total duration keeps a long run from winning on a low price alone. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **Accuracy** is confirmed over reported, as a percentage: how much of what the model said held up.