diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 1fde4e5..8ea87b9 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -256,6 +256,19 @@ Steps: ceiling for a run that never produces a turn or declines to drive (step 8): a decline counts as one of the two. + **A second run that is already worse than the first is stopped, not + finished.** Watch it against the first run's figures: once it has + been running longer than the first run's whole duration and its + report is not written yet, or its drive has not started by the time + the first run had finished, kill it by its PID — it can no longer + beat the first on duration, its cost is already spent for nothing, + and the twenty-to-forty minutes it still needs are better given to + the next model. The row is the first run; the pull request records + the second's launch time, the phase it was stopped in and its spend + to that point. Decided on 2026-09-20 by the maintainer after the + second run of one model passed the first's 29-minute total with its + report still being generated, at 49 minutes. + For each run: record the UTC timestamp **before** launching — step 7 needs it to identify the session. Then one headless run, from the repository root, on the work branch. @@ -378,10 +391,11 @@ Steps: opencode it is not**: `opencode run` stores the argument wrapped in one more pair of escaped quotes than it was given (verified on 2026-09-14 with a smoke argument: `"x"` passed, `"\"x\""` stored, after - decoding), so the argument to pass is the same 683-character string + decoding), so the argument to pass is the same 669-character string as under the other two CLIs, and the store's form passed verbatim reaches the run double-wrapped (one run of 2026-09-12 did, at 693 - characters). + characters, when the mission still named a depth and was 683 long — + #620 dropped the depth phrase, and every run since is 669). Add exactly three things to that line and nothing else: that the services' sources are under `.llms-benchmark/src/`; that you want @@ -547,6 +561,17 @@ Steps: result must equal it, and the transcript is `~/.claude/projects//.jsonl`. Anything else, stop and say so. + - **the observation must have run on the benchmarked model.** The + root session may dispatch `observe-run` through the Agent tool with + a `model` of its own choosing — on 2026-09-20 a `claude-sonnet-5` + root passed `model: 'opus'` on one run of two, and the subagent's + 89 requests, 90 % of the spend and the whole report were opus-5's. + Read `modelUsage`'s keys: the benchmarked model's key, the + `claude-haiku-4-5` background key, and nothing else carrying spend. + A run whose report was written by another model measured that + model: it is no row for this one, whatever it found — record it as + void in the pull request with its cost, and the other run is the + row (or re-run when it was the only one). - reconstruct the cost from the transcripts, root and `subagents/*.jsonl` together, at Anthropic's published list prices for the model: an `assistant` line is written once per content diff --git a/.claude/skills/test-plugin-harnessing/SKILL.md b/.claude/skills/test-plugin-harnessing/SKILL.md index c4f24d6..3b08e5a 100644 --- a/.claude/skills/test-plugin-harnessing/SKILL.md +++ b/.claude/skills/test-plugin-harnessing/SKILL.md @@ -47,9 +47,9 @@ the path of `/odd-observe`, `/odd-verify` or `/odd-status`. phase durations, its turn count and its **median turn**. Read it from `origin/main`, never from the working tree. - **A model whose median turn is small**, as the canonical - `vendor/name` id - default `deepseek/deepseek-v4.1-flash`; others - `google/gemini-3.7-flash`, `anthropic/claude-haiku-4.5`, - `openai/gpt-5.6-sol`: the scripts hand each CLI its own form. The published median is the instrument's + `vendor/name` id - default `openai/gpt-5.6-luna` on `copilot`; others + `deepseek/deepseek-v4.1-flash`, `google/gemini-3.7-flash`, + `anthropic/claude-haiku-4.5`: the scripts hand each CLI its own form. The published median is the instrument's precision: a model at 3 s per turn measures the harness, one at 20 s measures the provider. Prefer the fastest row in the table for that CLI, whatever its findings score — this is not a quality test. @@ -126,7 +126,7 @@ the path of `/odd-observe`, `/odd-verify` or `/odd-status`. taken another. Its whole surface, so `--help` has nothing to add: `--cli` (default - `opencode`), `--model` (default `deepseek/deepseek-v4.1-flash`), + `copilot`), `--model` (default `openai/gpt-5.6-luna`), `--tag`, `--phase`, one of `--prompt` / `--prompt-file`, `--out`, plus `--end-pattern` (a regular expression over the run's own lines — opencode's log, claude's transcripts, @@ -171,8 +171,9 @@ the path of `/odd-observe`, `/odd-verify` or `/odd-status`. rewritten `opencode.json` — a lab dirty in any other way is refused before the launch), the fake user scope is synced from the branch's deploy and checked identical (`--scope :`, - repeatable; opencode's two pairs are the default, the other CLIs - state theirs), `--scratch ` is cleared when given (the CLI's + repeatable; copilot syncs nothing - its deploy lives in the clone - + opencode's two pairs are known to the script, claude states its + own), `--scratch ` is cleared when given (the CLI's scratch directory; nothing outside the study is touched otherwise), the measurement above is launched (once more when it exits non-zero within 30 s — a launch that died measured nothing; `SAMPLE diff --git a/.claude/skills/test-plugin-harnessing/scripts/analyze_run.py b/.claude/skills/test-plugin-harnessing/scripts/analyze_run.py index 551bc2f..6d4fc3a 100755 --- a/.claude/skills/test-plugin-harnessing/scripts/analyze_run.py +++ b/.claude/skills/test-plugin-harnessing/scripts/analyze_run.py @@ -359,11 +359,11 @@ def main() -> int: cli, run_id, stdout, usage = args.cli, args.run_id, args.stdout, args.usage if args.record: record = json.loads(Path(args.record).read_text()) - cli = cli or record.get("cli") or "opencode" + cli = cli or record.get("cli") or "copilot" run_id = run_id or record.get("run_id") stdout = stdout or record.get("stdout") usage = usage or record.get("usage") - cli = cli or "opencode" + cli = cli or "copilot" if not run_id: print("no run id - pass --run-id or --record", file=sys.stderr) return 1 diff --git a/.claude/skills/test-plugin-harnessing/scripts/measure_phase.py b/.claude/skills/test-plugin-harnessing/scripts/measure_phase.py index 7450d8a..81f49b0 100755 --- a/.claude/skills/test-plugin-harnessing/scripts/measure_phase.py +++ b/.claude/skills/test-plugin-harnessing/scripts/measure_phase.py @@ -18,7 +18,7 @@ - a run that dies looks exactly like a run that thinks, so the watch fails loudly instead of returning a fast, wrong time. - measure_phase.py --cli opencode --model deepseek/deepseek-v4.1-flash --tag d1 \ + measure_phase.py --cli copilot --model openai/gpt-5.6-luna --tag l1 \ --phase preflight --prompt-file mission.txt --out /tmp/study The model is the canonical `vendor/name` id whatever the CLI; each CLI is @@ -305,13 +305,13 @@ def main() -> int: ap.add_argument( "--cli", choices=CLIS, - default="opencode", - help="the CLI the run is driven by (default opencode)", + default="copilot", + help="the CLI the run is driven by (default copilot)", ) ap.add_argument( "--model", - default="deepseek/deepseek-v4.1-flash", - help="the canonical vendor/name model id (default deepseek/deepseek-v4.1-flash)", + default="openai/gpt-5.6-luna", + help="the canonical vendor/name model id (default openai/gpt-5.6-luna)", ) ap.add_argument("--tag", required=True, help="names this measurement's files") ap.add_argument( diff --git a/.claude/skills/test-plugin-harnessing/scripts/run_samples.py b/.claude/skills/test-plugin-harnessing/scripts/run_samples.py index 1711ef7..ff4b5a1 100755 --- a/.claude/skills/test-plugin-harnessing/scripts/run_samples.py +++ b/.claude/skills/test-plugin-harnessing/scripts/run_samples.py @@ -2,7 +2,7 @@ """Run a study's samples in order, each on its lab branch, and journal them. python3 run_samples.py --lab --fake-home --out \\ - --cli opencode --model deepseek/deepseek-v4.1-flash --phase whole \\ + --cli copilot --model openai/gpt-5.6-luna --phase whole \\ base1=lab-main:mission.txt after1=lab-after:mission.txt \\ after2=lab-after:mission.txt base2=lab-main:mission.txt @@ -47,10 +47,13 @@ HERE = Path(__file__).resolve().parent RELAUNCH_WITHIN = 30 # seconds: a run gone before this measured nothing -# what the opencode deploy in the lab writes, and where the fake user scope -# expects it (launch-llms-benchmark step 3 states the scopes per CLI); the -# other CLIs' scopes are passed as --scope : +# what a CLI's deploy in the lab writes, and where the fake user scope +# expects it (launch-llms-benchmark step 3 states the scopes per CLI): +# copilot reads the deploy in the clone and syncs nothing, opencode also +# reads the user scope; claude's scopes are passed as +# --scope : DEFAULT_SCOPES = { + "copilot": [], "opencode": [ (".agents/skills", ".claude/skills"), (".opencode/agents", ".claude/agents"), @@ -218,20 +221,20 @@ def main() -> int: ap.add_argument("--lab", required=True, help="the lab clone the runs launch in") ap.add_argument("--fake-home", required=True, help="the HOME the runs see") ap.add_argument("--out", required=True, help="the study directory") - ap.add_argument("--cli", default="opencode", choices=CLIS) + ap.add_argument("--cli", default="copilot", choices=CLIS) ap.add_argument( "--scope", action="append", metavar="LAB_PATH:HOME_PATH", help="a deployed directory to sync into the fake home, both paths relative " - "(repeatable; the opencode pairs are the default, the other CLIs need theirs stated)", + "(repeatable; copilot syncs nothing, opencode's pairs are known, claude states its own)", ) ap.add_argument( "--scratch", metavar="DIR", help="the CLI's scratch directory to clear before each sample (none by default)", ) - ap.add_argument("--model", default="deepseek/deepseek-v4.1-flash") + ap.add_argument("--model", default="openai/gpt-5.6-luna") ap.add_argument( "--phase", default="whole", @@ -275,6 +278,7 @@ def main() -> int: raise SystemExit(f"no such mission file: {mission}") check_pair_order(samples) scopes = list(DEFAULT_SCOPES.get(args.cli, [])) + known = args.cli in DEFAULT_SCOPES for pair in args.scope or []: if ":" not in pair: raise SystemExit(f"--scope takes :, got {pair!r}") @@ -286,7 +290,7 @@ def main() -> int: f"never absolute, never through ..: got {pair!r}" ) scopes.append((src, dst)) - if not scopes: + if not scopes and not known: raise SystemExit(f"--cli {args.cli} needs its --scope pairs stated") branches = git(lab, "branch", "--format=%(refname:short)").splitlines() tips: dict[str, str] = {} diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 10edf25..b5a0b2f 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -15,40 +15,40 @@ One row per model and CLI, always its latest run. | Rank | Model | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3` | opencode | 1.12.0 | 18 / 19 | 12 / 4 / 3 | 12m23s | $1.98 | $0.110 | -| **#2** | `deepseek/deepseek-v4.1-flash` | opencode | 1.12.0 | **10 / 10** | 5 / 3 / 2 | 9m54s | $0.20 | $0.020 | -| **#3** | `z-ai/glm-5.3-flashx` | opencode | 1.12.0 | 12 / 14 | 3 / 6 / 3 | 16m02s | $0.42 | $0.035 | -| **#4** | `openai/gpt-5.6-luna` | copilot | 1.12.0 | 6 / 7 | 3 / 3 / 1 | **7m02s** | **$0.10** | **$0.016** | -| **#5** | `google/gemini-3.7-flash` | opencode | 1.12.0 | **7 / 7** | 3 / 3 / 1 | 8m34s | $0.87 | $0.124 | -| **#6** | `openai/gpt-5.6-terra` | copilot | 1.12.0 | 14 / 15 | 11 / 3 / 1 | 14m50s | $2.47 | $0.176 | -| **#7** | `openai/gpt-5.6-sol` | copilot | 1.12.0 | 11 / 12 | 8 / 3 / 1 | 9m39s | $3.12 | $0.284 | -| **#8** | `google/gemini-3.8-flash` | opencode | 1.12.0 | **10 / 10** | 5 / 4 / 1 | 14m59s | $1.96 | $0.196 | -| **#9** | `anthropic/claude-opus-5` | claude | 1.12.0 | 18 / 19 | 12 / 6 / 1 | 21m55s | $6.33 | $0.352 | -| **#10** | `anthropic/claude-fable-5.1` | claude | 1.12.0 | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#11** | `z-ai/glm-5.3-flash` | opencode | 1.12.0 | 13 / 14 | 7 / 4 / 2 | 66m43s | $0.24 | $0.018 | -| **#12** | `qwen/qwen3.8-27b` | opencode | 1.12.0 | 14 / 16 | 10 / 3 / 3 | 43m52s | $2.09 | $0.150 | -| **#13** | `qwen/qwen3.8-max-0902` | opencode | 1.12.0 | **14 / 14** | 8 / 3 / 3 | 66m09s | $2.88 | $0.206 | -| **#14** | `anthropic/claude-sonnet-5` | claude | 1.12.0 | **5 / 5** | 2 / 2 / 1 | 12m20s | $2.55 | $0.511 | +| **#1** | `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | +| **#2** | `openai/gpt-5.6-luna` | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | +| **#3** | `openai/gpt-5.6-terra` | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | +| **#4** | `openai/gpt-5.6-sol` | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | +| **#5** | `google/gemini-3.7-flash` | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | +| **#6** | `z-ai/glm-5.3` | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | +| **#7** | `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | +| **#8** | `google/gemini-3.8-flash` | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | +| **#9** | `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | +| **#10** | `anthropic/claude-opus-5` | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | +| **#11** | `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | +| **#12** | `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#13** | `anthropic/claude-sonnet-5` | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#14** | `qwen/qwen3.8-27b` | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens | Model | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3` | opencode | 1.12.0 | 1m28s | 2m01s | 8m54s | 46 | 6.1s | 4.2M | 139k | 4.0M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | opencode | 1.12.0 | 1m27s | 2m01s | 6m26s | 48 | 4.2s | 4.1M | 76k | 3.8M | 4/4 | -| `z-ai/glm-5.3-flashx` | opencode | 1.12.0 | 3m32s | 2m02s | 10m28s | 39 | 13.8s | 3.2M | 84k | 3.0M | 4/4 | -| `openai/gpt-5.6-luna` | copilot | 1.12.0 | 1m03s | 2m01s | 3m58s | 38 | 3.2s | 2.5M | 15k | 2.5M | 4/4 | -| `google/gemini-3.7-flash` | opencode | 1.12.0 | 2m11s | 2m02s | 4m21s | 71 | 4.3s | 4.2M | 31k | 3.5M | 4/4 | -| `openai/gpt-5.6-terra` | copilot | 1.12.0 | 1m39s | 2m00s | 11m11s | 64 | 3.9s | 6.5M | 54k | 6.5M | 4/4 | -| `openai/gpt-5.6-sol` | copilot | 1.12.0 | 1m38s | 2m00s | 6m01s | 53 | 4.6s | 3.9M | 22k | 3.8M | 4/4 | -| `google/gemini-3.8-flash` | opencode | 1.12.0 | 4m26s | 2m02s | 8m31s | 135 | 3.1s | 10.7M | 33k | 9.2M | 4/4 | -| `anthropic/claude-opus-5` | claude | 1.12.0 | 3m36s | 2m00s | 16m19s | 57 | 8.6s | 6.3M | 59k | 6.3M | 4/4 | -| `anthropic/claude-fable-5.1` | claude | 1.12.0 | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | opencode | 1.12.0 | 11m45s | 2m00s | 52m58s | 46 | 48.1s | 4.0M | 115k | 2.5M | 4/4 | -| `qwen/qwen3.8-27b` | opencode | 1.12.0 | 10m51s | 2m01s | 31m00s | 80 | 24.8s | 11.5M | 121k | 10.7M | 4/4 | -| `qwen/qwen3.8-max-0902` | opencode | 1.12.0 | 9m00s | 2m02s | 55m07s | 47 | 51.1s | 5.5M | 153k | 5.2M | 4/4 | -| `anthropic/claude-sonnet-5` | claude | 1.12.0 | 2m11s | 2m00s | 8m09s | 67 | 2.1s | 7.5M | 43k | 7.5M | 4/4 | +| `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-5.6-luna` | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `openai/gpt-5.6-terra` | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `openai/gpt-5.6-sol` | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `google/gemini-3.7-flash` | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | +| `z-ai/glm-5.3` | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `google/gemini-3.8-flash` | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | +| `anthropic/claude-opus-5` | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | +| `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `anthropic/claude-sonnet-5` | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `qwen/qwen3.8-27b` | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -58,9 +58,9 @@ design. **How to read the table** -- **Rank** weighs findings, cost and duration together. It is decided in each row's pull request, never computed: findings alone would rank a 67-minute run first, duration alone rewards whoever gives up soonest, cost alone rewards whoever barely looks. Adding a model re-sorts the whole table. +- **Rank** weighs findings, cost and duration together, cost and duration the heavier since the 1.13.0 campaign. It is decided in each row's pull request, never computed: findings alone would rank a 67-minute run first, duration alone rewards whoever gives up soonest, cost alone rewards whoever barely looks. Adding a model re-sorts the whole table. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. -- **Telemetry / Perf / Behavior** splits the reported findings by kind. +- **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. - **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. diff --git a/tests/skills/test-plugin-harnessing/test_run_samples.py b/tests/skills/test-plugin-harnessing/test_run_samples.py index 226065e..c1d866f 100644 --- a/tests/skills/test-plugin-harnessing/test_run_samples.py +++ b/tests/skills/test-plugin-harnessing/test_run_samples.py @@ -315,6 +315,14 @@ def test_a_missing_scope_source_is_refused_and_scratch_is_cleared_only_when_name lab, kit, out, f"s4=lab-main:{kit['mission']}", extra=("--cli", "claude") ) assert p.returncode == 1 and "--scope" in p.stderr + # copilot reads the deploy in the clone: no user-scope pair to sync, nothing refused + p = run_samples( + lab, kit, out, f"s5=lab-main:{kit['mission']}", extra=("--cli", "copilot") + ) + assert p.returncode == 0, p.stderr + p.stdout + assert "--cli copilot" in " ".join( + json.loads((out / "s5.record.json").read_text())["argv"] + ) def test_a_sample_spec_needs_all_three_parts():