Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 29 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,47 @@
[简体中文](README.zh-CN.md) · English

# amber-commandcode

> ⚠️ **更正(2026-10-02,另一项)**:防御轴的一案 A-d511f9e8 在所有车道上改记 NA(考场判的不是考生交付的文件,判分还要求了题面没写的事)。分母不变,**过案数不变**,每条道的总分都带 `'`。本仓各期成绩表里这一格请按 NA 读,其余内容保留原样,以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02-a-d511f9e8.md)为准。
> ⚠️ **Correction (2026-10-02, second)**: one defense-axis case, A-d511f9e8, is now NA on every lane (the exam room did not grade the file the candidate delivered, and the grader asks for something the task text does not say). The denominator and the **number of passed cases do not change**; every lane's total now carries `'`. In this repo's issue tables, read that cell as NA. Everything else stays as published; the [correction notice](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02-a-d511f9e8.en.md) governs.

> ⚠️ **更正(2026-10-02)**:以下考卷在作答时越出考卷、接触了判分材料,不计胜负。deepseek-v4.1-flash @ CommandCode(high 档) 有 2 张卷(A-61f7ad01、A-24bcf707)改记 NA,成绩 17/23∅ → **15'/23∅**;同道 none 档 有 1 张卷(A-a5608487)改记 NA,成绩 17/23 → **16'/23**;同道 medium 档 有 1 张卷(A-a5608487)改记 NA,成绩 15/23 → **14'/23**;mimo-v2.6-pro @ CommandCode 有 1 张卷(A-a5608487)改记 NA,成绩 17'/24 → **16'/24**;space-bunny-alpha @ CommandCode 有 2 张卷(A-a5608487、A-be92627f)改记 NA,成绩 15/24 → **13'/24**;grok-4.7 @ CommandCode(未完赛,不定级)的 A-be92627f、A-a5608487 两格由 ✓ 改记 NA;另有一格存疑、未能裁定:mimo-v2.6-flash 的 A-a5608487(发布为「过」),见声明。原因是考场隔离缺陷,责任在我们。本页其余内容保留原样,以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02.md)为准。
> ⚠️ **Correction (2026-10-02)**: the papers below were answered by a candidate that stepped outside its paper and touched grading material; they count neither as a pass nor as a fail. deepseek-v4.1-flash @ CommandCode (band high): 2 papers (A-61f7ad01, A-24bcf707) now NA, score 17/23∅ → **15'/23∅**; same lane, band none: 1 paper (A-a5608487) now NA, score 17/23 → **16'/23**; same lane, band medium: 1 paper (A-a5608487) now NA, score 15/23 → **14'/23**; mimo-v2.6-pro @ CommandCode: 1 paper (A-a5608487) now NA, score 17'/24 → **16'/24**; space-bunny-alpha @ CommandCode: 2 papers (A-a5608487, A-be92627f) now NA, score 15/24 → **13'/24**; the A-be92627f and A-a5608487 cells of grok-4.7 @ CommandCode (incomplete, unranked) change from ✓ to NA; one more cell is doubtful and could not be settled: A-a5608487 for mimo-v2.6-flash (published as a pass); see the notice. The cause was an isolation defect in our exam setup; the fault is ours. The rest of this page stays as published; where they differ, the [correction notice](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02.en.md) governs.

用私有题库 **AMBER** 实测 CommandCode(可从它这里购买各家模型的 API 额度,api.commandcode.ai)在售模型:只公开成绩,不公开题目。
English: [README.en.md](README.en.md)
Public periodic [AMBER](https://github.com/getaskclaw/amber) benchmark results of models served by CommandCode (api.commandcode.ai), where you buy API access to models from different vendors. **Cases stay private; results are public.**

## 这是什么
## What this is

我们定期让同一批模型考试,只发成绩单;考题和评分细节不公开。
We run the same private exam for models on a schedule and publish only the scorecard; the questions and scoring details stay private.

![最新一期榜单构成(2026-W39)](results/assets/2026-W39-board.zh.png?v=20260923)
![Latest board composition (2026-W39)](results/assets/2026-W39-board.en.png?v=20260923)

- 「道」= 一个模型名在某家平台的售卖入口(也可读作渠道);「案」= 一道题;「卷」= 一场考试记录;「档」= effort 档,思考力度档位。
- 每期一篇 `results/YYYY-Www.md`:同题、同考试程序(harness,跑考试并记分的程序),对目标模型跑全库;同名模型跨平台并排。
- 一期固定报告:题集规模与哈希、每案找茬分(d2 分:我们对模型犯错种类与严重度的打分,算法不公开;过/不过关另看每案通过线)与通过/失败、终端终态(程序跑完时的退出状态)、token 用量(若渠道上报)与时延、环境指纹、按证据纪律写的定性裁决。
- 题目、oracle(判分器)、transcript(答题全过程记录)、中间产物**永不公开**(见下「发布纪律」)。
- 姐妹仓:[amber-deepseek](https://github.com/getaskclaw/amber-deepseek)(DeepSeek 官方道)、[amber-opencode](https://github.com/getaskclaw/amber-opencode)(OpenCode Go 道)、[amber-gpt](https://github.com/getaskclaw/amber-gpt)、[amber-crof](https://github.com/getaskclaw/amber-crof)、[amber-ollama](https://github.com/getaskclaw/amber-ollama)、[amber-devin](https://github.com/getaskclaw/amber-devin)、[amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy)(WorkBuddy ACP 道)、[amber-doubao](https://github.com/getaskclaw/amber-doubao)、[amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato)、[amber-kimi](https://github.com/getaskclaw/amber-kimi)、[amber-stepfun](https://github.com/getaskclaw/amber-stepfun)。本仓的对照轴是**同名模型跨厂商对决**——同一个模型名在 CommandCode / OpenCode Go / DeepSeek 官方道上可能是不同端点,跨仓引用一律带日期与档位声明。
- AMBER 是 agentic 实战题库(施工/运维/审查/视觉/需求漂移——题中要求中途变化),规范与制题工具见 [getaskclaw/amber](https://github.com/getaskclaw/amber);考题本体私有。
- A "lane" is one vendor's shop/API for a model name; a "case" is one task; a "run" is one sitting; a "band" is the effort band, the thinking-effort setting.
- One `results/YYYY-Www.md` per issue: same cases, same exam program (the harness, which runs the exam and scores it), full library per model; the same model name across vendors side by side.
- Each issue pins: library size and hashes, per-case defect-hunt score and pass/fail (the d2 score — our score for mistake kind and severity; the algorithm is private, and pass/fail is decided by each case's pass bar), terminal states (how the run process exited), token usage (when the lane reports it) and latency, environment fingerprint, and a qualitative verdict written under evidence discipline.
- Cases, oracles (graders), transcripts (full answer logs) and intermediates are **never published**.
- Sister repos: [amber-deepseek](https://github.com/getaskclaw/amber-deepseek) (official DeepSeek lane), [amber-opencode](https://github.com/getaskclaw/amber-opencode) (OpenCode Go lane), [amber-gpt](https://github.com/getaskclaw/amber-gpt), [amber-crof](https://github.com/getaskclaw/amber-crof), [amber-ollama](https://github.com/getaskclaw/amber-ollama), [amber-devin](https://github.com/getaskclaw/amber-devin), [amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy) (WorkBuddy ACP lane), [amber-doubao](https://github.com/getaskclaw/amber-doubao), [amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato), [amber-kimi](https://github.com/getaskclaw/amber-kimi), [amber-stepfun](https://github.com/getaskclaw/amber-stepfun). This repo's comparison axis is **same-name cross-vendor duels** — the same model name on CommandCode / OpenCode Go / the official DeepSeek API can be a different endpoint, and every cross-repo citation carries an explicit date and band declaration.

## 发布纪律(红线)
## Publication red lines

1. 只发:分数与聚合、token 用量(若渠道上报)、速度、定性裁决。
2. 永不发:题目内容、oracle/判分器、transcript、考生工作区、任何能复原题面的中间产物。
3. 每期必钉:模型 ID、effort 档(思考力度档位)、日期(UTC)、harness 版本、每案内容哈希(bundle_sha,每题内容的哈希指纹)。哈希用于对照 [amber](https://github.com/getaskclaw/amber) 的公开哈希清单,自证题集未变。
4. 案号与题目结构属私有面:公开结果里案例只用稳定别名(A-xxxxxxxx,哈希派生)+ bundle 哈希作句柄;内部案号、变体名、题目描述永不出现。
5. 基调:这是社区实测,不是对厂商的攻击。数据说话,措辞克制。
1. Publish only: scores and aggregates, token usage (when reported), speed, qualitative verdicts.
2. Never publish: case content, oracles/graders, transcripts, candidate workspaces, anything that could reconstruct a case.
3. Every issue pins: model ID, effort band (the thinking-effort setting), date (UTC), harness version, per-case bundle hash — verifiable against the public hash index in [amber](https://github.com/getaskclaw/amber).
4. Case numbering is private: public matrices use stable aliases (A-xxxxxxxx, hash-derived) plus bundle hashes only.
5. Tone: community measurement, not vendor attacks.

## 一个方法论前提
## A methodological premise

同名模型、同 provider,两次跑也可能不同分——推理参数、负载、服务端版本都在漂;转发/聚合道还隔着一层上游路由,同名不一定是同一个端点。所以这里的一切结论都带日期与档位,且定期重测。单日数字是快照,不是定律。
Same model name, same provider, two runs can still score differently — inference parameters, load, and server-side versions drift. Relay/aggregator lanes add an upstream routing layer: the same name may not be the same endpoint. Every conclusion here is dated and banded, and we re-test periodically. A single day's number is a snapshot, not a law.

## 最新成绩 / Results index
## Latest results / Results index

成绩速读:「格」= 榜单里的一个计分格;「invalid」= 这场考试作废(多为考场侧问题),不计能力分;「挂起」= 暂不定论、待复核或重考;「案级计数」= 以案为单位汇总的成绩数(一道题可能有多卷);「三连考」= 一期连着考三条模型线;「腿」= 其中一条模型线;「车道冻结期」= 该渠道暂停补考的一段时间;「防御钉」= 防御类钉子题(刻意埋陷阱、专考模型犯错方式的难题);「五档天梯」= 同一模型在 5 个 effort 档上的成绩阶梯;「GA 当日」= 模型正式发布当天。
Reading aid: a "cell" is one scored box on the board; "invalid" means the sitting was voided (usually a bench-side problem) and does not count as a capability score; "held" means no verdict yet, pending review or re-exam; "case-level tally" counts by case (one case may have several runs); a "triple exam" runs three model legs back to back in one issue; a "leg" is one of those model lines; "lane freeze" means that lane is not taking re-exams for a while; "defense pins" are defense nail-down tasks (trap-laden tasks built to catch mistakes); the "five-band ladder" is the same model measured at five effort bands; "GA day" is the model's public release day.

| 期 | 内容 | 结论 |
| Issue | Content | Headline |
|---|---|---|
| [2026-W39](results/2026-W39.md)([EN](results/2026-W39.en.md)) | CommandCode 三连考:grok-4.7 / mimo-v2.6-pro / mimo-v2.6-flash;09-24 加餐 stealth/space-bunny-alpha | mimo-v2.6-pro **17/24**(1 invalid);mimo-v2.6-flash **13/24**(6 负;5 案 invalid 不定模型罪);grok-4.7 未完赛,7 案挂起待 09-29 原道重考;加餐 space-bunny-alpha **15/24**(stealth 免费窗,页内同端点并排) |
| [2026-W38 更正特刊](results/2026-W38-correction.md)([EN](results/2026-W38-correction.en.md)) | W38 全库复核:本仓 0 个计分格改判 · 14 个计分格挂起;另附三连考 mimo 两个模型成绩入档 | W37 五档天梯 14 卷挂起(high 档 2 格 + 案级计数),车道冻结期内不补考;加餐:mimo-v2.6-pro 判过 17 / 判负 6 / 挂起 1,mimo-v2.6-flash 判过 13 / 判负 6 / 挂起 5(A-be92627f 同一案在两个模型因不同原因失败,分别记账) |
| [2026-W37](results/2026-W37.md) | deepseek/deepseek-v4.1-flash 全库首考(23 案,GA 当日) | **17/23**;UI 搭建卷满分(第五个公开通过该案的成绩);防御钉 8/9 刷新全员纪录;同名跨厂商三家核对,确认都是真 v4.1;审查/视觉面是弱项 |
| [2026-W39](results/2026-W39.en.md)([中文](results/2026-W39.md)) | CommandCode triple exam: grok-4.7 / mimo-v2.6-pro / mimo-v2.6-flash; 09-24 extra sitting: stealth/space-bunny-alpha | mimo-v2.6-pro **17/24** (1 invalid); mimo-v2.6-flash **13/24** (6 fails; 5 invalid, no fault to the model); grok-4.7 incomplete, 7 cases held for the 09-29 same-lane re-exam; extra sitting space-bunny-alpha **15/24** (stealth free window, side by side in the page) |
| [2026-W38 correction notice](results/2026-W38-correction.en.md)([中文](results/2026-W38-correction.md)) | W38 full-library review: 0 cells reversed · 14 cells held here; plus the triple-exam mimo model scores filed | 14 W37 ladder papers held (2 high-band cells + case-level tallies), no re-exam while the lane is frozen; addendum: mimo-v2.6-pro 17 pass / 6 fail / 1 held, mimo-v2.6-flash 13 pass / 6 fail / 5 held (A-be92627f failed for different reasons on the two models — recorded separately) |
| [2026-W37](results/2026-W37.md) | deepseek/deepseek-v4.1-flash full-library debut (23 cases, GA day) | **17/23**; full marks on the UI-build paper (the fifth public pass on that case); 8/9 on the defense case that used to kill everyone; all three same-name vendors checked out as genuine v4.1; adversarial-review and vision are its weak faces |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

W37 row links only the Chinese results/2026-W37.md with no language marker (W39/W38 rows have one); mark it 中文 or add an .en page.


## 免责
## Disclaimer

与 CommandCode、DeepSeek(深度求索)无任何隶属/赞助关系。分数是特定周、特定档位的快照,不构成采购建议。
Not affiliated with or sponsored by CommandCode or DeepSeek. Scores are dated, band-specific snapshots, not purchasing advice.
Loading