diff --git a/README.md b/README.md index 652b6a8..86c6929 100644 --- a/README.md +++ b/README.md @@ -1,97 +1,99 @@ -# AMBER — 琥珀式封存的历史回放评测 +[简体中文](README.zh-CN.md) · English -`封进琥珀，重做当时的题。` +# AMBER — Sealed-in-Amber Historical Replay Evaluation -**一句话**：拿真实发生过的事故和任务，让 AI 模型只用当时手里的信息重做一遍。比如一次真实的运维故障：模型只能拿当时的监控和日志找根因，事后才揭晓的证据一律封存不给看。题目永不公开，但每题有哈希指纹，分数和指纹全部公开，谁都能核对题没被换、分不是编的。 +`Seal the scene in amber. Retake the exam of that moment.` -**案集** 24 案 · **规范** v0.2.2（草案） · **哈希索引** [v2026-09](hash-index/v2026-09.md) · **成绩仓** × 13 · **纪律** 只发分数，不发题 · **官网榜单** [askclaw.dev](https://askclaw.dev/) · English: [README.en.md](README.en.md) +**In one line**: we take real past incidents and tasks and have AI models redo them using only what people had in hand at the time. For example, a real operations outage: the model gets only the monitoring and logs of the time, and the evidence revealed afterwards stays sealed. The questions are never published, but every question has a hash fingerprint, and scores and fingerprints are all public, so anyone can check that the questions were not swapped and the scores were not made up. -> ⚠️ **更正**（原文与受影响车道清单见[榜单全文](docs/leaderboard-notes.md)） -> - **[2026-10-02](docs/corrections-2026-10-02.md)**：22 张已发布的考卷在作答时越出考卷、接触了判分材料，改记 NA（不计胜负）。责任在我们；2026-10-01 起已改为隔离容器作答。 -> - **[2026-10-02（另一项）](docs/corrections-2026-10-02-a-d511f9e8.md)**：防御轴的一案 A-d511f9e8 在所有车道上改记 NA。分母不变，**过案数不变**，每条道的总分都带 `'`。 -> - **[2026-10-07](docs/corrections-2026-10-07-a-24bcf707.md)**：运维轴的一案 A-24bcf707 在七条车道上由「未过」改记 NA（判分要求比题面多了一条）。分母不变，**总分和名次档不变**，不重考任何场次。 -> - 更早：[2026-09-18](docs/corrections-2026-09-18.md)（swe-2-low 实为 swe-2-high）· [2026-W38](docs/corrections-2026-W38.md)（全库复核，逐仓特刊） +**Library** 24 cases · **Spec** v0.2.2 (draft) · **Hash index** [v2026-09](hash-index/v2026-09.md) · **Result repos** × 13 · **Rule** scores always public, cases never · **Live board** [askclaw.dev](https://askclaw.dev/en/) -## 现在谁领先 +> ⚠️ **Corrections** (original text and the list of affected lanes: [leaderboard in full](docs/leaderboard-notes.en.md)) +> - **[2026-10-02](docs/corrections-2026-10-02.en.md)**: 22 published papers whose candidate stepped outside the paper and touched grading material are now NA (neither a pass nor a fail). The fault is ours; since 2026-10-01 papers are answered in isolated containers. +> - **[2026-10-02 (second)](docs/corrections-2026-10-02-a-d511f9e8.en.md)**: one defense-axis case, A-d511f9e8, is now NA on every lane. The denominator and the **number of passed cases do not change**; every lane's total now carries `'`. +> - **[2026-10-07](docs/corrections-2026-10-07-a-24bcf707.en.md)**: one ops-axis case, A-24bcf707, changes from a loss to NA on seven lanes (the grading asked for one more thing than the task text says). The denominator, **totals and rank tiers do not change**, and no sitting is re-run. +> - Earlier: [2026-09-18](docs/corrections-2026-09-18.en.md) (swe-2-low was actually swe-2-high) · [2026-W38](docs/corrections-2026-W38.en.md) (full-library review, per-repo notices) -![当前前五 2026-W41（含 2026-10-02 更正、2026-10-02 考试、2026-10-04 WorkBuddy 直连四条道、2026-10-06 五场新考、2026-10-07 品牌案重考一案与 A-d511f9e8 全车道挂起）](docs/images/top5-2026-w41b.png?v=20261007) +## Who leads now -| # | 总分 | 模型 @ 端点 | +![Current top five 2026-W41, including the 2026-10-02 correction, the 2026-10-02 tests, the four WorkBuddy direct lanes of 2026-10-04, the five new sittings of 2026-10-06, one brand-case re-sit of 2026-10-07 and A-d511f9e8 on hold for every lane](docs/images/top5-2026-w41b.en.png?v=20261007) + +| # | Total | Model @ endpoint | |---|---|---| -| 1 | 19'/24 | swe-2-max @ Devin
claude-sonnet-5-5 @ Anthropic 订阅道
claude-opus-5-5 @ Anthropic 订阅道 | -| 2 | 18'/24 | glm-5.3-flash @ Ollama Cloud
k3 @ Kimi
hy4-preview-f @ WorkBuddy ACP（W37）
deepseek-v4.1-flash @ WorkBuddy 直连（W40）
step-5-preview @ StepFun（W41） | -| 3 | 17'/24 | gpt-6-sol-900k @ OpenAI Codex
swe-2-high @ Devin
doubao-seed-evolving @ 火山方舟
glm-5.3-flash @ WorkBuddy 直连（W40）
gpt-5.6-luna @ OpenAI Codex | -| 4 | 16'/24 | gpt-5.6-luna-900k（high）@ OpenAI Codex
gpt-6-astra-900k @ OpenAI Codex
gpt-6.1-sol @ OpenAI Codex
claude-fable-5-1 @ Anthropic 订阅道
deepseek-v4-flash:0731 @ Ollama Cloud
swe-2-medium @ Devin
deepseek-v4.1-flash @ Ollama Cloud ▼
deepseek-flash @ DeepSeek ▼
mimo-v2.6-pro @ CommandCode ▼
hy4-preview-f @ WorkBuddy 直连（W40）
minimax-m3 @ WorkBuddy 直连（W40）
gpt-5.6-luna-900k @ OpenAI Codex（W41）
gpt-6-luna @ OpenAI Codex | -| 5 | 15'/24 | kimi-for-coding（K2.8）@ Kimi
gpt-6-luna-900k @ OpenAI Codex
swe-1-7-medium @ Devin
Qwen3.8-27B @ GoldenPotato 自部署
deepseek-flash @ OpenCode Go ▼ | +| 1 | 19'/24 | swe-2-max @ Devin
claude-sonnet-5-5 @ Anthropic subscription lane
claude-opus-5-5 @ Anthropic subscription lane | +| 2 | 18'/24 | glm-5.3-flash @ Ollama Cloud
k3 @ Kimi
hy4-preview-f @ WorkBuddy ACP (W37)
deepseek-v4.1-flash @ WorkBuddy direct (W40)
step-5-preview @ StepFun (W41) | +| 3 | 17'/24 | gpt-6-sol-900k @ OpenAI Codex
swe-2-high @ Devin
doubao-seed-evolving @ Volcengine Ark
glm-5.3-flash @ WorkBuddy direct (W40)
gpt-5.6-luna @ OpenAI Codex | +| 4 | 16'/24 | gpt-5.6-luna-900k (high) @ OpenAI Codex
gpt-6-astra-900k @ OpenAI Codex
gpt-6.1-sol @ OpenAI Codex
claude-fable-5-1 @ Anthropic subscription lane
deepseek-v4-flash:0731 @ Ollama Cloud
swe-2-medium @ Devin
deepseek-v4.1-flash @ Ollama Cloud ▼
deepseek-flash @ DeepSeek ▼
mimo-v2.6-pro @ CommandCode ▼
hy4-preview-f @ WorkBuddy direct (W40)
minimax-m3 @ WorkBuddy direct (W40)
gpt-5.6-luna-900k @ OpenAI Codex (W41)
gpt-6-luna @ OpenAI Codex | +| 5 | 15'/24 | kimi-for-coding (K2.8) @ Kimi
gpt-6-luna-900k @ OpenAI Codex
swe-1-7-medium @ Devin
Qwen3.8-27B @ GoldenPotato self-hosted
deepseek-flash @ OpenCode Go ▼ | -`'` = 其中有案暂不计分（NA），不算输；现在每条道都带，因为 A-d511f9e8 在所有车道上暂不计分。同名模型换个端点可能是另一个脑，所以成绩一律按「端点 × 名字」记。claude-opus-5-5 取 W40 重考一场（W39 首考 17'/24）：两场各考一次、考场不同，**不据此判断强弱**，见 [amber-claude W40](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)。▼ = 经 2026-10-02 更正下调。WorkBuddy 直连（W40）的四条道是 2026-10-04 隔离考场的新场次，和 W37（ACP 道）、W39（直连道场次）不可逐格对比；hy4-preview-f 在榜上有 ACP（W37，18'）和直连（W40，16'）两行，是两条不同的道、不同的考场，**不据此判断强弱**，见 [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.md)。2026-10-06（W41）的五场新考也在隔离考场：gpt-5.6-luna、gpt-5.6-luna-900k（W41）、gpt-6-luna 是三条新入榜的道；step-5-preview 与 gpt-6-luna-900k 榜上改取 2026-10-06 重考一场：gpt-6-luna-900k 是 15'/24；step-5-preview 当场是 17'/24，品牌案 A-d9b79b46 的题面修订后（见 [hash-index](hash-index/v2026-09.md) 的 2026-10-07 换版说明），该案于 2026-10-07 单案重考通过，现为 18'/24（3 NA：A-d511f9e8 挂起，另两案撞考场时间上限）。step-5-preview 在 W38 首考的 15'（经 2026-10-02 更正下调、曾标 ▼）不再作榜上成绩，所以不再带 ▼。这些场次和 W37–W39 的场次不可逐格对比；step-5-preview 与 gpt-6-luna-900k 的新旧两场各考一次、考场不同，**不据此判断强弱**。榜上有两条 gpt-5.6-luna-900k，是两条不同的道：「（high）」一条是 W37 基线（high 档）的成绩，「（W41）」一条是 2026-10-06 隔离考场的新场次。W41 各场的逐案成绩见 [amber-gpt W41](https://github.com/getaskclaw/amber-gpt/blob/main/results/2026-W41.md)（gpt-5.6-luna、gpt-5.6-luna-900k、gpt-6-luna、gpt-6-luna-900k 四场）和 [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.md)（step-5-preview）。各车道出处、脚注、冻结道：[榜单全文](docs/leaderboard-notes.md)。 +`'` = at least one case is NA (not scored; it does not count as a loss). Every total carries it now, because A-d511f9e8 is NA on every lane. The same model name on a different endpoint can be a different brain, so scores are always recorded per endpoint × name. claude-opus-5-5 is shown with its W40 re-test (its W39 first test was 17'/24): each test was taken once, in a different exam room, so this does **not** show that it got stronger or weaker; see [amber-claude W40](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.en.md). ▼ = moved down by the 2026-10-02 correction. The four WorkBuddy direct (W40) lanes are new sittings of 2026-10-04 in an isolated exam room, not comparable cell by cell with W37 (ACP lane) or W39 (direct-lane sitting); hy4-preview-f has two rows on the board, ACP (W37, 18') and direct (W40, 16'): different lanes in different exam rooms, so this does **not** show that it got stronger or weaker; see [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.en.md). The five new sittings of 2026-10-06 (W41) were also taken in the isolated exam room: gpt-5.6-luna, gpt-5.6-luna-900k (W41) and gpt-6-luna are three new lanes on the board; for step-5-preview and gpt-6-luna-900k the board now takes the 2026-10-06 re-sit: gpt-6-luna-900k is 15'/24; step-5-preview scored 17'/24 in that sitting, and after the prompt of the brand case A-d9b79b46 was revised (see the 2026-10-07 re-issue note in the [hash index](hash-index/v2026-09.md)) that case was re-sat on 2026-10-07 and passed, so it now stands at 18'/24 (3 NA: A-d511f9e8 on hold, and two cases that hit the exam-room time cap). The 15' that step-5-preview scored at its W38 first test (moved down by the 2026-10-02 correction, previously marked ▼) is no longer its board entry, so it no longer carries ▼. These sittings are not comparable cell by cell with those of W37 to W39; for step-5-preview and gpt-6-luna-900k, each of the two sittings was taken once, in a different exam room, so this does **not** show that either got stronger or weaker. There are two gpt-5.6-luna-900k rows, and they are different lanes: the one marked (high) is the W37 baseline (high band) result, the one marked (W41) is a new 2026-10-06 sitting in the isolated exam room. Case-by-case results of the W41 sittings: [amber-gpt W41](https://github.com/getaskclaw/amber-gpt/blob/main/results/2026-W41.en.md) (gpt-5.6-luna, gpt-5.6-luna-900k, gpt-6-luna and gpt-6-luna-900k) and [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.en.md) (step-5-preview). Sources per lane, footnotes and frozen lanes: [leaderboard in full](docs/leaderboard-notes.en.md). -## 同分不同款 +## Same score, different shape -总分相同的车道，强项和短板可以完全不同。点越大、颜色越深，完成度越高；NA = 该轴没有可计分的案，不是零分。 +Lanes with the same total can have very different strengths and weak spots. A bigger, darker dot = higher completion; NA = no scorable case on that axis, not a zero. -![完成度矩阵：榜上总分 18 以上的 8 条车道，十轴，24 案，2026-10-07](docs/images/completion-matrix-top8-2026-w41.png?v=20261007) +![Completion matrix: the 8 lanes with 18+ on the board, ten axes, 24 cases, 2026-10-07](docs/images/completion-matrix-top8-2026-w41.en.png?v=20261007) -*图中 8 条 = 榜上总分 18 以上的全部车道（2026-10-07 的库）。hy4-preview-f 取 ACP（W37）一行；同名的直连（W40，16'）总分不到 18，不在图内。deepseek-v4.1-flash @ WorkBuddy 直连（W40）的 UI 轴只有一案 A-d9b79b46，该案挂起、记 NA，所以那一格是 NA 不是零分；它的视觉格取考场修复后的重考卷，见 [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.md)。防御轴按 A-d511f9e8 在所有车道上 NA 的口径算，各格只剩另一案的读数（两案都没有可计分结果的格是 NA）。step-5-preview（W41）的 UI 格是 1.00：品牌案 A-d9b79b46 在题面修订后于 2026-10-07 单案重考通过（12/12），见 [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.md)；它的防御和归因两格是 NA，因为这两轴各有一案撞考场时间上限、没有可计分结果。* +*The 8 lanes = every lane with 18 or more on the board (database of 2026-10-07). hy4-preview-f is its ACP (W37) row; the direct (W40, 16') lane of the same name has a total below 18 and is not in the figure. On deepseek-v4.1-flash @ WorkBuddy direct (W40), the UI axis holds one case, A-d9b79b46, which is on hold and counted as NA, so that cell is NA, not a zero; its vision cell is taken from the re-sit paper after the exam-room fix, see [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.en.md). The defense axis is computed with A-d511f9e8 as NA on every lane, so each cell there reflects only the other case (a cell where neither case has a scorable result is NA). On step-5-preview (W41), the UI cell is 1.00: the brand case A-d9b79b46 was re-sat alone on 2026-10-07 after its prompt was revised and passed (12/12), see [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.en.md); its defense and attribution cells are NA because on each of those two axes one case hit the exam-room time cap and has no scorable result.* -**读法**：动手干活的「施工面」几乎全满，差别在「判断面」。每格 = 该轴全部案子的完成度（0–1）。官方榜 [askclaw.dev](https://askclaw.dev/) 也用这十个轴。 +**How to read it**: the hands-on "building" axes are almost full for everyone; the differences are on the "judging" side. Each cell = the lane's completion (0–1) across that axis's cases. The official board on [askclaw.dev](https://askclaw.dev/en/) uses the same ten axes. -| 面 | 轴 | 考什么 | +| Side | Axis | What it tests | |---|---|---| -| 施工面 | 编码 | 照需求把功能写对 | -| | 交付 | 做完要交卷，没产出就是 0 | -| | 运维 | 照规程干脏活，一步不省 | -| | 需求 | 客户要 A 不要 B，交上去得是 A | -| | 收敛 | 真干完，还是绕圈装忙 | -| 判断面 | UI | 照设计稿做页面，像素级验收 | -| | 视觉 | 给真截图挑毛病 | -| | 防御 | 堵死校验器的漏洞，还不误伤好人 | -| | 归因 | 把每条毛病对到正确根因 | -| | 审查 | 给别人的交付物挑错，漏看和冤枉都扣分 | +| Building | Coding | implement the spec correctly | +| | Delivery | done ≠ handed in: no artifact means 0 | +| | Ops | follow the runbook, no skipped steps | +| | Requirements | the client asked for A, not B: ship A | +| | Convergence | real finish, or busywork loops | +| Judging | UI | build the page to the mock, pin-level | +| | Vision | spot defects in real screenshots | +| | Defense | plug validator holes without turning away legit input | +| | Attribution | pin each defect to the right root cause | +| | Review | find real defects in someone's work; misses and false alarms both cost | -各轴的详细读法、每条车道的数字：[十个轴与完成度矩阵](docs/axes.md)。 +Per-axis reading and each lane's numbers: [the ten axes and the completion matrix](docs/axes.en.md). -## 它怎么运作 +## How it works -私有题库 + 公开成绩：题目永不公开，分数与哈希永远可查。评分标准在看到任何输出之前就已冻结，全程留档。 +A private case library plus public scores: the cases never go public, the scores and hashes always do. The rubric is frozen before anyone sees the output, and everything is archived. -![AMBER 是什么](docs/images/what-is-amber-2026-10.png?v=20261003) +![What is AMBER](docs/images/what-is-amber-2026-10.en.png?v=20261003) -一期成绩像一场考试：出卷封存、净室应考、逐卷验脑、脱敏发布、人人可核对。 +A result is produced like an exam: papers sealed at authoring, sat in a clean room, audited paper by paper, published redacted, checkable by anyone. -![一期成绩怎么炼成](docs/images/trust-chain.png?v=20260915b) +![How a result is produced](docs/images/trust-chain.en.png?v=20260915b) -另有三个发现（思考档位、token 账单、耗时与分数的关系）：[docs/findings-effort-and-speed.md](docs/findings-effort-and-speed.md)。 +Three more findings (effort bands, token bills, wall time vs. score): [docs/findings-effort-and-speed.en.md](docs/findings-effort-and-speed.en.md). -## 成绩仓 +## Result repos -成绩与题目分开发布：结果公开，题目永不公开。别名 + bundle 哈希逐案对照本仓的公开哈希索引。 +Results are public, cases never are. Aliases and bundle hashes are checked case by case against this repo's public hash index. -[amber-claude](https://github.com/getaskclaw/amber-claude)（Claude）· [amber-gpt](https://github.com/getaskclaw/amber-gpt)（GPT）· [amber-devin](https://github.com/getaskclaw/amber-devin)（Devin）· [amber-kimi](https://github.com/getaskclaw/amber-kimi)（Kimi）· [amber-deepseek](https://github.com/getaskclaw/amber-deepseek)（DeepSeek）· [amber-ollama](https://github.com/getaskclaw/amber-ollama)（Ollama Cloud）· [amber-opencode](https://github.com/getaskclaw/amber-opencode)（OpenCode Go）· [amber-commandcode](https://github.com/getaskclaw/amber-commandcode)（CommandCode）· [amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy)（WorkBuddy）· [amber-doubao](https://github.com/getaskclaw/amber-doubao)（豆包）· [amber-stepfun](https://github.com/getaskclaw/amber-stepfun)（阶跃星辰）· [amber-nous](https://github.com/getaskclaw/amber-nous)（Nous Portal）· [amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato)（社区自部署 Qwen3.8-27B） +[amber-claude](https://github.com/getaskclaw/amber-claude) (Claude) · [amber-gpt](https://github.com/getaskclaw/amber-gpt) (GPT) · [amber-devin](https://github.com/getaskclaw/amber-devin) (Devin) · [amber-kimi](https://github.com/getaskclaw/amber-kimi) (Kimi) · [amber-deepseek](https://github.com/getaskclaw/amber-deepseek) (DeepSeek) · [amber-ollama](https://github.com/getaskclaw/amber-ollama) (Ollama Cloud) · [amber-opencode](https://github.com/getaskclaw/amber-opencode) (OpenCode Go) · [amber-commandcode](https://github.com/getaskclaw/amber-commandcode) (CommandCode) · [amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy) (WorkBuddy) · [amber-doubao](https://github.com/getaskclaw/amber-doubao) (Doubao) · [amber-stepfun](https://github.com/getaskclaw/amber-stepfun) (StepFun) · [amber-nous](https://github.com/getaskclaw/amber-nous) (Nous Portal) · [amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato) (community self-hosted Qwen3.8-27B) -## 常见问题 +## FAQ -**题目不公开，凭什么信分数？** 靠可验证的过程：净室考场、逐卷验脑（替身 = 整卷作废）、收卷闸、别名 + 哈希公开、harness 版本每期钉死（现役 [Hermes](https://github.com/NousResearch/hermes-agent)）。考场条件的变更逐条公开，最近一次：[方法变更 2026-10-01](docs/method-change-2026-10-01.md)。 +**If cases are private, why trust the scores?** Trust comes from a verifiable process: clean-room exams, per-paper wire audits (a substitute call voids the paper), a closing gate, alias + hash publishing, and a harness pin in every issue (current: [Hermes](https://github.com/NousResearch/hermes-agent)). Changes to exam conditions are published one by one; the latest: [method change 2026-10-01](docs/method-change-2026-10-01.en.md). -**为什么不公开题目？** 公开题库会被训练数据吃掉，分数通胀、无法审计。题目私有，泄漏状态才可核查。 +**Why keep cases private at all?** Public corpora get eaten by training data; scores inflate and become unauditable. Private cases make leak status checkable. -**两个模型的分能直接比吗？** 只在同档、同题集版本、尽量同日时可比。跨周对比必须带日期和档位。 +**Can I compare two models' scores directly?** Only at the same band and library version, ideally the same day. Cross-week comparisons must carry the date and band. -**能复现或加入吗？** 目前不能：仅凭公仓还跑不通一次合规运行（见 [PLAN.md](PLAN.md)）。追结果请订阅成绩仓；方法论问题欢迎开 issue。 +**Can I reproduce or join?** Not from the public repos alone yet (see [PLAN.md](PLAN.md)). Follow the result repos for scores; methodology questions are welcome as issues. -## 规范与文档 +## Spec and docs -- [AMBER-Core-Specification.md](AMBER-Core-Specification.md) — 规范本体（English） -- [protocols/distribution.md](protocols/distribution.md) · [protocols/stability.md](protocols/stability.md) — 分发协议、稳定性协议 -- [hash-index/v2026-09.md](hash-index/v2026-09.md) — 公开哈希索引（24 案） -- [PLAN.md](PLAN.md) · [CONTRIBUTING.md](CONTRIBUTING.md) — 路线图；贡献规则（**本仓绝不接收案件内容**） -- [docs/](docs/README.md) — 全部文档索引（榜单全文、十个轴、研究笔记、更正声明、案件清单 schema 与校验器） +- [AMBER-Core-Specification.md](AMBER-Core-Specification.md) — the normative spec +- [protocols/distribution.md](protocols/distribution.md) · [protocols/stability.md](protocols/stability.md) — distribution and stability protocols +- [hash-index/v2026-09.md](hash-index/v2026-09.md) — the public hash index (24 cases) +- [PLAN.md](PLAN.md) · [CONTRIBUTING.md](CONTRIBUTING.md) — roadmap; contribution rules (**this repo never accepts case content**) +- [docs/](docs/README.en.md) — the full documentation index (leaderboard in full, the ten axes, research notes, correction notices, manifest schema and validator) -- **状态**：草案 v0.2.2。Core 规范已稳定；`profiles/` 和建案/运行工具尚未发布。 -- **两条绕不开的警告**：① 运行时密封 ≠ 训练数据纯净——我们封存的是运行时证据，模型训练时有没有见过未来，我们证明不了。② 历史结果是证据，不是唯一正解。 -- **为什么需要它**：公开基准是静态题库，污染普遍且无法审计，也测不出多步诊断、弃权、拒绝、在不确定中升级。AMBER 回放来源受控的真实事件，与公开基准互补，不取代它们。 +- **Status**: draft v0.2.2. The Core spec is stable; `profiles/` and the case-building/runner tooling are not yet published. +- **Two caveats that won't go away**: (1) runtime sealing ≠ training-data purity: we seal runtime evidence and cannot prove the model never saw the future in training. (2) A historical outcome is evidence, not the one correct answer. +- **Why it exists**: public benchmarks are frozen corpora; contamination is rampant and unauditable, and they cannot measure multi-step diagnosis, abstention, refusal or escalation under uncertainty. AMBER replays provenance-controlled real events; it runs alongside public benchmarks and does not replace them. -## 许可 +## License -Apache-2.0 — 见 [LICENSE](LICENSE)。 +Apache-2.0 — see [LICENSE](LICENSE). diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..16b0f14 --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,99 @@ +[English](README.md) · 简体中文 + +# AMBER — 琥珀式封存的历史回放评测 + +`封进琥珀，重做当时的题。` + +**一句话**：拿真实发生过的事故和任务，让 AI 模型只用当时手里的信息重做一遍。比如一次真实的运维故障：模型只能拿当时的监控和日志找根因，事后才揭晓的证据一律封存不给看。题目永不公开，但每题有哈希指纹，分数和指纹全部公开，谁都能核对题没被换、分不是编的。 + +**案集** 24 案 · **规范** v0.2.2（草案） · **哈希索引** [v2026-09](hash-index/v2026-09.md) · **成绩仓** × 13 · **纪律** 只发分数，不发题 · **官网榜单** [askclaw.dev](https://askclaw.dev/) + +> ⚠️ **更正**（原文与受影响车道清单见[榜单全文](docs/leaderboard-notes.md)） +> - **[2026-10-02](docs/corrections-2026-10-02.md)**：22 张已发布的考卷在作答时越出考卷、接触了判分材料，改记 NA（不计胜负）。责任在我们；2026-10-01 起已改为隔离容器作答。 +> - **[2026-10-02（另一项）](docs/corrections-2026-10-02-a-d511f9e8.md)**：防御轴的一案 A-d511f9e8 在所有车道上改记 NA。分母不变，**过案数不变**，每条道的总分都带 `'`。 +> - **[2026-10-07](docs/corrections-2026-10-07-a-24bcf707.md)**：运维轴的一案 A-24bcf707 在七条车道上由「未过」改记 NA（判分要求比题面多了一条）。分母不变，**总分和名次档不变**，不重考任何场次。 +> - 更早：[2026-09-18](docs/corrections-2026-09-18.md)（swe-2-low 实为 swe-2-high）· [2026-W38](docs/corrections-2026-W38.md)（全库复核，逐仓特刊） + +## 现在谁领先 + +![当前前五 2026-W41（含 2026-10-02 更正、2026-10-02 考试、2026-10-04 WorkBuddy 直连四条道、2026-10-06 五场新考、2026-10-07 品牌案重考一案与 A-d511f9e8 全车道挂起）](docs/images/top5-2026-w41b.png?v=20261007) + +| # | 总分 | 模型 @ 端点 | +|---|---|---| +| 1 | 19'/24 | swe-2-max @ Devin
claude-sonnet-5-5 @ Anthropic 订阅道
claude-opus-5-5 @ Anthropic 订阅道 | +| 2 | 18'/24 | glm-5.3-flash @ Ollama Cloud
k3 @ Kimi
hy4-preview-f @ WorkBuddy ACP（W37）
deepseek-v4.1-flash @ WorkBuddy 直连（W40）
step-5-preview @ StepFun（W41） | +| 3 | 17'/24 | gpt-6-sol-900k @ OpenAI Codex
swe-2-high @ Devin
doubao-seed-evolving @ 火山方舟
glm-5.3-flash @ WorkBuddy 直连（W40）
gpt-5.6-luna @ OpenAI Codex | +| 4 | 16'/24 | gpt-5.6-luna-900k（high）@ OpenAI Codex
gpt-6-astra-900k @ OpenAI Codex
gpt-6.1-sol @ OpenAI Codex
claude-fable-5-1 @ Anthropic 订阅道
deepseek-v4-flash:0731 @ Ollama Cloud
swe-2-medium @ Devin
deepseek-v4.1-flash @ Ollama Cloud ▼
deepseek-flash @ DeepSeek ▼
mimo-v2.6-pro @ CommandCode ▼
hy4-preview-f @ WorkBuddy 直连（W40）
minimax-m3 @ WorkBuddy 直连（W40）
gpt-5.6-luna-900k @ OpenAI Codex（W41）
gpt-6-luna @ OpenAI Codex | +| 5 | 15'/24 | kimi-for-coding（K2.8）@ Kimi
gpt-6-luna-900k @ OpenAI Codex
swe-1-7-medium @ Devin
Qwen3.8-27B @ GoldenPotato 自部署
deepseek-flash @ OpenCode Go ▼ | + +`'` = 其中有案暂不计分（NA），不算输；现在每条道都带，因为 A-d511f9e8 在所有车道上暂不计分。同名模型换个端点可能是另一个脑，所以成绩一律按「端点 × 名字」记。claude-opus-5-5 取 W40 重考一场（W39 首考 17'/24）：两场各考一次、考场不同，**不据此判断强弱**，见 [amber-claude W40](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)。▼ = 经 2026-10-02 更正下调。WorkBuddy 直连（W40）的四条道是 2026-10-04 隔离考场的新场次，和 W37（ACP 道）、W39（直连道场次）不可逐格对比；hy4-preview-f 在榜上有 ACP（W37，18'）和直连（W40，16'）两行，是两条不同的道、不同的考场，**不据此判断强弱**，见 [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.md)。2026-10-06（W41）的五场新考也在隔离考场：gpt-5.6-luna、gpt-5.6-luna-900k（W41）、gpt-6-luna 是三条新入榜的道；step-5-preview 与 gpt-6-luna-900k 榜上改取 2026-10-06 重考一场：gpt-6-luna-900k 是 15'/24；step-5-preview 当场是 17'/24，品牌案 A-d9b79b46 的题面修订后（见 [hash-index](hash-index/v2026-09.md) 的 2026-10-07 换版说明），该案于 2026-10-07 单案重考通过，现为 18'/24（3 NA：A-d511f9e8 挂起，另两案撞考场时间上限）。step-5-preview 在 W38 首考的 15'（经 2026-10-02 更正下调、曾标 ▼）不再作榜上成绩，所以不再带 ▼。这些场次和 W37–W39 的场次不可逐格对比；step-5-preview 与 gpt-6-luna-900k 的新旧两场各考一次、考场不同，**不据此判断强弱**。榜上有两条 gpt-5.6-luna-900k，是两条不同的道：「（high）」一条是 W37 基线（high 档）的成绩，「（W41）」一条是 2026-10-06 隔离考场的新场次。W41 各场的逐案成绩见 [amber-gpt W41](https://github.com/getaskclaw/amber-gpt/blob/main/results/2026-W41.md)（gpt-5.6-luna、gpt-5.6-luna-900k、gpt-6-luna、gpt-6-luna-900k 四场）和 [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.md)（step-5-preview）。各车道出处、脚注、冻结道：[榜单全文](docs/leaderboard-notes.md)。 + +## 同分不同款 + +总分相同的车道，强项和短板可以完全不同。点越大、颜色越深，完成度越高；NA = 该轴没有可计分的案，不是零分。 + +![完成度矩阵：榜上总分 18 以上的 8 条车道，十轴，24 案，2026-10-07](docs/images/completion-matrix-top8-2026-w41.png?v=20261007) + +*图中 8 条 = 榜上总分 18 以上的全部车道（2026-10-07 的库）。hy4-preview-f 取 ACP（W37）一行；同名的直连（W40，16'）总分不到 18，不在图内。deepseek-v4.1-flash @ WorkBuddy 直连（W40）的 UI 轴只有一案 A-d9b79b46，该案挂起、记 NA，所以那一格是 NA 不是零分；它的视觉格取考场修复后的重考卷，见 [amber-workbuddy W40](https://github.com/getaskclaw/amber-workbuddy/blob/main/results/2026-W40.md)。防御轴按 A-d511f9e8 在所有车道上 NA 的口径算，各格只剩另一案的读数（两案都没有可计分结果的格是 NA）。step-5-preview（W41）的 UI 格是 1.00：品牌案 A-d9b79b46 在题面修订后于 2026-10-07 单案重考通过（12/12），见 [amber-stepfun W41](https://github.com/getaskclaw/amber-stepfun/blob/main/results/2026-W41.md)；它的防御和归因两格是 NA，因为这两轴各有一案撞考场时间上限、没有可计分结果。* + +**读法**：动手干活的「施工面」几乎全满，差别在「判断面」。每格 = 该轴全部案子的完成度（0–1）。官方榜 [askclaw.dev](https://askclaw.dev/) 也用这十个轴。 + +| 面 | 轴 | 考什么 | +|---|---|---| +| 施工面 | 编码 | 照需求把功能写对 | +| | 交付 | 做完要交卷，没产出就是 0 | +| | 运维 | 照规程干脏活，一步不省 | +| | 需求 | 客户要 A 不要 B，交上去得是 A | +| | 收敛 | 真干完，还是绕圈装忙 | +| 判断面 | UI | 照设计稿做页面，像素级验收 | +| | 视觉 | 给真截图挑毛病 | +| | 防御 | 堵死校验器的漏洞，还不误伤好人 | +| | 归因 | 把每条毛病对到正确根因 | +| | 审查 | 给别人的交付物挑错，漏看和冤枉都扣分 | + +各轴的详细读法、每条车道的数字：[十个轴与完成度矩阵](docs/axes.md)。 + +## 它怎么运作 + +私有题库 + 公开成绩：题目永不公开，分数与哈希永远可查。评分标准在看到任何输出之前就已冻结，全程留档。 + +![AMBER 是什么](docs/images/what-is-amber-2026-10.png?v=20261003) + +一期成绩像一场考试：出卷封存、净室应考、逐卷验脑、脱敏发布、人人可核对。 + +![一期成绩怎么炼成](docs/images/trust-chain.png?v=20260915b) + +另有三个发现（思考档位、token 账单、耗时与分数的关系）：[docs/findings-effort-and-speed.md](docs/findings-effort-and-speed.md)。 + +## 成绩仓 + +成绩与题目分开发布：结果公开，题目永不公开。别名 + bundle 哈希逐案对照本仓的公开哈希索引。 + +[amber-claude](https://github.com/getaskclaw/amber-claude)（Claude）· [amber-gpt](https://github.com/getaskclaw/amber-gpt)（GPT）· [amber-devin](https://github.com/getaskclaw/amber-devin)（Devin）· [amber-kimi](https://github.com/getaskclaw/amber-kimi)（Kimi）· [amber-deepseek](https://github.com/getaskclaw/amber-deepseek)（DeepSeek）· [amber-ollama](https://github.com/getaskclaw/amber-ollama)（Ollama Cloud）· [amber-opencode](https://github.com/getaskclaw/amber-opencode)（OpenCode Go）· [amber-commandcode](https://github.com/getaskclaw/amber-commandcode)（CommandCode）· [amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy)（WorkBuddy）· [amber-doubao](https://github.com/getaskclaw/amber-doubao)（豆包）· [amber-stepfun](https://github.com/getaskclaw/amber-stepfun)（阶跃星辰）· [amber-nous](https://github.com/getaskclaw/amber-nous)（Nous Portal）· [amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato)（社区自部署 Qwen3.8-27B） + +## 常见问题 + +**题目不公开，凭什么信分数？** 靠可验证的过程：净室考场、逐卷验脑（替身 = 整卷作废）、收卷闸、别名 + 哈希公开、harness 版本每期钉死（现役 [Hermes](https://github.com/NousResearch/hermes-agent)）。考场条件的变更逐条公开，最近一次：[方法变更 2026-10-01](docs/method-change-2026-10-01.md)。 + +**为什么不公开题目？** 公开题库会被训练数据吃掉，分数通胀、无法审计。题目私有，泄漏状态才可核查。 + +**两个模型的分能直接比吗？** 只在同档、同题集版本、尽量同日时可比。跨周对比必须带日期和档位。 + +**能复现或加入吗？** 目前不能：仅凭公仓还跑不通一次合规运行（见 [PLAN.md](PLAN.md)）。追结果请订阅成绩仓；方法论问题欢迎开 issue。 + +## 规范与文档 + +- [AMBER-Core-Specification.md](AMBER-Core-Specification.md) — 规范本体（English） +- [protocols/distribution.md](protocols/distribution.md) · [protocols/stability.md](protocols/stability.md) — 分发协议、稳定性协议 +- [hash-index/v2026-09.md](hash-index/v2026-09.md) — 公开哈希索引（24 案） +- [PLAN.md](PLAN.md) · [CONTRIBUTING.md](CONTRIBUTING.md) — 路线图；贡献规则（**本仓绝不接收案件内容**） +- [docs/](docs/README.md) — 全部文档索引（榜单全文、十个轴、研究笔记、更正声明、案件清单 schema 与校验器） + + +- **状态**：草案 v0.2.2。Core 规范已稳定；`profiles/` 和建案/运行工具尚未发布。 +- **两条绕不开的警告**：① 运行时密封 ≠ 训练数据纯净——我们封存的是运行时证据，模型训练时有没有见过未来，我们证明不了。② 历史结果是证据，不是唯一正解。 +- **为什么需要它**：公开基准是静态题库，污染普遍且无法审计，也测不出多步诊断、弃权、拒绝、在不确定中升级。AMBER 回放来源受控的真实事件，与公开基准互补，不取代它们。 + +## 许可 + +Apache-2.0 — 见 [LICENSE](LICENSE)。