From 98a9c892b52a2f43608f3d56018ca152e0a9bcde Mon Sep 17 00:00:00 2001 From: Lumen Date: Wed, 7 Oct 2026 05:02:06 +0000 Subject: [PATCH] README: English-first facade (README.md en + README.zh-CN.md zh) --- README.md | 122 ++++++++++++++++++++++++------------------------ README.zh-CN.md | 101 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+), 61 deletions(-) create mode 100644 README.zh-CN.md diff --git a/README.md b/README.md index 21f1925..cb9efe0 100644 --- a/README.md +++ b/README.md @@ -1,101 +1,101 @@ -# amber-stepfun +[简体中文](README.zh-CN.md) · English -> ⚠️ **更正（2026-10-02，另一项）**：防御轴的一案 A-d511f9e8 在所有车道上改记 NA（考场判的不是考生交付的文件，判分还要求了题面没写的事）。分母不变，**过案数不变**，每条道的总分都带 `'`。本仓各期成绩表里这一格请按 NA 读，其余内容保留原样，以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02-a-d511f9e8.md)为准。 +# amber-stepfun -> ⚠️ **更正（2026-10-02）**：以下考卷在作答时越出考卷、接触了判分材料，不计胜负。step-5-preview @ stepfun plan 端点 有 1 张卷（A-61f7ad01）改记 NA，榜上成绩 16'/24 → **15'/24**。原因是考场隔离缺陷，责任在我们。本页其余内容保留原样，以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02.md)为准。 +> ⚠️ **Correction (2026-10-02, second)**: one defense-axis case, A-d511f9e8, is now NA on every lane (the exam room did not grade the file the candidate delivered, and the grader asks for something the task text does not say). The denominator and the **number of passed cases do not change**; every lane's total now carries `'`. In this repo's issue tables, read that cell as NA. Everything else stays as published; the [correction notice](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02-a-d511f9e8.en.md) governs. -> 周次更正：全库考试在 W38，发文与收敛补测在 W39，组合成绩仍为 16'/24。见 [更正](results/2026-W38-correction-20260923.md)。已有挂起标记不因本次改周解除。 +> ⚠️ **Correction (2026-10-02)**: the papers below were answered by a candidate that stepped outside its paper and touched grading material; they count neither as a pass nor as a fail. step-5-preview @ stepfun plan endpoint: 1 paper (A-61f7ad01) now NA, board score 16'/24 → **15'/24**. The cause was an isolation defect in our exam setup; the fault is ours. The rest of this page stays as published; where they differ, the [correction notice](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02.en.md) governs. -> **W41 起换隔离考场**：本仓从 2026-W41 起的考试在隔离考场里进行，所以 W41 与 W38、W39 的各格跨期不可逐格对比（详见 [2026-W41 期文](results/2026-W41.md)）。W38、W39 的页面保持原样。 +> Week correction: full test in W38; publication and convergence make-up in W39. The combined score stays 16'/24. [Correction](results/2026-W38-correction-20260923.en.md). Existing held-case notices remain in force. -> **2026-10-07 更新**：品牌题 A-d9b79b46（UI）由 NA（挂起）改记为过：10-07 该案题面修改、驱动改用中性工作目录后，只重考了这一格，判分器 12/12 通过。step-5-preview 17'/24 → **18'/24**（过案 17→18，NA 4→3）。其他车道这一案的格子仍是旧题面下的成绩，只有 claude-fable-5-1 于 2026-10-07 在修改后的题面下重考过一次（负，11/12，见 [amber-claude 的 W40 页](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)）；四条 WorkBuddy 直连道这一案的格子没有重考。见 [2026-W41 期文](results/2026-W41.md)。 +> **From W41 the exam room is isolated**: this repo's sittings from 2026-W41 on are taken in an isolated room, so W41 cells are not comparable cell by cell with W38 or W39 (see the [2026-W41 issue](results/2026-W41.en.md)). The W38 and W39 pages stay as published. -' = contested（安全拒答挂起）或 invalid（基建相关（考场 harness 或判分环境）的挂起、作废或待重评），均不计胜负；所有含 NA 的道都带撇号，包括冻结展示行；挂起不表示死因已定。StepFun（W41）：3 个 NA 格（1 个挂起、2 个超时作废），不计胜负。 +> **Update 2026-10-07**: brand case A-d9b79b46 (UI) changed from NA (on hold) to a pass: after its prompt was revised and the driver switched to a neutral working directory on 10-07, only this cell was re-taken and the grader scored 12/12. step-5-preview 17'/24 → **18'/24** (wins 17→18, NA 4→3). The same case on other lanes still stands on the old prompt, except claude-fable-5-1, which was re-sat once on 2026-10-07 under the revised prompt (loss, 11/12; see the [amber-claude W40 page](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.en.md)); the four WorkBuddy direct lanes' cells for this case have not been re-sat. See the [2026-W41 issue](results/2026-W41.en.md). +' = contested (held for safety refusal) or invalid (infrastructure-related (test harness or scoring environment) cases: held, void or awaiting re-scoring); neither counts as a win or a loss. Every lane with NA carries an apostrophe, including frozen display rows; a hold does not settle the cause. StepFun (W41): 3 NA cells (1 held, 2 timed out as invalid), neither wins nor losses. -用私有题库 **AMBER** 实测 StepFun 阶跃星辰在 **stepfun plan 端点**上服务的新旗舰 `step-5-preview`，只公开结果，不公开题目。 -English: [README.en.md](README.en.md) +Weekly AMBER measurement of StepFun's new flagship `step-5-preview` as served on the **stepfun plan endpoint**. Results are public; cases never are. -> **一句话**：step-5-preview 在 stepfun plan 端点的第二次考试（W41，2026-10-06，隔离考场；品牌题 10-07 重考）：24 案过 **18'/24**（18 胜 · 3 负 · 3 NA）。动手面的运维、交付、需求、收敛全过，编码 5/6；UI 1/1（品牌题，重考成绩）；审查 1/2、看图 0/1；防御、归因两轴本次全是 NA，没有读数。 +> **In one line**: step-5-preview's second sitting on the stepfun plan endpoint (W41, 2026-10-06, isolated room; brand case re-taken on 10-07): **18'/24** on 24 cases (18 wins · 3 losses · 3 NA). On the build side ops, delivery, requirements and convergence are all passed and coding is 5/6; UI is 1/1 (the brand case, a re-sit result); review is 1/2 and vision 0/1; defense and attribution are all NA in this sitting, so there is no reading on them. > -> 分数后的 `'` 表示其中有几案暂不计分（NA），既不算过也不算没过；NA 的原因写在期文里。W38 那一场是旧考场、23 案题集，两场不逐格对比，也不据此说模型变强或变弱。 +> The `'` after a score means some cases are not scored (NA): neither a pass nor a fail; the reasons are in the issue. The W38 sitting was in the old room on a 23-case set; the two are not compared cell by cell, and nothing here says the model got stronger or weaker. -## 成绩一览 +## Scoreboard -![amber-stepfun 成绩一览：step-5-preview 逐轴过案数](results/assets/scoreboard.zh.png?v=20261007) +![amber-stepfun scoreboard: cases passed per axis for step-5-preview](results/assets/scoreboard.en.png?v=20261007) -| 大类 | 轴 | 考什么 | step-5-preview · [W41](results/2026-W41.md) | +| Group | Axis | What it tests | step-5-preview · [W41](results/2026-W41.en.md) | |---|---|---|:-:| -| 施工面 | 编码 | 照着需求把功能写对 | 5/6 | -| | 交付 | 做完还得交得出东西 | 3/3 | -| | 运维 | 照规程干脏活 | 6/6 | -| | 需求 | 客户要 A 不要 B | 1/1 | -| | 收敛 | 真干完，不绕圈装忙 | 1/1 | -| 判断面 | UI | 照设计稿做页面 | 1/1 | -| | 视觉 | 给真截图挑毛病 | 0/1 | -| | 防御 | 堵死校验器的漏网口 | 0/2 · 2 NA | -| | 归因 | 毛病对到正确根因 | 0/1 · 1 NA | -| | 审查 | 给别人的交付物挑错 | 1/2 | -| | **合计** | | **18'/24** | - -每格 = 过了几案/该轴共几案（案 = 一道计分题）。NA = 这一案作废或暂停计分，不算过也不算没过；总分带 `'` 表示其中有 NA。多数轴只有 1–2 案，差一案读数就变，所以别把小差距当结论。各列考试周次相同（W41），具体日期可能不同，数字是当期快照。 +| Building | Coding | Implement the spec correctly | 5/6 | +| | Delivery | Done means handed in | 3/3 | +| | Ops | Follow the runbook | 6/6 | +| | Requirements | Ship A when A was asked | 1/1 | +| | Convergence | Finish, don't spin | 1/1 | +| Judging | UI | Build the page to the mock | 1/1 | +| | Vision | Spot defects in screenshots | 0/1 | +| | Defense | Plug every hole in the validator | 0/2 · 2 NA | +| | Attribution | Pin defects to their root cause | 0/1 · 1 NA | +| | Review | Inspect someone else's work | 1/2 | +| | **Total** | | **18'/24** | + +Each cell = cases passed / cases on that axis (a case is one scored task). NA = the case was voided or put on hold; it counts as neither a pass nor a fail, and a total carrying `'` contains at least one NA. Most axes hold only 1–2 cases, so one case moves the reading: do not over-read small gaps. All columns are from the same week (W41) and the test dates may differ; every number is a snapshot. -## 这是什么 +## What this is -- 「道」= 同一个模型名在不同家的卖场/接口；「案」= 一道题，「卷」= 一场考试记录（一案多卷 = 一道题的几个变体场次）。 +- A 'lane' is one vendor's shop/API for a model name; a 'case' is one task, a 'run' is one sitting (a multi-variant case has several runs). -- 每期 `results/YYYY-Www.md`：同题、同 harness（跑考试并记分的程序），对目标模型跑全库（W38 为 23 案 / 26 卷，W41 起为 24 案 / 28 卷（含一卷重考））。 -- 一期固定报告：题集规模与哈希、每案得分与通过/失败、终端终态（程序跑完时的退出状态）、token 用量与时延、环境指纹、按证据纪律写的定性裁决。 -- 题目、oracle（判分器）、transcript（答题全过程记录）、中间产物**永不公开**（见下「发布纪律」）。 -- 姐妹仓：[amber-gpt](https://github.com/getaskclaw/amber-gpt)、[amber-crof](https://github.com/getaskclaw/amber-crof)、[amber-ollama](https://github.com/getaskclaw/amber-ollama)、[amber-devin](https://github.com/getaskclaw/amber-devin)、[amber-deepseek](https://github.com/getaskclaw/amber-deepseek)、[amber-commandcode](https://github.com/getaskclaw/amber-commandcode)、[amber-opencode](https://github.com/getaskclaw/amber-opencode)、[amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy)、[amber-kimi](https://github.com/getaskclaw/amber-kimi)、[amber-doubao](https://github.com/getaskclaw/amber-doubao)、[amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato)。 -- AMBER 是 agentic 实战题库（施工/运维/审查/视觉/需求漂移——题中要求中途变化），规范与制题工具见 [getaskclaw/amber](https://github.com/getaskclaw/amber)；考题本体私有。 +- Every issue `results/YYYY-Www.md`: same cases, same harness (the program that runs the exam and scores it), full-library run (23 cases / 26 papers in W38, 24 cases / 28 papers from W41, including one re-sit paper). +- A fixed report shape: library size + hashes, per-case pass/fail, terminal states (how the run process exited), token usage and wall clock, environment fingerprint, and qualitative verdicts written to evidence discipline. +- Cases, oracles, transcripts (full answer logs)s and intermediate artifacts are **never published** (see "Publishing rules" below). +- Sister repos: [amber-gpt](https://github.com/getaskclaw/amber-gpt), [amber-crof](https://github.com/getaskclaw/amber-crof), [amber-ollama](https://github.com/getaskclaw/amber-ollama), [amber-devin](https://github.com/getaskclaw/amber-devin), [amber-deepseek](https://github.com/getaskclaw/amber-deepseek), [amber-commandcode](https://github.com/getaskclaw/amber-commandcode), [amber-opencode](https://github.com/getaskclaw/amber-opencode), [amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy), [amber-kimi](https://github.com/getaskclaw/amber-kimi), [amber-doubao](https://github.com/getaskclaw/amber-doubao), [amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato). +- AMBER is an agentic real-world task library (build / ops / review / vision / requirement-drift (the requirements change mid-task)). The spec and case-forging tooling live in [getaskclaw/amber](https://github.com/getaskclaw/amber); the cases themselves are private. -## 渠道说明（本仓的特殊性） +## Lane notes (what is special about this repo) -本仓考的是 StepFun 阶跃星辰官方 **plan 端点**（OpenAI 兼容面）上的新旗舰 `step-5-preview`（600B/27B MoE，官方声明 1M 上下文 + 视觉）。 +This repo measures StepFun's official **plan endpoint** (OpenAI-compatible) serving the new flagship `step-5-preview` (600B/27B MoE; the vendor advertises 1M context + vision). -因此本仓成绩带三条额外口径： +Three extra disclosures therefore ride with the scores: -1. **燃烧量不可验**：端点 usage **不返回 `reasoning_tokens`**，故「请求 high」只是请求标签，实际跑的是端点默认思维预算。本期不与其它仓做「同档」对比——band 列如实标为请求档 + 不可验。 -2. **视觉面端点事故**：发车前视觉请求在本端点零字节挂起（同端点另一模型同图正常答对），故相关案先判 not run（非失败），端点复测恢复后单案补考取真值。事故与补考过程写进 Findings。 -3. **加时帽收敛**：核验三案在放大墙钟帽后才落定，全卷墙钟因此偏重——跨仓比 wall 时须带此口径。 +1. **Reasoning burn is unverifiable.** The endpoint's usage payload **does not report `reasoning_tokens`**, so "high" is only a request label — the run actually used the endpoint's default thinking budget. No same-band comparison is offered against other repos; the band column is honestly marked "requested, unverifiable". +2. **Vision face: endpoint incident.** Before launch, vision requests hung on this endpoint with zero bytes (a different model on the same endpoint answered the same image correctly in 6.8s). That case was first recorded `not run` (not a failure); after the endpoint recovered it was re-taken and the real value recorded. Both the incident and the makeup are written into Findings. +3. **Extended wall-clock caps.** The three verify cases only settled after the runner's time caps were doubled, so the whole-issue wall clock runs heavy — carry this caveat when comparing wall time across repos. -## 一分钟看懂 W41 +## W41 in one minute -stepfun plan 端点、请求 high（**燃烧量不可验**）、隔离考场、24 案（品牌题 A-d9b79b46 的题面 10-07 改过，重考卷是新哈希，其余同哈希）：**step-5-preview 18'/24**（18 胜 · 3 负 · 3 NA）。W38 的更正里公开为「挂起待补考」的三格本期重考：A-1fd3683a 过（2/2）；A-cdc3d11a（审查）和 A-ea80d793（看图）仍记负，这两案里模型只回了一句打算先做什么的话（A-cdc3d11a 里还带着写成文本的工具调用），回合就结束了，没有给出结论。3 个 NA：A-d511f9e8 在所有车道上记 NA（见更正声明）；A-a317e74b 和 A-be92627f 各试两次都撞到时间上限。品牌题 A-d9b79b46（UI）：10-06 首次作答时模型把工具调用写成文本、没有交出文件，题面与考场不一致，当时沿用 10-02 的先例记 NA；10-07 题面修改、驱动改用中性工作目录后只重考这一格，交出两个文件，12/12 通过，记过；其他车道这一案的格子仍是旧题面下的成绩，只有 claude-fable-5-1 于 2026-10-07 在修改后的题面下重考过一次（负，11/12，见 [amber-claude 的 W40 页](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)）；四条 WorkBuddy 直连道这一案的格子没有重考。逐案矩阵、考试条件与各 NA 的说明见 [2026-W41 期文](results/2026-W41.md)。 +stepfun plan endpoint, high requested (**burn not verifiable**), isolated room, the same 24 cases (the prompt of the brand case A-d9b79b46 was revised on 10-07, so its re-sit paper has a new hash; the others have the same hashes): **step-5-preview 18'/24** (18 wins · 3 losses · 3 NA). The three cells that the W38 correction showed as "held, re-sit pending" were re-taken: A-1fd3683a passed (2/2); A-cdc3d11a (review) and A-ea80d793 (vision) stay recorded as losses; on both the model answered with one sentence about what it would do first (on A-cdc3d11a with tool calls written out as text), the turn ended, and no answer was given. The 3 NA cells: A-d511f9e8 is NA on every lane (see the correction notice); A-a317e74b and A-be92627f hit the time cap on both tries. The brand case A-d9b79b46 (UI): in the first paper on 10-06 the model wrote a tool call out as text and handed in no files, the text and the room did not match, and it was recorded NA following the 10-02 precedent; after the prompt revision and the switch to a neutral working directory on 10-07, only this cell was re-taken, the model delivered both files, 12/12, recorded as a pass; the same case on other lanes still stands on the old prompt, except claude-fable-5-1, which was re-sat once on 2026-10-07 under the revised prompt (loss, 11/12; see the [amber-claude W40 page](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.en.md)); the four WorkBuddy direct lanes' cells for this case have not been re-sat. The case-by-case matrix, the exam conditions and the notes on each NA are in the [2026-W41 issue](results/2026-W41.en.md). -上面「渠道说明」的第 1 条（燃烧量不可验）对 W41 同样适用；第 2、3 条是 W38 当时的端点事故和加时帽，W41 没有放大时间帽重考，超时记 NA。 +Disclosure 1 in "Lane notes" above (burn not verifiable) applies to W41 as well; disclosures 2 and 3 are the endpoint incident and the extended caps of W38; there was no raised-cap re-run in W41, and timeouts are recorded NA. -## W38 基线与 W39 补测 +## W38 base + W39 make-up -![W38 全库基线：step-5-preview 15/23；W39 补测另列](docs/images/w38-face-profile-correction.png?v=corrections-20260924-r2) +![W38 full-test base: step-5-preview 15/23; W39 make-up shown separately](docs/images/w38-face-profile-correction.en.png?v=corrections-20260924-r2) -stepfun plan 端点、请求 high（**燃烧量不可验**）、23 案同哈希：**step-5-preview 15/23**（公共 21 案子集 13/21）——**施工面顶级**：运维 6/6 全清 + 需求漂移四变体全过 + 编码 5/6（含全库唯一硬区分器 A-442d4aab 7/7 满分）+ 交付满分；**判断面掉队**：归因轴 0.800 反超对照锚 k3 的 0.667（同一案 12/15 对 10/15），但核验 0/3、审查净 −2、视觉 −2、前端废卷。三条口径须随行：① 端点不报 `reasoning_tokens`，行按端点默认档记；② 视觉面发车前在本端点零字节挂死（同端点另一模型同图正常），该案先判 not run、端点恢复后补考取真值；③ 核验三案撞标准帽后放大帽收敛（7200/10800/3600s），墙钟是本期最重的成本。逐案矩阵与车道账本见 [W38 基线与 W39 补测更正](results/2026-W38-correction-20260923.md)（原始期文保留旧 URL）。图源与 PNG 同目录（`docs/images/`，Vega-Lite）。 +stepfun plan endpoint, requested high band (**reasoning burn unverifiable**), same 23 cases same hashes: **step-5-preview 15/23** (13/21 on the public 21-case subset) — **build side top-tier**: ops 6/6 clean + req-drift all four variants + coding 5/6 (including the library's only hard discriminator A-442d4aab at a perfect 7/7) + delivery perfect; **judgement side trails**: the attribution axis beats the reference anchor k3 (0.800 vs 0.667 — the same case 12/15 vs 10/15), but verify 0/3, review net −2, vision −2, ui-build void. Three disclosures ride with the row: (1) the endpoint does not report `reasoning_tokens`, so the row is the endpoint's default band; (2) the vision face hung on this endpoint at launch with zero bytes (a different model on the same endpoint answered the same image fine), so that case was first `not run` and re-taken after recovery; (3) the three verify cases converged only under doubled wall-clock caps. Per-case matrix and lane ledger in the [W38 base / W39 make-up correction](results/2026-W38-correction-20260923.en.md) (original report kept at its old URL). Spec sources sit beside the PNGs (`docs/images/`, Vega-Lite). -## 发布纪律（红线） +## Publication discipline (red lines) -1. 只发：分数与聚合、token 用量、速度、定性裁决。 -2. 永不发：题目内容、oracle/判分器、transcript、考生工作区、任何能复原题面的中间产物、端点访问凭证。 -3. 每期必钉：模型 ID、effort 档（思考力度档位）、日期（UTC）、harness 版本、每案内容哈希（bundle_sha，每题内容的哈希指纹）。哈希用于对照 [amber](https://github.com/getaskclaw/amber) 的公开哈希清单，自证题集未变。 -4. 案号与题目结构属私有面：公开结果里案例只用稳定别名（A-xxxxxxxx，哈希派生）+ bundle 哈希作句柄；内部案号、变体名、题目描述永不出现。 -5. 基调：这是对公开端点的实测，不是对任何厂商的攻击。数据说话，措辞克制。 +1. Publish only: scores and aggregates, token usage, speed, qualitative verdicts. +2. Never publish: case content, oracles/scorers, transcripts, candidate workspaces, any intermediate artifact that can reconstruct a case, endpoint credentials. +3. Every issue pins: model id, effort band (the thinking-effort setting), date (UTC), harness version, per-case content hash (bundle_sha (per-case content-hash fingerprint)). The hashes are checkable against [amber](https://github.com/getaskclaw/amber)'s public hash index, self-proving the library did not change. +4. Case numbers and case structure are private: public results reference cases only by stable alias (A-xxxxxxxx, hash-derived) plus bundle hash. Internal case ids, variant names and case descriptions never appear. +5. Tone: this is measurement of a public endpoint, not an attack on any vendor. Let the data speak; keep the wording restrained. -## 一个方法论前提 +## One methodological caveat -同一模型、同一端点，两次跑也可能不同分——推理参数、负载、服务端版本都在漂。所以这里的一切结论都带日期与档位。单日数字是快照，不是定律。 +The same model on the same endpoint can score differently across runs — inference parameters, load and server version all drift. Every conclusion here therefore carries its date and band. A single-day number is a snapshot, not a law. -## 结果索引 +## Results index -| 期 | 内容 | 结论 | +| Issue | Content | Verdict | |---|---|---| -| [2026-W41](results/2026-W41.md) | step-5-preview，2026-10-06 第二次全库考试（隔离考场，24 案；品牌题 10-07 重考） | **18'/24**（18 胜 · 3 负 · 3 NA）；W38 挂起的 A-1fd3683a 过，A-cdc3d11a、A-ea80d793 仍记负；品牌题 A-d9b79b46 10-07 题面修改后重考，过；防御、归因两轴全是 NA。与 W38 那一场不逐格对比 | -| [2026-W38 基线](results/2026-W38-correction-20260923.md) | step-5-preview，2026-09-20 全库考试 | 基线 15/23；W39 收敛补测增加 1 案通过，组合 16'/24。端点实际思考用量不可验。[原始期文](results/2026-W39.md) | -| [此前全库复核](results/2026-W38-correction.md) | 历史更正：改判 0 格，挂起 4 格 | 旧文 W39 指旧期文名称；全库考试实属 W38，本次不解除挂起 | +| [2026-W41](results/2026-W41.en.md) | step-5-preview, second full-library sitting on 2026-10-06 (isolated room, 24 cases; brand case re-taken 10-07) | **18'/24** (18 wins · 3 losses · 3 NA); A-1fd3683a, held in W38, passes; A-cdc3d11a and A-ea80d793 stay losses; brand case A-d9b79b46 passes in a 10-07 re-sit after its prompt was revised; defense and attribution are all NA. Not compared cell by cell with the W38 sitting | +| [2026-W38 base](results/2026-W38-correction-20260923.en.md) | step-5-preview, full test on 2026-09-20 | Base 15/23; W39 convergence make-up adds 1 pass, giving 16'/24. The endpoint does not report actual thinking use. [Old report](results/2026-W39.md) | +| [Earlier full-library review](results/2026-W38-correction.en.md) | Historical notice: 0 cells reversed, 4 held | Its W39 label refers to the old report name. The full test was in W38; the held marks are not cleared here. | -## 免责 +## Disclaimer -与 StepFun 阶跃星辰团队无任何隶属/赞助关系。分数是特定日期、特定负载下的快照，不构成任何选型建议。 +No affiliation with or sponsorship from the StepFun team. Scores are snapshots under a particular date and load, and constitute no purchasing advice. diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..37d057e --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,101 @@ +[English](README.md) · 简体中文 + +# amber-stepfun + +> ⚠️ **更正（2026-10-02，另一项）**：防御轴的一案 A-d511f9e8 在所有车道上改记 NA（考场判的不是考生交付的文件，判分还要求了题面没写的事）。分母不变，**过案数不变**，每条道的总分都带 `'`。本仓各期成绩表里这一格请按 NA 读，其余内容保留原样，以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02-a-d511f9e8.md)为准。 + +> ⚠️ **更正（2026-10-02）**：以下考卷在作答时越出考卷、接触了判分材料，不计胜负。step-5-preview @ stepfun plan 端点 有 1 张卷（A-61f7ad01）改记 NA，榜上成绩 16'/24 → **15'/24**。原因是考场隔离缺陷，责任在我们。本页其余内容保留原样，以[更正声明](https://github.com/getaskclaw/amber/blob/main/docs/corrections-2026-10-02.md)为准。 + +> 周次更正：全库考试在 W38，发文与收敛补测在 W39，组合成绩仍为 16'/24。见 [更正](results/2026-W38-correction-20260923.md)。已有挂起标记不因本次改周解除。 + +> **W41 起换隔离考场**：本仓从 2026-W41 起的考试在隔离考场里进行，所以 W41 与 W38、W39 的各格跨期不可逐格对比（详见 [2026-W41 期文](results/2026-W41.md)）。W38、W39 的页面保持原样。 + +> **2026-10-07 更新**：品牌题 A-d9b79b46（UI）由 NA（挂起）改记为过：10-07 该案题面修改、驱动改用中性工作目录后，只重考了这一格，判分器 12/12 通过。step-5-preview 17'/24 → **18'/24**（过案 17→18，NA 4→3）。其他车道这一案的格子仍是旧题面下的成绩，只有 claude-fable-5-1 于 2026-10-07 在修改后的题面下重考过一次（负，11/12，见 [amber-claude 的 W40 页](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)）；四条 WorkBuddy 直连道这一案的格子没有重考。见 [2026-W41 期文](results/2026-W41.md)。 + +' = contested（安全拒答挂起）或 invalid（基建相关（考场 harness 或判分环境）的挂起、作废或待重评），均不计胜负；所有含 NA 的道都带撇号，包括冻结展示行；挂起不表示死因已定。StepFun（W41）：3 个 NA 格（1 个挂起、2 个超时作废），不计胜负。 + +用私有题库 **AMBER** 实测 StepFun 阶跃星辰在 **stepfun plan 端点**上服务的新旗舰 `step-5-preview`，只公开结果，不公开题目。 + +> **一句话**：step-5-preview 在 stepfun plan 端点的第二次考试（W41，2026-10-06，隔离考场；品牌题 10-07 重考）：24 案过 **18'/24**（18 胜 · 3 负 · 3 NA）。动手面的运维、交付、需求、收敛全过，编码 5/6；UI 1/1（品牌题，重考成绩）；审查 1/2、看图 0/1；防御、归因两轴本次全是 NA，没有读数。 +> +> 分数后的 `'` 表示其中有几案暂不计分（NA），既不算过也不算没过；NA 的原因写在期文里。W38 那一场是旧考场、23 案题集，两场不逐格对比，也不据此说模型变强或变弱。 + +## 成绩一览 + + + +![amber-stepfun 成绩一览：step-5-preview 逐轴过案数](results/assets/scoreboard.zh.png?v=20261007) + +| 大类 | 轴 | 考什么 | step-5-preview · [W41](results/2026-W41.md) | +|---|---|---|:-:| +| 施工面 | 编码 | 照着需求把功能写对 | 5/6 | +| | 交付 | 做完还得交得出东西 | 3/3 | +| | 运维 | 照规程干脏活 | 6/6 | +| | 需求 | 客户要 A 不要 B | 1/1 | +| | 收敛 | 真干完，不绕圈装忙 | 1/1 | +| 判断面 | UI | 照设计稿做页面 | 1/1 | +| | 视觉 | 给真截图挑毛病 | 0/1 | +| | 防御 | 堵死校验器的漏网口 | 0/2 · 2 NA | +| | 归因 | 毛病对到正确根因 | 0/1 · 1 NA | +| | 审查 | 给别人的交付物挑错 | 1/2 | +| | **合计** | | **18'/24** | + +每格 = 过了几案/该轴共几案（案 = 一道计分题）。NA = 这一案作废或暂停计分，不算过也不算没过；总分带 `'` 表示其中有 NA。多数轴只有 1–2 案，差一案读数就变，所以别把小差距当结论。各列考试周次相同（W41），具体日期可能不同，数字是当期快照。 + + + +## 这是什么 + +- 「道」= 同一个模型名在不同家的卖场/接口；「案」= 一道题，「卷」= 一场考试记录（一案多卷 = 一道题的几个变体场次）。 + +- 每期 `results/YYYY-Www.md`：同题、同 harness（跑考试并记分的程序），对目标模型跑全库（W38 为 23 案 / 26 卷，W41 起为 24 案 / 28 卷（含一卷重考））。 +- 一期固定报告：题集规模与哈希、每案得分与通过/失败、终端终态（程序跑完时的退出状态）、token 用量与时延、环境指纹、按证据纪律写的定性裁决。 +- 题目、oracle（判分器）、transcript（答题全过程记录）、中间产物**永不公开**（见下「发布纪律」）。 +- 姐妹仓：[amber-gpt](https://github.com/getaskclaw/amber-gpt)、[amber-crof](https://github.com/getaskclaw/amber-crof)、[amber-ollama](https://github.com/getaskclaw/amber-ollama)、[amber-devin](https://github.com/getaskclaw/amber-devin)、[amber-deepseek](https://github.com/getaskclaw/amber-deepseek)、[amber-commandcode](https://github.com/getaskclaw/amber-commandcode)、[amber-opencode](https://github.com/getaskclaw/amber-opencode)、[amber-workbuddy](https://github.com/getaskclaw/amber-workbuddy)、[amber-kimi](https://github.com/getaskclaw/amber-kimi)、[amber-doubao](https://github.com/getaskclaw/amber-doubao)、[amber-goldenpotato](https://github.com/getaskclaw/amber-goldenpotato)。 +- AMBER 是 agentic 实战题库（施工/运维/审查/视觉/需求漂移——题中要求中途变化），规范与制题工具见 [getaskclaw/amber](https://github.com/getaskclaw/amber)；考题本体私有。 + +## 渠道说明（本仓的特殊性） + +本仓考的是 StepFun 阶跃星辰官方 **plan 端点**（OpenAI 兼容面）上的新旗舰 `step-5-preview`（600B/27B MoE，官方声明 1M 上下文 + 视觉）。 + +因此本仓成绩带三条额外口径： + +1. **燃烧量不可验**：端点 usage **不返回 `reasoning_tokens`**，故「请求 high」只是请求标签，实际跑的是端点默认思维预算。本期不与其它仓做「同档」对比——band 列如实标为请求档 + 不可验。 +2. **视觉面端点事故**：发车前视觉请求在本端点零字节挂起（同端点另一模型同图正常答对），故相关案先判 not run（非失败），端点复测恢复后单案补考取真值。事故与补考过程写进 Findings。 +3. **加时帽收敛**：核验三案在放大墙钟帽后才落定，全卷墙钟因此偏重——跨仓比 wall 时须带此口径。 + +## 一分钟看懂 W41 + +stepfun plan 端点、请求 high（**燃烧量不可验**）、隔离考场、24 案（品牌题 A-d9b79b46 的题面 10-07 改过，重考卷是新哈希，其余同哈希）：**step-5-preview 18'/24**（18 胜 · 3 负 · 3 NA）。W38 的更正里公开为「挂起待补考」的三格本期重考：A-1fd3683a 过（2/2）；A-cdc3d11a（审查）和 A-ea80d793（看图）仍记负，这两案里模型只回了一句打算先做什么的话（A-cdc3d11a 里还带着写成文本的工具调用），回合就结束了，没有给出结论。3 个 NA：A-d511f9e8 在所有车道上记 NA（见更正声明）；A-a317e74b 和 A-be92627f 各试两次都撞到时间上限。品牌题 A-d9b79b46（UI）：10-06 首次作答时模型把工具调用写成文本、没有交出文件，题面与考场不一致，当时沿用 10-02 的先例记 NA；10-07 题面修改、驱动改用中性工作目录后只重考这一格，交出两个文件，12/12 通过，记过；其他车道这一案的格子仍是旧题面下的成绩，只有 claude-fable-5-1 于 2026-10-07 在修改后的题面下重考过一次（负，11/12，见 [amber-claude 的 W40 页](https://github.com/getaskclaw/amber-claude/blob/main/results/2026-W40.md)）；四条 WorkBuddy 直连道这一案的格子没有重考。逐案矩阵、考试条件与各 NA 的说明见 [2026-W41 期文](results/2026-W41.md)。 + +上面「渠道说明」的第 1 条（燃烧量不可验）对 W41 同样适用；第 2、3 条是 W38 当时的端点事故和加时帽，W41 没有放大时间帽重考，超时记 NA。 + +## W38 基线与 W39 补测 + +![W38 全库基线：step-5-preview 15/23；W39 补测另列](docs/images/w38-face-profile-correction.png?v=corrections-20260924-r2) + +stepfun plan 端点、请求 high（**燃烧量不可验**）、23 案同哈希：**step-5-preview 15/23**（公共 21 案子集 13/21）——**施工面顶级**：运维 6/6 全清 + 需求漂移四变体全过 + 编码 5/6（含全库唯一硬区分器 A-442d4aab 7/7 满分）+ 交付满分；**判断面掉队**：归因轴 0.800 反超对照锚 k3 的 0.667（同一案 12/15 对 10/15），但核验 0/3、审查净 −2、视觉 −2、前端废卷。三条口径须随行：① 端点不报 `reasoning_tokens`，行按端点默认档记；② 视觉面发车前在本端点零字节挂死（同端点另一模型同图正常），该案先判 not run、端点恢复后补考取真值；③ 核验三案撞标准帽后放大帽收敛（7200/10800/3600s），墙钟是本期最重的成本。逐案矩阵与车道账本见 [W38 基线与 W39 补测更正](results/2026-W38-correction-20260923.md)（原始期文保留旧 URL）。图源与 PNG 同目录（`docs/images/`，Vega-Lite）。 + +## 发布纪律（红线） + +1. 只发：分数与聚合、token 用量、速度、定性裁决。 +2. 永不发：题目内容、oracle/判分器、transcript、考生工作区、任何能复原题面的中间产物、端点访问凭证。 +3. 每期必钉：模型 ID、effort 档（思考力度档位）、日期（UTC）、harness 版本、每案内容哈希（bundle_sha，每题内容的哈希指纹）。哈希用于对照 [amber](https://github.com/getaskclaw/amber) 的公开哈希清单，自证题集未变。 +4. 案号与题目结构属私有面：公开结果里案例只用稳定别名（A-xxxxxxxx，哈希派生）+ bundle 哈希作句柄；内部案号、变体名、题目描述永不出现。 +5. 基调：这是对公开端点的实测，不是对任何厂商的攻击。数据说话，措辞克制。 + +## 一个方法论前提 + +同一模型、同一端点，两次跑也可能不同分——推理参数、负载、服务端版本都在漂。所以这里的一切结论都带日期与档位。单日数字是快照，不是定律。 + +## 结果索引 + +| 期 | 内容 | 结论 | +|---|---|---| +| [2026-W41](results/2026-W41.md) | step-5-preview，2026-10-06 第二次全库考试（隔离考场，24 案；品牌题 10-07 重考） | **18'/24**（18 胜 · 3 负 · 3 NA）；W38 挂起的 A-1fd3683a 过，A-cdc3d11a、A-ea80d793 仍记负；品牌题 A-d9b79b46 10-07 题面修改后重考，过；防御、归因两轴全是 NA。与 W38 那一场不逐格对比 | +| [2026-W38 基线](results/2026-W38-correction-20260923.md) | step-5-preview，2026-09-20 全库考试 | 基线 15/23；W39 收敛补测增加 1 案通过，组合 16'/24。端点实际思考用量不可验。[原始期文](results/2026-W39.md) | +| [此前全库复核](results/2026-W38-correction.md) | 历史更正：改判 0 格，挂起 4 格 | 旧文 W39 指旧期文名称；全库考试实属 W38，本次不解除挂起 | + +## 免责 + +与 StepFun 阶跃星辰团队无任何隶属/赞助关系。分数是特定日期、特定负载下的快照，不构成任何选型建议。