Skip to content

fix(release): probe real external-wait fallback decision - #5023

Merged
huangruiteng merged 1 commit into
mainfrom
codex/release-host-prompt-probe
Sep 24, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/release-host-prompt-probe

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

The release-only host prompt probe could reject a correct agent decision because its external-wait case supplied an external_wait_observation field that the production quota result never emits. A model could advance the already selected fallback Todo and still fail the probe's synthetic external_wait oracle.

Replace that case with the existing typed external-wait fallback scenario and the compact CLI quota projection. The independent oracle now expects work on the selected fallback, a user notification, and no premature Goal closure. Keep the two independent attempts per thin/brief mode and the no-retry-until-pass rule. The release testing guide describes the actual case.

Validation on this head: 8 focused pytest cases, Ruff, mypy, public boundary scan of the three changed files, and the opt-in live Doubao host prompt probe (20/20 decisions). The risk-based premerge canary is required before merge and is reported in the exact-head review.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

PR #5023,精确 head 3082b2a76e7695979ac92cf7eb82441238eb999f。

动机

v1.2.0 的宿主提示词发布验收有一条假阴性:第五个“外部等待”用例加入了生产配额结果从未输出的 external_wait_observation 字段,却要求模型再次创建等待。真实语义是主任务已被监控、独立后续任务已被选中;模型推进后续任务并通知用户才正确。旧用例在 20 次真实调用中造成一项失败,持续重试不会修好验收依据。此 PR 修复一个独立的发布门槛,整版发布仍需随后完成精确提交验收。

改动思路

保留现有的四个静态 work、wait、replan 用例以及 thin/brief 双模式与逐次失败计数。第五个用例直接使用项目已有的外部等待 fallback 场景,经生产 CLI 配额压缩器形成宿主真正能收到的数据;预期答案独立写成 work + notify + 不关闭 Goal,不发给模型。配额仍是 selected Todo、执行义务和通知的唯一裁决者;发布探针只观察模型决策,不写 Goal、不改调度或权限。对比过“删除第五个用例”和“修改生产提示词”:前者失去真实等待回归,后者修错层,故都没有采用。

具体改动

loopx/control_plane/testing/host_prompt_behavior.py 删除虚构的 external_wait_observation 分支和对应布尔位,复用 external_wait_fallback_scenario_source() 与 compact_quota_should_run_cli_payload() 组成第五个用例。tests/control_plane/test_host_prompt_behavior.py 固定真实 selected_todo、must_attempt_work、NOTIFY,并证明虚构字段不存在;原有错误动作、错误类型和“后一次成功不得覆盖前一次失败”的负例继续执行。docs/development/testing-and-quality.md 把发布探针所测的第五种决策和其非执行边界写清楚。

关键代码讲解

  • cases() 是唯一改动的决策输入生产者:四个基础场景仍走原路径;第五个从真实 quota 构造可执行 fallback,隐藏预期答案为 work/notify/false。
  • test_probe_uses_current_production_prompts_and_hidden_independent_oracle() 先断言第五个包的关键类型事实,再验证两个模式都使用当前生产宿主提示且不泄露预期。
  • run_probe() 没改:每次答案仍独立解析与计数,任何一次失败都使汇总失败,探针结果只保留哈希和通过状态。

对主干的风险

主要风险是误把“决策模拟通过”当成真实宿主执行或整版发布通过;此 PR 和文档均明确限定为发布前的模型决策探针。另一风险是大一些的真实 quota 包导致调用变慢或输出波动:精确 head 的两种模式各重复两次,总共 20/20 通过,单次超时上限未放宽。相同脚本化模型答案在基线第五用例两次失败、当前 head 两次通过,说明修的是错误 oracle;8 项定向测试、Ruff、mypy、三文件公开边界扫描、有效质量回执及 11 项最终预合并检查均通过。首次预合并只因新 worktree 缺 npm 依赖导致 TS 解析器无法启动;补齐依赖后整套重跑通过,未把初次失败算成通过。目标 Goal 设置不等待 PR CI;本次未查询 PR CI。没有持久状态迁移或普通运行时入口变化。

我的整体评价

这是必要且比例合适的发布验收修复:复用现有 quota 事实,删掉伪状态,保留严格重复与失败门槛。它改善长期 fallback 工作不被误判为空转,普通用户路径与默认零模型调用保持一致。精确 head 已有真实模型、负例和风险检查;我给出 APPROVE 结论。合并后仍须对新的 main 提交重跑整版模型、测试、打包安装与发布回读,不能沿用旧提交的通过记录。用户已明确授权本批修复采用 admin bypass,合并还需以当前 head 的 readiness 结果为准。

English verdict: APPROVE - exact head 3082b2a; real typed fallback probe, 8 focused tests, 20/20 live decisions and 11 risk checks passed; final merged-release qualification remains separate.

@huangruiteng
huangruiteng merged commit 563de3b into main Sep 24, 2026
23 of 24 checks passed
@huangruiteng
huangruiteng deleted the codex/release-host-prompt-probe branch September 24, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant