Skip to content

feat: controlled residual reclamation for deployment operators - #116

Merged
EricWvi merged 15 commits into
ora-space:mainfrom
creegon2:feat/382-controlled-residual-reclaim
Oct 21, 2026
Merged

EricWvi merged 15 commits into
ora-space:mainfrom
creegon2:feat/382-controlled-residual-reclaim

Conversation

@creegon2

@creegon2 creegon2 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

残留回收 ADR 的部署受控回收(Cloud 侧):部署运维显式登记回收意图,Cloud 按原计划的每个 sandbox ID 派发终止,拿到逐目标双确认才算完成;删数据是单独的显式命令;无法物理确认时可以外部确认并转交责任。

第 1 步账本已由 #112 合入 main(35b536f)。本 PR 仅包含第 2 步的受控回收意图与管理入口,新增迁移 0058_agent_run_controlled_reclamation.sql;第 1 步的 0057_agent_run_residual_reclamation.sql 已在 main,不属于本 PR 的 diff。

  • 运维入口:cloudctl residual list/get/terminate/delete-data/confirm-external,用部署数据库凭据在受控主机上运行,认证主体取 PostgreSQL 的 session_user;--operator-label 只是人工自述。没有新增 HTTP 路由、端口、角色或租户管理员入口,隐藏 Run Workspace 的公开 API 仍返回 404。
  • 数据:新增三张表:受控意图、逐 sandbox 目标、按 (tenant, key) 唯一的命令回执。D4/D5 的责任、目标并集和事实仍以第 1 步的账本为准;受控根意图在同一事务里取得 controlled 仲裁位。自动层已持有 owned_delete 时受控命令返回冲突;受控先取得时,自动层只收集事实、不能另建删除责任。
  • 两阶段:
    • 每个资源只有一个 terminate 根意图(registered → terminating → succeeded),完成后账本以 terminated_data_retained 闭环;
    • 删数据是单独的子意图,绑定唯一一个固定的 delete_workspace operation,删完追加 released,首次的 cleared_at 不变。
  • 完成条件:账本里每个计划 sandbox ID(包括从未 ensure 落地的)都要拿到 terminated + lateEnsureFenced,删数据还要当前 epoch 的 workspace_data_delete 成功证据。404、不可达、超时、许可过期都不算。
  • 幂等与冲突:同租户同键、相同输入的重放返回原意图;同键不同输入先提交冲突审计再报错。
  • 外部确认:只能关闭未完成的意图,要求说明、逐个未闭环目标的外部引用和显式责任转交,以 externally_confirmed 闭环,不伪造物理成功证据。
  • D8 历史残留:按原 Workspace、失败的 delete operation 和 sandbox 身份显式接管,不回填账本;旧 operation 的 failed/node_unavailable、Run 的业务终态和结算历史保持原值。
  • 真实登记意图后的 clone 冲突:TestControlledResidualCloneReceiptConflictThroughRegisteredIntentsOverGRPC 通过 ManageResidual 登记意图,再从真实 TakeOverNodeEvent 提交冲突回执,分别覆盖 source=legacy_d8 与 source=residual_ledger。它断言原回执不变、追加冲突审计,并阻止后续 terminate/delete-data,不以手写 intent 行代替受控入口。
  • 保留 main 的跳过交付行为:controlledRootForWorkspace 先用 validID 排除非 UUID。非法 Workspace ID 不可能命中受控意图,因此不应先触发 PostgreSQL 22P02;main 的 TestDeliveryWithoutAGitServerIsSkipped 恢复原来的 workspaceId: "w" 夹具,已跳过交付仍返回 skipped。此前改用真实 Workspace ID 只绕开了数据库错误,本轮改为修复保护查询,并新增真实 PostgreSQL 下通过公开 Store.EnqueueExecutionWork 的直接回归 TestControlledResidualSkippedDeliveryWithInvalidWorkspaceID。
  • 不新增 RPC / proto:受控目标在现有 RuntimeControlService 里按根意图 ID 分流,映射到现有 ListForceStops / GetEffectPermit / ConfirmForceStop 的外形;不写伪造的 runtime_force_stops 记录,也不调用普通强停的业务收尾。Controller 收到的各字段含义见下表。
  • 受控测试提速:只在串行受控测试内复用已迁移的独立 PostgreSQL schema 和只读 Git source,每个场景仍重置业务状态、新建 Store / 服务栈;竞争屏障、两种提交顺序、真实 RPC 和原断言保留。没有生产代码、workflow、Taskfile、超时或 Skip 改动,也没有修改 TestResidual*。本地 -race 下 24 个顶层测试合计 136.23 → 48.06 秒;包括包清理的两包 wall time 合计 152.600 → 58.676 秒。
  • 迁移:仅新增 0058_agent_run_controlled_reclamation.sql,以 main 已有的 0057 账本为基础;实际共有 60 个 SQL。本轮合入 main 35b536f,未改迁移 SQL 或编号。
  • 残留回收、cloudctl、core、migrations 的中英文说明已同步。
Controller 收到的字段 受控回收中的值
RuntimeForceStopPlan.id / version / workspace_id terminate 根意图 UUID / 意图当前版本 / 原隐藏 Run Workspace。首次列出时意图进入 terminating。
effects[].id 持久目标行 UUID(与 sandbox ID 分开),重试不换。
effects[].request.sandbox_terminate 原 project_id、workspace_id 和原计划的 sandbox_instance_id,没有 sandbox 实例行也保留。
GetEffectPermit force_stop_id=根意图 ID、effect_id=目标行 ID;只有未完成意图的 planned 目标能拿到许可,有效期最多 10 秒且不超过 lease 截止。
ConfirmForceStop 两个布尔值都为 true 才计入;completed 表示根意图已全部闭环。
删数据 走原 operation 的 claim/plan/result/advance 协议(DELETE_WORKSPACE / cleanup / WORKSPACE_DATA_DELETE),不包装成新的强停。

Controller 一侧按任意强停计划处理这些目标,见 ora-space/desktop#710;Sandbox Server 的墓碑和迟到 ensure 拒绝见 ora-space/cluster#35。

ADR 状态:decisions/cloud/issue-run/20261018-residual-agent-run-reclamation.md 保持 approved。核心用例九项中六项的 Cloud 范围有直接测试;"逐目标双确认""只在运维显式决定时重试与升级""D8 按身份接管"三项保持 Partial,物理执行与联合故障演练在 #251 第 4 步。

Refs ora-space/circus#382

Evidence

当前 HEAD 9178923:合入 main 35b536f(cloud#112 已合并)后的验证。Windows、真实 PostgreSQL,TEST_DATABASE_URL 去掉外层引号并设置 REQUIRE_POSTGRES=1;计数为 PASS / SKIP / FAIL,括号内含子测试。原始结果为 merged112-step2-*,没有加 -timeout。

Gate 结果
go run ./cmd/checkformat 退出 0
task lint 退出 0,0 issues
go test -p 1 -json ./internal/core/... -count=1 338 / 0 / 0(817 / 0 / 0),退出 0
TestControlledResidual*、residual/clone integration、全部 TestMigration*、0056 invocation migration、TestDeliveryWithoutAGitServerIsSkipped 63 / 0 / 1(126 / 0 / 1),退出 1
回执 integration(含真实登记受控意图,-race -count=1) 6 / 0 / 0(15 / 0 / 0),退出 0,无 race 报告

本地唯一失败是 TestGitServerGrantsACloneExecutionReadOnlyDuringItsStep:git server requires git http-backend,属于已有 Windows 限制;没有将失败计为通过。没有 S3 SKIP。

相对 upstream/main 的 PR diff:43 个文件,3165 行新增、91 行删除,迁移仅新增 0058;不含 0057。合并没有冲突。

Backend CI(当前 HEAD):39561110022,success;integration 包 ok,wall time 527.086 秒。沿用仓库 CI 命令,未增加 -timeout。

Backend CI 已通过;未重触发或增加空提交。

#114 先创建真实 WorkflowRun,再建立引用型 workflow IssueRun;agent dispatcher 和释放扫描跳过这种引用,状态投影不写 agent phase、Workspace 绑定或残留账本。残留仍要求 agent run 与 issue_run_id 绑定的 Workspace,WorkflowRun 的删除路径继续受未清残留保护。检查未发现需额外修复的交互。

已合入 #119 的 WorkflowRun Workspace release:它记录 workflow_runs 的释放状态,不创建或结清 agent_run_residuals;共用 delete_workspace 完成入口按 Workspace 的 Run 绑定分流,残留责任仍匹配原 reclaim_operation_id。新释放逻辑复用 D8 判断,但不接管 D8 历史残留。检查未发现需额外修复的语义冲突。

提速测量(历史 HEAD 8f5b495,本轮保留其实现):本地真实 PostgreSQL,-race -count=1 -v,受控测试前后都为 24 个顶层、99 个含子测试 PASS,0 SKIP / FAIL。core 顶层 95.89 → 31.90 秒,integration 顶层 40.34 → 16.16 秒,合计 136.23 → 48.06 秒(35.3%,约三分之一);包含包清理的两包 Go wall time 合计 152.600 → 58.676 秒。它是提速修订前后的本地测量,不冒充合并新 main 后的 CI 时间。复用迁移完成的独立 schema 与只读 Git source;每例重置业务状态并新建 Store / 服务栈。没有减场景、断言、真实 RPC、两事务屏障或提交顺序,没有改生产代码、TestResidual*、workflow、Taskfile、Skip 或超时。

六组真实事务竞争及两种提交顺序仍保留;早期 -race -count=3 18 次顶层通过属于历史证据,不代表这次本地又重跑。TestControlledResidualCloneReceiptConflictThroughRegisteredIntentsOverGRPC 的 legacy_d8 / residual_ledger 两个真实登记意图场景在本轮普通集成和回执 -race 均覆盖。

Merge Danger

Door: one-way。新增迁移 0058(三张表与约束),不改旧迁移和 proto。

Blast Radius: 残留回收与强停派发。只有运维显式登记意图后才会派发受控终止;自动对账的行为不变。cloudctl residual 凭部署数据库凭据工作,如果 owner 要求独立的 HTTP 运维身份,需要另定认证方案(已在 circus#382 提出)。

🤖 Generated with Claude Code

creegon2 and others added 4 commits October 20, 2026 07:22
Step 2 builds the controlled reclamation intent on the step 1 residual ledger (cloud#112), so
this branch starts from it; once cloud#112 merges, main is merged in instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Persist deployment intents, tenant-wide command receipts and per-sandbox termination
targets on the existing residual responsibility. Separate termination, explicit
data deletion and external disposition, including identity-based ledgerless D8
takeover. Preserve business terminals and the first retained-data completion.

Project targets through the existing force-stop RPCs and permit fencing without
new protobufs, tenant routes or synthetic force-stop history. Keep automatic and
controlled authority mutually exclusive, and retain late facts after completion.

Validation: controlled DB, migration and bufconn tests passed in the scoped suite;
six real two-transaction race tests passed with -race -count=3 (138 test events).
Six affected Project/Workflow tests passed after fixing optional Workspace scope.
Formatting, lint and cloudctl build passed. The full scoped suite and focused
repairs retain the two documented Windows git http-backend exceptions.
Add residual list, get, terminate, delete-data and confirm-external commands using
the deployment database credential. Record session_user as the authenticated
actor and keep the optional operator label self-reported. Require explicit
termination, data-deletion and external-responsibility confirmations.

Keep migration/bootstrap syntax compatible, reject ambiguous flags and prevent
infrastructure errors from printing connection details. Document the operator
boundary and separate commands in Chinese and English.

Validation: all four cloudctl tests passed in the scoped suite; cloudctl build,
format check and lint passed.
The background-loop section still said controlled operator reclamation and
D8 takeover were unimplemented; link the cloudctl residual entry instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@ElephantGit ElephantGit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查提交:59a244371797b2e1362966ee49ca178d9edbee64。本步增量范围为 de18d5e..59a2443 的三个提交。

新增的受控回收代码未发现可确认的新阻断问题;已核对终止与删数据分离、单一责任仲裁、逐目标双确认、幂等冲突审计、外部责任转交、D8 历史身份保护,以及 cloudctl 的输入和查询范围。

当前整体仍为 Request changes:此 head 继承了 #112 尚未修复的 clone receipt 冲突审计缺口。同一 execution/sequence/result 携带不同 event 时,cloneResult 仍会经旧 receipt_conflict 路径回滚,无法持久化冲突证据和阻断标记。具体意见已在 #112 的 review 中给出,这里不重复新增行内评论。请先完成该修复并同步进本分支,再按新的 head 请求复审;#112 合并前继续保持本 PR 不合并。

验证:已核对本 head 的 Backend CI,格式、lint、proto 检查、完整 go test -race -count=1 ... ./...(真实 PostgreSQL / S3)与构建均成功。本机未另行运行测试。Cloud 测试中的终止双确认仍是注入的结果,不能据此证明 Substrate 物理终止与墓碑;PR 已明确保留跨仓库联合验证及 ADR 的 approved 状态。

Register controlled termination through the management entry for legacy_d8
and residual_ledger roots before submitting a conflicting clone receipt.
Check the durable intent or ledger audit, preserved canonical receipt and
result, and blocked follow-up terminate and delete-data requests.

Verified checkformat, lint, receipt takeover tests with -race, core tests,
and residual/clone integration tests including controlled reclamation,
with PostgreSQL required.
Merge step 1 after cloud main a9c5f20, preserving the residual clone receipt
conflict path on Workspace clone steps. Move controlled reclamation to 0056
after the residual ledger's 0055 and update both migration READMEs and tests.

Adapt controlled late-delivery verification tests to Git delivery refs and
session history while preserving conflict auditing and business terminals.
Use the real run Workspace for skipped-delivery replay so the controlled
scope guard still runs before a missing Git Server skips the delivery.
Validate workspace IDs before querying controlled reclamation intents. The earlier fixture change from main's target 'w' to a real Workspace ID hid this guard's PostgreSQL 22P02 error; it did not fix skipped-delivery compatibility. Restore main's fixture and exercise the public enqueue entry with a durable skip and a non-UUID target.

Verification after merging the updated residual ledger branch: checkformat, lint (0 issues), core tests, controlled/residual/clone integration, and receipt race tests. Targeted integration retains the known Windows git http-backend failure.
Reuse migrated PostgreSQL schemas and immutable Git sources only within the serial controlled suite. Reset business rows between cases and retain fresh Store hooks, service stacks, real RPCs, transaction barriers and both commit orders. Seed unrelated project prerequisites without dispatching a second workspace clone.

Controlled core and integration tests with -race: 136.23s to 48.06s summed top-level duration, all 24 tests and 99 tests including subtests passed. Checkformat and lint pass (0 issues); core packages pass 318 top-level tests. Remaining targeted integration: 30 pass, one existing Windows git http-backend failure. No production, workflow, Taskfile or TestResidual changes.
Merge the updated residual-ledger branch, including upstream main df9799c and migration 0057. Keep WorkflowRun deletion settlement together with the existing controlled-operation guard. Rename unchanged controlled SQL to 0058; synchronize upgrade coverage and the actual 59-file migration index, reserving 0056 for cloud#114. Preserve the controlled fixture speedup and every scenario.
Merge the updated residual-ledger branch including main 0db4230. Preserve migrations 0056, 0057 and 0058 in order, remove the expired cloud#114 reservation and count the actual 60 SQL files in both indexes. Keep controlled reclamation and its fixture speedup unchanged.

@ElephantGit ElephantGit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审核对 head 6257136:上轮要求的 clone receipt 冲突修复已带入,新增真实登记 legacy_d8 / residual_ledger 意图后的 gRPC 回归覆盖持久审计、原回执不变与后续命令阻断;workspaceId 的 validID 修复也已带入。

当前仍不能通过合并门禁:Backend run 39556403820 的 integration 包在 600.281 秒超时(panic: test timed out after 10m0s),当前 head 没有完整通过的 CI。请按 circus#382 最新通知合入 main(本次已拉取 35b536f),更新已过时的 #112 未合并提示,在新 head 上取得完整 CI 通过后再请求复审。

迁移 0057 与本次 main 内容一致;新增 0058 目前接在 main 的 0057 后,合并前仍须重新核号。本轮为上次 review 后的增量与门禁核对,本机未运行测试,不给出新 head 的完整批准。

@ElephantGit ElephantGit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已复审 head 9178923。本轮是对上一轮两项要求的增量复审:已合入 main 35b536f,受控回收 core/cloudctl 与迁移相对上一轮 head 无新增变更;PR diff 不再包含 0057,只新增 0058,描述中的等待 #112 提示已清理。当前 main 921e73e 最新迁移仍是 0057,0058 未撞号。git diff --check 通过。

已直接核对当前 head 的 Backend run 39561110022:格式、lint、proto 检查、完整 go test -race(真实 PostgreSQL/S3)和构建均成功。本轮未在本机重跑测试。结合前两轮代码审查,本轮未发现新增阻断问题,同意本 head。

ADR 保持 approved,物理执行与跨仓联合验收及最终 implemented 同步由后续运行承接;本批准不替代该验收。合并由 owner 决定,合并前请再次确认迁移编号。

@EricWvi
EricWvi merged commit 6fb86d1 into ora-space:main Oct 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants