Repository navigation
feat: controlled residual reclamation for deployment operators - #116
Conversation
Step 2 builds the controlled reclamation intent on the step 1 residual ledger (cloud#112), so this branch starts from it; once cloud#112 merges, main is merged in instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Persist deployment intents, tenant-wide command receipts and per-sandbox termination targets on the existing residual responsibility. Separate termination, explicit data deletion and external disposition, including identity-based ledgerless D8 takeover. Preserve business terminals and the first retained-data completion. Project targets through the existing force-stop RPCs and permit fencing without new protobufs, tenant routes or synthetic force-stop history. Keep automatic and controlled authority mutually exclusive, and retain late facts after completion. Validation: controlled DB, migration and bufconn tests passed in the scoped suite; six real two-transaction race tests passed with -race -count=3 (138 test events). Six affected Project/Workflow tests passed after fixing optional Workspace scope. Formatting, lint and cloudctl build passed. The full scoped suite and focused repairs retain the two documented Windows git http-backend exceptions.
Add residual list, get, terminate, delete-data and confirm-external commands using the deployment database credential. Record session_user as the authenticated actor and keep the optional operator label self-reported. Require explicit termination, data-deletion and external-responsibility confirmations. Keep migration/bootstrap syntax compatible, reject ambiguous flags and prevent infrastructure errors from printing connection details. Document the operator boundary and separate commands in Chinese and English. Validation: all four cloudctl tests passed in the scoped suite; cloudctl build, format check and lint passed.
The background-loop section still said controlled operator reclamation and D8 takeover were unimplemented; link the cloudctl residual entry instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ElephantGit
left a comment
There was a problem hiding this comment.
审查提交:59a244371797b2e1362966ee49ca178d9edbee64。本步增量范围为 de18d5e..59a2443 的三个提交。
新增的受控回收代码未发现可确认的新阻断问题;已核对终止与删数据分离、单一责任仲裁、逐目标双确认、幂等冲突审计、外部责任转交、D8 历史身份保护,以及 cloudctl 的输入和查询范围。
当前整体仍为 Request changes:此 head 继承了 #112 尚未修复的 clone receipt 冲突审计缺口。同一 execution/sequence/result 携带不同 event 时,cloneResult 仍会经旧 receipt_conflict 路径回滚,无法持久化冲突证据和阻断标记。具体意见已在 #112 的 review 中给出,这里不重复新增行内评论。请先完成该修复并同步进本分支,再按新的 head 请求复审;#112 合并前继续保持本 PR 不合并。
验证:已核对本 head 的 Backend CI,格式、lint、proto 检查、完整 go test -race -count=1 ... ./...(真实 PostgreSQL / S3)与构建均成功。本机未另行运行测试。Cloud 测试中的终止双确认仍是注入的结果,不能据此证明 Substrate 物理终止与墓碑;PR 已明确保留跨仓库联合验证及 ADR 的 approved 状态。
Register controlled termination through the management entry for legacy_d8 and residual_ledger roots before submitting a conflicting clone receipt. Check the durable intent or ledger audit, preserved canonical receipt and result, and blocked follow-up terminate and delete-data requests. Verified checkformat, lint, receipt takeover tests with -race, core tests, and residual/clone integration tests including controlled reclamation, with PostgreSQL required.
Merge step 1 after cloud main a9c5f20, preserving the residual clone receipt conflict path on Workspace clone steps. Move controlled reclamation to 0056 after the residual ledger's 0055 and update both migration READMEs and tests. Adapt controlled late-delivery verification tests to Git delivery refs and session history while preserving conflict auditing and business terminals. Use the real run Workspace for skipped-delivery replay so the controlled scope guard still runs before a missing Git Server skips the delivery.
Validate workspace IDs before querying controlled reclamation intents. The earlier fixture change from main's target 'w' to a real Workspace ID hid this guard's PostgreSQL 22P02 error; it did not fix skipped-delivery compatibility. Restore main's fixture and exercise the public enqueue entry with a durable skip and a non-UUID target. Verification after merging the updated residual ledger branch: checkformat, lint (0 issues), core tests, controlled/residual/clone integration, and receipt race tests. Targeted integration retains the known Windows git http-backend failure.
Reuse migrated PostgreSQL schemas and immutable Git sources only within the serial controlled suite. Reset business rows between cases and retain fresh Store hooks, service stacks, real RPCs, transaction barriers and both commit orders. Seed unrelated project prerequisites without dispatching a second workspace clone. Controlled core and integration tests with -race: 136.23s to 48.06s summed top-level duration, all 24 tests and 99 tests including subtests passed. Checkformat and lint pass (0 issues); core packages pass 318 top-level tests. Remaining targeted integration: 30 pass, one existing Windows git http-backend failure. No production, workflow, Taskfile or TestResidual changes.
Merge the updated residual-ledger branch, including upstream main df9799c and migration 0057. Keep WorkflowRun deletion settlement together with the existing controlled-operation guard. Rename unchanged controlled SQL to 0058; synchronize upgrade coverage and the actual 59-file migration index, reserving 0056 for cloud#114. Preserve the controlled fixture speedup and every scenario.
Merge the updated residual-ledger branch including main 0db4230. Preserve migrations 0056, 0057 and 0058 in order, remove the expired cloud#114 reservation and count the actual 60 SQL files in both indexes. Keep controlled reclamation and its fixture speedup unchanged.
ElephantGit
left a comment
There was a problem hiding this comment.
复审核对 head 6257136:上轮要求的 clone receipt 冲突修复已带入,新增真实登记 legacy_d8 / residual_ledger 意图后的 gRPC 回归覆盖持久审计、原回执不变与后续命令阻断;workspaceId 的 validID 修复也已带入。
当前仍不能通过合并门禁:Backend run 39556403820 的 integration 包在 600.281 秒超时(panic: test timed out after 10m0s),当前 head 没有完整通过的 CI。请按 circus#382 最新通知合入 main(本次已拉取 35b536f),更新已过时的 #112 未合并提示,在新 head 上取得完整 CI 通过后再请求复审。
迁移 0057 与本次 main 内容一致;新增 0058 目前接在 main 的 0057 后,合并前仍须重新核号。本轮为上次 review 后的增量与门禁核对,本机未运行测试,不给出新 head 的完整批准。
ElephantGit
left a comment
There was a problem hiding this comment.
已复审 head 9178923。本轮是对上一轮两项要求的增量复审:已合入 main 35b536f,受控回收 core/cloudctl 与迁移相对上一轮 head 无新增变更;PR diff 不再包含 0057,只新增 0058,描述中的等待 #112 提示已清理。当前 main 921e73e 最新迁移仍是 0057,0058 未撞号。git diff --check 通过。
已直接核对当前 head 的 Backend run 39561110022:格式、lint、proto 检查、完整 go test -race(真实 PostgreSQL/S3)和构建均成功。本轮未在本机重跑测试。结合前两轮代码审查,本轮未发现新增阻断问题,同意本 head。
ADR 保持 approved,物理执行与跨仓联合验收及最终 implemented 同步由后续运行承接;本批准不替代该验收。合并由 owner 决定,合并前请再次确认迁移编号。
Summary
残留回收 ADR 的部署受控回收(Cloud 侧):部署运维显式登记回收意图,Cloud 按原计划的每个 sandbox ID 派发终止,拿到逐目标双确认才算完成;删数据是单独的显式命令;无法物理确认时可以外部确认并转交责任。
第 1 步账本已由 #112 合入 main(
35b536f)。本 PR 仅包含第 2 步的受控回收意图与管理入口,新增迁移0058_agent_run_controlled_reclamation.sql;第 1 步的0057_agent_run_residual_reclamation.sql已在 main,不属于本 PR 的 diff。cloudctl residual list/get/terminate/delete-data/confirm-external,用部署数据库凭据在受控主机上运行,认证主体取 PostgreSQL 的session_user;--operator-label只是人工自述。没有新增 HTTP 路由、端口、角色或租户管理员入口,隐藏 Run Workspace 的公开 API 仍返回 404。(tenant, key)唯一的命令回执。D4/D5 的责任、目标并集和事实仍以第 1 步的账本为准;受控根意图在同一事务里取得controlled仲裁位。自动层已持有owned_delete时受控命令返回冲突;受控先取得时,自动层只收集事实、不能另建删除责任。registered → terminating → succeeded),完成后账本以terminated_data_retained闭环;delete_workspaceoperation,删完追加released,首次的cleared_at不变。terminated+lateEnsureFenced,删数据还要当前 epoch 的workspace_data_delete成功证据。404、不可达、超时、许可过期都不算。externally_confirmed闭环,不伪造物理成功证据。failed/node_unavailable、Run 的业务终态和结算历史保持原值。TestControlledResidualCloneReceiptConflictThroughRegisteredIntentsOverGRPC通过ManageResidual登记意图,再从真实TakeOverNodeEvent提交冲突回执,分别覆盖source=legacy_d8与source=residual_ledger。它断言原回执不变、追加冲突审计,并阻止后续 terminate/delete-data,不以手写 intent 行代替受控入口。controlledRootForWorkspace先用validID排除非 UUID。非法 Workspace ID 不可能命中受控意图,因此不应先触发 PostgreSQL22P02;main 的TestDeliveryWithoutAGitServerIsSkipped恢复原来的workspaceId: "w"夹具,已跳过交付仍返回skipped。此前改用真实 Workspace ID 只绕开了数据库错误,本轮改为修复保护查询,并新增真实 PostgreSQL 下通过公开Store.EnqueueExecutionWork的直接回归TestControlledResidualSkippedDeliveryWithInvalidWorkspaceID。RuntimeControlService里按根意图 ID 分流,映射到现有ListForceStops/GetEffectPermit/ConfirmForceStop的外形;不写伪造的runtime_force_stops记录,也不调用普通强停的业务收尾。Controller 收到的各字段含义见下表。TestResidual*。本地-race下 24 个顶层测试合计 136.23 → 48.06 秒;包括包清理的两包 wall time 合计 152.600 → 58.676 秒。0058_agent_run_controlled_reclamation.sql,以 main 已有的 0057 账本为基础;实际共有 60 个 SQL。本轮合入 main35b536f,未改迁移 SQL 或编号。RuntimeForceStopPlan.id/version/workspace_idterminating。effects[].ideffects[].request.sandbox_terminateproject_id、workspace_id和原计划的sandbox_instance_id,没有 sandbox 实例行也保留。GetEffectPermitforce_stop_id=根意图 ID、effect_id=目标行 ID;只有未完成意图的 planned 目标能拿到许可,有效期最多 10 秒且不超过 lease 截止。ConfirmForceStopcompleted表示根意图已全部闭环。DELETE_WORKSPACE/cleanup/WORKSPACE_DATA_DELETE),不包装成新的强停。Controller 一侧按任意强停计划处理这些目标,见 ora-space/desktop#710;Sandbox Server 的墓碑和迟到 ensure 拒绝见 ora-space/cluster#35。
ADR 状态:
decisions/cloud/issue-run/20261018-residual-agent-run-reclamation.md保持approved。核心用例九项中六项的 Cloud 范围有直接测试;"逐目标双确认""只在运维显式决定时重试与升级""D8 按身份接管"三项保持 Partial,物理执行与联合故障演练在 #251 第 4 步。Refs ora-space/circus#382
Evidence
当前 HEAD
9178923:合入 main35b536f(cloud#112 已合并)后的验证。Windows、真实 PostgreSQL,TEST_DATABASE_URL去掉外层引号并设置REQUIRE_POSTGRES=1;计数为 PASS / SKIP / FAIL,括号内含子测试。原始结果为merged112-step2-*,没有加-timeout。go run ./cmd/checkformattask lintgo test -p 1 -json ./internal/core/... -count=1TestControlledResidual*、residual/clone integration、全部TestMigration*、0056 invocation migration、TestDeliveryWithoutAGitServerIsSkipped-race -count=1)本地唯一失败是
TestGitServerGrantsACloneExecutionReadOnlyDuringItsStep:git server requires git http-backend,属于已有 Windows 限制;没有将失败计为通过。没有 S3 SKIP。相对
upstream/main的 PR diff:43 个文件,3165 行新增、91 行删除,迁移仅新增 0058;不含 0057。合并没有冲突。Backend CI(当前 HEAD):39561110022,
success;integration 包ok,wall time 527.086 秒。沿用仓库 CI 命令,未增加-timeout。Backend CI 已通过;未重触发或增加空提交。
#114 先创建真实 WorkflowRun,再建立引用型 workflow IssueRun;agent dispatcher 和释放扫描跳过这种引用,状态投影不写 agent phase、Workspace 绑定或残留账本。残留仍要求 agent run 与 issue_run_id 绑定的 Workspace,WorkflowRun 的删除路径继续受未清残留保护。检查未发现需额外修复的交互。
已合入 #119 的 WorkflowRun Workspace release:它记录
workflow_runs的释放状态,不创建或结清agent_run_residuals;共用delete_workspace完成入口按 Workspace 的 Run 绑定分流,残留责任仍匹配原reclaim_operation_id。新释放逻辑复用 D8 判断,但不接管 D8 历史残留。检查未发现需额外修复的语义冲突。提速测量(历史 HEAD
8f5b495,本轮保留其实现):本地真实 PostgreSQL,-race -count=1 -v,受控测试前后都为 24 个顶层、99 个含子测试 PASS,0 SKIP / FAIL。core 顶层 95.89 → 31.90 秒,integration 顶层 40.34 → 16.16 秒,合计 136.23 → 48.06 秒(35.3%,约三分之一);包含包清理的两包 Go wall time 合计 152.600 → 58.676 秒。它是提速修订前后的本地测量,不冒充合并新 main 后的 CI 时间。复用迁移完成的独立 schema 与只读 Git source;每例重置业务状态并新建 Store / 服务栈。没有减场景、断言、真实 RPC、两事务屏障或提交顺序,没有改生产代码、TestResidual*、workflow、Taskfile、Skip 或超时。六组真实事务竞争及两种提交顺序仍保留;早期
-race -count=318 次顶层通过属于历史证据,不代表这次本地又重跑。TestControlledResidualCloneReceiptConflictThroughRegisteredIntentsOverGRPC的legacy_d8/residual_ledger两个真实登记意图场景在本轮普通集成和回执-race均覆盖。Merge Danger
Door: one-way。新增迁移 0058(三张表与约束),不改旧迁移和 proto。
Blast Radius: 残留回收与强停派发。只有运维显式登记意图后才会派发受控终止;自动对账的行为不变。
cloudctl residual凭部署数据库凭据工作,如果 owner 要求独立的 HTTP 运维身份,需要另定认证方案(已在 circus#382 提出)。🤖 Generated with Claude Code