Repository navigation
feat(core): residual agent run reclamation ledger and automatic reconciliation - #112
Conversation
Add forward-only reclaim responsibility, evidence, target and disposition columns with constraints and scan indexes. Preserve legacy responsibility and fail clearly on historical cleared_at values. Reserve migration 0051 after upstream main 0050; update both migration READMEs and add normal migration-helper tests.
Append identity-bound provisioning, execution, delivery and runtime receipts without rewriting historical responsibility. Rescue original verified deliveries after business settlement and cleanup. Commit contradictions and a durable block before returning residual_fact_conflict; keep canonical facts and idempotent replays. Add real PostgreSQL and bufconn intake coverage using existing RPCs only.
Union original targets and require verified delivery plus real execution or per-target termination evidence before system authorization. Bind exactly one cleanup operation, recheck plan/permit/completion under the shared transaction lock, and atomically clear only after all targets and data removal close. Preserve public and project guards and retain the same retry_wait operation after quiesce refusal. Reserve controlled arbitration without exposing an operator command.
Scan reconciling and reclaiming rows fairly in bounded batches, preserve original responsibility on restart, and join the existing server cancellation lifecycle. Add a validated one-minute reconciliation interval and paired Chinese/English documentation. Document missing explicit abandonment and existing evidence-discovery limits; add no RPC, proto regeneration, physical termination mechanism or controlled management endpoint.
Exercise the real Cloud owned-delete topology, negative evidence, target union, dual termination, atomic rollback and restart without changing business settlement. Add controlled two-connection PostgreSQL interleavings for both commit orders across authorization, mode arbitration, completion, new uncovered executions, restart and cleared replays. The six D2 test functions passed with -race -count=3; keep physical and operator acceptance out of scope.
Treat only failed effects bound to the original residual cleanup operation as retryable observations, preserving both failure and later success. Keep successful conclusions and historical provisioning failures immutable. Accept completed cleanup receipts through the existing effect-result intake without restoring command authority or changing operation versions. Avoid UUID queries for absent Workspace scopes. Add both cleanup retry/epoch tests and real NULL-scope intake coverage; focused clone regressions, lint, formatting and the D2 race suite pass.
The existing Controller requests blocked for unconfirmed termination, but hidden run Workspaces cannot use the public retry action. Normalize only active unconflicted owned cleanup deferrals to retry_wait, retaining the same operation, error and delay while plans and permits recheck evidence. Extend the real Controller retry test to request blocked and prove the same operation resumes under the next epoch.
ElephantGit
left a comment
ElephantGit
left a comment
There was a problem hiding this comment.
审查基于 c7f4d1e22f0659afcf3bb33040b45d59f9bbbc7e，对照 specs main ffd6b5b222eda395e039560ccca02fb32e7b4e8e 的残留回收 ADR 与核心用例。
发现 1 项 P2：clone 的同序号异内容事件回执仍走旧的事务回滚路径，未持久化残留冲突审计与资源阻断，详见行内评论。请补齐该入口并增加覆盖 reconciling / reclaiming / cleared 的回归用例后复审。
已核查账本资格、owned delete 推进、事实接管、普通删除/Project 删除保护及 D2 并发测试。未发现另一项足够确定的阻断问题。42 个改动文件均为代码、测试或正式文档；本 PR 只实现 ADR 的部分范围，保持 approved 的说明清楚。
验证边界：已核实该 head 的 Backend CI 成功，包含格式、lint、proto 检查、真实 PostgreSQL 与 S3 配置下的全量 go test -race 及构建。本机没有可用 Go/PostgreSQL 工具链，本轮未重跑测试，缺陷由源码控制流确认。D2 用例证明数据库事务竞争，不等同于完整 Controller/跨仓故障验收。
当前 main 迁移最高为 0051，本 PR 的 0052 目前顺延正确；合并前仍需按届时 main 再核对。
| if !acceptResidualExecutionFact(t, e, result) { | ||
| return e | ||
| } |
There was a problem hiding this comment.
[P2] 将 clone 事件回执冲突也接入持久审计与阻断。已有残留账本的 clone 首次成功接管 (execution=X, sequence=N, result=R, event=A) 后，再用新的 submission 提交同一执行/序号/结果但 event=B：这里把 R 视为幂等事实，随后 287–288 行的 receipt_conflict 使整笔事务回滚，账本没有冲突审计或 reclaim_conflicted_at，后续回收仍可继续。这不满足残留回收 ADR D2 的冲突先落审计并阻断后续处置；同类 node execution 已调用 acceptResidualReceipt，但该 helper 只查 node_event_receipts，clone 尚未覆盖。请对 clone_event_receipts 增加对应的提交后报错路径，并用带回执的 TakeOverNodeEvent 覆盖 reconciling / reclaiming / cleared；现有 RecordProvisioningFact clone 测试不经过该回执分支。
Audit conflicting clone event receipts in the residual ledger before returning residual_fact_conflict, preserving the canonical result and receipt. Keep the ordinary receipt_conflict rollback and the node-execution path unchanged. Cover receipt takeover over gRPC in reconciling, reclaiming and cleared states, including committed conflict blocking, replay stability and the no-ledger path. Validation: checkformat, lint, race-enabled receipt regressions, internal/core tests and residual/clone-related integration tests passed with PostgreSQL.
Check residual clone receipts before importing execution facts so a rejected receipt only persists its conflict audit. Share the node and clone receipt helper, keeping node evidence keys stable and using a separate clone prefix. Add a gRPC regression for results held only in legacy responsibility facts. Verified checkformat, lint, receipt takeover tests with -race, core tests, and the residual/clone integration tests with PostgreSQL required.
Merge cloud main a9c5f20, keeping snapshot validation and residual receipt conflict auditing on the Workspace clone step. Move the residual migration after main's 0054 as 0055 and update upgrade coverage and both READMEs. Adapt residual Git delivery and clone fixtures to the remaining contracts, and remove the retired tenant-clone fixture field from transaction tests.
Restore main's bilingual 0050 migration explanation and document the intentional clone validation order and RPC fixture boundary. Verification: checkformat and lint (0 issues); core 301 pass; receipt race tests 5 pass; targeted integration 30 pass, with one known Windows git http-backend failure.
Remove the duplicate 0050 upgrade bullets introduced by the residual branch, leaving main's original paragraphs and only residual-specific migration additions. SQL and Go code are unchanged from the completed round-two gates.
Merge upstream main df9799c without rebasing. Keep both migration README additions, rename the unchanged residual SQL to 0057, synchronize migration tests and the actual 58-file count, and reserve 0056 for cloud#114 without a placeholder. WorkflowRun release records remain separate from the IssueRun residual ledger.
Merge upstream main 0db4230 without rebasing. Keep main's 0056 migration row and upgrade explanation alongside residual migration 0057, remove the obsolete cloud#114 reservation, and update both migration indexes to the actual 59 SQL files. Workflow invocation projection remains separate from agent residual ownership and deletion.
ElephantGit
left a comment
ElephantGit
left a comment
There was a problem hiding this comment.
增量复审通过，审查 commit：c8fabb8205740fdb545aae5eb87c8af8a933edf6。
上轮 clone 回执冲突问题已修复：回执检查先于结果接管；有残留账本时提交冲突审计及阻断后返回 residual_fact_conflict，保留原结果、回执与责任；无账本时保留 receipt_conflict 的回滚行为。新增真实 gRPC 用例直接覆盖 reconciling/reclaiming/cleared、同/新 submission 重放、后续 PlanEffect 阻断，以及 legacy lateFacts 不被拒绝的回执导入。未发现新的阻断问题。
已核对 main 合并带来的 Workspace clone / Git delivery 与 WorkflowRun 引用边界；43 个改动文件均为交付物。当前 cloud main 48b77fc 的迁移最高为 0056，本 PR 的 0057 目前无冲突，合并前仍须复核。ADR 只完成第 1 步，保持 approved，后续受控回收和跨仓验证边界已披露。
依据：specs main dcca919d53569b8d8710d8c73d8510db3b8715fa 的残留回收 ADR/核心用例。验证：该 head 的 Backend CI 39555677132 成功，含格式、lint、proto、全量 race 测试及构建；git diff --check 通过。本机无 Go/PostgreSQL，本轮未重跑测试。此结论不替代后续受控回收、真实跨仓语义验证和联合故障演练。
Refs ora-space/circus#348
#251 落地顺序的第 1 步（Cloud）：残留 Agent Run 账本状态机与自动对账层。依据是 specs main 上已批准的 残留回收 ADR 和 核心用例，不重开已批准的语义。实施方案与 A1–A6 的取舍已在 circus#348 留痕。
改动
0057_agent_run_residual_reclamation.sql，agent_run_residual.go）：agent_run_residuals只增列：reclaim_state（reconciling/reclaiming/cleared，存量默认reconciling）、disposition、单一责任reclaim_id/reclaim_mode/reclaim_operation_id/reclaim_authorization、原目标并集reclaim_targets、追加证据reclaim_evidence、reclaim_version、reclaim_checked_at、reclaim_conflicted_at。cleared_at时，迁移明确报错并回滚（A2）；不改旧reason、responsibility，也不改业务结算。agent_run_residual_fact.go，以及control.go、node.go、node_execution.go、clone.go、revision.go、runtime_control_delivery.go的既有入口）：RecordProvisioningFact、执行与交付事件、effect 结果、runtime ack 这些已有入口，迟到事实在三种状态下都追加进账本。residual_fact_conflict（A4）。clone_receipt:，与 Node 的receipt:分开，因为两张执行表没有跨表唯一约束。main 的base_commit_mismatch校验仍先于回执检查；结果本身不合法时直接拒绝，不写冲突审计。agent_run_residual_reclaim.go）：delete_workspaceoperation，按 quiesce → terminate → cleanup 推进。blocked，都留在同一 operation 的retry_wait；effect 失败属于可以重试的观察结果。重启、换 lease、重试都不会产生第二个 operation 或 effect，也不复活历史上失败的 operation（A3）。terminated和lateEnsureFenced，数据删除要removed。额外的历史 ID 缺少确认时，责任保持打开（A5）。cleared/released、把 Workspace 软删除；不改 Run 的业务终态。agent_run_residual_loop.go，cmd/server、internal/config）：独立于运行结算，按配置的间隔和批量推进；重启后从账本续上。deliver_revision真实登记并通过 main 的 Git 交付 ref 与 history 核验的 Revision，才能作为交付资格。失败、跳过、超时、没有结果都不算明确放弃（A1），这类条目保持reconciling。docs/agent-runs/residual-reclamation{,.en}.md；同步更新 IssueRun、core / config / server README、迁移 README 和 docs 索引。验证
当前 HEAD
c8fabb8：合入 main0db4230（#114）后的验证。Windows、真实 PostgreSQL，TEST_DATABASE_URL去掉外层引号并设置REQUIRE_POSTGRES=1；计数为 PASS / SKIP / FAIL，括号内含子测试。原始结果为main114-step1-*，没有加-timeout。go run ./cmd/checkformattask lintgo test -p 1 -json ./internal/core/... -count=1TestMigration*、0056 invocation migration-race -count=1）本地唯一失败是
TestGitServerGrantsACloneExecutionReadOnlyDuringItsStep：git server requires git http-backend，属于已有 Windows 限制；没有将失败计为通过。本轮筛选也保留TestDeliveryWithoutAGitServerIsSkipped；0056、0057 的迁移测试实际运行，没有 S3 SKIP。Backend CI（本次 HEAD）：39555677132，
success；integration 包ok，wall time 409.699 秒。使用仓库原有 CI 命令，未加-timeout。相对 main
0db4230，按顶层Test*函数名集合差集统计，本 PR 新增 14 个 core、10 个 integration 用例，不含子例，不把本 PR 内迁移改号另算一份测试。四个迁移用例的当前名称是：TestMigration0057ResidualReclamationFreshAndConstraintsTestMigration0057ResidualReclamationUpgradePreservesHistoryTestMigration0057ResidualReclamationReplayPreservesProgressTestMigration0057ResidualReclamationRejectsLegacyClearedRows新增 integration 包括
TestResidualCloneReceiptConflictDoesNotImportLegacyResultOverGRPC：通过真实TakeOverNodeEvent拒绝冲突回执，只新增 conflict，原结果和 lateFacts 不变。合并 main #183、#119、#114 的处理
本分支已合入 main
0db4230（含 #183、#119、#114），保留 main 的租户级 clone 退役与 Git 交付协议。已合入 #119 的 WorkflowRun Workspace release：它记录workflow_runs的释放状态，不创建或结清agent_run_residuals；共用delete_workspace完成入口按 Workspace 的 Run 绑定分流，残留责任仍匹配原reclaim_operation_id。新释放逻辑复用 D8 判断，但不接管 D8 历史残留。检查未发现需额外修复的语义冲突。clone.go保留base_commit_mismatch，并有意放在残留回执检查之前；只有合法结果才进入回执冲突检查，之后才接受执行事实。<0057的迁移。RecordDispatch；删除退役的legacyCloneFixture，不恢复租户级 clone。交付测试先推送 Git delivery ref，再检查 ref/history 核验、事务失败后的重试、重放不再访问 Git/对象存储，并断言 Revision 没有旧 bundle 字段。早期的四组 D2 直接事务竞争验证（保留历史证据，本轮未重复运行）：真实 PostgreSQL，两个独立连接上的两个事务，用 channel 屏障控制。B 在同一事务里先用 try-lock 观察到 A 未提交的锁，再阻塞等待生产使用的 advisory lock；不靠 sleep，也不是顺序调用。
TestResidualRaceAuthorizationVersusLateFactsBothCommitOrdersTestResidualRaceAutomaticVersusControlledClaimBothCommitOrdersTestResidualRaceCompletionVersusLateDeliveryBothCommitOrders、TestResidualRaceCompletionVersusNewUnclosedExecutionTargetsTestResidualRaceRestartRebuildsTwoStoresWithoutNewResponsibility、TestResidualRaceClearedReplayVersusLateFactBothCommitOrders-race -count=3下 6 个函数各 3 次通过（含子例 66 PASS / 0 SKIP / 0 FAIL），没有 race 报告。核心用例对应：账本状态机、迁移、自动对账、系统授权与保护、清理前后的迟到事实，这些都有本步的直接测试。下面几项只覆盖了 Cloud 这一段，后续步骤承接：
不在本 PR
approved；转implemented的 specs PR 由最后的落地运行负责。迁移编号
已合入 main
0db4230的0056_workflow_invocation_reference.sql，保留其表格行和升级说明；本 PR 的残留账本仍为0057_agent_run_residual_reclamation.sql，第 2 步 cloud#116 仍为 0058。两份 README 已删除过期说明，当前实际为 59 个 SQL 文件；本轮没有修改或重命名迁移 SQL。#114 先创建真实 WorkflowRun，再建立引用型 workflow IssueRun；agent dispatcher 和释放扫描跳过这种引用，状态投影不写 agent phase、Workspace 绑定或残留账本。残留仍要求 agent run 与 issue_run_id 绑定的 Workspace，WorkflowRun 的删除路径继续受未清残留保护。检查未发现需额外修复的交互。
🤖 Generated with Claude Code