feat(core): residual agent run reclamation ledger and automatic reconciliation

cmd/server/README.en.md

@@ -16,6 +16,8 @@ The production entrypoint composes configuration, databases, HTTP/gRPC adapters
 
 `main.go` owns loops with one cancellation context and WaitGroup: optional catalog sync and workflow asset imports, Agent dispatch/startup, blocked windows, Thread end/cancellation, delivery/release recovery and Workflow scheduling. Core methods still decide state; loops do not make process-local state authoritative.
 
+The residual-ledger recovery loop calls `ReconcileAgentRunResidualsOnce` immediately at startup and then at `issue_runs.residual_reconcile_interval` (default `1m`), sharing that cancellation context and WaitGroup. It retains recovery responsibility after the business run ends and consumes existing fact entrypoints without adding a Controller discovery RPC or contract. Missing evidence leaves the ledger open; timeouts, reconnection and forced stops do not establish successful cleanup.
+
 Cleanup differs between shutdown and failure paths:
 
 - Signal-exit branch (SIGINT/SIGTERM): cancel background tasks and send the Controller drain signal; stop gRPC gracefully, forcing it to stop at the shared 10-second shutdown deadline, then call HTTP Shutdown with that same deadline, join background loops, close the database and flush logs.

cmd/server/README.md

@@ -16,6 +16,8 @@
 
 `main.go` 用同一个取消 context 与 WaitGroup 拥有循环：可选市场同步和工作流资产导入、Agent 派发/启动、blocked 窗口、Thread 结束/取消、交付/释放恢复，以及 Workflow 调度。具体状态判定仍由各 Core 方法负责；循环不把进程内状态当作权威。
 
+残留账本恢复循环调用 `ReconcileAgentRunResidualsOnce`，启动即运行一次，之后按 `issue_runs.residual_reconcile_interval`（缺省 `1m`）重试，并参与同一 context 的取消与 WaitGroup 等待。它在业务运行结束后继续承担恢复责任，只消费既有事实入口，不新增 Controller 发现 RPC 或契约；证据收不到就保留 open，不按超时、重连或强停推定清理成功。
+
 停机与失败路径的清理范围不同：
 
 - 信号退出分支（SIGINT/SIGTERM）：取消后台任务，发送 Controller drain 信号；gRPC 先优雅停止，到达与 HTTP 共用的 10 秒停机 deadline 时强制停止，然后调用 HTTP Shutdown，等待后台循环结束，再关闭数据库和刷新日志。

cmd/server/main.go

@@ -263,6 +263,15 @@ func run() (runErr error) {
 			pluginmarket.RunSyncLoop(ctx, store.GiveUpStaleWorkspaceReleasesOnce, agentDeliveryInterval, pluginmarket.ContextSleep, log)
 		}()
 	}
+	// Residual recovery remains independent of a run's business terminal state. Cloud consumes
+	// existing, identity-bound facts; a missing fact leaves its ledger open rather than treating
+	// a timer or reconnection as cleanup proof. This loop shares cancellation and shutdown joining
+	// with the delivery/release passes above and has no disable switch.
+	syncGroup.Add(1)
+	go func() {
+		defer syncGroup.Done()
+		pluginmarket.RunSyncLoop(ctx, store.ReconcileAgentRunResidualsOnce, cfg.IssueRuns.ResidualReconcileInterval, pluginmarket.ContextSleep, log)
+	}()
 	// Workflow advance loop: the Cloud-side scheduler (specs#99 D1 条件②/④, landing order step 3).
 	// A bounded rescan of pending and running workflow runs, each advanced in its own short
 	// transaction; the ready set is recomputed from committed state only, so the pass is idempotent

config.toml.template

@@ -8,6 +8,11 @@
 # `task setup` (cmd/devsetup) only turns the values below into what those services already accept:
 # CLOUD_*/GATEWAY_* environment overrides and TEST_DATABASE_URL in .local/dev.env, which
 # Taskfile.yml loads for every task, plus the client secret file the Gateway reads.
+# Cloud's residual recovery pass runs every 1m by default. Configure
+# issue_runs.residual_reconcile_interval in the Cloud YAML, or set
+# CLOUD_ISSUE_RUNS_RESIDUAL_RECONCILE_INTERVAL in the server environment. This TOML setup
+# template does not accept an [issue_runs] section. The interval must be positive; it is not
+# a cleanup-success deadline and there is no disable switch.
 
 [database]
 # PostgreSQL DSN shared by Cloud, the Gateway, `task migrate` and the test suite. Any PostgreSQL 17

docs/README.en.md

@@ -50,6 +50,7 @@ package live in that directory's `README.md`; the authoritative source for busin
 - [Agent IssueRuns: an Agent run from start to finish](agent-runs/issue-runs.en.md)
 - [Threads: the conversation of an Agent run](agent-runs/threads.en.md)
 - [Bounded startup: waiting limits and settling unknown outcomes](agent-runs/startup-settlement.en.md)
+- [Agent Run residual reclamation: Cloud authority and evidence](agent-runs/residual-reclamation.en.md)
 - [Revisions: delivery, verified registration and resume](agent-runs/revisions.en.md)
 - [Session reads: session lists, archives and model request statistics](agent-runs/session-reads.en.md)
 

docs/README.md

@@ -48,6 +48,7 @@
 - [Agent IssueRun：一次 Agent 运行的全过程](agent-runs/issue-runs.md)
 - [Thread：Agent 运行的对话](agent-runs/threads.md)
 - [有界启动：等待上限与未知结果的收尾](agent-runs/startup-settlement.md)
+- [Agent Run 残留回收：Cloud 责任与证据](agent-runs/residual-reclamation.md)
 - [Revision：交付、校验登记与续接](agent-runs/revisions.md)
 - [会话读取：会话列表、归档与模型请求统计](agent-runs/session-reads.md)
 

docs/agent-runs/issue-runs.en.md

@@ -20,6 +20,7 @@ This document is the overview; individual parts are covered in:
 | Creation (@Agent, the Project gate, the input snapshot) | [Issue collaboration](../collaboration/interactions.en.md#task-mode-agent) |
 | Thread entries, user turns, idleness and ending | [Threads](threads.en.md) |
 | Startup waiting limits and settling unknown outcomes | [Bounded startup](startup-settlement.en.md) |
+| D4/D5 residual ledger, system cleanup responsibility and evidence boundaries | [Residual reclamation](residual-reclamation.en.md) |
 | Object storage, upload grants, Revision registration and resume | [Revisions](revisions.en.md) |
 | Session lists, archives and model request statistics | [Session reads](session-reads.en.md) |
 
@@ -67,6 +68,12 @@ and it accepts no other run or interactive session.
 
 ## Background loops
 
+After D4/D5 business settlement, independent ledger reconciliation continues and grants one durable system owned-delete responsibility only
+when evidence qualifies. Retries keep the same operation; cleanup does not settle the Run again. The current scope is
+[Cloud step 1 of residual reclamation](residual-reclamation.en.md). Controlled operator endpoints, retained-data/external dispositions,
+Controller/Sandbox Server physical acceptance, joint acceptance and D8 historical takeover remain incomplete. Neither `done` nor a completed
+reconciliation pass proves that resources were reclaimed.
+
 Besides the synchronous transitions taken while evidence is taken over, the Cloud server process runs a
 set of loops every 10 seconds for the progress and settlement that have no external event to follow:
 

docs/agent-runs/issue-runs.md

@@ -15,6 +15,7 @@ Cloud 校验并登记之后删除运行 Workspace，释放沙盒和数据。
 | 怎么创建（@Agent、Project 关口、输入快照） | [Issue 协作](../collaboration/interactions.md#task-modeagent) |
 | Thread 条目、用户轮次、空闲与结束 | [Thread](threads.md) |
 | 启动的等待上限与未知结果收尾 | [有界启动](startup-settlement.md) |
+| D4/D5 残留账本、系统清理责任与证据边界 | [残留回收](residual-reclamation.md) |
 | 对象存储、上传授权、Revision 登记与续接 | [Revision](revisions.md) |
 | 会话列表、归档与模型请求统计 | [会话读取](session-reads.md) |
 
@@ -49,6 +50,10 @@ Cloud 在接管上一阶段证据的事务里写入。
 
 ## 后台循环
 
+D4/D5 业务收尾后的残留由独立账本对账继续处理，证据足够才授予一份持久的系统 owned-delete 责任；重试保持同一 operation，
+完成回收不重新结算 Run。当前仅覆盖 [残留回收的 Cloud 第 1 步](residual-reclamation.md)：受控运维入口、保留数据/外部确认、
+Controller/Sandbox Server 物理验收、跨仓联合验收和 D8 历史接管仍未完成；`done` 或对账已运行均不证明资源已回收。
+
 除了接管证据时的同步转换，Cloud server 进程里有一组每 10 秒执行一次的循环，负责那些没有外部事件可跟随的推进和收尾：
 
 | 循环 | 作用 |

docs/agent-runs/residual-reclamation.en.md

@@ -0,0 +1,106 @@
+# Agent Run residual reclamation: Cloud authority and evidence
+
+English | [中文](residual-reclamation.md)
+
+D4/D5 [bounded startup settlement](startup-settlement.en.md) can finish the business Run while retaining unresolved resources in
+`agent_run_residuals`. Cloud continues collecting evidence on that original ledger and only grants system cleanup authority when the
+evidence qualifies. Deletion follows the existing owned-delete path. It neither settles the [IssueRun](issue-runs.en.md) again nor reopens its Workspace.
+
+This page describes the Cloud step 1 implementation boundary. It is not evidence of physical termination by the Controller or Sandbox Server,
+deployment acceptance, or joint fault acceptance. The
+[residual reclamation ADR](https://github.com/ora-space/specs/blob/main/decisions/cloud/issue-run/20261018-residual-agent-run-reclamation.md)
+remains `approved`; this step does not declare the whole ADR implemented.
+
+## Ledger states and one durable responsibility
+
+| `reclaim_state` | Meaning | Authority fields |
+|---|---|---|
+| `reconciling` | Collect original-identity facts; deletion is not yet authorized | ID, mode and operation are NULL; authorization is `{}` |
+| `reclaiming` | One durable responsibility has been granted; continue or await evidence | Fixed `reclaim_id`, mode and nonempty authorization; `owned_delete` binds exactly one operation |
+| `cleared` | Disposition under the original responsibility is complete | Keep that responsibility, with both `cleared_at` and disposition |
+
+Step 1 production authorization uses only `owned_delete` and completes only as `released`. `controlled` reserves the shared internal arbitration
+point; it is neither a controlled management endpoint nor operator authority. The schema's `terminated_data_retained` and `externally_confirmed`
+dispositions are reserved for later controlled paths.
+
+Automatic responsibility stays bound to the same `delete_workspace` operation. Restarts, lease changes, retries and quiesce refusals do not
+create another responsibility or operation, and do not revive a failed historical operation. Ordinary deletion, dispatch and permit paths retain
+residual guards; an internal system responsibility is not a public deletion flag. The transaction rechecks the ledger, Run, Workspace, Project and
+operation tenant, original bindings and generation; a UUID foreign key alone does not establish that scope.
+
+The [migration](../../internal/core/migrations/README.en.md) only adds columns. Defaults initialize existing D4/D5 rows as `reconciling`, without
+inferring completion, backfilling D8, or changing old `reason`, `responsibility` or business settlement. A legacy row with non-null `cleared_at`
+causes an explicit error and rollback pending compatibility handling (A2).
+
+## Original-identity evidence and deletion eligibility
+
+`reclaim_targets` is a union of original plans and fact targets: original operations/effects, preallocated sandbox ensure IDs, sandboxes, Nodes,
+executions, clones, work and data. Late facts may add original targets; they cannot replace an old target with the current generation or drop an
+unknown target because a query did not find it.
+
+New facts append to `reclaim_evidence` with logical event keys, facts, sources and database receipt times, instead of changing old
+`responsibility.lateFacts`. An identical replay reuses its receipt; new evidence, additional targets or authority transitions increment
+`reclaim_version`. Evidence intake, authorization, permits and completion use Cloud's existing serialized transaction boundary. Completion must
+therefore reread the full target set and latest facts in its own transaction.
+
+System authorization requires located original identities, closed admission, no conflict, no blocking Project operation or other activity, and
+qualifying delivery and execution evidence. Execution evidence includes input closure and an explicit unfinished-execution list for the same
+runtime control identity, together with original execution terminals. Per-target termination evidence can also establish physical closure.
+A no-target proof applies only when original plans and execution relationships truly do not exist; missing rows, NULL results and elapsed time
+do not establish that proof.
+
+Step 1 accepts delivery eligibility only through a genuinely registered and verified Revision for the original `deliver_revision` execution;
+every other actual delivery execution needs its own completed verification. There is currently no unified existing durable source for explicit
+delivery abandonment. `failed`, `skipped`, business timeouts, absent object storage and missing results do not imply explicit abandonment (A1).
+Entries with insufficient evidence remain open.
+
+Authorized cleanup follows existing quiesce → terminate → cleanup. A quiesce refusal leaves the original operation in `retry_wait`, retaining
+the original authority and guards for evidence collection and retry (A3). It neither restores new execution eligibility, escalates to force-stop,
+nor creates another operation.
+When the existing Controller requests `blocked` for unconfirmed termination, this system responsibility instead persists claimable `retry_wait`;
+actual plans and permits still recheck evidence.
+
+A failed `sandbox_terminate` / `workspace_data_delete` effect bound to this responsibility is a retryable observation, not a disposal conclusion.
+After fresh evidence, the same effect may advance to success while retaining both observations. Historical provisioning/execution failures,
+external identity bindings and successful conclusions remain protected against contradictions.
+After completion, the existing effect-result intake only preserves original bound facts or audits conflicts; it cannot advance the operation,
+change its version or grant execution authority again.
+
+Every original sandbox/ensure ID requires a corresponding successful termination result containing both `terminated` and `lateEnsureFenced`.
+`terminated_at`, a failed ensure, a 404 or elapsed time alone is insufficient. The current safe target can reuse owned delete; an additional
+historical ID without both confirmations remains open, without replacing its identity or dispatching another termination effect (A5).
+Data-deletion planning, permits and completion recheck target coverage. One transaction confirms successful data deletion, succeeds the operation,
+soft-deletes the Workspace and writes `cleared/released`, without changing the Run's phase/status/result or repeating business settlement activity.
+
+## Existing fact intake and conflicts
+
+Cloud uses existing `runtime_ack`, creation-fact, execution-result and Revision intake. This step adds no RPC, proto or Controller discovery
+mechanism. After checking the original control epoch/version, Node, sandbox and generation, `runtime_ack` records input closure and the full
+`unfinishedExecutionIds` list on the ledger. Session terminals and delivery results are accepted under their original execution/work/resource
+identity. A done Run or cleared ledger is not a reason to reject a consistent late fact. Output rescue uses existing Revision verification;
+it does not open admission or start a new session or execution.
+
+Consistent replay preserves the original conclusion. A contradictory terminal first appends a conflict audit and sets `reclaim_conflicted_at`;
+the audit and block commit before the stable `409 residual_fact_conflict` is returned (A4). Canonical facts, existing `cleared_at` and disposition
+remain intact. Further deletion or reuse is blocked; this step provides no conflict-resolution endpoint.
+
+## Reconciliation, recovery and remaining scope
+
+`ReconcileAgentRunResidualsOnce` scans `reconciling/reclaiming` ledger rows independently of whether the business Run is still `releasing`.
+Bounded passes rotate by `reclaim_checked_at NULLS FIRST, created_at, run_id`; a checked time is scheduling metadata, not completion evidence.
+Each entry rereads targets, eligibility and original bindings. An existing responsibility resumes its original operation, subject to the current
+Controller lease and effect permits. Query failures, insufficient evidence and temporary refusals neither clear the ledger nor expand authority.
+
+If a failed create's original effects have no existing active-discovery path, they continue waiting for verifiable original-identity facts (A6).
+Cloud's evidence intake and responsibility scheduling do not establish new Controller discovery capability, or production validation of real
+containers, late-ensure fencing or physical data deletion.
+
+| Later scope | Step 1 boundary |
+|---|---|
+| Step 2: controlled operator endpoints, retained data, external confirmation | Not implemented; internal `controlled` arbitration does not supply authentication, intent or commands |
+| Step 3: Controller / Sandbox Server physical contracts and per-target evidence | This page states Cloud evidence requirements, not completed physical acceptance |
+| Step 4: cross-repository joint fault and recovery acceptance | Incomplete; single-repository implementation or database tests are not a substitute |
+| Taking over D8 historical deletion abandonment and residuals without ledger rows | Not implemented; no ledger backfill, automatic retry or historical business-result rewrite |
+
+See the [core code map](../../internal/core/README.en.md) for implementation entry points. Test results require their execution artifacts;
+this page does not claim migration, concurrency or end-to-end tests have passed.