Repository navigation
Fence restore termination and partial delivery on original stop evidence - #712
Conversation
EricWvi
left a comment
There was a problem hiding this comment.
Request changes on 01f56f77d563b73b6ea76a0766b3c8b3ba9d1c31.
Blocking: the v10 backfill fences completed historical restores forever.
restore.sql marks every node_executions row with payload.spec.prior_revision as legacy_unknown, with no state filter. Nothing ever moves a row out of legacy_unknown (set_restore_phase only allows git_admitted → restored → agent_admitted), and two fences read it:
unfinished_runtime_executions(crates/node-db/src/runtime_control.rs:113) includes everylegacy_unknownrow, so accepting a new runtime binding that is not input-closed returnsResourceConflictpermanently.- The checkout conflict check (
crates/node-db/src/session.rs:117) blocks any new session on thatcheckout_execution_idpermanently.
In-flight legacy executions are already covered by state<>'completed' in both queries, so the only extra effect of the backfill is on completed history. The DB lives at home_directory/ora-node.sqlite3 and is never rebuilt, so after upgrading, any Node that has ever restored a prior revision can no longer accept a new runtime binding. The ADR's D6 fence is for executions whose stop evidence is unknown, and it hands release to the residual-reclaim ADR (specs#126: real evidence, or explicit operator confirmation). This PR adds no such exit for legacy_unknown.
Please either:
- limit the backfill to executions that are not yet completed (old terminals stay as history and do not fence), or
- if completed legacy restores really must fence, add the release path specs#126 requires (evidence or explicit operator confirmation), and a test that an upgraded DB with completed prior-revision history can still accept a runtime binding, or explicitly documents why it cannot.
Either way, please add an upgrade test with a completed prior-revision execution and assert what the runtime-binding and checkout fences do.
Non-blocking:
ExecutionStatus.stop_unconfirmedis additive withserde(default); please confirm the Controller/Cloud decoders on main tolerate the new field (no strictdeny_unknown_fieldson that path).
ae65096 to
616ef10
Compare
creegon2
left a comment
There was a problem hiding this comment.
复审对象:desktop#712 @ 616ef100(PR 基于 main 391e62a4),cloud#117 @ c707072b(PR 基于 main 9814cacd)。依据:specs main 911c976 中的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md(D1/D4/D6)。本轮改动(01f56f77..616ef100 去掉 rebase 带入的 main 提交后只有 616ef100 一个提交,range-diff 确认前两个提交与上轮内容相同)和两份 PR 的全部内容都看过。
结论:Request changes
两项阻断:一是和最新 main 冲突(main 已用掉 schema v10,恢复也已改走 Git Server);二是升级时还在进行的旧 prior-revision 会话会被永久拦住,而且拦的是整个 Node。另有一项应修:已完成的旧会话如果交付还没做完,升级后交付会被判为最终失败。
P1 阻断 — 与最新 main 冲突:v10 已被占用,恢复路径已更换
- 现状:desktop main 现在是
ff9de8f9,已合入 desktop#681(内部 Git 线)和 desktop#709。gh pr view 712 --json mergeable返回CONFLICTING / DIRTY。用git merge-tree --write-tree ff9de8f9 616ef100检查,有 24 个文件冲突,包括crates/node-db/src/schema.rs、apps/ora-node/src/revision/restore.rs、revision/restore/{download,git}.rs、revision/snapshot.rs、managed/{delivery,transport}.rs、service/{agents,revisions,worker}.rs、session/{driver,ports,resume}.rs和四份 docs。 - main 的
crates/node-db/src/schema.rs:44已经允许版本1..=10,:82、:192两处的 v10 都是branch_heads.sql。本 PR 的restore.sql也写成 v10,合并后同一个版本号会对应两种 schema。 - main 的
apps/ora-node/src/revision/restore.rs已经从 bundle 下载改成 Git Server 拉取加传输仓库。本 PR 的 D1/D6/D4 是在 bundle 路径上实现的(download.rs、bundle 临时目录、ora-restore-*.bundle清理)。PR 描述里写的"Git Server 路径的接入留给后续",现在已经不能作为合并时的前提。ADR D5 要求截止、取消和停止确认的语义对两种介质都成立,而 main 上现在只剩 Git Server 这一种。 - 要求:rebase 到最新 main,schema 改成 v11;把 D1(全程截止和命令预算裁剪)、D6(原 Scope/Run 持久化、停止确认、上限、未知停止路径)、D4(未完成恢复只读 HEAD)接到 Git Server 恢复路径上,包括传输仓库
.ora-transport/的启动清理门禁(ADR D5、D6 的清理规则);在新 head 上重新跑本轮测试和 CI。补充一点:616ef100的 CI(run 39530595183,02:00:46Z 开始)是在 desktop#709 合入(02:02:37Z)之前跑的,#709 改的正好是 Scope 关闭确认的协调逻辑,rebase 后的 CI 必须覆盖它。
P1 阻断 — 升级时还在进行的旧 prior-revision 会话会被永久拦住,范围是整个 Node
-
代码:
crates/node-db/src/restore.sql:15-17:所有state<>'completed'的带 prior 输入的旧执行都回填为legacy_unknown。apps/ora-node/src/service/agents.rs:82-85:reconcile遇到legacy_unknown直接continue,不会结束这个会话。main 在启动时会以 Interrupted 结束所有未完成的会话(main 版service/agents.rs:180的recover_interrupted加end_session),本 PR 把这一步去掉了。crates/node-db/src/session/events.rs:67-80:Record::Ended对legacy_unknown一律返回ResourceConflict,所以没有任何路径能给它写终态。crates/node-db/src/session/restore.rs:44-58:set_restore_phase只允许git_admitted → restored → agent_admitted,没有离开legacy_unknown的转换,也没有运维释放的入口。crates/node-db/src/runtime_control.rs:112-113:unfinished_runtime_executions同时包含phase='legacy_unknown'和这个永远不会 completed 的node_executions行。这个列表一旦非空,bind_runtime(runtime_control.rs:73-98)在新 runtime generation 或输入未关闭时都会拒绝;所有带 permit 的准入也会拒绝,见session.rs:121、plugin.rs:73、repository.rs:91、revision.rs:129。同一 checkout 的新会话还会被session.rs:117拒绝。
-
失败场景:v9 Node 上有一个带
prior_revision的会话,恢复早已完成,Agent 已经在运行并产生了 thread 记录。Node 为升级而重启。升级后:- 这个会话永远不会结束,它的交付也被拒绝(
service/revisions.rs:178,以及inputs()要求会话已 Completed); - 这个 Node 之后所有受控准入都返回
ResourceConflict或InvalidTransition,包括 runtime 绑定、会话、插件、仓库和 Revision; - 重启 Node 不会改变任何一项。
按 v9 的执行顺序,恢复在 Agent 启动之前同步完成(
session/driver.rs:129-138),所以对这类会话来说,"恢复 Git 进程可能还在写入"并不成立,ADR D6 拦截的理由不适用于它们。 - 这个会话永远不会结束,它的交付也被拒绝(
-
上轮 owner 说"在途的遗留执行本来就被
state<>'completed'覆盖"。这句话在 main 上成立,因为 main 启动时会把这些执行结束掉。本 PR 同时去掉了这个结束出口,所以在途的旧行和上轮被指出的已完成旧行是同一个问题:legacy_unknown没有出口。PR 自带的upgrade_keeps_inflight_legacy_restore_fenced只断言了拦截存在,没有任何测试说明拦截怎样解除。 -
验证:在 review worktree 里加了一个临时测试。先用 v10 代码建库,接受并启动一个带 prior 的会话,追加一条 thread 事件,然后用 PR 自带的
reopen_after_v9_downgrade退回 v9 再重新打开。输出:phase=Some(LegacyUnknown),end_session=Err(ResourceConflict),unfinished=["session-execution"],bind_runtime=Err(ResourceConflict);关闭数据库再打开一次,end_session和bind_runtime结果不变。测试已删除。 -
要求,二选一:
- 按 specs#126 给
legacy_unknown一个出口(真实停止证据,或运维显式确认后释放),并测试升级后的在途行最终能解除这两处拦截; - 缩小回填范围,把旧记录能证明恢复已经完成的会话排除在外(例如已经有 Agent 产生的 thread 事件;请作者核实这个判据在 v9/v10 上是否可靠),其余仍按未知处理。
无论选哪种,都要补一个升级测试:输入是一条升级前已有 Agent 记录、还未完成的 prior 会话,断言升级并启动 reconcile 之后,会话怎样结束、两处拦截怎样解除。rebase 后这里是 v10→v11 的升级,main 上的 v10 会话同样没有恢复 journal,所以这个问题在 rebase 后依然存在。
- 按 specs#126 给
P2 应修 — 已完成的旧 prior 会话,升级后交付被判为最终失败
- 代码:
apps/ora-node/src/service/revisions.rs:294-306:本轮改动之后,已完成的旧 prior 会话在let mode = match (session.command.payload.spec.prior_revision.is_some(), phase) { (true, Some(Pending | GitAdmitted)) => SnapshotMode::UnstartedRestore, (true, Some(Restored | AgentAdmitted)) | (false, None) => SnapshotMode::Worktree, _ => return Ok(Inputs::Failed(RevisionFailureCode::SessionNotSettled)), };
session_restores里没有行,restore_phase返回None(PR 自带测试upgrade_keeps_completed_legacy_restore_out_of_admission_fences断言的就是restore_phase == None),于是(true, None)落入兜底分支。SessionNotSettled在crates/node-protocol/src/domain/revision.rs:167-171中定义为最终失败("Definitive delivery failures")。 - 失败场景:一个带 prior 的会话在升级前正常结束,交付已被 Node 接受、还停在
Preparing;Node 为升级重启。升级后recoverable_deliveries再次调用inputs(),交付以SessionNotSettled失败,这次运行的成果交付不出去。main 上这条交付走的是Worktree,会正常完成。升级后 Cloud 才派发的交付也是同样结果。 - 说明:上轮 head
01f56f77把这类会话标成legacy_unknown,同样会落到兜底分支,所以这个问题不是本轮引入的,但本轮的说明"已完成的旧终态只作为历史保留"在交付这条路径上并不成立。 - 要求:
(true, None)应走SnapshotMode::Worktree,与"已完成恢复不留进程责任"的回填说明一致。补一个经过inputs()或Revisions恢复路径的测试。现在inputs()的模式选择没有任何测试覆盖,revision/tests/snapshot.rs是直接设置job.mode。
P3 建议 — 每次执行状态查询都在 worker 线程上对所有未完成会话做一次交接证明
- 代码:
apps/ora-node/src/service/worker.rs:423-432。GetExecutionStatus先对所有recoverable_sessions执行reconcile(.., StopCheck::Handover),用的是block_on。managed/restore.rs:99-104在 Handover 下每个 attempt 给满cleanup_timeout_ms,prove(:123-131)还要先等transport::close一个cleanup_timeout_ms。 - 场景:Node 上有一个停止未知的会话,它的 host 能连上,但 Scope 一直不报
Closed。之后 Controller 对任何一个执行发状态查询,worker 都要阻塞最长两倍cleanup_timeout_ms;有 N 个这样的会话就是 N 倍。阻塞期间 worker 不处理其他控制消息。 - 建议:只对被查询的那个执行做交接证明,或者给一次查询设总预算。
P3 建议 — reconcile 中任何一条记录出错,都会中断启动或让 worker 退出
- 代码:
apps/ora-node/src/service/agents.rs:77-109对每条记录都用?返回。调用方:启动时agents.rs:275(open()失败,Node 起不来);周期检查worker.rs:203-210(map_err(..)?,worker 退出)。 - 场景:某个未开始会话的 history 路径已经存在,但不是普通文件。这时
seal_unstarted_history(session/model.rs:63-90)每次都返回history_unavailable,Node 每次启动都失败。main 在启动时不做 seal,所以这是新增的失败方式。 - 建议:单条记录出错时记日志,跳过这一条,下一轮再试。
确认没有问题的部分(owner 上轮的非阻塞问题)
ExecutionStatus.stop_unconfirmed(crates/node-protocol/src/message/execution.rs:43-44)带#[serde(default, skip_serializing_if = "Vec::is_empty")],这个结构体没有deny_unknown_fields,旧版本的 Controller 收到会忽略这个字段。Cloud 不解码ExecutionStatus。- 以下几项看过,没有发现问题:
- 停止上限只在
first_stop_at_ms IS NULL时锚定一次;poll读的是落盘的stop_bound_at_ms,不按当前配置重新计算。 - 停止未知时,冻结计划的上传和
Preparing都会被拦住(revisions.rs:175-180、:288-294)。 UnstartedRestore只读 HEAD。- 启动清理只处理已经完成、非
legacy_unknown、没有 pending attempt 的执行(agents.rs:129-133)。 - 写首次停止时间失败时仍然发出 Close(
managed/restore.rs:68-77、worker.rs:371-378),这符合 ADR D6"持久化失败时以停止优先"的规定。
- 停止上限只在
验证
- 环境:Windows 可用提交内存 19.79 GiB,满足 6 GiB 的要求。两个隔离 worktree 是
desktop/.worktrees/review-712(FETCH_HEAD616ef100)和cloud/.worktrees/review-117(FETCH_HEADc707072b),其中的 CLAUDE.md 已换成占位并设了 skip-worktree。临时改动都已还原,两个 worktree 的git status都是干净的。 - desktop,在 WSL Ubuntu-24.04(用户 hanli)里运行,
CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2:cargo test --locked --offline -p ora-node-db --lib tests::session::restore -- --nocapture:6 个通过,其中 PR 自带 5 个、我的临时测试 1 个。临时测试的输出见上面第二个 P1。- 变异检查:去掉
restore.sql中的AND state<>'completed'后,运行cargo test ... tests::session::restore::upgrade_keeps,3 个测试中upgrade_keeps_completed_legacy_restore_out_of_admission_fences失败(restore.rs:185的restore_phase断言),说明本轮新增的测试在没有修复时会失败。已还原。
- cloud,环境变量
REQUIRE_POSTGRES=1,TEST_DATABASE_URL取自本机 dev.env,数据库是容器ora-cloud-158-postgres-1:go test -p 1 -count=1 ./internal/core -run 'TestRestoreStopUnknownAddsOneResidualObservationWithoutReclaimAuthority|TestStartup|Abandon' -v:7 个通过。go test -p 1 -count=1 ./integration -run 'TestMigration0053|TestMigration0052' -v:2 个通过(0053 的空库/升级/重放测试,以及 main 上的 0052 测试)。- 变异检查:去掉证据写入后,上面的 core 测试失败。已还原。
- CI:
- desktop#712 @
616ef100:必需检查全部通过(Linux/macOS/Windows crates、Tauri、e2e、ci-ok;frontend 按改动过滤跳过)。但这次 CI 是在 #709 合入之前跑的,而且 PR 现在与 main 冲突。 - cloud#117 @
c707072b:新一轮 Backend check(run 39531436022)已经跑完,02:09:59Z 开始,02:21:20Z 结束,结论是 success。同一个 head 上更早的一轮(run 39530974711)被取消。PR 现在与 main 冲突,rebase 后需要再跑一次。
- desktop#712 @
没有验证的部分
- 没有跑
ora-nodecrate 的测试(deadline、restore、agents、真实 host/guardian/Git 用例),也没有跑完整测试套件。上面第二个 P1 中会话永不结束、交付被拒这一段,以及交付 P2 和两个 P3,是根据代码得出的结论,没有在服务层实际复现。第二个 P1 中会话无法写终态、Node 拦截准入这两点,在 node-db 层实际复现过。 - 没有在 rebase 后的代码上做任何验证。rebase 会改动 24 个冲突文件,需要整体再审一轮。
- 没有做 Cloud/Node 联合窗口验证,也没有调用真实模型。
🤖 Generated with Claude Code
The v10 migration marked every prior-revision execution as `legacy_unknown`, including executions that had already reached a terminal. That phase has no exit, and both admission fences read it, so an upgraded Node that had ever restored a prior revision could never accept another runtime binding or reuse the checkout. Limit the backfill to executions that are not yet completed. Old terminals stay as history without fencing later admissions; in-flight pre-upgrade restores still keep both fences until original stop evidence or the residual-reclaim path resolves them. Document the split in both languages and cover the upgraded database from both sides: a completed prior-revision execution binds a runtime and accepts the next controlled session on the same checkout, while an accepted one keeps rejecting both.
Rebase the restore ledger onto Git Server restores: bind every workload command to its original session, keep the deployment-sized restore and stop-confirmation budgets, gate transport cleanup on confirmed stop, deliver pre-journal restores as ordinary worktree snapshots, and release upgraded sessions the Agent proved had restored.
616ef10 to
2496d64
Compare
Expose the stop-confirmation budget as deployment configuration with its greater-than-deadline invariant, drop the unused cleanup hook left from bundle staging, and group restore composition inputs so the startup and resume entry points stay within the argument limit.
The deployment policy is the only source of the two restore budgets, so the unused constant policy is removed and the test passes its own budgets explicitly.
Teach the fixture runner the deadline-aware restore trait, pass command budgets to the host cutoff test, and give the new protocol fields to the session fixtures.
The stalled-host case must clip the restore cutoff against a shorter command budget, not replace that budget with the restore deadline.
The restore ledger moves the current schema to v11, so the migration fixtures check 11 instead of 10.
creegon2
left a comment
There was a problem hiding this comment.
复审对象:desktop#712 @ a6b394e8d2cdea030bcfcd70677467810a772e25(基于 main ff9de8f9)。依据:specs main 6aad4f9 的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md(D1/D4/D5/D6,与上轮 911c976 相比没有改动)。ff9de8f9..a6b394e8 的 9 个提交全部看过。
结论:Request changes
上轮的 P2、两条 P3、schema v11 和 rebase 都已改好,CI 在新 head 上全绿。还有两项阻断:
- 新的回填判据读的是
execution_events,这张表在 Controller ACK 后会删行。所以在线 Node 上已有 Agent 记录、事件也已被 ACK 的会话,升级后仍会被标成legacy_unknown,整个 Node 仍被永久拦住。上轮的失败场景在最常见的情况下仍然成立。 - Git Server 恢复路径中,以 Node 身份执行的那一半命令没有接入 D6,传输仓库
.ora-transport/restore-*的删除也没有按停止确认放行:运行中删除和 Node 启动时删除都不看证据。回复里"在任何 host 命令前持久化原 Scope/Run"和"传输仓库只在取得原停止确认后删除"这两句,与代码不符。
P1 阻断 — 回填判据依赖会被 ACK 删除的出站事件,在线 Node 升级后仍会被永久拦住
-
代码:
crates/node-db/src/restore.sql:16-24:未完成的 prior 执行,只要execution_events里没有message_type='thread_event'的行,就回填为legacy_unknown。crates/node-db/src/session/events.rs:122-134:acknowledge_session收到 ACK 后执行DELETE FROM execution_events WHERE execution=?1 AND sequence=?2。生产路径apps/ora-node/src/service/worker.rs:443对每个EventAck都会调用它。execution_events是出站队列,不是历史记录。
-
失败场景:一个带
prior_revision的会话,恢复已完成,Agent 已运行并产生了 thread 事件,Controller 在线并已 ACK 这些事件,所以execution_events里这个会话一行都没有。Node 为升级重启:- 回填把它标成
legacy_unknown; service/agents.rs:85的reconcile跳过它,Record::Ended返回ResourceConflict,会话永远不会结束;unfinished_runtime_executions永远包含它,bind_runtime和所有带 permit 的准入都被拒绝,范围是整个 Node,重启后不变。
- 回填把它标成
-
验证:在 review worktree 里加了一个临时 node-db 测试:接受并启动一个 prior 会话,追加一条 thread 事件,按生产路径
NodeDatabase::acknowledge确认它,然后用 PR 自带的reopen_after_v10_downgrade退回 v10 再打开。输出:events_rows_before_upgrade=0 last_sequence=1 phase=Some(LegacyUnknown) end_session=Err(ResourceConflict) unfinished=["session-execution"] bind_runtime=Err(ResourceConflict)PR 自带的
upgrade_releases_inflight_restore_proven_by_agent_thread_event能通过,是因为它没有 ACK 事件。 -
可行的判据:
node_executions.last_sequence。main 上只有record_event会增加它(session/events.rs:42-45,102-105),ACK 不会减少;未完成的会话只能写 thread 事件,而 thread 事件只在 Agent 启动后才写(mainsession/host.rs:307;mainsession/driver.rs:119-130先等恢复结束再启动 Agent)。所以state<>'completed' AND last_sequence>0可以证明恢复已经完成。我临时把NOT EXISTS (...)换成AND last_sequence=0后重跑:上面的探针变成phase=None end_session=Ok(..) unfinished=[] bind_runtime=Ok([]),PR 自带的 6 个tests::session::restore用例也全部通过。改动已还原。 -
要求:
- 回填改用
last_sequence或其他在 ACK 后仍保留的事实,不能用execution_events; - 升级测试要覆盖"事件已 ACK"的情况,最好经过启动时的
reconcile,断言会话以interrupted结束、两处拦截解除。现在的测试是直接调用end_session。
- 回填改用
P1 阻断 — Git Server 路径中 Node 身份的命令没有接入 D6,传输仓库的删除不看停止确认
ADR 依据:
- D1:恢复的每条受管 Git 命令(含"传输仓库建立与拉取")都要在取得停止确认后才能发终态;
- D5 和 D6"清理与证据分级":Node 启动清理
.ora-transport/时,只能清理原执行已取得停止确认的目录,未确认的要保留。
代码里有三处不满足:
(a) Node 身份的命令不记录原 Run,Close 失败后照样发终态。
-
apps/ora-node/src/service/agents.rs:50-55:只有工作负载 runner 调用了runner.track_restore(stops),传给RevisionRestorer的 Node 身份 runnerAsNode(host()?)没有调用。 -
apps/ora-node/src/managed/delivery.rs:40中,AsNode走的是run_scoped(.., Budget::Command)。restore_context为None,所以既不执行record_restore_attempt(:168-182),Close 失败时也只是返回错误(:214)。in-flight 的 host 交换走的是transport::execute,没有execute_before的截止裁剪。 -
受影响的步骤见
apps/ora-node/src/revision/restore.rs:175-213:TransportRepository::create(git init、git config)、negotiate_from、Git Serverfetch、commit_of,都在 Node 身份下执行。 -
失败场景:截止到期时,正在执行的是一条慢的 Git Server 拉取,ADR D1 举的就是这个例子。这时 guardian 不可达,Close 没有拿到
Closed。fetch返回Unavailable后,session/driver.rs:65调用confirm_stop。managed/restore.rs:53-119的poll查不到任何 attempt,直接返回true。随后传输仓库被删除,end_session写入agent_failed/prior_revision_unavailable。也就是说,在原 Git 进程停止未知时发出了终态,这正是 ADR D6 否定的"上一版"做法。 -
验证:临时测试按
agents::restorer的组合构造RevisionRestorer:工作负载 runner 做了 track,Node runner 没有。假 host 接受CreateScope/Start,Run 一直不结束,Close一律返回GuardianUnavailable。输出:restore=Some(Unavailable) host_starts=1 host_close_rejections=1 phase=Some(Pending) pending_attempts=0 confirm_stop=true end_session=Ok(Sequence(1))host 已启动一个 Run,Close 被拒绝,但账本里没有这个 Run,终态照常写入。
(b) 运行中的恢复结束时,传输仓库在停止确认之前就被删除。
apps/ora-node/src/git_server/transport.rs:211-216:TransportRepository的Drop会删除目录。revision/restore.rs:174-236:transport是spawn_blocking闭包里的局部变量,从来没有调用keep()。闭包无论成功还是失败,返回时目录都会被删除,这发生在confirm_stop(:132-145)之前。confirm_stop里那次"确认后删除"实际上什么也删不到。- 失败场景:截止到期时,正在执行的是工作负载的
git fetch -- <transport> refs/ora/prior,它从传输仓库读取对象。Close 没有确认,run_scoped返回restore stop is unconfirmed,闭包随即返回,传输仓库被删除,而原进程可能还在读它。
(c) Node 启动时,所有 restore 传输仓库在 reconcile 之前就被清空。
-
apps/ora-node/src/repository.rs:36-51:configure_clone调用purge_transports,只保留 clone 自己的传输仓库。service/worker.rs:53先执行它,agents::open(:89,其中的reconcile和sweep_restores)在后面才执行,门禁检查时目录早已不在。 -
验证:临时测试先建一个 prior 会话,并记录一条未确认的 restore attempt;再建好
.ora-transport/restore-<hash>,然后调用node.configure_clone(..)。输出:pending_attempts=1 session_completed=false transport_before=true transport_after_configure_clone=false -
测试缺口:
service/agents/tests.rs:108的restore_cleanup_requires_original_terminal_stop_evidence用node.clone_config()取传输仓库路径。但这个测试只调用了Node::open,没有调用configure_clone,所以clone_config()是None,关于传输仓库的断言从来没有执行过。
要求:
- Node 身份的恢复命令和工作负载命令使用同一套机制:先记录原 Run,按截止裁剪包括 in-flight 交换在内的执行,到期或取消时按 D6 停止确认;取不到确认时不发终态。
- 传输仓库在恢复期间
keep(),只在confirm_stop或cleanup_restore取得停止确认后删除。 configure_clone的启动清理保留以下会话的 restore 传输仓库:未完成的、legacy_unknown的、还有 pending attempt 的。- 补测试,并且测试必须调用真实的
configure_clone。需要两个:- 一个 Node 身份 Run 的 Close 未确认时,不写终态;
- 有 pending attempt 时,启动后传输仓库还在。
非阻断说明与建议
- 修好回填判据后,剩下的
legacy_unknown范围。 只剩"升级时带 prior、未完成、且last_sequence=0"的会话,也就是升级那一刻恢复可能正在进行的会话。它们仍会拦住整个 Node(bind_runtime、所有带 permit 的准入、同 checkout 的新会话),Node 侧没有出口,因为没有原 Run 身份可以交接。这和 ADR D6"未取得证据不释放"、specs#126"由残留回收的真实证据或运维显式确认释放"一致,也和 main 上process_attempts cleaned=0拦住整个 Node 的现有做法一致,所以我不把它列为阻断。两点建议:state='accepted'的行也可以排除。service/agents.rs:454-455写明 Accepted 状态"never started an actor",这时不可能有恢复进程。- 在 PR 描述或 docs 里写明运维怎样清除这类行,例如由残留回收替换这个 Node 的状态。
- 建议:Node 重启发生在
record_restore_attempt已提交、Start还没送达 host 之间时,prove只认QueryRun返回Run且直接进程已退出。这个窗口里的 Run 可能永远无法确认。我没有实测,请作者确认 host 对未知 Run 的应答,看是否应当把"Scope 已 Closed 且 host 没有这个 Run"当作停止确认。
已确认改好的部分
- P2:
service/revisions.rs:412-426的restore_mode把(true, None)映射到Worktree,legacy_unknown和不一致的组合返回None。inputs()在:356-367使用它,并先拒绝有 pending attempt 的会话。单测restore_mode_isolates_partial_restores覆盖了每个 phase。recoverable_deliveries也跳过legacy_unknown和有 pending attempt 的会话(:213-220)。 - P3:
GetExecutionStatus只对被查询的执行做交接证明,见worker.rs:426-432、agents.rs:76-80。reconcile中单条记录的 defer、poll、seal、end 失败时,都记录execution_id和错误后跳过,下一个recovery_interval_ms周期重试,失败不是静默的。只有整表读取这类数据库错误还用?,可以接受。
- D1(工作负载一侧):
transport::execute_before(managed/transport.rs:63-84)用timeout_at裁剪 in-flight 交换;每次交换前用check_exchange拒绝已过期的准入;Close 不受截止裁剪。- Git Server
fetch沿用TimeLimit::Until(deadline),命令执行时长受截止限制。
- D6(工作负载一侧):
- host 交换前先执行
record_restore_attempt(delivery.rs:168-182); - Close 之前先写首次停止时间(
managed/restore.rs:70的request在prove之前),测试unknown_stop_matrix...里的假 host 断言了这个顺序; - 两处
request_*只在first_stop_at_ms IS NULL时写入,上限一旦锚定就不再变; EndSession被接受后,worker 立即写首次停止时间(worker.rs:362-378);执行者在取消前也会通过request_stop再写一次(session/resume.rs),两处都在 Close 之前。
- host 交换前先执行
- 配置:
RestoreConfig::policy(revision/restore/config.rs:34-56)拒绝 0、不可表示的值和stop_confirmation_ms <= deadline_ms;serve在打开任何状态之前就调用它(service.rs:160),测试serve_refuses_zero_restore_budget_before_opening_node_state覆盖了这一点。
- D4:
Pending/GitAdmitted只读 HEAD(revision/prepare.rs的SnapshotMode::UnstartedRestore),这个模式跳过own_tree_no_follow和工作区快照。 - rebase:
- schema 改为 v11(
schema.rs:44,85,208-218),v10 仍是branch_heads.sql; - desktop#709 的 4 个文件,本 PR 都没有改动;
- desktop#681 的改动只在恢复改接的地方被替换,main 原来的
recover_interrupted和新reconcile生成的 Interrupted 记录相同。
- schema 改为 v11(
- CI:run 39541300448(head
a6b394e8,03:56Z 开始,在 #681 和 #709 合入之后)的 Linux/macOS/Windows crates、tauri、e2e、ci-ok 全部通过,frontend 按改动过滤跳过。
验证命令(WSL Ubuntu-24.04,CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2)
cargo test --locked --offline -p ora-node-db --lib tests::session::restore -- --nocapture:PR 自带的 6 个用例加我的 1 个探针,共 7 个,全部通过,探针输出见第一个 P1。把判据换成last_sequence=0后再跑一次,7 个都通过。cargo test --locked --offline -p ora-node --lib review_probe -- --nocapture:两个临时探针,输出见第二个 P1 的 (a) 和 (c)。- 所有临时测试和 SQL 改动都已删除,worktree 是干净的。
The v11 backfill read execution_events, the outbound queue the Controller deletes rows from when it acknowledges them. A Node upgraded with a session whose events were already acknowledged therefore fenced itself permanently. Use node_executions.last_sequence instead: v9 and v10 ran the restore synchronously before the Agent started, only Agent Thread events advance the counter, and acknowledgements never reset it. An Accepted session never started an actor, so no restore process can exist for it; keep the pending phase its admission would have written rather than fencing it.
The restore Half that runs as the Node (transport repository creation, the Git Server fetch and commit_of) never recorded its original Run, so a failed Close still produced a terminal result, and its host exchanges were not clipped by the restore cutoff. Bind both runners to the session, run Node-identity commands through the deadline-aware runner, and record their original Run before the host sees them. The transport repository was also deleted when the restore closure returned, before stop confirmation, and startup cleanup cleared every restore repository before recovery ran. Prepare it before any Git runs, keep it while an original command is unconfirmed, and let configure_clone keep the repository of every unsettled restore session.
…keep inputs One case runs the production restorer composition against a host whose guardian never answers, so the only Start is the Node's own command: the attempt is journaled, no terminal event exists, and the transport repository stays. Two more drive startup: recovery ends an upgraded session whose acknowledged Thread event survived only as its sequence, and the real configure_clone keeps restore inputs with a pending attempt and releases them once the stop is confirmed.
…fence State that both identities' restore commands are journaled under one cutoff, that a transport repository outlives its restore until every original command confirms its stop, and which pre-upgrade rows stay legacy_unknown and how the residual-reclaim contract releases them. Remove a duplicated ADR sentence fragment from the Chinese page.
creegon2
left a comment
There was a problem hiding this comment.
复审对象:desktop#712 @ b9d92c9394f9402d0262dbdb90b4b1609d091492(基于 main ff9de8f9)。上一轮看的是 a6b394e8,本轮看了它之后的 4 个提交:e045c0cd、bc4fc451、aef5bbb4、b9d92c93。依据:specs main 6aad4f9 的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md。
结论:Approve
上轮的两项 P1 都已修好。我把上轮的三个临时测试改成适配新代码的版本重跑了一遍,结果都从"复现缺陷"变成了"行为正确"。没有发现新的阻断。
P1 回填判据:已修好
-
crates/node-db/src/restore.sql的回填现在分两种情况:state='accepted'的行写入pending。这类会话从未启动过执行者,不可能有恢复进程。state='running' AND last_sequence=0的行写入legacy_unknown。
last_sequence只由record_event增加,ACK 不会把它清零,所以这个判据在 ACK 之后仍然可靠。 -
升级后,
accepted行在启动时的reconcile里会被这样处理:phase 为Pending、没有 attempt,所以poll返回 true,接着封存 history、以interrupted结束。Record::Ended允许"Pending且没有 attempt"的组合。 -
测试:
- node-db 的
upgrade_releases_inflight_restore_proven_after_events_were_acknowledged走生产的acknowledge路径; - ora-node 的
startup_recovery_ends_a_restore_proven_by_the_sequence_the_controller_acked经过启动时的reconcile。
- node-db 的
-
我的探针(启动会话、写入事件、ACK、退回 v10 再打开)现在输出:
phase=None end_session=Ok(Sequence(2)) unfinished=[] bind_runtime=Ok([])。
P1 Git Server 路径接入 D6:已修好
-
(a) Node 身份的命令现在也记录原 Run:
service/agents.rs:50-58给 Node runner 也调用了track_restore;managed/delivery.rs为AsNode实现了RestoreGitRunner,走run_scoped(.., RunAs::Node, Budget::Restore(deadline));revision/restore.rs在开跑前对两个 runner 都调用bind_restore。建传输仓库、negotiate_from、Git Serverfetch、commit_of都通过RestoreCommands执行,因此都会先写record_restore_attempt,再走execute_before,命令结束后用poll取停止确认。
我的探针用同样的组合,假 host 接受 Start、Run 不结束、Close 一律被拒。输出:
host_starts=1 phase=Some(GitAdmitted) pending_attempts=1 confirm_stop=false unconfirmed=1 end_session=Err(ResourceConflict) transport_after=true -
(b) 传输仓库不再被
Drop提前删除:TransportRepository::create拆成了prepare(只建目录)和init(执行git init/git config);- 恢复闭包结束时,只要
RestoreStops::retains_inputs显示还有原命令未确认,就调用keep(); - 其他使用方(clone 的
repository/snapshot.rs、交付的revision/push.rs、ora_git/operations.rs)仍调用create,create等于prepare加init,失败时照旧由Drop删除,行为没有变化。
-
(c) 启动清理不再提前删除:
repository.rs的configure_clone在 keep 列表里加上了"恢复会话尚未完成"的传输仓库。cleanup_restore的删除条件是"已完成,且不是legacy_unknown,且没有 pending attempt"。keep 条件"未完成"在字面上不是它的反面,但在能出现的状态里两者等价:Record::Ended拒绝legacy_unknown,也拒绝有 pending attempt 的会话,所以已完成的会话不可能还带着这两种状态;record_restore_attempt要求会话处于Running,所以已完成的会话不会再新增 attempt。
- 不会无限期保留:会话一结束,
reconcile/cleanup_restore就会删除它的传输仓库,未保留的残留由下一次启动清理。真正长期保留的只有两类:legacy_unknown会话,和停止确认一直取不到的会话。这和 ADR D6 一致。
我的探针在有 pending attempt 时调用真实的
configure_clone,输出:
transport_after_configure_clone=true unowned_after=false原来的清理测试现在也调用了
configure_clone,关于传输仓库的断言会实际执行。
非阻断说明
- 剩余的
legacy_unknown范围:只剩"升级时 running、带 prior、last_sequence=0"的会话,也就是升级那一刻恢复可能正在进行。它们会拦住整个 Node,Node 侧没有出口,由 specs#126 的残留回收释放。docs/node/revisions*.md已写明,和 ADR D6 一致。 - Git Server
fetch失败后的重试:fetch的首次尝试如果因为"restore stop is unconfirmed"失败,GitServerAccess会按Unavailable重试。重试时record_restore_attempt会因为已有 pending attempt 而被拒绝,所以不会启动第二个进程,只是会在截止之前多等一段重试间隔。不需要改。 - 作者提出的未决问题:
record_restore_attempt已提交,但Start没有送达 host。这时QueryRun返回UnknownIdentity或NotStarted,本 PR 按 ADR D6 的字面定义不把它算作停止确认,这个做法正确。是否放宽需要修改 ADR,我会在 circus 上单独问 owner,不影响本 PR。
验证(WSL Ubuntu-24.04,CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2)
cargo test --locked --offline -p ora-node-db --lib:59 个用例全部通过,其中 58 个是 PR 自带的,1 个是我的临时探针。cargo test --locked --offline -p ora-node --lib -- service::agents::tests managed::restore_tests revision::tests::restore加两个临时探针:全部通过。original_restore_run_survives_host_loss_and_is_settled_after_bound需要先cargo build -p ora-process-host -p ora-process-guardian,补齐之后,它和两个restore_cutoff用例都通过。- CI:run 39548181456(head
b9d92c93)的 crates(Linux/macOS/Windows)、tauri、e2e、ci-ok 全部通过。 - 和 desktop#710 当前 head
fbe37e24做git merge-tree,没有冲突。 - 临时测试都已删除,worktree 是干净的。
Abort prior-Revision restoration without turning unknown stop evidence or mixed checkout files into a terminal or deliverable result. Persist original host intent before admission, retain a non-renewing stop-confirmation clock, and require original Scope closure plus direct Run exit before terminal release, transport cleanup or checkout reuse. A stopped, incomplete restore with no Agent admission reads HEAD alone and delivers Unchanged.
Refs ora-space/circus#390
This combines the D1 deadline slice from #711 with D6 stop ownership and D4 delivery isolation, rebased onto the Git Server restore path. Cloud companion: ora-space/cloud#117.
ADR:
decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.mdremains approved. This implements the Node D1/D6/D4 paths on the only remaining restore transport (Git Server). Joint Cloud/Node window validation, the complete cancellation/checkout interruption matrix and real-deployment fault evidence remain follow-up work in the linked implementation run. The full ADR is not yet implemented.Rebase onto
ff9de8f9:branch_heads.sql."restore": { "deadline_ms": 300000 }; invalid values refuse startup. Restore command budgets are clipped to the remaining deadline, and an in-flight host exchange is bounded by the same cutoff instead of the longer transport timeout.stop_confirmation_ms(default ten minutes, must exceed the restore deadline) is frozen on first request. Transport repositories (.ora-transport/restore-*) and v9 staging directories are removed only after original stop confirmation; startup cleanup and crash handover consume the same evidence, and unknown ownership is retained as named residue.legacy_unknownentirely.Storage: Node schema v11 records restore admission, original Scope/Run and actual containment, first stop/bound timestamps and deferred terminal intention. The v11 upgrade backfills
legacy_unknownonly for running pre-journal restores whose sequence counter is still zero.execution_eventsis the outbound queue — the Controller deletes a row when it acknowledges it — so the durablelast_sequencedecides instead: v9 and v10 ran the restore synchronously before the Agent started, only Agent Thread events advance that counter, and an acknowledged event never resets it. Completed historical restores stay terminal history, and a session that was accepted but never started keeps a pending phase because no actor could have run a restore. The rows that remainlegacy_unknownare exactly the restores that may have been running at the upgrade: they have no Node-side exit and are released by the residual-reclaim contract (specs#126). Linux BestEffort cleanup does not claim Strong quiescence.Third review round in this head:
node_executions.last_sequenceinstead ofexecution_events. New upgrade tests cover the acknowledged case at the ledger level (accept + start, append a Thread event, acknowledge it through the productionNodeDatabase::acknowledge, downgrade, reopen) and through startupreconcile, where the session endsinterruptedand both admission fences clear.AsNodeimplements the restore runner,agents::restorertracks the Node runner exactly like the workload runner, and transport creation, the Git Server fetch andcommit_ofnow record their original Run before the host sees them and run under the restore cutoff, in-flight exchanges included. A workload-sideCloseand a Node-side one now share the same stop evidence.confirm_stop/startup cleanup remove it once that evidence arrives while an unconfirmed one keeps it on disk.configure_cloneno longer clears restore transport repositories before recovery runs: it keeps the repository of every session whose restore is not settled and still clears the rest.configure_clonekeeps inputs with a pending attempt and releases them after confirmation. The existing startup-cleanup test now configures clone, so its transport assertions actually execute instead of being skipped.restore_modemaps(prior=true, phase=None)toWorktreeand is unit-tested per phase, an execution-status query proves only the execution it names, andreconciledefers a failing record instead of aborting startup.Validation at
b9d92c9(WSL2 Ubuntu, Linux,CARGO_TARGET_DIRoutside the checkout):cargo fmt --all --checkpassed.cargo clippy --workspace --exclude ora-desktop --exclude ora-desktop-tests -- -D warningspassed.cargo test --workspace --exclude ora-desktop --exclude ora-desktop-tests --no-runbuilt every test target.cargo test -p ora-node-db --lib: 58 passed, 0 failed.cargo test -p ora-node --lib: 110 passed, 0 failed, including the real host/guardian restore fixtures and the Git Server restore cases.Open point, left to the owner rather than widened silently: a restore whose Run the host never recorded (the Node died between the durable attempt and the host's
Start), or one the host reports as never started, can never produce "direct process exit". ADR D6 accepts only original ScopeClosedplus direct-Run exit, so such an attempt staysstop_unconfirmedand keeps its inputs; widening that evidence is a specs change. The exact windows and the host answers (HostRejection::UnknownIdentity,DirectProcessState::NotStarted) are recorded in circus#390.No deployment or real-model acceptance is included. The complete ADR stays approved until remaining integration/fault evidence is finished.