Skip to content

Fence restore termination and partial delivery on original stop evidence - #712

Merged
EricWvi merged 13 commits into
ora-space:mainfrom
wanglongan587:feat/390-restore-stop-isolation
Oct 21, 2026
Merged

EricWvi merged 13 commits into
ora-space:mainfrom
wanglongan587:feat/390-restore-stop-isolation

Conversation

@wanglongan587

@wanglongan587 wanglongan587 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Abort prior-Revision restoration without turning unknown stop evidence or mixed checkout files into a terminal or deliverable result. Persist original host intent before admission, retain a non-renewing stop-confirmation clock, and require original Scope closure plus direct Run exit before terminal release, transport cleanup or checkout reuse. A stopped, incomplete restore with no Agent admission reads HEAD alone and delivers Unchanged.

Refs ora-space/circus#390

This combines the D1 deadline slice from #711 with D6 stop ownership and D4 delivery isolation, rebased onto the Git Server restore path. Cloud companion: ora-space/cloud#117.

ADR: decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md remains approved. This implements the Node D1/D6/D4 paths on the only remaining restore transport (Git Server). Joint Cloud/Node window validation, the complete cancellation/checkout interruption matrix and real-deployment fault evidence remain follow-up work in the linked implementation run. The full ADR is not yet implemented.

Rebase onto ff9de8f9:

  • The restore ledger moved from v10 to v11; main's v10 remains branch_heads.sql.
  • D1 wraps the Git Server fetch, the workload import and every supervised command in one absolute deadline. The deployment sets "restore": { "deadline_ms": 300000 }; invalid values refuse startup. Restore command budgets are clipped to the remaining deadline, and an in-flight host exchange is bounded by the same cutoff instead of the longer transport timeout.
  • D6 records the original Scope/Run and containment before the host sees any restore command. The first stop request is anchored before Close; stop_confirmation_ms (default ten minutes, must exceed the restore deadline) is frozen on first request. Transport repositories (.ora-transport/restore-*) and v9 staging directories are removed only after original stop confirmation; startup cleanup and crash handover consume the same evidence, and unknown ownership is retained as named residue.
  • D4 is selected by the durable phase: a prior input whose restore never completed and whose Agent was never admitted reads HEAD alone and delivers Unchanged; delivery refuses legacy_unknown entirely.

Storage: Node schema v11 records restore admission, original Scope/Run and actual containment, first stop/bound timestamps and deferred terminal intention. The v11 upgrade backfills legacy_unknown only for running pre-journal restores whose sequence counter is still zero. execution_events is the outbound queue — the Controller deletes a row when it acknowledges it — so the durable last_sequence decides instead: v9 and v10 ran the restore synchronously before the Agent started, only Agent Thread events advance that counter, and an acknowledged event never resets it. Completed historical restores stay terminal history, and a session that was accepted but never started keeps a pending phase because no actor could have run a restore. The rows that remain legacy_unknown are exactly the restores that may have been running at the upgrade: they have no Node-side exit and are released by the residual-reclaim contract (specs#126). Linux BestEffort cleanup does not claim Strong quiescence.

Third review round in this head:

  • Backfill criterion. The upgrade now reads node_executions.last_sequence instead of execution_events. New upgrade tests cover the acknowledged case at the ledger level (accept + start, append a Thread event, acknowledge it through the production NodeDatabase::acknowledge, downgrade, reopen) and through startup reconcile, where the session ends interrupted and both admission fences clear.
  • Node-identity restore commands. AsNode implements the restore runner, agents::restorer tracks the Node runner exactly like the workload runner, and transport creation, the Git Server fetch and commit_of now record their original Run before the host sees them and run under the restore cutoff, in-flight exchanges included. A workload-side Close and a Node-side one now share the same stop evidence.
  • Transport retention. The transport repository is prepared before any Git runs, so its handle survives a failed first command. It is removed only when every original restore command has confirmed its stop, and confirm_stop/startup cleanup remove it once that evidence arrives while an unconfirmed one keeps it on disk.
  • Startup cleanup. configure_clone no longer clears restore transport repositories before recovery runs: it keeps the repository of every session whose restore is not settled and still clears the rest.
  • Tests. A Node-identity Run whose Close stays unconfirmed records the attempt, starts only that command, writes no terminal event and keeps the transport repository. The real configure_clone keeps inputs with a pending attempt and releases them after confirmation. The existing startup-cleanup test now configures clone, so its transport assertions actually execute instead of being skipped.
  • Earlier review items stay: restore_mode maps (prior=true, phase=None) to Worktree and is unit-tested per phase, an execution-status query proves only the execution it names, and reconcile defers a failing record instead of aborting startup.

Validation at b9d92c9 (WSL2 Ubuntu, Linux, CARGO_TARGET_DIR outside the checkout):

  • cargo fmt --all --check passed.
  • cargo clippy --workspace --exclude ora-desktop --exclude ora-desktop-tests -- -D warnings passed.
  • cargo test --workspace --exclude ora-desktop --exclude ora-desktop-tests --no-run built every test target.
  • cargo test -p ora-node-db --lib: 58 passed, 0 failed.
  • cargo test -p ora-node --lib: 110 passed, 0 failed, including the real host/guardian restore fixtures and the Git Server restore cases.
  • Required CI on this head (run 39548181456): Linux/macOS/Windows crates, Tauri, e2e and ci-ok passed; frontend was skipped by the change filter.

Open point, left to the owner rather than widened silently: a restore whose Run the host never recorded (the Node died between the durable attempt and the host's Start), or one the host reports as never started, can never produce "direct process exit". ADR D6 accepts only original Scope Closed plus direct-Run exit, so such an attempt stays stop_unconfirmed and keeps its inputs; widening that evidence is a specs change. The exact windows and the host answers (HostRejection::UnknownIdentity, DirectProcessState::NotStarted) are recorded in circus#390.

No deployment or real-model acceptance is included. The complete ADR stays approved until remaining integration/fault evidence is finished.

@EricWvi EricWvi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on 01f56f77d563b73b6ea76a0766b3c8b3ba9d1c31.

Blocking: the v10 backfill fences completed historical restores forever.

restore.sql marks every node_executions row with payload.spec.prior_revision as legacy_unknown, with no state filter. Nothing ever moves a row out of legacy_unknown (set_restore_phase only allows git_admitted → restored → agent_admitted), and two fences read it:

  • unfinished_runtime_executions (crates/node-db/src/runtime_control.rs:113) includes every legacy_unknown row, so accepting a new runtime binding that is not input-closed returns ResourceConflict permanently.
  • The checkout conflict check (crates/node-db/src/session.rs:117) blocks any new session on that checkout_execution_id permanently.

In-flight legacy executions are already covered by state<>'completed' in both queries, so the only extra effect of the backfill is on completed history. The DB lives at home_directory/ora-node.sqlite3 and is never rebuilt, so after upgrading, any Node that has ever restored a prior revision can no longer accept a new runtime binding. The ADR's D6 fence is for executions whose stop evidence is unknown, and it hands release to the residual-reclaim ADR (specs#126: real evidence, or explicit operator confirmation). This PR adds no such exit for legacy_unknown.

Please either:

  1. limit the backfill to executions that are not yet completed (old terminals stay as history and do not fence), or
  2. if completed legacy restores really must fence, add the release path specs#126 requires (evidence or explicit operator confirmation), and a test that an upgraded DB with completed prior-revision history can still accept a runtime binding, or explicitly documents why it cannot.

Either way, please add an upgrade test with a completed prior-revision execution and assert what the runtime-binding and checkout fences do.

Non-blocking:

  • ExecutionStatus.stop_unconfirmed is additive with serde(default); please confirm the Controller/Cloud decoders on main tolerate the new field (no strict deny_unknown_fields on that path).

@wanglongan587
wanglongan587 force-pushed the feat/390-restore-stop-isolation branch 3 times, most recently from ae65096 to 616ef10 Compare October 21, 2026 02:23

@creegon2 creegon2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审对象:desktop#712 @ 616ef100(PR 基于 main 391e62a4),cloud#117 @ c707072b(PR 基于 main 9814cacd)。依据:specs main 911c976 中的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md(D1/D4/D6)。本轮改动(01f56f77..616ef100 去掉 rebase 带入的 main 提交后只有 616ef100 一个提交,range-diff 确认前两个提交与上轮内容相同)和两份 PR 的全部内容都看过。

结论:Request changes

两项阻断:一是和最新 main 冲突(main 已用掉 schema v10,恢复也已改走 Git Server);二是升级时还在进行的旧 prior-revision 会话会被永久拦住,而且拦的是整个 Node。另有一项应修:已完成的旧会话如果交付还没做完,升级后交付会被判为最终失败。

P1 阻断 — 与最新 main 冲突:v10 已被占用,恢复路径已更换

  • 现状:desktop main 现在是 ff9de8f9,已合入 desktop#681(内部 Git 线)和 desktop#709。gh pr view 712 --json mergeable 返回 CONFLICTING / DIRTY。用 git merge-tree --write-tree ff9de8f9 616ef100 检查,有 24 个文件冲突,包括 crates/node-db/src/schema.rs、apps/ora-node/src/revision/restore.rs、revision/restore/{download,git}.rs、revision/snapshot.rs、managed/{delivery,transport}.rs、service/{agents,revisions,worker}.rs、session/{driver,ports,resume}.rs 和四份 docs。
  • main 的 crates/node-db/src/schema.rs:44 已经允许版本 1..=10,:82、:192 两处的 v10 都是 branch_heads.sql。本 PR 的 restore.sql 也写成 v10,合并后同一个版本号会对应两种 schema。
  • main 的 apps/ora-node/src/revision/restore.rs 已经从 bundle 下载改成 Git Server 拉取加传输仓库。本 PR 的 D1/D6/D4 是在 bundle 路径上实现的(download.rs、bundle 临时目录、ora-restore-*.bundle 清理)。PR 描述里写的"Git Server 路径的接入留给后续",现在已经不能作为合并时的前提。ADR D5 要求截止、取消和停止确认的语义对两种介质都成立,而 main 上现在只剩 Git Server 这一种。
  • 要求:rebase 到最新 main,schema 改成 v11;把 D1(全程截止和命令预算裁剪)、D6(原 Scope/Run 持久化、停止确认、上限、未知停止路径)、D4(未完成恢复只读 HEAD)接到 Git Server 恢复路径上,包括传输仓库 .ora-transport/ 的启动清理门禁(ADR D5、D6 的清理规则);在新 head 上重新跑本轮测试和 CI。补充一点:616ef100 的 CI(run 39530595183,02:00:46Z 开始)是在 desktop#709 合入(02:02:37Z)之前跑的,#709 改的正好是 Scope 关闭确认的协调逻辑,rebase 后的 CI 必须覆盖它。

P1 阻断 — 升级时还在进行的旧 prior-revision 会话会被永久拦住,范围是整个 Node

  • 代码:

    • crates/node-db/src/restore.sql:15-17:所有 state<>'completed' 的带 prior 输入的旧执行都回填为 legacy_unknown。
    • apps/ora-node/src/service/agents.rs:82-85:reconcile 遇到 legacy_unknown 直接 continue,不会结束这个会话。main 在启动时会以 Interrupted 结束所有未完成的会话(main 版 service/agents.rs:180 的 recover_interrupted 加 end_session),本 PR 把这一步去掉了。
    • crates/node-db/src/session/events.rs:67-80:Record::Ended 对 legacy_unknown 一律返回 ResourceConflict,所以没有任何路径能给它写终态。
    • crates/node-db/src/session/restore.rs:44-58:set_restore_phase 只允许 git_admitted → restored → agent_admitted,没有离开 legacy_unknown 的转换,也没有运维释放的入口。
    • crates/node-db/src/runtime_control.rs:112-113:unfinished_runtime_executions 同时包含 phase='legacy_unknown' 和这个永远不会 completed 的 node_executions 行。这个列表一旦非空,bind_runtime(runtime_control.rs:73-98)在新 runtime generation 或输入未关闭时都会拒绝;所有带 permit 的准入也会拒绝,见 session.rs:121、plugin.rs:73、repository.rs:91、revision.rs:129。同一 checkout 的新会话还会被 session.rs:117 拒绝。
  • 失败场景:v9 Node 上有一个带 prior_revision 的会话,恢复早已完成,Agent 已经在运行并产生了 thread 记录。Node 为升级而重启。升级后:

    1. 这个会话永远不会结束,它的交付也被拒绝(service/revisions.rs:178,以及 inputs() 要求会话已 Completed);
    2. 这个 Node 之后所有受控准入都返回 ResourceConflict 或 InvalidTransition,包括 runtime 绑定、会话、插件、仓库和 Revision;
    3. 重启 Node 不会改变任何一项。

    按 v9 的执行顺序,恢复在 Agent 启动之前同步完成(session/driver.rs:129-138),所以对这类会话来说,"恢复 Git 进程可能还在写入"并不成立,ADR D6 拦截的理由不适用于它们。

  • 上轮 owner 说"在途的遗留执行本来就被 state<>'completed' 覆盖"。这句话在 main 上成立,因为 main 启动时会把这些执行结束掉。本 PR 同时去掉了这个结束出口,所以在途的旧行和上轮被指出的已完成旧行是同一个问题:legacy_unknown 没有出口。PR 自带的 upgrade_keeps_inflight_legacy_restore_fenced 只断言了拦截存在,没有任何测试说明拦截怎样解除。

  • 验证:在 review worktree 里加了一个临时测试。先用 v10 代码建库,接受并启动一个带 prior 的会话,追加一条 thread 事件,然后用 PR 自带的 reopen_after_v9_downgrade 退回 v9 再重新打开。输出:phase=Some(LegacyUnknown),end_session=Err(ResourceConflict),unfinished=["session-execution"],bind_runtime=Err(ResourceConflict);关闭数据库再打开一次,end_session 和 bind_runtime 结果不变。测试已删除。

  • 要求,二选一:

    1. 按 specs#126 给 legacy_unknown 一个出口(真实停止证据,或运维显式确认后释放),并测试升级后的在途行最终能解除这两处拦截;
    2. 缩小回填范围,把旧记录能证明恢复已经完成的会话排除在外(例如已经有 Agent 产生的 thread 事件;请作者核实这个判据在 v9/v10 上是否可靠),其余仍按未知处理。

    无论选哪种,都要补一个升级测试:输入是一条升级前已有 Agent 记录、还未完成的 prior 会话,断言升级并启动 reconcile 之后,会话怎样结束、两处拦截怎样解除。rebase 后这里是 v10→v11 的升级,main 上的 v10 会话同样没有恢复 journal,所以这个问题在 rebase 后依然存在。

P2 应修 — 已完成的旧 prior 会话,升级后交付被判为最终失败

  • 代码:apps/ora-node/src/service/revisions.rs:294-306:
    let mode = match (session.command.payload.spec.prior_revision.is_some(), phase) {
        (true, Some(Pending | GitAdmitted)) => SnapshotMode::UnstartedRestore,
        (true, Some(Restored | AgentAdmitted)) | (false, None) => SnapshotMode::Worktree,
        _ => return Ok(Inputs::Failed(RevisionFailureCode::SessionNotSettled)),
    };
    本轮改动之后,已完成的旧 prior 会话在 session_restores 里没有行,restore_phase 返回 None(PR 自带测试 upgrade_keeps_completed_legacy_restore_out_of_admission_fences 断言的就是 restore_phase == None),于是 (true, None) 落入兜底分支。SessionNotSettled 在 crates/node-protocol/src/domain/revision.rs:167-171 中定义为最终失败("Definitive delivery failures")。
  • 失败场景:一个带 prior 的会话在升级前正常结束,交付已被 Node 接受、还停在 Preparing;Node 为升级重启。升级后 recoverable_deliveries 再次调用 inputs(),交付以 SessionNotSettled 失败,这次运行的成果交付不出去。main 上这条交付走的是 Worktree,会正常完成。升级后 Cloud 才派发的交付也是同样结果。
  • 说明:上轮 head 01f56f77 把这类会话标成 legacy_unknown,同样会落到兜底分支,所以这个问题不是本轮引入的,但本轮的说明"已完成的旧终态只作为历史保留"在交付这条路径上并不成立。
  • 要求:(true, None) 应走 SnapshotMode::Worktree,与"已完成恢复不留进程责任"的回填说明一致。补一个经过 inputs() 或 Revisions 恢复路径的测试。现在 inputs() 的模式选择没有任何测试覆盖,revision/tests/snapshot.rs 是直接设置 job.mode。

P3 建议 — 每次执行状态查询都在 worker 线程上对所有未完成会话做一次交接证明

  • 代码:apps/ora-node/src/service/worker.rs:423-432。GetExecutionStatus 先对所有 recoverable_sessions 执行 reconcile(.., StopCheck::Handover),用的是 block_on。managed/restore.rs:99-104 在 Handover 下每个 attempt 给满 cleanup_timeout_ms,prove(:123-131)还要先等 transport::close 一个 cleanup_timeout_ms。
  • 场景:Node 上有一个停止未知的会话,它的 host 能连上,但 Scope 一直不报 Closed。之后 Controller 对任何一个执行发状态查询,worker 都要阻塞最长两倍 cleanup_timeout_ms;有 N 个这样的会话就是 N 倍。阻塞期间 worker 不处理其他控制消息。
  • 建议:只对被查询的那个执行做交接证明,或者给一次查询设总预算。

P3 建议 — reconcile 中任何一条记录出错,都会中断启动或让 worker 退出

  • 代码:apps/ora-node/src/service/agents.rs:77-109 对每条记录都用 ? 返回。调用方:启动时 agents.rs:275(open() 失败,Node 起不来);周期检查 worker.rs:203-210(map_err(..)?,worker 退出)。
  • 场景:某个未开始会话的 history 路径已经存在,但不是普通文件。这时 seal_unstarted_history(session/model.rs:63-90)每次都返回 history_unavailable,Node 每次启动都失败。main 在启动时不做 seal,所以这是新增的失败方式。
  • 建议:单条记录出错时记日志,跳过这一条,下一轮再试。

确认没有问题的部分(owner 上轮的非阻塞问题)

  • ExecutionStatus.stop_unconfirmed(crates/node-protocol/src/message/execution.rs:43-44)带 #[serde(default, skip_serializing_if = "Vec::is_empty")],这个结构体没有 deny_unknown_fields,旧版本的 Controller 收到会忽略这个字段。Cloud 不解码 ExecutionStatus。
  • 以下几项看过,没有发现问题:
    • 停止上限只在 first_stop_at_ms IS NULL 时锚定一次;poll 读的是落盘的 stop_bound_at_ms,不按当前配置重新计算。
    • 停止未知时,冻结计划的上传和 Preparing 都会被拦住(revisions.rs:175-180、:288-294)。
    • UnstartedRestore 只读 HEAD。
    • 启动清理只处理已经完成、非 legacy_unknown、没有 pending attempt 的执行(agents.rs:129-133)。
    • 写首次停止时间失败时仍然发出 Close(managed/restore.rs:68-77、worker.rs:371-378),这符合 ADR D6"持久化失败时以停止优先"的规定。

验证

  • 环境:Windows 可用提交内存 19.79 GiB,满足 6 GiB 的要求。两个隔离 worktree 是 desktop/.worktrees/review-712(FETCH_HEAD 616ef100)和 cloud/.worktrees/review-117(FETCH_HEAD c707072b),其中的 CLAUDE.md 已换成占位并设了 skip-worktree。临时改动都已还原,两个 worktree 的 git status 都是干净的。
  • desktop,在 WSL Ubuntu-24.04(用户 hanli)里运行,CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2:
    • cargo test --locked --offline -p ora-node-db --lib tests::session::restore -- --nocapture:6 个通过,其中 PR 自带 5 个、我的临时测试 1 个。临时测试的输出见上面第二个 P1。
    • 变异检查:去掉 restore.sql 中的 AND state<>'completed' 后,运行 cargo test ... tests::session::restore::upgrade_keeps,3 个测试中 upgrade_keeps_completed_legacy_restore_out_of_admission_fences 失败(restore.rs:185 的 restore_phase 断言),说明本轮新增的测试在没有修复时会失败。已还原。
  • cloud,环境变量 REQUIRE_POSTGRES=1,TEST_DATABASE_URL 取自本机 dev.env,数据库是容器 ora-cloud-158-postgres-1:
    • go test -p 1 -count=1 ./internal/core -run 'TestRestoreStopUnknownAddsOneResidualObservationWithoutReclaimAuthority|TestStartup|Abandon' -v:7 个通过。
    • go test -p 1 -count=1 ./integration -run 'TestMigration0053|TestMigration0052' -v:2 个通过(0053 的空库/升级/重放测试,以及 main 上的 0052 测试)。
    • 变异检查:去掉证据写入后,上面的 core 测试失败。已还原。
  • CI:
    • desktop#712 @ 616ef100:必需检查全部通过(Linux/macOS/Windows crates、Tauri、e2e、ci-ok;frontend 按改动过滤跳过)。但这次 CI 是在 #709 合入之前跑的,而且 PR 现在与 main 冲突。
    • cloud#117 @ c707072b:新一轮 Backend check(run 39531436022)已经跑完,02:09:59Z 开始,02:21:20Z 结束,结论是 success。同一个 head 上更早的一轮(run 39530974711)被取消。PR 现在与 main 冲突,rebase 后需要再跑一次。

没有验证的部分

  • 没有跑 ora-node crate 的测试(deadline、restore、agents、真实 host/guardian/Git 用例),也没有跑完整测试套件。上面第二个 P1 中会话永不结束、交付被拒这一段,以及交付 P2 和两个 P3,是根据代码得出的结论,没有在服务层实际复现。第二个 P1 中会话无法写终态、Node 拦截准入这两点,在 node-db 层实际复现过。
  • 没有在 rebase 后的代码上做任何验证。rebase 会改动 24 个冲突文件,需要整体再审一轮。
  • 没有做 Cloud/Node 联合窗口验证,也没有调用真实模型。

🤖 Generated with Claude Code

wanglongan587 and others added 4 commits October 21, 2026 11:02
The v10 migration marked every prior-revision execution as `legacy_unknown`,
including executions that had already reached a terminal. That phase has no
exit, and both admission fences read it, so an upgraded Node that had ever
restored a prior revision could never accept another runtime binding or reuse
the checkout.

Limit the backfill to executions that are not yet completed. Old terminals
stay as history without fencing later admissions; in-flight pre-upgrade
restores still keep both fences until original stop evidence or the
residual-reclaim path resolves them. Document the split in both languages and
cover the upgraded database from both sides: a completed prior-revision
execution binds a runtime and accepts the next controlled session on the same
checkout, while an accepted one keeps rejecting both.
Rebase the restore ledger onto Git Server restores: bind every workload command to its original session, keep the deployment-sized restore and stop-confirmation budgets, gate transport cleanup on confirmed stop, deliver pre-journal restores as ordinary worktree snapshots, and release upgraded sessions the Agent proved had restored.
@wanglongan587
wanglongan587 force-pushed the feat/390-restore-stop-isolation branch from 616ef10 to 2496d64 Compare October 21, 2026 03:21
Expose the stop-confirmation budget as deployment configuration with its greater-than-deadline invariant, drop the unused cleanup hook left from bundle staging, and group restore composition inputs so the startup and resume entry points stay within the argument limit.
The deployment policy is the only source of the two restore budgets, so the unused constant policy is removed and the test passes its own budgets explicitly.
Teach the fixture runner the deadline-aware restore trait, pass command budgets to the host cutoff test, and give the new protocol fields to the session fixtures.
The stalled-host case must clip the restore cutoff against a shorter command budget, not replace that budget with the restore deadline.
The restore ledger moves the current schema to v11, so the migration fixtures check 11 instead of 10.

@creegon2 creegon2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审对象:desktop#712 @ a6b394e8d2cdea030bcfcd70677467810a772e25(基于 main ff9de8f9)。依据:specs main 6aad4f9 的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md(D1/D4/D5/D6,与上轮 911c976 相比没有改动)。ff9de8f9..a6b394e8 的 9 个提交全部看过。

结论:Request changes

上轮的 P2、两条 P3、schema v11 和 rebase 都已改好,CI 在新 head 上全绿。还有两项阻断:

  1. 新的回填判据读的是 execution_events,这张表在 Controller ACK 后会删行。所以在线 Node 上已有 Agent 记录、事件也已被 ACK 的会话,升级后仍会被标成 legacy_unknown,整个 Node 仍被永久拦住。上轮的失败场景在最常见的情况下仍然成立。
  2. Git Server 恢复路径中,以 Node 身份执行的那一半命令没有接入 D6,传输仓库 .ora-transport/restore-* 的删除也没有按停止确认放行:运行中删除和 Node 启动时删除都不看证据。回复里"在任何 host 命令前持久化原 Scope/Run"和"传输仓库只在取得原停止确认后删除"这两句,与代码不符。

P1 阻断 — 回填判据依赖会被 ACK 删除的出站事件,在线 Node 升级后仍会被永久拦住

  • 代码:

    • crates/node-db/src/restore.sql:16-24:未完成的 prior 执行,只要 execution_events 里没有 message_type='thread_event' 的行,就回填为 legacy_unknown。
    • crates/node-db/src/session/events.rs:122-134:acknowledge_session 收到 ACK 后执行 DELETE FROM execution_events WHERE execution=?1 AND sequence=?2。生产路径 apps/ora-node/src/service/worker.rs:443 对每个 EventAck 都会调用它。execution_events 是出站队列,不是历史记录。
  • 失败场景:一个带 prior_revision 的会话,恢复已完成,Agent 已运行并产生了 thread 事件,Controller 在线并已 ACK 这些事件,所以 execution_events 里这个会话一行都没有。Node 为升级重启:

    1. 回填把它标成 legacy_unknown;
    2. service/agents.rs:85 的 reconcile 跳过它,Record::Ended 返回 ResourceConflict,会话永远不会结束;
    3. unfinished_runtime_executions 永远包含它,bind_runtime 和所有带 permit 的准入都被拒绝,范围是整个 Node,重启后不变。
  • 验证:在 review worktree 里加了一个临时 node-db 测试:接受并启动一个 prior 会话,追加一条 thread 事件,按生产路径 NodeDatabase::acknowledge 确认它,然后用 PR 自带的 reopen_after_v10_downgrade 退回 v10 再打开。输出:

    events_rows_before_upgrade=0 last_sequence=1 phase=Some(LegacyUnknown)
    end_session=Err(ResourceConflict) unfinished=["session-execution"] bind_runtime=Err(ResourceConflict)
    

    PR 自带的 upgrade_releases_inflight_restore_proven_by_agent_thread_event 能通过,是因为它没有 ACK 事件。

  • 可行的判据:node_executions.last_sequence。main 上只有 record_event 会增加它(session/events.rs:42-45,102-105),ACK 不会减少;未完成的会话只能写 thread 事件,而 thread 事件只在 Agent 启动后才写(main session/host.rs:307;main session/driver.rs:119-130 先等恢复结束再启动 Agent)。所以 state<>'completed' AND last_sequence>0 可以证明恢复已经完成。我临时把 NOT EXISTS (...) 换成 AND last_sequence=0 后重跑:上面的探针变成 phase=None end_session=Ok(..) unfinished=[] bind_runtime=Ok([]),PR 自带的 6 个 tests::session::restore 用例也全部通过。改动已还原。

  • 要求:

    • 回填改用 last_sequence 或其他在 ACK 后仍保留的事实,不能用 execution_events;
    • 升级测试要覆盖"事件已 ACK"的情况,最好经过启动时的 reconcile,断言会话以 interrupted 结束、两处拦截解除。现在的测试是直接调用 end_session。

P1 阻断 — Git Server 路径中 Node 身份的命令没有接入 D6,传输仓库的删除不看停止确认

ADR 依据:

  • D1:恢复的每条受管 Git 命令(含"传输仓库建立与拉取")都要在取得停止确认后才能发终态;
  • D5 和 D6"清理与证据分级":Node 启动清理 .ora-transport/ 时,只能清理原执行已取得停止确认的目录,未确认的要保留。

代码里有三处不满足:

(a) Node 身份的命令不记录原 Run,Close 失败后照样发终态。

  • apps/ora-node/src/service/agents.rs:50-55:只有工作负载 runner 调用了 runner.track_restore(stops),传给 RevisionRestorer 的 Node 身份 runner AsNode(host()?) 没有调用。

  • apps/ora-node/src/managed/delivery.rs:40 中,AsNode 走的是 run_scoped(.., Budget::Command)。restore_context 为 None,所以既不执行 record_restore_attempt(:168-182),Close 失败时也只是返回错误(:214)。in-flight 的 host 交换走的是 transport::execute,没有 execute_before 的截止裁剪。

  • 受影响的步骤见 apps/ora-node/src/revision/restore.rs:175-213:TransportRepository::create(git init、git config)、negotiate_from、Git Server fetch、commit_of,都在 Node 身份下执行。

  • 失败场景:截止到期时,正在执行的是一条慢的 Git Server 拉取,ADR D1 举的就是这个例子。这时 guardian 不可达,Close 没有拿到 Closed。fetch 返回 Unavailable 后,session/driver.rs:65 调用 confirm_stop。managed/restore.rs:53-119 的 poll 查不到任何 attempt,直接返回 true。随后传输仓库被删除,end_session 写入 agent_failed / prior_revision_unavailable。也就是说,在原 Git 进程停止未知时发出了终态,这正是 ADR D6 否定的"上一版"做法。

  • 验证:临时测试按 agents::restorer 的组合构造 RevisionRestorer:工作负载 runner 做了 track,Node runner 没有。假 host 接受 CreateScope/Start,Run 一直不结束,Close 一律返回 GuardianUnavailable。输出:

    restore=Some(Unavailable) host_starts=1 host_close_rejections=1 phase=Some(Pending)
    pending_attempts=0 confirm_stop=true end_session=Ok(Sequence(1))
    

    host 已启动一个 Run,Close 被拒绝,但账本里没有这个 Run,终态照常写入。

(b) 运行中的恢复结束时,传输仓库在停止确认之前就被删除。

  • apps/ora-node/src/git_server/transport.rs:211-216:TransportRepository 的 Drop 会删除目录。
  • revision/restore.rs:174-236:transport 是 spawn_blocking 闭包里的局部变量,从来没有调用 keep()。闭包无论成功还是失败,返回时目录都会被删除,这发生在 confirm_stop(:132-145)之前。confirm_stop 里那次"确认后删除"实际上什么也删不到。
  • 失败场景:截止到期时,正在执行的是工作负载的 git fetch -- <transport> refs/ora/prior,它从传输仓库读取对象。Close 没有确认,run_scoped 返回 restore stop is unconfirmed,闭包随即返回,传输仓库被删除,而原进程可能还在读它。

(c) Node 启动时,所有 restore 传输仓库在 reconcile 之前就被清空。

  • apps/ora-node/src/repository.rs:36-51:configure_clone 调用 purge_transports,只保留 clone 自己的传输仓库。service/worker.rs:53 先执行它,agents::open(:89,其中的 reconcile 和 sweep_restores)在后面才执行,门禁检查时目录早已不在。

  • 验证:临时测试先建一个 prior 会话,并记录一条未确认的 restore attempt;再建好 .ora-transport/restore-<hash>,然后调用 node.configure_clone(..)。输出:

    pending_attempts=1 session_completed=false transport_before=true transport_after_configure_clone=false
    
  • 测试缺口:service/agents/tests.rs:108 的 restore_cleanup_requires_original_terminal_stop_evidence 用 node.clone_config() 取传输仓库路径。但这个测试只调用了 Node::open,没有调用 configure_clone,所以 clone_config() 是 None,关于传输仓库的断言从来没有执行过。

要求:

  • Node 身份的恢复命令和工作负载命令使用同一套机制:先记录原 Run,按截止裁剪包括 in-flight 交换在内的执行,到期或取消时按 D6 停止确认;取不到确认时不发终态。
  • 传输仓库在恢复期间 keep(),只在 confirm_stop 或 cleanup_restore 取得停止确认后删除。
  • configure_clone 的启动清理保留以下会话的 restore 传输仓库:未完成的、legacy_unknown 的、还有 pending attempt 的。
  • 补测试,并且测试必须调用真实的 configure_clone。需要两个:
    • 一个 Node 身份 Run 的 Close 未确认时,不写终态;
    • 有 pending attempt 时,启动后传输仓库还在。

非阻断说明与建议

  • 修好回填判据后,剩下的 legacy_unknown 范围。 只剩"升级时带 prior、未完成、且 last_sequence=0"的会话,也就是升级那一刻恢复可能正在进行的会话。它们仍会拦住整个 Node(bind_runtime、所有带 permit 的准入、同 checkout 的新会话),Node 侧没有出口,因为没有原 Run 身份可以交接。这和 ADR D6"未取得证据不释放"、specs#126"由残留回收的真实证据或运维显式确认释放"一致,也和 main 上 process_attempts cleaned=0 拦住整个 Node 的现有做法一致,所以我不把它列为阻断。两点建议:
    • state='accepted' 的行也可以排除。service/agents.rs:454-455 写明 Accepted 状态"never started an actor",这时不可能有恢复进程。
    • 在 PR 描述或 docs 里写明运维怎样清除这类行,例如由残留回收替换这个 Node 的状态。
  • 建议:Node 重启发生在 record_restore_attempt 已提交、Start 还没送达 host 之间时,prove 只认 QueryRun 返回 Run 且直接进程已退出。这个窗口里的 Run 可能永远无法确认。我没有实测,请作者确认 host 对未知 Run 的应答,看是否应当把"Scope 已 Closed 且 host 没有这个 Run"当作停止确认。

已确认改好的部分

  • P2:service/revisions.rs:412-426 的 restore_mode 把 (true, None) 映射到 Worktree,legacy_unknown 和不一致的组合返回 None。inputs() 在 :356-367 使用它,并先拒绝有 pending attempt 的会话。单测 restore_mode_isolates_partial_restores 覆盖了每个 phase。recoverable_deliveries 也跳过 legacy_unknown 和有 pending attempt 的会话(:213-220)。
  • P3:
    • GetExecutionStatus 只对被查询的执行做交接证明,见 worker.rs:426-432、agents.rs:76-80。
    • reconcile 中单条记录的 defer、poll、seal、end 失败时,都记录 execution_id 和错误后跳过,下一个 recovery_interval_ms 周期重试,失败不是静默的。只有整表读取这类数据库错误还用 ?,可以接受。
  • D1(工作负载一侧):
    • transport::execute_before(managed/transport.rs:63-84)用 timeout_at 裁剪 in-flight 交换;每次交换前用 check_exchange 拒绝已过期的准入;Close 不受截止裁剪。
    • Git Server fetch 沿用 TimeLimit::Until(deadline),命令执行时长受截止限制。
  • D6(工作负载一侧):
    • host 交换前先执行 record_restore_attempt(delivery.rs:168-182);
    • Close 之前先写首次停止时间(managed/restore.rs:70 的 request 在 prove 之前),测试 unknown_stop_matrix... 里的假 host 断言了这个顺序;
    • 两处 request_* 只在 first_stop_at_ms IS NULL 时写入,上限一旦锚定就不再变;
    • EndSession 被接受后,worker 立即写首次停止时间(worker.rs:362-378);执行者在取消前也会通过 request_stop 再写一次(session/resume.rs),两处都在 Close 之前。
  • 配置:
    • RestoreConfig::policy(revision/restore/config.rs:34-56)拒绝 0、不可表示的值和 stop_confirmation_ms <= deadline_ms;
    • serve 在打开任何状态之前就调用它(service.rs:160),测试 serve_refuses_zero_restore_budget_before_opening_node_state 覆盖了这一点。
  • D4:Pending/GitAdmitted 只读 HEAD(revision/prepare.rs 的 SnapshotMode::UnstartedRestore),这个模式跳过 own_tree_no_follow 和工作区快照。
  • rebase:
    • schema 改为 v11(schema.rs:44,85,208-218),v10 仍是 branch_heads.sql;
    • desktop#709 的 4 个文件,本 PR 都没有改动;
    • desktop#681 的改动只在恢复改接的地方被替换,main 原来的 recover_interrupted 和新 reconcile 生成的 Interrupted 记录相同。
  • CI:run 39541300448(head a6b394e8,03:56Z 开始,在 #681 和 #709 合入之后)的 Linux/macOS/Windows crates、tauri、e2e、ci-ok 全部通过,frontend 按改动过滤跳过。

验证命令(WSL Ubuntu-24.04,CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2)

  • cargo test --locked --offline -p ora-node-db --lib tests::session::restore -- --nocapture:PR 自带的 6 个用例加我的 1 个探针,共 7 个,全部通过,探针输出见第一个 P1。把判据换成 last_sequence=0 后再跑一次,7 个都通过。
  • cargo test --locked --offline -p ora-node --lib review_probe -- --nocapture:两个临时探针,输出见第二个 P1 的 (a) 和 (c)。
  • 所有临时测试和 SQL 改动都已删除,worktree 是干净的。

The v11 backfill read execution_events, the outbound queue the Controller deletes rows from when it acknowledges them. A Node upgraded with a session whose events were already acknowledged therefore fenced itself permanently. Use node_executions.last_sequence instead: v9 and v10 ran the restore synchronously before the Agent started, only Agent Thread events advance the counter, and acknowledgements never reset it.

An Accepted session never started an actor, so no restore process can exist for it; keep the pending phase its admission would have written rather than fencing it.
The restore Half that runs as the Node (transport repository creation, the Git Server fetch and commit_of) never recorded its original Run, so a failed Close still produced a terminal result, and its host exchanges were not clipped by the restore cutoff. Bind both runners to the session, run Node-identity commands through the deadline-aware runner, and record their original Run before the host sees them.

The transport repository was also deleted when the restore closure returned, before stop confirmation, and startup cleanup cleared every restore repository before recovery ran. Prepare it before any Git runs, keep it while an original command is unconfirmed, and let configure_clone keep the repository of every unsettled restore session.
…keep inputs

One case runs the production restorer composition against a host whose guardian never answers, so the only Start is the Node's own command: the attempt is journaled, no terminal event exists, and the transport repository stays. Two more drive startup: recovery ends an upgraded session whose acknowledged Thread event survived only as its sequence, and the real configure_clone keeps restore inputs with a pending attempt and releases them once the stop is confirmed.
…fence

State that both identities' restore commands are journaled under one cutoff, that a transport repository outlives its restore until every original command confirms its stop, and which pre-upgrade rows stay legacy_unknown and how the residual-reclaim contract releases them. Remove a duplicated ADR sentence fragment from the Chinese page.

@creegon2 creegon2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审对象:desktop#712 @ b9d92c9394f9402d0262dbdb90b4b1609d091492(基于 main ff9de8f9)。上一轮看的是 a6b394e8,本轮看了它之后的 4 个提交:e045c0cd、bc4fc451、aef5bbb4、b9d92c93。依据:specs main 6aad4f9 的 decisions/node/revision/20261018-complete-restore-deadline-and-cancel-confirmation.md。

结论:Approve

上轮的两项 P1 都已修好。我把上轮的三个临时测试改成适配新代码的版本重跑了一遍,结果都从"复现缺陷"变成了"行为正确"。没有发现新的阻断。

P1 回填判据:已修好

  • crates/node-db/src/restore.sql 的回填现在分两种情况:

    • state='accepted' 的行写入 pending。这类会话从未启动过执行者,不可能有恢复进程。
    • state='running' AND last_sequence=0 的行写入 legacy_unknown。

    last_sequence 只由 record_event 增加,ACK 不会把它清零,所以这个判据在 ACK 之后仍然可靠。

  • 升级后,accepted 行在启动时的 reconcile 里会被这样处理:phase 为 Pending、没有 attempt,所以 poll 返回 true,接着封存 history、以 interrupted 结束。Record::Ended 允许"Pending 且没有 attempt"的组合。

  • 测试:

    • node-db 的 upgrade_releases_inflight_restore_proven_after_events_were_acknowledged 走生产的 acknowledge 路径;
    • ora-node 的 startup_recovery_ends_a_restore_proven_by_the_sequence_the_controller_acked 经过启动时的 reconcile。
  • 我的探针(启动会话、写入事件、ACK、退回 v10 再打开)现在输出:
    phase=None end_session=Ok(Sequence(2)) unfinished=[] bind_runtime=Ok([])。

P1 Git Server 路径接入 D6:已修好

  • (a) Node 身份的命令现在也记录原 Run:

    • service/agents.rs:50-58 给 Node runner 也调用了 track_restore;
    • managed/delivery.rs 为 AsNode 实现了 RestoreGitRunner,走 run_scoped(.., RunAs::Node, Budget::Restore(deadline));
    • revision/restore.rs 在开跑前对两个 runner 都调用 bind_restore。建传输仓库、negotiate_from、Git Server fetch、commit_of 都通过 RestoreCommands 执行,因此都会先写 record_restore_attempt,再走 execute_before,命令结束后用 poll 取停止确认。

    我的探针用同样的组合,假 host 接受 Start、Run 不结束、Close 一律被拒。输出:
    host_starts=1 phase=Some(GitAdmitted) pending_attempts=1 confirm_stop=false unconfirmed=1 end_session=Err(ResourceConflict) transport_after=true

  • (b) 传输仓库不再被 Drop 提前删除:

    • TransportRepository::create 拆成了 prepare(只建目录)和 init(执行 git init/git config);
    • 恢复闭包结束时,只要 RestoreStops::retains_inputs 显示还有原命令未确认,就调用 keep();
    • 其他使用方(clone 的 repository/snapshot.rs、交付的 revision/push.rs、ora_git/operations.rs)仍调用 create,create 等于 prepare 加 init,失败时照旧由 Drop 删除,行为没有变化。
  • (c) 启动清理不再提前删除:

    • repository.rs 的 configure_clone 在 keep 列表里加上了"恢复会话尚未完成"的传输仓库。
    • cleanup_restore 的删除条件是"已完成,且不是 legacy_unknown,且没有 pending attempt"。keep 条件"未完成"在字面上不是它的反面,但在能出现的状态里两者等价:
      • Record::Ended 拒绝 legacy_unknown,也拒绝有 pending attempt 的会话,所以已完成的会话不可能还带着这两种状态;
      • record_restore_attempt 要求会话处于 Running,所以已完成的会话不会再新增 attempt。
    • 不会无限期保留:会话一结束,reconcile/cleanup_restore 就会删除它的传输仓库,未保留的残留由下一次启动清理。真正长期保留的只有两类:legacy_unknown 会话,和停止确认一直取不到的会话。这和 ADR D6 一致。

    我的探针在有 pending attempt 时调用真实的 configure_clone,输出:
    transport_after_configure_clone=true unowned_after=false

    原来的清理测试现在也调用了 configure_clone,关于传输仓库的断言会实际执行。

非阻断说明

  • 剩余的 legacy_unknown 范围:只剩"升级时 running、带 prior、last_sequence=0"的会话,也就是升级那一刻恢复可能正在进行。它们会拦住整个 Node,Node 侧没有出口,由 specs#126 的残留回收释放。docs/node/revisions*.md 已写明,和 ADR D6 一致。
  • Git Server fetch 失败后的重试:fetch 的首次尝试如果因为"restore stop is unconfirmed"失败,GitServerAccess 会按 Unavailable 重试。重试时 record_restore_attempt 会因为已有 pending attempt 而被拒绝,所以不会启动第二个进程,只是会在截止之前多等一段重试间隔。不需要改。
  • 作者提出的未决问题:record_restore_attempt 已提交,但 Start 没有送达 host。这时 QueryRun 返回 UnknownIdentity 或 NotStarted,本 PR 按 ADR D6 的字面定义不把它算作停止确认,这个做法正确。是否放宽需要修改 ADR,我会在 circus 上单独问 owner,不影响本 PR。

验证(WSL Ubuntu-24.04,CARGO_TARGET_DIR=/home/hanli/cargo-target/review-712,CARGO_BUILD_JOBS=2)

  • cargo test --locked --offline -p ora-node-db --lib:59 个用例全部通过,其中 58 个是 PR 自带的,1 个是我的临时探针。
  • cargo test --locked --offline -p ora-node --lib -- service::agents::tests managed::restore_tests revision::tests::restore 加两个临时探针:全部通过。original_restore_run_survives_host_loss_and_is_settled_after_bound 需要先 cargo build -p ora-process-host -p ora-process-guardian,补齐之后,它和两个 restore_cutoff 用例都通过。
  • CI:run 39548181456(head b9d92c93)的 crates(Linux/macOS/Windows)、tauri、e2e、ci-ok 全部通过。
  • 和 desktop#710 当前 head fbe37e24 做 git merge-tree,没有冲突。
  • 临时测试都已删除,worktree 是干净的。

@EricWvi
EricWvi merged commit bb349ee into ora-space:main Oct 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants