Skip to content

fix(nodeswap): run the staged binary's own node-swap, and cross-drive/cross-fs installs - #481

Merged
dmmdea merged 1 commit into
mainfrom
fix/node-swap-runner-and-xdrive
Sep 24, 2026
Merged

dmmdea merged 1 commit into
mainfrom
fix/node-swap-runner-and-xdrive

Conversation

@dmmdea

@dmmdea dmmdea commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Summary

Two node-swap defects found by c23b5c6f's own first real production rollout (2026-09-24), fixed independently:

  • Launcher runner selection (setup/linux-node-swap-launch.sh, setup/windows-node-swap-launch.ps1): both detached launchers defaulted the exe that runs node-swap to the currently-installed binary, not the newly-staged one. A fix landing inside node-swap's own engine code could therefore never take effect on the very deploy that staged it — this is what actually happened to the Lenovo/binxarn nodes in the rollout (they ran the OLD, still-buggy deps_other.go logic via the old installed binary, even though the staged build already had it fixed; the rollout doc's own diagnosis of that failure — "gate FindProcessesByExe on runtime.GOOS" — was therefore wrong, since deps_other.go already returns nil, nil on non-Windows in c23b5c6f). Both launchers now default the runner to the staged binary, hash-verified by the launcher itself before it is ever executed, with --runner-exe/-RunnerExe still available as an explicit override, and a fallback to the installed binary (logged clearly) only when the staged build provably does not support node-swap at all.
  • Cross-drive/cross-filesystem install (internal/nodeswap): the install-new step used plain os.Rename, which fails outright when the staged file and the target live on different volumes (Windows: a different drive letter; Linux: EXDEV) — hit for real on the Aorus node (staged at C:\tmp\ against a D:\ target): it rolled back safely, but the swap never happened. installNewBinary now falls back to copying the staged binary into the target's own directory, re-verifying its sha256 before trusting it, and renaming from there (now same-device); the temp file is cleaned up on any failure.

Both launchers' runner-selection logic is refactored to be independently unit-testable with fakes (the bash launcher's body moved into main(), only run when executed directly, so a new self-test can source it; the PowerShell launcher's existing -SelfTest convention gained matching assertions).

Review findings addressed

Two independent reviews ran on this diff (offload_review_diff on a free local seat, and a pinned Sonnet pr-review-toolkit:code-reviewer):

  • The node-swap-support probe's exit-code classification was too broad (rc != 2 = "supported"), which would misread a crashed/corrupt/wrong-architecture staged binary as safe to run. Tightened to a narrow allowlist (exit 0 or 1 only) in both launchers, with new self-test cases covering signal-death-shaped exit codes (bash) and a real crash exit code (PowerShell).
  • The PowerShell probe (Test-NodeSwapSupport) had no try/catch: a genuinely non-PE staged file (corrupt/wrong-arch) makes CreateProcess fail before $LASTEXITCODE is ever set, which PowerShell surfaces as a terminating ApplicationFailedException — uncaught under the script's $ErrorActionPreference = 'Stop', this aborted the whole launcher instead of falling back to the installed binary. Fixed with try/catch, and pinned with a new self-test using an actual text-file-renamed-.exe (the only way to exercise the real CreateProcess-level failure, which the .cmd exit-code tests can't reach).
  • One reviewer finding (PowerShell hash comparison being "case-sensitive") was checked empirically and found to be a false positive — PowerShell's -eq/-ne are case-insensitive by default, confirmed live.
  • Two minor/optional notes were left as-is per the reviewer: the real live-cross-drive-rename Go test never runs in CI (Windows-only, gated behind an env var for manual local runs — CI has no Windows Go test job), and a theoretical fixed---backup-suffix collision on the temp-file naming (pre-existing pattern, not introduced by this change).

How tested

  • go build/vet/test ./... — all green except one pre-existing, unrelated, timing-flaky test in internal/gpuactivity (confirmed pre-existing: passes 5/5 in isolation; package untouched by this diff).
  • bash setup/linux-node-swap-launch.tests.sh and powershell -File setup/windows-node-swap-launch.tests.ps1 — both green, now wired into this PR's CI (build job gained one new step for the bash self-test; installer-windows already ran the PowerShell one).
  • Every new guard broken once (red) and restored (green) by hand: the launcher hash checks, the exit-code classification, the PowerShell try/catch, and the Go-side cross-device detection/corruption-check.
  • docs/systems/node-swap.md updated to document both fixes and the new invariants.

Risk

Low. Both fixes are additive fallback paths (same-device rename and a build that already supports node-swap both take the exact same fast path as before); the only behavior change on the happy path is which binary a launcher runs node-swap from, which is now hash-verified before execution either way.

🤖 Generated with Claude Code

…/cross-fs installs

The 2026-09-24 c23b5c6 rollout's own first real use found two defects in the
swap tooling itself:

1. Both launchers (setup/linux-node-swap-launch.sh,
   setup/windows-node-swap-launch.ps1) defaulted the exe that RUNS
   `node-swap` to the currently-INSTALLED binary, not the newly-staged one.
   A fix landing inside node-swap's own engine code could therefore never
   take effect on the deploy that staged it: two Linux nodes hit exactly
   this, running the OLD deps_other.go stub via the old installed binary
   even though the staged build already had the fix. Both launchers now
   default the runner to the STAGED binary, hash-verified by the launcher
   itself before it is ever executed, falling back to the installed binary
   (with a clear log line) only when the staged build provably lacks the
   node-swap subcommand. Support detection uses a narrow exit-code allowlist
   (0 or 1 only, never a broad "anything but 2") and the PowerShell probe is
   wrapped in try/catch, so a corrupt/wrong-arch/non-PE staged file is never
   misread as safe to run and never aborts the launcher outright — both
   gaps found by code review and pinned with new self-test cases (a real
   non-PE file for the PowerShell CreateProcess-failure path, signal-death
   exit codes for the bash path).

2. node-swap's install-new step used plain os.Rename, which fails outright
   when the staged file and the target live on different volumes/
   filesystems (Windows: a different drive letter; Linux/macOS: EXDEV) —
   hit for real staging at C:\tmp\ against a D:\ target: the tool rolled
   back safely, but the swap never happened. installNewBinary now falls
   back to copying the staged binary into the target's own directory,
   re-verifying its sha256 before trusting it, and renaming from there
   (same-device, so the fast path applies); the temp file is cleaned up on
   any failure along that path.

Both launchers' runner-selection logic (resolve_runner_exe /
Resolve-RunnerExe) is now independently unit-testable with fakes: the bash
launcher's body moved into main(), only run when executed directly, so
setup/linux-node-swap-launch.tests.sh (new, wired into CI) can source it;
the PowerShell launcher's existing -SelfTest convention gained matching
assertions.

Tests: Go unit tests with fakes for the cross-device fallback (happy path,
copy failure, corruption caught before install, ordinary failures never
taking the fallback, nil-hooks degrading safely) plus real isCrossDeviceRenameErr/copyFile
coverage (synthetic EXDEV, synthetic and a real live cross-drive rename on
Windows); bash and PowerShell self-tests for runner selection and support
detection. Every new guard broken once (red) and restored (green) by hand
during review.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dmmdea
dmmdea merged commit 0b4a5fd into main Sep 24, 2026
5 checks passed
@dmmdea
dmmdea deleted the fix/node-swap-runner-and-xdrive branch September 24, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant