daemon: per-peer path watchdog — heal a single dead peer path in place#401
Merged
Conversation
2026-07-17 incident: one peer's NAT/relay session died (list-agents unreachable for hours) while 8 other peers stayed healthy — so the global rx-watchdog never fired (any live peer keeps PktsRecv moving), and the only remedy was a full daemon restart, which drops EVERY session and costs minutes of re-establishment on a NAT'd link. Nothing health-checks an individual peer mapping: the tunnel keepalive is fire-and-forget (the receiver silently swallows it), so it keeps NAT mappings warm but verifies nothing. Add pathwatch.go (L4): every Ready peer already refreshes lastInboundDecrypt at least every ~30s via bidirectional keepalives. A peer silent past 55s gets up to 3 pong-soliciting probes; still silent → that ONE peer's path is reset in place via the same sequence as `pilotctl prefer-direct` (drop cached resolve + tunnel, preserve relay-active, clear rekey cooldowns, proactively push a fresh PILA) — extracted from the IPC handler into the shared resetPeerPath. Worst- case detection ≈85s, recovery ~1 RTT, healthy sessions untouched. Reset is rate-limited per peer (2min cooldown); disable with -no-path-watch. Events: tunnel.path_suspect / tunnel.path_recovered. Backward compatible with the whole deployed fleet — no wire changes: the probe is a plain PortPing control frame with a non-empty payload. Verified against the v1.10.0, v1.10.0-rc5 and -rc7 trees (the fleet runs rc6-dev): their isTunnelKeepalive only swallows EMPTY frames, so the probe reaches handleControlPacket and gets a pong; old peers' own keepalives independently keep refreshing our liveness signal, so an old peer is never falsely reset. The compat invariant is pinned by TestPathProbeIsNotSwallowedByOldKeepaliveFilter. Tests: probe/reset/recovery/cooldown/prune state machine, relay-state preservation in resetPeerPath, keepalive-filter compat. pkg/daemon green under -race for the new tests and under the -short CI gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TeoSlayer
added a commit
that referenced
this pull request
Jul 19, 2026
…estart (#405) 2026-07-19 incident (pilot-director, Lambda box): a systemd unit running `pilotctl daemon start --foreground` crash-looped for HOURS with 'another daemon start is in progress (PID file locked)' after the daemon died overnight. Root cause chain: 1. The O_EXCL pidfile claim writes a '0' placeholder, and the --foreground path syscall.Exec's the daemon WITHOUT ever writing the real PID — so the file holds '0' for the daemon's whole life. 2. After any crash, the next start reads pid=0, skips the pid>0 staleness cleanup, and dies on the O_EXCL claim. Every systemd respawn hits the same wall: an unrecoverable restart loop that also silently defeats the rx-watchdog's exit-86 'supervisor will respawn us' recovery (#369/#387/#401). Fixes, all paths: - foreground: write os.Getpid() before exec (exec keeps the PID), so the real daemon PID is recorded; `daemon status` now reports systemd-managed daemons correctly instead of 'pid file stale'. - start: a pidfile with no usable PID (empty/garbage/'0') is never a live daemon — remove it instead of failing the claim. - new pidLooksLikeDaemon(): verifies the PID's executable base name is actually pilot-daemon/daemon (via /proc cmdline, ps fallback) — Signal(0) alone matches ANY process that inherited a recycled PID, which made start refuse ('already running') and stop SIGTERM/SIGKILL an innocent bystander. Used by the start gate, stop, and status. Tests: pidLooksLikeDaemon rejects self/sleep/dead pids; readPID treats '0'/empty/garbage as unusable while a real PID round-trips. Live repro verified: a poisoned '0' pidfile no longer blocks start. Full cmd/pilotctl suite green; gosec finding count unchanged vs main. Co-authored-by: Teodor Calin <teodor@vulturelabs.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (2026-07-17 incident)
One peer's NAT/relay session died (list-agents unreachable for hours) while 8 other peers stayed healthy. The global rx-watchdog never fires in this shape — any single live peer keeps
PktsRecvmoving — and nothing health-checks an individual peer mapping: the tunnel keepalive is fire-and-forget (receiver silently swallows it; it warms the NAT mapping but verifies nothing). The only remedy was a full daemon restart — which drops every session and costs minutes of re-establishment on a NAT'd link (measured: mechanical restart 4s, functional recovery to service agents 4-15+ min).Fix
New
pathwatch.go(L4), per-peer feedback loop:lastInboundDecrypt≤~30s via bidirectional keepalives (all daemon versions). Silent > 55s → up to 3 pong-soliciting probes (one per 10s tick).pilotctl prefer-directsequence, now extracted as the sharedresetPeerPath(drop cached resolve + tunnel, preserve relay-active / clear pin, clear rekey cooldowns, proactively push fresh PILA). Healthy sessions untouched. Worst-case detection ≈ 85s; recovery ~1 RTT.-no-path-watchto disable; eventstunnel.path_suspect/tunnel.path_recovered; panic boundary per tick.Backward compatibility (the hard requirement)
Zero wire changes. The probe is a plain
ProtoControl/PortPingframe with a non-empty payload:rc6-dev, between the two): theirisTunnelKeepaliveonly swallows empty frames, so the probe passes the filter, reacheshandleControlPacket, and gets a pong (present in all three trees).TestPathProbeIsNotSwallowedByOldKeepaliveFilteras a tripwire.Tests
State machine (healthy/probe/reset/cooldown/prune), relay-state preservation in
resetPeerPath, keepalive-filter compat. New tests green under-race; fullpkg/daemon/...green under the-shortCI gate. (TestIPCServer_MaxClientsCap/TestWriteLoopExitsOnWriteDeadlinefail locally on clean main and v1.12.7 too — pre-existing non-short-only, unrelated.)Post-merge validation plan
Release, update this laptop, then reproduce the original failure live (wifi→hotspot switch mid-session) and confirm list-agents reachability self-heals within ~85s with zero manual restarts.
🤖 Generated with Claude Code