[privateip]

Summary

Edge uplinks on Broadcom NetXtreme NICs can stop receiving on a queue for good while Galactic's XDP programs run in native mode, and a staging run in generic mode saw no stalls or queue discards over seven minutes of NAT traffic.

Galactic now picks one XDP mode per host from its NIC drivers, generic when any NIC uses the Broadcom driver and native otherwise. A node already running the shared dispatcher in native mode is moved over in place, one bond member at a time, keeping every datapath's state.

A missing target interface or an unreadable target driver stops startup instead of guessing a mode, and a failed move restores the old mode and retries with backoff.

Rollout

A cluster gets this only through a release and a separate infra pin bump. Soak it on the staging labs, which track main, including a node still running the dispatcher in native mode, before any production site takes the tag.

Test plan

• A Broadcom edge node logs generic mode, and every bond member shows its program in generic mode
• Gateway and egress shard report ready, with NAT64, NAT66 and VIP traffic passing
• The stalled receive queue alert stays quiet over a day of normal traffic
• Lint, unit tests, root kernel tests and e2e pass

Related to #673

🤖 Generated with Claude Code

[privateip]

Where the selection lives and what it changes:

internal/plumbing/ebpf/xdpmode   Select(required...): generic if any /sys/class/net/*/device/driver is bnxt_en
                                 error on a missing target, an unreadable target driver, or an unreadable
                                 driver link; a non-target NIC with no driver link is skipped with a warning
xdpattach                        Attach/NewSet/ReleaseIdleDispatcher take the mode
                                 ClearDispatcherSlots uses OpenForSlots, which records no mode
                                 native-support check only in native mode; Set coverage reads IFLA_XDP_ATTACHED
xdpdispatch                      Open(ctx, dir, mode); Linked/Coverage mode-aware (ErrModeMismatch)
                                 EnsureLink migrates a pinned link in the other mode (detach, attach, pin)
                                 restore on failure, per-ifindex backoff 30s..10m, host_mode pin + ErrHostModeChanged
cmd/galactic-gateway, -nat       select at startup, log mode and reason, pass it to every attach
docs/xdp-attach-mode.md          mitigation, performance, rollback, staging validation

The kernel cannot change a bpf_link's mode with a link update, and it refuses native and generic programs on one netdev at once, so migration has to detach first. It runs behind the existing bond guard under the dispatch lock, because a link in the other mode is not counted as linked. The interface carries no program between the detach and the new attach.

Test results on this head:

golangci-lint run ./...                                    0 issues
task build test:unit                                       pass
task test:e2e                                              pass (kind)
sudo go test ./internal/plumbing/ebpf/{xdpdispatch,xdpattach,xdpmode,edgeattach,natattach}/ \
             ./cmd/galactic-{gateway,nat}/ -count=1        pass

The root tests cover both modes on a real veth: migration keeping root, slot, lease and role; an injected generic attach failure restoring native and backing off; a stale process refusing to move a link back; a foreign generic program left alone; and a generic attach reading back as SKB mode, so a native Set does not count it.

Build:

task build
docker build --network=host --build-context network=../network -t galactic-gateway:<tag> -f containers/galactic-gateway/Dockerfile .
docker build --network=host --build-context network=../network -t galactic-nat:<tag> -f containers/galactic-nat/Dockerfile .

Staging validation, once a main build with this reaches the staging lab's edge node. That node already runs the dispatcher in generic mode by hand, so it needs no migration and sees no bounce.

1. Both pods log Selected the XDP attach mode for this host with mode=generic and a reason naming the four bnxt_en ports.
2. ip -d link show on the node shows xdpgeneric and prog/xdp ... name xdp_dispatch on all four bond members.
3. Gateway readiness passes, the EgressShard is Ready, and galactic_nat_uplink_covered is 1 on every uplink.
4. NAT64 and NAT66 TCP, UDP and ping, plus VIP traffic, pass.
5. Over a day, GalacticNatUplinkRxQueueStalled stays quiet, every queue's receive counter keeps rising, and queue discards stay at 0.

Rollback: revert the image. In dispatch mode an older release accepts the pinned generic links as they are, so the node stays generic. To return it to native, run hack/xdp-dispatch-release.sh on the node, then restart galactic-gateway and galactic-nat. Each uplink bounces once.

The 408-second validation figure comes from the earlier staging run. Its logs were not available here, so this PR does not reproduce them.