perf(net): reduce health-check probe and reconfiguration overhead

[Patrick W. Healy (phealy)]

Summary

Third production-only efficiency layer, based on #744.

Stack: #731 -> #743 (status collection) -> #744 (status transport/controller memory) -> #745 (this PR).

• Spread initial probes and transmit-interval changes using a deterministic directed-peer phase instead of synchronized starts.
• Update settings in live health-check sessions rather than recreating goroutines/timers and losing health state, counters, RTT, uptime, or flap history.
• Pass freshly resolved health profiles into tunnel reconciliation. An explicitly disabled governing profile prevents fallback.
• Preserve fix(net): validate OIDC pod bindings against informer caches #731's routing rollback, localCIDR handling, and GatewayPoolPeering protocol precedence.
• Include focused unit/race coverage, CLI/API documentation, and regenerated CRD descriptions. No simulator or e2e files are included.

Intentional behavior change

Default transmit and receive intervals increase from 1s to 15s, reducing default probe traffic and processing. With the unchanged detect multiplier of 3, the nominal detection timeout increases from 3s to 45s; timer checks can add another half-timeout. Explicit interval overrides remain honored for deployments requiring faster detection.

Settings-only changes preserve live session state. Shortening a healthy session's timeout provides a bounded transition window for a reply under the new cadence. Changing the overlay IP still replaces the session.

Validation

• Scoped formatting, lint, actionlint, normal tests, race tests, and builds passed before stacking.
• Regenerated the five consuming Site/net CRDs using the pinned controller-gen; repeat generation was idempotent and schema changes are documentation-only.
• After rebasing onto perf(net): reduce status transport and controller memory overhead #744, combined race-enabled tests passed for healthcheck, node, controller, shared status, net CLI, and both affected API packages.
• Combined scoped lint and node/controller/kubectl builds passed with a clean worktree.

This PR does not claim a measured whole-cluster CPU or RSS reduction. It reduces probe cadence and session churn while documenting the responsiveness tradeoff.

[Copilot]

🟡 Changes recommended

Successive reconfigurations can retain stale detection grace beyond the current settings, and public reference documentation remains inconsistent.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Reduces network health-check overhead while preserving live session state during reconfiguration.

Changes:

• Raises default probe intervals to 15s and phases probe scheduling.
• Applies profile updates to active sessions with validation and fallback handling.
• Updates tests, CLI help, API descriptions, CRDs, and documentation.

File summaries

File	Description
internal/net/healthcheck/types.go	Updates defaults and validates settings.
internal/net/healthcheck/session.go	Reconfigures live timers and sessions.
internal/net/healthcheck/reconfiguration_test.go	Tests live reconfiguration.
internal/net/healthcheck/probe_phase.go	Computes deterministic probe phases.
internal/net/healthcheck/probe_phase_test.go	Tests phase distribution and timing.
internal/net/healthcheck/manager.go	Preserves sessions during updates.
docs/net/configuration.md	Documents health-check behavior.
docs/net/architecture.md	Updates architecture defaults.
deploy/net/crd/net.unbounded-cloud.io_sitepeerings.yaml	Updates generated interval descriptions.
deploy/net/crd/net.unbounded-cloud.io_sitegatewaypoolassignments.yaml	Updates generated interval descriptions.
deploy/net/crd/net.unbounded-cloud.io_gatewaypools.yaml	Updates generated interval descriptions.
deploy/net/crd/net.unbounded-cloud.io_gatewaypoolpeerings.yaml	Updates generated interval descriptions.
deploy/machina/crd/unbounded-cloud.io_sites.yaml	Updates generated interval descriptions.
cmd/unbounded-net-node/wireguard_config.go	Applies fresh profiles to WireGuard peers.
cmd/unbounded-net-node/tunnel_config.go	Applies profiles to shared tunnels.
cmd/unbounded-net-node/site_watch_reconcile.go	Resolves disabled and fresh profiles.
cmd/unbounded-net-node/site_routing_reconcile_test.go	Adapts reconciliation tests.
cmd/unbounded-net-node/reconciliation_helpers.go	Updates profile comparison and disabling.
cmd/unbounded-net-node/reconciliation_helpers_test.go	Updates profile expectations.
cmd/unbounded-net-node/peer_healthcheck.go	Resolves and registers profile settings.
cmd/unbounded-net-node/peer_healthcheck_test.go	Tests profile precedence and freshness.
cmd/unbounded-net-node/main_update_test.go	Adapts configuration test hooks.
cmd/unbounded-net-node/gateway_pool_peering_protocol_test.go	Adapts protocol tests.
cmd/unbounded-net-node/cni_reconcile_test.go	Adapts CNI reconciliation tests.
cmd/kubectl-unbounded/app/net/create.go	Documents inherited CLI defaults.
cmd/kubectl-unbounded/app/net/create_test.go	Tests CLI default preservation.
api/net/v1alpha1/types.go	Documents API interval defaults.

Review details

Suppressed comments (1)

docs/net/configuration.md:383

• The newly documented precedence and disabled-fallback behavior conflicts with both custom-resource references: docs/net/custom-resources.md:565-579 and docs/content/reference/networking/custom-resources.md:390-398 still describe the old gateway fallback order and omit the explicit-disable rule. Update those pages with the same resolved-scope semantics.

For mesh peers, an explicit gateway-pool or pool-peering profile wins, followed by a `SiteGatewayPoolAssignment`, a `SitePeering`, and then the peer's `Site`.
For node-to-gateway peers, the node's site/pool assignment governs; gateway nodes use their explicit peer profile or the peer's pool profile, with a remote-site/pool assignment as fallback.
Shared-tunnel gateway peers may fall back to the local site's profile when no governing association exists; WireGuard gateway peers do not use that site fallback.
An explicitly disabled governing profile blocks every fallback.

• Files reviewed: 27/27 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.