Skip to content

feat(router): Apply traffic policy bindings on each node - #870

Merged
privateip merged 1 commit into
feat/traffic-policy-ingressfrom
feat/traffic-policy-router
Oct 15, 2026
Merged

privateip merged 1 commit into
feat/traffic-policy-ingressfrom
feat/traffic-policy-router

Conversation

@privateip

Copy link
Copy Markdown
Collaborator

Summary

The datapath can enforce interface traffic policy, but nothing on a node yet turns a tenant's bindings into the rules it runs, so every interface could only hold the built-in deny.

The node router now compiles each attachment's bound policies, installs them per direction, reports what it applied in the binding's per-node status, and serves the policy counters to the existing metrics scrape.

A rebind that cannot be installed, a missing policy, or a reserved-name conflict leaves the direction denying rather than keeping old grants, and a restart keeps counts by adopting what is still installed.

The feature is off by default and does not enforce anything by itself, since whether the datapath enforces stays the CNI setting.

Test plan

  • Against a real API server and real BPF maps, bindings install, rebinds and deletions fail closed, and counts survive a restart
  • Counter series keep their rows across rule edits and start fresh on a rename, with the documented labels
  • Resyncs and attachments on other nodes write no status
  • Build, lint, unit, kernel and e2e tests pass

Stacked on #869. Related to #691

🤖 Generated with Claude Code

@privateip privateip self-assigned this Oct 15, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Detail for reviewers, in suggested review order.

  1. internal/trafficpolicy/node/reconciler.go is the core. Requests are keyed by VPCAttachment. Each direction resolves its reference (none means the built-in default-deny-all, identity builtin:default-deny-all:v1), compiles, and publishes a snapshot. Counter rows are keyed by (rule name, action) within one policy UID and interface token, freed only after the outer-map update returns, and rolled back on a failed publish. A rebind whose publish fails is invalidated; a failed update of the same UID keeps its last snapshot.
  2. Restart adoption: open() reads every bound snapshot and reserves its rows, or keeps nothing. adopt requires token, UID and generation to match and the bound entries to equal what the policy compiles to today (RuleRows), so a compiler change republishes. Snapshots on interfaces no attachment claims within resync plus 5s are unbound; interfaces this node has reconciled are never swept.
  3. Status: server-side apply with field manager galactic-router/<node> (hashed past 128 bytes). writeStatus reads the cached binding and patches only when this node's entry would change, after the datapath lock is released, so resyncs and other nodes' attachments cost no writes.
  4. metrics.go: galactic_cni_traffic_policy_*, the names the design fixed. galactic-router serves them because it owns the rows and their labels; the design doc now says so. Only an attachment's current ifindex is exported, so a direction stranded on an old ifindex never duplicates labels. Unreadable rows are left out and counted in galactic_cni_traffic_policy_counter_read_errors.
  5. cmd/galactic-router/policy.go: policy explain (file or cluster, evaluator plus compiled entries, fails on disagreement) and policy show.
  6. usid.c/tpmap: the snapshot header gains generation and policy_uid (pad 48, header still 88 bytes); the datapath does not read them. Verifier cost unchanged (usid_ingress 21,098, usid_egress 12,454 on 7.1).

Checks run on 870e5a1 with base 5399e67: task build test:unit test:e2e pass; golangci-lint clean; root tests (-race -p 1) for the node reconciler, tpmap, prog, cnibgp pass. The reconciler test was tamper-checked: removing the rebind invalidation, the no-entry skip, the unchanged-entry skip, row reuse, or the sweep guard each fails it.

Gaps: no containerlab traffic yet (PR 8). After a restart, an invalid edit of an adopted same-UID policy invalidates rather than keeping the adopted snapshot, since adopted snapshots carry no rule names; this is fail-closed and documented in the notes. A pre-push review's request-changes findings (rebind leaving grants, cluster-wide status writes, partial startup) are fixed in this commit.

@privateip

Copy link
Copy Markdown
Collaborator Author

Two notes from the conventions review. The new grants sit in the ClusterRole the router and the route reflector share, like the existing service binding grants, so the reflector holds read access and status patch it never uses because it refuses the feature at startup. #643 also edits the router setup, its RBAC and go.mod, so whichever of the two lands second rebases.

@privateip
privateip added this pull request to stack #872 October 15, 2026 17:15
@privateip
privateip marked this pull request as ready for review October 15, 2026 17:16
@privateip
privateip requested a review from a team as a code owner October 15, 2026 17:16
ecv
ecv previously approved these changes Oct 15, 2026
galactic-router gains the node side of interface traffic policy, behind
GALACTIC_ROUTER_TRAFFIC_POLICY (off by default, refused on a route
reflector). For every VPCAttachment on its node it resolves the binding's
ingress and egress references, compiles each policy, and publishes it into
the uSID datapath; a direction with no reference gets the built-in
default-deny-all.

- Counter rows: each rule keeps its row while its name and action stay the
  same within one policy UID and interface token, so editing a match keeps
  its counts. Replaced rows are freed only after the outer-map update
  returns; a failed publish frees what it allocated.
- Fail closed: a missing or reserved-conflict reference, and a rebind that
  cannot be installed, unbind the direction (policy_unavailable). A failed
  update or an invalid edit of the same policy keeps its last snapshot.
- Restart: snapshots record the policy UID and generation beside the token.
  A restarted router adopts a snapshot only when it holds exactly what the
  policy compiles to today, so counts continue and an older compiler's
  output is republished. Startup reads every bound snapshot or nothing, and
  a snapshot nothing claims within a resync is unbound and its rows freed.
- Status: this node's entry in the binding's status.nodes, applied with
  server-side apply, written only when it changes, after the datapath lock
  is released.
- Metrics: galactic_cni_traffic_policy_* on the router's metrics port,
  labeled from what the router installed, plus a gauge of counter rows a
  scrape could not read. The design said galactic-cni would serve them; the
  router allocates the rows and knows their rule and interface, so it does,
  under the names the design fixed. docs/metrics.md documents them.
- Diagnostics: `galactic-router policy explain` decides a packet with the
  Go evaluator and the compiled entries and fails if they disagree;
  `policy show` prints what the node holds for one attachment.

RBAC gains read access to policies and bindings and patch on binding
status. The root unit job gains envtest assets so the reconciler's
envtest-and-BPF test runs in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@privateip
privateip merged commit 2c562cf into main Oct 15, 2026
25 of 26 checks passed
@privateip
privateip deleted the feat/traffic-policy-router branch October 15, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants