Repository navigation
feat(ebpf): Pick an egress shard per tenant address in usid_egress - #835
Merged
Merged
Conversation
A tenant VRF's egress route names exactly one shard, so every compute node sends a metro's egress to the first shard that resolves and keeps the rest as standby. Losing that shard loses every session in the metro, and the other shards' capacity sits idle. An egress_route_table entry can now name a shard group instead: link_ifindex 0xFFFFFFFF, the group ID in the first four bytes of sid and the tenant's Argument where a SID carries it. usid_egress hashes the tenant's source address (or, per group, the 5-tuple) with FNV-1a, follows a live pin in egress_shard_pin or else the group's 1021-slot Maglev table, completes the chosen shard's base SID with the Argument and encapsulates exactly as for an ordinary entry. An empty or disabled group drops the packet and counts it in egress_shard_stats rather than failing open into the main table. Each group is one immutable snapshot (Maglev table plus every member's SID, next hop, flags and slot generation) in a one-entry inner map of egress_shard_groups, an ARRAY_OF_MAPS. egressroutemap publishes a change by creating a new inner map, filling it and swapping the outer slot in one update, so a packet sees the old snapshot or the new one, never a mix. Updating a published value in place produced 17 torn packets out of 54,777 on a kernel; the swap produced 0 out of 122,312 across 400 concurrent swaps. A pin is honoured only while its slot still carries the generation the pin recorded. Generations come from next_slot_generation in the published snapshot plus an in-process high-water mark that also counts failed publications, so a reused slot never inherits another shard's pins, even after a publication that failed half way. egress_shard_group_template, an unused instance of the inner map type, makes clang before 21 emit the value's full BTF. There is one group per translation class: group 0 for NAT66 and groups 1-3 for up to three NAT64 prefixes. CNI ADD (attachreg.installVRFEgress) and the datapath repair route each class to its group when the group is open and has a reachable shard, and otherwise keep today's ordered path. A group marked CLOSING still forwards but takes no new routes, and ADD re-reads the groups after writing and falls back to the ordered install if they closed meanwhile. Refresh, given WithShardGroup, moves shard-bound entries onto the groups and, without it, moves group routes back to the first resolvable shard of the static list, returning every error. Nothing changes until something publishes a group: with every group disabled, ADD and repair behave as before. Every piece is a new map and no existing map changes layout, so attach.Load reuses every pinned row on upgrade; it removes the pins of two group maps that only pre-release lab builds created. The maps grow the datapath from about 59Mi to 67Mi, inside galactic-cni's 128Mi limit. The privileged tests run usid_egress under BPF_PROG_RUN against maps the Go side published and check every placement against ShardHash and internal/maglev: the spread, a failure moving only its shard's tenants, pins through added and draining shards, slot reuse, the drop paths, flow hashing with fragments, IPv4, publication faults at each step, and concurrent swaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
scotwells
approved these changes
Oct 10, 2026
1 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each tenant VRF's egress route names one egress shard, so a compute node sends all of its metro's egress to the first shard that resolves and keeps the others idle. Losing that shard drops every session in the metro.
This is the datapath half of active/active egress, the first of three stacked PRs. An
egress_route_tableentry can now name a shard group instead of a shard.usid_egresshashes the tenant's source address (or the 5-tuple), follows a live pin or else the group's Maglev table, and encapsulates toward the chosen shard as it does today.ARRAY_OF_MAPSslot in one update. A packet sees the old snapshot or the new one, never a mix: 0 mixed packets of 122,312 across 400 concurrent swaps, against 17 of 54,777 when a published value was rewritten in place.Nothing reads a group until something publishes one, which only the next PR's hashed mode does. With every group disabled, CNI ADD and repair behave as before. All the maps are new and no existing map changes layout, so upgrades keep every pinned row. Datapath memory grows from about 59Mi to 67Mi, inside galactic-cni's 128Mi limit.
Stack
The feature's review and re-review are in the third PR. Three re-review findings are still open and block enabling hashed mode. Two of them touch this PR's code: CNI ADD skips a NAT64 prefix that no shard serves, so it falls through to the NAT66 route (R03), and closing the groups has no barrier against an in-flight ADD (R01, repaired within one sweep). Neither affects ordered mode, the default.
Test plan
go test -race -p 1)Running the root-gated packages in parallel, as
task test:unit-rootdoes, can failinternal/cnibgpandinternal/cnitests that collide on host interfaces. Run one at a time they pass, here and on main.🤖 Generated with Claude Code