Repository navigation
feat(cni): Spread egress across all EgressShards in hashed mode - #836
Merged
Merged
Conversation
privateip
changed the title
This was referenced Oct 10, 2026
…mode
galactic-cni run now owns how tenant VRFs reach the egress shards. GALACTIC_CNI_EGRESS_MODE=ordered, the default, keeps today's behaviour: each VRF uses the first reachable shard in GALACTIC_CNI_EGRESS_SHARD_SIDS. GALACTIC_CNI_EGRESS_MODE=hashed makes every eligible EgressShard in the cluster a member of the shard groups usid_egress picks from, so each shard carries about 1/N of the metro's egress and losing one costs only its sessions.
In hashed mode the installer watches EgressShards. A shard is a candidate once its status carries a shardSID with Programmed and Ready true; it is draining while spec.drain is set, and a terminating shard is left out. It joins a translation class's group only if its status can translate that class: an IPv6 address for NAT66; an IPv4 address and the matching nat64Prefix, or translatesWellKnownPrefix for 64:ff9b::/96, for each NAT64 prefix. More than three NAT64 prefixes in hashed mode stops the container at startup. A candidate takes tenants only while its SID resolves to an uplink neighbor, so a dead node's stale status does not matter. A netlink route and neighbor watch rebuilds the groups as soon as a route moves, and the 30s sweep stays as a backstop. Until the watch has listed the cluster once, the groups keep their last members.
Going back to ordered mode publishes every enabled group as CLOSING, moves every route back to the static list, and disables the groups only after a sweep that read and wrote every route without error and found none still naming a group, other namespaces' routes included. A failed sweep leaves the groups forwarding and is retried.
GALACTIC_CNI_EGRESS_HASH picks source (default, one public address per tenant address) or flow hashing, GALACTIC_CNI_EGRESS_PIN_IDLE sets how long a tenant address stays pinned to its shard after its last packet (default 2h4m, the shard's established-TCP timeout; 0 turns pins off), and GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE sets the floor the pool alert compares against. An invalid value stops the container at startup.
New metrics: galactic_cni_egress_shard_{packets,bytes}_total{shard_sid}, galactic_cni_egress_shard_selections_total{result}, galactic_cni_egress_pool_members{class,state} (with an ineligible state), galactic_cni_egress_pool_min_active, galactic_cni_egress_pool_enabled{class}, and the control loop's own state: galactic_cni_egress_mode, galactic_cni_egress_shard_watch_synced, galactic_cni_egress_group_routes{class}, galactic_cni_egress_sweep_errors_total{stage} and the last successful sweep's timestamp. A node told to run hashed mode that never published a group says so instead of exporting nothing.
galactic-cni's ClusterRole gains get, list and watch on egressshards. go.datum.net/network moves to the commit that adds EgressShard spec.drain (datum-cloud/network#32); it moves to the release once that is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
privateip
force-pushed
the
feat/egress-aa-cni
branch
from
October 12, 2026 16:05
90d6a75 to
9c13287
Compare
privateip
marked this pull request as ready for review
October 12, 2026 16:06
scotwells
approved these changes
Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second of three stacked PRs for active/active egress, on top of #835. It adds
GALACTIC_CNI_EGRESS_MODEto galactic-cni.ordered, the default, keeps today's behaviour.hashedspreads tenant addresses across every eligibleEgressShardin the cluster through the groups #835 added, so each shard carries about 1/N of the metro's egress and losing one costs only its sessions.spec.drainis set. A terminating shard is left out. It joins a translation class only if its status can translate that class. A netlink route watch rebuilds the groups when a route moves, and the 30s sweep is the backstop.orderedmarks every group closing, moves every route back to the static list, and disables the groups only after a sweep that wrote every move and found no route still naming a group. Any error leaves the groups forwarding, and the next sweep tries again.GALACTIC_CNI_EGRESS_HASH(source or flow),GALACTIC_CNI_EGRESS_PIN_IDLE(default 2h4m) andGALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE. Invalid values, or more than three NAT64 prefixes in hashed mode, stop the container at startup.egressshards.Merge order
go.modpins datum-cloud/network#32's commit by pseudo-version. Keep this a draft until that PR is merged and released, then re-pin to the release tag.Three findings from the feature's re-review (recorded in the third PR) are open and block enabling hashed mode; this PR holds two of them:
Ordered mode is unaffected.
Test plan
go test -race -p 1)verify:nat-active-activecheck passed against this code (see the third PR)🤖 Generated with Claude Code