From 6e44785a9a7ff4b2d25a0b4c5ea01fdd076ee39d Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Mon, 12 Oct 2026 06:50:41 -0400 Subject: [PATCH 1/3] fix(cni): Stop charging galactic-cni for empty service policy maps The private service route maps added about 52 MiB of eBPF maps, and the kernel charges a map's memory to the memory cgroup of the process that creates it. galactic-cni's credential-refresh container creates them, so a node upgrading from v0.5.4, which has none of them pinned, charges the new pod about 59 MiB of maps at start. Five of those maps are plain hashes preallocated at max_entries, about 28 MiB, although production holds no service policy at all. On the busiest production compute node that puts the upgraded container at about 87% of its 128Mi limit. service_route_table, service_access_table, service_deny_table, service_remote_grant_table and attachment_identity_table are now BPF_F_NO_PREALLOC. Only userspace writes them and the datapath only looks rows up, so the per-packet path never allocates. An empty map costs only its bucket array, about 3.5 MiB for the five, and the maps created at start drop from about 59.6 MiB to about 35.3 MiB (the kernel's sizing formula applied to the compiled spec, which reproduces the 59 MiB #808 measured). Rows are still charged to galactic-cni's cgroup as they are written (galactic-router writes the four policy maps, galactic-cni the identity table), so a full set costs what it did before; a row that does not fit now fails that write, which its writer retries, instead of the container's start. service_reverse_table keeps its preallocation: the datapath writes it, and the kernel requires preallocation for an LRU hash. Resizing it is a capacity decision left for later. The flag change makes the five pinned maps incompatible on nodes that already have them, which today means staging only. galactic-cni unpins and recreates them at start, and galactic-router sees the new map IDs and rebuilds its rows, the path the service route maps were built for. Related to #808 Co-Authored-By: Claude Opus 5.5 --- config/galactic-cni/README.md | 21 +++++++++++++++++---- internal/plumbing/ebpf/prog/usid.c | 15 +++++++++++++++ 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/config/galactic-cni/README.md b/config/galactic-cni/README.md index 4653b5e9..e6971130 100644 --- a/config/galactic-cni/README.md +++ b/config/galactic-cni/README.md @@ -69,9 +69,20 @@ node running this manifest. The rationale for each capability: The kernel charges eBPF map memory to the memory cgroup of the process that creates the map. `credential-refresh` creates the uSID datapath's maps when a node has none pinned: a new node, or one upgraded from a build that lacks a -map. Those maps take about 59Mi, most of it `service_reverse_table`. When the -container's limit leaves too little room for them, the map create fails with -`cannot allocate memory` and the pod crash-loops (#808). +map. Those maps take about 35Mi when created, most of it the preallocated +`service_reverse_table`. When the container's limit leaves too little room for +them, the map create fails with `cannot allocate memory` and the pod +crash-loops (#808). + +The five service policy maps that only userspace writes +(`service_route_table`, `service_access_table`, `service_deny_table`, +`service_remote_grant_table` and `attachment_identity_table`) are not +preallocated. Created empty, they cost only their bucket arrays, about 3.5Mi +together. Their rows are still charged to this container's memory cgroup as +they are written (by `galactic-router` for the four policy maps, by +`galactic-cni` for `attachment_identity_table`), up to about 28Mi more if all +five filled to `max_entries`. A row that doesn't fit fails that write, which +its writer retries, rather than failing the container's start. `TestUsid_MapMemoryFitsGalacticCNILimit` in `internal/plumbing/ebpf/prog` loads the maps, sums the size the kernel reports for each, adds 40Mi for the @@ -81,7 +92,9 @@ fails there, so raise the limit in the same change. The test reads that size from each map's fdinfo `memlock` line. It is the kernel's estimate of the map, not the memory cgroup's charge, and it grows -slightly with CPU count. The 40Mi allowance doubles as margin for that gap. +slightly with CPU count. For a map that isn't preallocated it counts only the +rows present, so the test covers the cost of creating the maps, not of filling +the policy maps. The 40Mi allowance doubles as margin for that gap. The end-to-end check is loading the datapath in a memory cgroup: with a 32M limit it fails at `service_reverse_table` with `cannot allocate memory`, and with 96M it loads. diff --git a/internal/plumbing/ebpf/prog/usid.c b/internal/plumbing/ebpf/prog/usid.c index 89d2a82b..0a0d49ac 100644 --- a/internal/plumbing/ebpf/prog/usid.c +++ b/internal/plumbing/ebpf/prog/usid.c @@ -1033,8 +1033,19 @@ struct { // Node-local private-service routes are exact attachment/service matches. They // remain separate from fabric routes so policy never creates a Linux route. +// +// This map and the other policy maps below (attachment_identity_table, +// service_access_table, service_deny_table, service_remote_grant_table) are +// BPF_F_NO_PREALLOC. Only userspace writes them, and most nodes hold few or no +// rows, so a preallocated map would charge its full size (about 28 MiB for the +// five) to galactic-cni's memory cgroup, which creates it, whether or not any +// policy exists (#808). Without preallocation an empty map costs only its +// bucket array, and rows are charged as they are written. The datapath only +// looks rows up, so it never allocates. service_reverse_table stays +// preallocated: the datapath writes it, and the kernel requires it for an LRU. struct { __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); __uint(max_entries, 32768); __type(key, struct service_route_key); __type(value, struct service_route_value); @@ -1045,6 +1056,7 @@ struct { // the token before attaching the service classifier. struct { __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); __uint(max_entries, 32768); __type(key, __u32); __type(value, __u64); @@ -1052,6 +1064,7 @@ struct { struct { __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); __uint(max_entries, 65536); __type(key, struct service_access_key); __type(value, struct service_access_value); @@ -1059,6 +1072,7 @@ struct { struct { __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); __uint(max_entries, 32768); __type(key, struct service_deny_key); __type(value, struct service_deny_value); @@ -1078,6 +1092,7 @@ struct { // immediately even while an LRU row remains. struct { __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); __uint(max_entries, 65536); __type(key, struct service_remote_grant_key); __type(value, struct service_remote_grant_value); From 6a05012abc791aed3b65e5baf653dab5df5130f2 Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Mon, 12 Oct 2026 07:01:52 -0400 Subject: [PATCH 2/3] docs(cni): Shorten the policy map allocation comment The root unit test measured the uSID maps galactic-cni creates at 35 MiB, against a 128 MiB container limit. The comment on the policy maps ran long and repeated the reasoning, so it now says only why they are not preallocated and why the reverse table is. The README states the measured size. Co-Authored-By: Claude Haiku 5.5 --- config/galactic-cni/README.md | 4 ++-- internal/plumbing/ebpf/prog/usid.c | 12 +++--------- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/config/galactic-cni/README.md b/config/galactic-cni/README.md index e6971130..559e53ce 100644 --- a/config/galactic-cni/README.md +++ b/config/galactic-cni/README.md @@ -69,8 +69,8 @@ node running this manifest. The rationale for each capability: The kernel charges eBPF map memory to the memory cgroup of the process that creates the map. `credential-refresh` creates the uSID datapath's maps when a node has none pinned: a new node, or one upgraded from a build that lacks a -map. Those maps take about 35Mi when created, most of it the preallocated -`service_reverse_table`. When the container's limit leaves too little room for +map. Those maps take about 35Mi when created (the root unit test measures it), most +of it the preallocated `service_reverse_table`. When the container's limit leaves too little room for them, the map create fails with `cannot allocate memory` and the pod crash-loops (#808). diff --git a/internal/plumbing/ebpf/prog/usid.c b/internal/plumbing/ebpf/prog/usid.c index 0a0d49ac..fda30a6b 100644 --- a/internal/plumbing/ebpf/prog/usid.c +++ b/internal/plumbing/ebpf/prog/usid.c @@ -1034,15 +1034,9 @@ struct { // Node-local private-service routes are exact attachment/service matches. They // remain separate from fabric routes so policy never creates a Linux route. // -// This map and the other policy maps below (attachment_identity_table, -// service_access_table, service_deny_table, service_remote_grant_table) are -// BPF_F_NO_PREALLOC. Only userspace writes them, and most nodes hold few or no -// rows, so a preallocated map would charge its full size (about 28 MiB for the -// five) to galactic-cni's memory cgroup, which creates it, whether or not any -// policy exists (#808). Without preallocation an empty map costs only its -// bucket array, and rows are charged as they are written. The datapath only -// looks rows up, so it never allocates. service_reverse_table stays -// preallocated: the datapath writes it, and the kernel requires it for an LRU. +// Only userspace writes service_route_table, attachment_identity_table, service_access_table, +// service_deny_table and service_remote_grant_table, so they are BPF_F_NO_PREALLOC and an +// empty one costs only its buckets. service_reverse_table stays preallocated: the datapath writes it. struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(map_flags, BPF_F_NO_PREALLOC); From d338b57dbae499274457e09b2c63eb9dca0fbaf1 Mon Sep 17 00:00:00 2001 From: Peter Sprygada Date: Mon, 12 Oct 2026 07:06:43 -0400 Subject: [PATCH 3/3] style(cni): Wrap the policy map comment and README at 80 columns Co-Authored-By: Claude Opus 5.5 --- config/galactic-cni/README.md | 8 ++++---- internal/plumbing/ebpf/prog/usid.c | 7 ++++--- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/config/galactic-cni/README.md b/config/galactic-cni/README.md index 559e53ce..78a3b4f4 100644 --- a/config/galactic-cni/README.md +++ b/config/galactic-cni/README.md @@ -69,10 +69,10 @@ node running this manifest. The rationale for each capability: The kernel charges eBPF map memory to the memory cgroup of the process that creates the map. `credential-refresh` creates the uSID datapath's maps when a node has none pinned: a new node, or one upgraded from a build that lacks a -map. Those maps take about 35Mi when created (the root unit test measures it), most -of it the preallocated `service_reverse_table`. When the container's limit leaves too little room for -them, the map create fails with `cannot allocate memory` and the pod -crash-loops (#808). +map. Those maps take about 35Mi when created (the root unit test measures it), +most of it the preallocated `service_reverse_table`. When the container's limit +leaves too little room for them, the map create fails with `cannot allocate +memory` and the pod crash-loops (#808). The five service policy maps that only userspace writes (`service_route_table`, `service_access_table`, `service_deny_table`, diff --git a/internal/plumbing/ebpf/prog/usid.c b/internal/plumbing/ebpf/prog/usid.c index fda30a6b..1be49558 100644 --- a/internal/plumbing/ebpf/prog/usid.c +++ b/internal/plumbing/ebpf/prog/usid.c @@ -1034,9 +1034,10 @@ struct { // Node-local private-service routes are exact attachment/service matches. They // remain separate from fabric routes so policy never creates a Linux route. // -// Only userspace writes service_route_table, attachment_identity_table, service_access_table, -// service_deny_table and service_remote_grant_table, so they are BPF_F_NO_PREALLOC and an -// empty one costs only its buckets. service_reverse_table stays preallocated: the datapath writes it. +// Only userspace writes service_route_table, service_access_table, +// service_deny_table, service_remote_grant_table and attachment_identity_table, +// so they are BPF_F_NO_PREALLOC and an empty one costs only its buckets. +// service_reverse_table stays preallocated: the datapath writes it. struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(map_flags, BPF_F_NO_PREALLOC);