Repository navigation
Operators can't see or alert on NAT egress health #722
Description
Summary
The egress shards count every drop by reason and track their session table, but the monitoring bundle ships scrape configs and alert rules for the two routers only, not for NAT.
A shard that stops translating, fills its session table, or starts dropping replies is invisible until a tenant reports broken egress. The #684 design calls these SLIs out as missing before implementation.
Acceptance criteria
- Deploying the monitoring bundle collects NAT drop counters and session table use from every shard
- An operator is alerted when a shard stops translating or its drop rate climbs well past its baseline
- An operator is alerted before a shard's session table fills
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Medium
Activity
privateip commentedon Oct 30, 2026
Verdict: valid, partly. Checked against
mainat 7493aa0.mainconfig/monitoring/podmonitor-galactic-nat.yaml(76c9339, #741) scrapes every shard's metrics port.GalacticNatNotTranslatingandGalacticNatForwardDropRatio(7261dd4), plusGalacticNatNotProgrammed,GalacticNatUplinkUncoveredandGalacticNatMetricsDown.On the last criterion: the only session-table rule,
GalacticNatSessionTableEvictingLive, fires when the table is over 90% full and its oldest row is younger than the 7440 s TCP timeout. At that point the LRU is already evicting live sessions, so the alert reports the failure rather than warning ahead of it. Raw row fill (galactic_nat_conns) can't give an early warning, because expired rows stay until evicted and a busy table sits near full by design.galactic_nat_sessions(live sessions only, from #739) is the right signal, but no rule expression inconfig/monitoring/prometheusrule.yamlreads it. It appears only in one annotation's text.privateip commentedon Oct 30, 2026
Closing this as complete. The monitoring bundle on
mainalready scrapes every shard (#741) and alerts when a shard stops translating or its drop rate climbs (7261dd4). For the session table we're keepingGalacticNatSessionTableEvictingLiveas the signal. Because expired rows fill the table by design, the useful event is the LRU starting to evict live sessions, and that rule fires on exactly that. A separate earlier warning on live-session fill can be raised as its own issue if it turns out to be needed.Please reopen if a shard's egress health still goes unseen on the latest version.