Skip to content

report why a Node is left out of the target groups - #199

Open
m1ron0xFF wants to merge 1 commit into
fix/skip-empty-provider-id-nodesfrom
feat/report-uninitialized-nodes
Open

report why a Node is left out of the target groups#199
m1ron0xFF wants to merge 1 commit into
fix/skip-empty-provider-id-nodesfrom
feat/report-uninitialized-nodes

Conversation

@m1ron0xFF

Copy link
Copy Markdown
Contributor

What

Improves the report about Nodes that target group synchronization leaves out.

Says which of two causes an empty spec.providerID is. The two need completely
different remediation, and skipping the Node without naming the cause turns a real
capacity loss into a silent one. The node.cloudprovider.kubernetes.io/uninitialized
taint tells them apart at no cost — it sits on the Node object, so no cloud API call
is involved:

Node state What it means Reported as
Taint present kubelet did hand the Node to a cloud provider and cloud-node has not initialized it yet, normally a matter of seconds awaiting cloud-node initialization
Taint absent the Node was never offered to a cloud provider: either a genuine static Node, or a cloud VM whose kubelet is missing --cloud-provider=external no <taint> taint, never handed to a cloud provider

Simplifies the bookkeeping behind that report. lastSkippedNodes goes from a
mapset.Set to the rendered string, so the accumulation loop, the nil guard and a
duplicate strings.Join disappear, and the zero value means "nothing skipped before"
on its own. Reasons are sorted, which is what makes the comparison correct — Node
order from the informer is not stable — and also keeps the logged line stable between
reports.

hasTaint is shared with nodeEligibleForLoadBalancer, which carried the same loop.

Why not backfill the ProviderID instead

#195 hit the second case in production and proposed resolving the Instance by Node
name and patching spec.providerID. That was not taken:

  • spec.providerID is immutable once set, and the value would come from a Node-name
    lookup, so a wrong match becomes permanent — unfixable without deleting the Node
    object.
  • cloud-node already owns this: it sets spec.ProviderID when empty, using the same
    name-based fallback. Doing it again from the load balancer path races that controller.
  • The root cause is cluster configuration, which also affects Node addresses, zones and
    instance types — far more than load balancers.

Reporting which cause it is, is the part that belongs in the target group syncer.

Base

Based on fix/skip-empty-provider-id-nodes (#198), not on master: this changes
partitionNodesByProviderID, which #198 introduces. GitHub will retarget this PR to
master once #198 merges.

Testing

go build ./..., go vet ./pkg/..., go test ./pkg/....

TestPartitionNodesByProviderIDDistinguishesUninitializedFromStaticNodes asserts the
two causes produce different messages. TestNewlySkippedNodesReportsOnlyOnChange
covers the dedup, including that reasons are sorted and that a reshuffled slice is
treated as unchanged. Both were verified by mutation — inverting the taint check and
dropping the sort each make the corresponding test fail.

An empty spec.providerID hides two causes that need different fixes, and
skipping the Node without saying which one it is turns a real capacity loss
into a silent one. The node.cloudprovider.kubernetes.io/uninitialized taint
tells them apart at no cost:

  - taint present: kubelet did hand the Node to a cloud provider and
    cloud-node has not initialized it yet, normally a matter of seconds;
  - taint absent: the Node was never offered to a cloud provider, so it is
    either a genuine static Node or a cloud VM whose kubelet is missing
    --cloud-provider=external, and it stays out of the target groups until
    that is fixed.

Raised while reviewing #195, which hit the second case in production and
proposed backfilling providerID instead. Backfilling writes an immutable
field from a Node-name lookup, so a wrong match becomes permanent, and the
root cause is cluster configuration that also affects Node addresses, zones
and instance types. Reporting which cause it is, is the part that belongs
in the target group syncer.

The bookkeeping behind that report is a rendered string rather than a set,
so the accumulation loop, the nil guard and a duplicate join disappear and
the zero value means "nothing skipped before" on its own. Reasons are
sorted, which is what makes the comparison correct at all - Node order from
the informer is not stable - and keeps the logged line stable between
reports. cleanUpTargetGroups resets it alongside lastVisitedNodes: both
describe target groups that have just been removed.

Signed-off-by: Egor Balakin <14162703+m1ron0xFF@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@m1ron0xFF
m1ron0xFF force-pushed the feat/report-uninitialized-nodes branch from 56bf457 to d775554 Compare August 20, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant