nri agent: clear own config status entry on clean stop

[TonyxSun]

Summary

Every NRI resource-policy agent MergePatches a shared config CR
(TopologyAwarePolicy / balloons / template) at
status.nodes[<nodeName>] when it applies config. That map only loses a
key when the config resource name changes for the same agent
(NodeStatusPatch(node, nil) on the previous name). It is never
cleared when the agent process exits.

On clusters with node replacement, that could leave
permanent orphans: hosts that no longer exist still appear as
Success in status, the CR keeps growing, and every live agent rewrites
a larger object on each status PATCH.

This change, on clean Agent.Stop(), merge-patches
status.nodes[<thisNode>] = null for the current config CR (with a 10s
timeout) before closing the stop channel, while the kube client is
still usable. Graceful DaemonSet stops and SIGTERM then remove that
node's status key instead of leaving it forever. Use the same existing
NodeStatusPatch(node, nil).

What this does not fix

• Hard kill / OOM / node power loss (no Stop())
• Hostname replace without a graceful stop of the old agent

Why it matters (evidence)

Observed on a cluster running topology aware nri plugin.

# Shared policy: count status.nodes keys
kubectl --context <cluster> -n kube-system get topologyawarepolicies default \
  -o jsonpath='{.status.nodes}' | jq 'keys | length'
# example: 135

# Live nodes
kubectl --context <cluster> get nodes --no-headers | wc -l
# example: 126

[askervin]

Hi @TonyxSun! Thank you for bringing out this issue and submitting the patch!

As you pointed out by "what this does not fix", agent shutdown sequence is quite unreliable location for clean up: there is no guarantee that it gets executed or that it has enough time to finish before the agent dies.

Very recently, we had somewhat similar issue with extended resources published via the agent. There we ended up not even trying to clean them up at exit, but running a reconciliation loop that keeps published resources up-to-date, which sometimes means removing them if the policy is no more publishing resources.

This case is not exactly the same because there will never be agents that would reconcile (clean up) status of nodes that are forever gone.

If this solution eases the problem in your case, I'm fine to approve this change as is before we get a more complete solution.

One possibility could be that the agent configuration would have a "removeStatusesOfMissingNodes" flag. With that flag set in a node-specific policy configuration, there could be then a single node where the agent would be responsible for comparing published config statuses in the CR to list of nodes, and updating it accordingly. This would avoid every agent trying to do the same thing.

Let's hear thoughts from @klihub, @marquiz and @kad, too.

[TonyxSun]

a single node where the agent would be responsible for comparing published config statuses in the CR to list of nodes

Is there already leader election mechanism in the agent? Feels like it could be overengineering, unless, it is part of a bigger scope of clean up (I do see #709 on this).

avoid every agent trying to do the same thing.

But yes, on, for example ds deletion, all agents patching the status will likely not all succeed whereas a dedicated clean up loop might be more effective.

running a reconciliation loop

solution eases the problem in your case

It is not a big deal to have stale resources, and we could run a loop to clean up as well, so happy to close / abandon this if there is something more comprehensive (less best-effort) planned.