Kill dead pods in node restore - #6985
Conversation
[static] Signed-off-by: Oriol Muñoz <oriol.munoz@digitalasset.com>
Signed-off-by: Oriol Muñoz <oriol.munoz@digitalasset.com>
OriolMunoz-da
changed the title
There was a problem hiding this comment.
Pull request overview
This PR updates the node restore workflow to proactively remove “dead” Kubernetes pods after scaling down components, preventing wait_down from blocking indefinitely on pods stuck in bad states (e.g., evicted/unknown).
Changes:
- Add a
kill_dead_podshelper to identify and delete pods in problematic states before waiting for scale-down completion. - Invoke the new cleanup step in
main()after scaling down components and beforewait_down.
Suppressed comments (1)
cluster/scripts/node-restore.sh:506
kill_dead_podsis called without any scoping, so it may delete unrelated “bad” pods in the namespace (e.g., from components not being restored). Sincewait_downwaits onapp=$deployment_name, it’s safer to scope pod cleanup to the same set ofapplabels derived from the components being restored.
kill_dead_pods "$namespace"
for component in "${components[@]}"; do
wait_down "$namespace" "$component" "$migration_id"
done
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| function kill_dead_pods() { | ||
| local -r namespace=$1 | ||
| local -r bad_pods=$( | ||
| kubectl get pods -n "$namespace" -o json | \ | ||
| jq -r '.items[] | | ||
| select( | ||
| (.status.phase == "Unknown" and .status.reason == "ContainerStatusUnknown") or | ||
| (.status.reason == "Evicted") or | ||
| (.status.containerStatuses[]?.state.terminated? | .reason == "Error" and .exitCode == 137) or | ||
| (.status.initContainerStatuses[]?.state.waiting?.reason == "ContainerStatusUnknown") | ||
| ) | .metadata.name' | sort -u | ||
| ); | ||
| if [ -z "$bad_pods" ]; then | ||
| echo "No bad pods found in $namespace. Skipping."; | ||
| return | ||
| fi | ||
| echo "Found bad pods in $namespace: $bad_pods"; | ||
| for pod_name in $bad_pods; do | ||
| echo "Attempting to delete pod $namespace/$pod_name"; | ||
| if kubectl delete pod -n "$namespace" "$pod_name"; then | ||
| echo "Successfully deleted $namespace/$pod_name"; | ||
| else | ||
| echo "Failed to delete $namespace/$pod_name"; | ||
| fi | ||
| done |
There was a problem hiding this comment.
(1) accepting an optional label selector so callers can scope deletions to the workloads being restored
doesn't seem worth it
--ignore-not-found=true
that sounds reasonable enough
--wait=false
this less so: I do want it to be stuck here because it's much easier to debug than it waiting in wait_down
Signed-off-by: Oriol Muñoz <oriol.munoz@digitalasset.com>
|
/cluster_test |
|
Deploy cluster test triggered for Commit 357aefceeea706078cabba0d41f3d490ef7bed93 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/80000 |
Fixes https://github.com/DACH-NY/cn-test-failures/issues/9618
See https://github.com/DACH-NY/cn-test-failures/issues/9618#issuecomment-5427692351 and linked logs