[CI] alpha.211 installer and upgrade blocked by hermes-agent CrashLoopBackOff

[khalildai]

Failure

Cloud Installer And Upgrade Test Ref:v2.3.0-alpha.211 run 30972437222 failed all three GitHub-hosted matrices:

• upgrade: job 92199515843
• fresh K3s v1.33: job 92199515848
• fresh K3s v1.22: job 92199515856

In each matrix, the surrounding KB Cloud pods become ready while hermes-agent alternates between RunContainerError and CrashLoopBackOff until the bounded readiness wait fails. Fresh v1.22 reaches at least 7 restarts; upgrade reaches at least 8.

The workflow then runs kubectl logs, but the captured hermes-agent log is empty. It does not preserve the container state reason/message, exit code, command/args, image digest, events, or previous-container log, so the first actionable runtime error is currently lost.

This is scoped to the alpha.211 product input: alpha.209 run 30890210350 and alpha.210 run 30895858238 both succeeded, and alpha.210 used the same apecloud-cd workflow head 69779e56bdf56764e941e1311362f5e0f6c7475e. It is not a self-hosted runner or workflow-code regression signal.

Closeout

1. Resolve and record the exact alpha.211 hermes-agent image tag and digest before install/upgrade.
2. On RunContainerError or CrashLoopBackOff, capture sanitized kubectl describe pod, container current/last state reason and message, exit code, command/args, image ID, events, and both current and --previous logs.
3. Fix the image entrypoint/runtime/configuration defect identified by that first error; do not hide it by extending the readiness timeout.
4. Validate the corrected immutable image through both fresh matrices and the upgrade matrix.
5. Keep alpha.210 as a known-good comparison and do not silently fall back to it.

Patrol did not rerun/cancel the workflow or modify images, charts, workflow, secrets, clusters, release state, or production.

[khalildai]

Patrol update for the next release inputs on the same workflow head 69779e56bdf56764e941e1311362f5e0f6c7475e:

• alpha.212 run 30981983923 reproduced the same failure in all three independent GitHub-hosted matrices: fresh K3s v1.22 job 92228211175, fresh K3s v1.33 job 92228211142, and upgrade job 92228211155. In each case, hermes-agent remained in RunContainerError / CrashLoopBackOff until the bounded readiness check failed; the captured container log was empty.
• alpha.213 run 30984159719 then completed naturally with all three matrices green: v1.22 job 92235033293, v1.33 job 92235033386, and upgrade job 92235033448.

This narrows the product-input regression window to alpha.211-alpha.212 and provides a green successor at alpha.213. It does not close the workflow diagnostic gap: the failed runs still did not preserve the kubelet current/last-state message, exit code, image digest, events, or previous-container log, so the exact alpha.211/212 container-start cause cannot be recovered from Actions evidence alone.

No rerun/cancel, image/chart/workflow mutation, or release action was performed by patrol.