fix(doctor): don't treat an assigned-but-Pending training pod as a running job (backend#3247)

[aptracebloc]

What

checkNodeFit counted every non-terminal job-name pod that had a NodeName into the "a running job holds the room" sum — including pods still Pending (ImagePullBackOff / ContainerCreating). A training pod scheduled to a node but wedged on an image pull was therefore reported as HeldByRunningJob, and because the same pod also showed up stuck‑Pending in Pod health, summarizeDoctor rolled it up to "a training is already running, wait for it to finish" at exit 0. The operator was told to wait on a pod that was never going to run.

Closes tracebloc/backend#3247. This is the Bugbot (High) finding that blocked the develop→staging hop (cli#641, thread #641 (comment)).

Fix

A Pending pod (even with a NodeName) is not running:

• checkNodeFit routes a job-name pod into the running‑job sums only when its phase is genuinely Running. An assigned‑but‑Pending one feeds neither the job nor the steady‑state sums, so it can no longer emit HeldByRunningJob.
• Past the same grace window checkPods uses, such a pod is recorded and surfaced as an actionable Fail under a new StuckJobPod prefix that names its real state (ImagePullBackOff / ContainerCreating / Pending) and tells the operator to inspect the pod, not wait. A run legitimately waiting for a running job has no NodeName, so it is never mistaken for a wedged one.
• summarizeDoctor gains a matching arm placed above the inferred stuck‑Pending arm (a measured cause beats an inference), so a wedged pod fails at exit 2 with the right remedy instead of the "wait for the running job" Warn at exit 0.

The genuinely‑running case is unchanged.

Tests

• internal/doctor — TestCheckNodeFitStuckJobPod pins each pod state: Running (→ HeldByRunningJob, unchanged), assigned‑Pending past grace (→ StuckJobPod Fail naming ImagePullBackOff / ContainerCreating / bare Pending), within‑grace (→ neither, node OK), a non‑Job Pending pod (scope guard), and a stuck pod co‑existing with a running job (the stuck Fail wins).
• internal/cli — summarizeDoctor pins that a scheduled‑but‑stuck training pod is a Fail (exit 2), never the "wait for a running job" Warn, both with and without Pod health also flagging it.
• Copy‑catalog golden regenerated for the new user‑facing strings.
• go build ./... && go vet ./... && go test ./... all green.

Version‑bump gate

No VERSION change: develop already carries the pending unreleased bump (0.10.24; the latest release tag is v0.10.23), so this PR ships published‑path changes under an untagged version — the gate's documented pass. Hand‑bumping would be a forbidden manual version bump.

— drafted with Claude Code

---

Note

Medium Risk
Changes doctor verdict and exit codes for a specific cluster state (scheduled training pods wedged on pull/create); logic is heavily tested but misclassification would still mislead operators on training readiness.

Overview
Fixes backend#3247: tracebloc doctor could exit 0 and tell users to wait for a “running” training job when the pod was actually scheduled but stuck (e.g. image pull backoff).

Node capacity (checkNodeFit) now counts batch job pods toward “room held by a running job” only when phase is Running. Assigned-but-Pending job pods with kubelet-reported wedged reasons (ImagePullBackOff, ErrImagePull, create errors, etc., past the same grace as Pod health) surface a new StuckJobPod Fail with inspect-the-pod guidance—not capacity or wait advice. Still-pulling/creating states stay on the existing age-based path so cold large pulls don’t false-positive.

Rollup (summarizeDoctor) adds an arm for that measured Fail (above generic stuck-Pending inference), so readiness is Not ready — stuck starting at exit 2 instead of the transient “wait for the running job” Warn.

Tests cover node-fit classification, rollup behavior, and updated user-facing string golden file.

^{Reviewed by Cursor Bugbot for commit 0e28595. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saqlainsyed007]

Core fix is sound and well-tested — routing genuinely-Running Job pods into the capacity sums while carrying assigned-but-Pending ones out to a dedicated StuckJobPod Fail correctly stops doctor telling operators to "wait for a running job" on a wedged pod (backend#3247). Verdict-switch ordering (checkNodeFit and summarizeDoctor) is internally consistent, the NodeName/terminal guards are in place, and the tests are derived-not-restated and mutation-proof. Two points before approval: (1) doctor.go:545 is the only summarizeDoctor rollup line using kubectl, breaking the plain-terms invariant this function documents three times and its own comment restates — trim to end at "…names the pod." (2) the escalation labels still-progressing ContainerCreating/Pending as "Waiting will not clear it," which can misfire on a large first-pull on a cold node. Neither is a correctness crash; happy to approve once addressed or waved.

[aptracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0e28595. Configure here.}

[saqlainsyed007]

Both findings from my earlier review are genuinely resolved in 0e28595. Approving.

Finding 1 — kubectl in the summarizeDoctor rollup line (plain-terms invariant): Fixed. The rolled-up ready line now ends at "Waiting will not clear it — %s doctor --verbose names the pod." — the kubectl describe pod phrase is gone, so summarizeDoctor holds the no-Kubernetes-vocabulary contract it documents three times. The kubectl form correctly remains one --verbose away in the granular checkNodeFit remedy. A test now guards the rolled-up remedy against kubectl and asserts it points at --verbose, so the invariant won't silently regress.

Finding 2 — "Waiting will not clear it" misfiring on a still-progressing pull: Fixed correctly by scoping the measured Fail to genuinely-wedged reasons via the new stuckReasonWedged allowlist (ImagePullBackOff, ErrImagePull, ErrImageNeverPull, InvalidImageName, CreateContainerConfigError, CreateContainerError). A pod still ContainerCreating/Pulling/bare Pending past grace is left to checkPods' age-based inference, so a large first pull on a cold node no longer trips the "will not clear it" assertion — the escalation fires only where the kubelet actually reported a failure. TestCheckNodeFitStuckJobPod pins wedged vs still-progressing states separately, plus within-grace, non-Job scope, and stuck-outranks-running-job cases.

Node-fit logic is sound: non-Running job pods feed neither resource sum, so a wedged pod can no longer emit HeldByRunningJob; the StuckJobPod Fail sits below the measured capacity Fails and above the transient Warn. CI fully green. LGTM.