Expose complete workload lifecycle status - #30
Kevin Cho (chokevin) wants to merge 3 commits into
Conversation
…oad-status # Conflicts: # cli/internal/cli/run_lifecycle.go # cli/internal/cli/run_lifecycle_test.go
| out.Reason = issue.summary() | ||
| } else { | ||
| out.Reason = outputFailureReason(snapshot) | ||
| out.Degraded = out.Reason != "" || WorkloadFailed(snapshot) |
There was a problem hiding this comment.
P1 deriveRayJobState can return Failed from jobDeploymentStatus while rayJobStatusFailed returns false when a stale nonterminal jobStatus is present. This then emits state: Failed with degraded: false and no reason. Derive degradation from the normalized state, or make the failure helpers use the same field precedence.
| var out []containerItem | ||
| rayJob := snapshotRayJob(s) | ||
| for _, pod := range s.Pods { | ||
| teardown := podDisplayPhase(rayJob, pod) == "Teardown" |
There was a problem hiding this comment.
P1 A same-name Job/RayJob snapshot can contain pods from both workloads, but this applies the single RayJob state to every merged pod. A successful RayJob can mark a failed batch pod as Teardown and suppress its container failure; unrelated Ray pods can also degrade the batch Job. Preserve pod ownership when hydrating and classify/filter health against the workload that owns each pod.
Summary
Researchers currently need
kubectlto verify Kueue admission, node placement, container restarts, and termination causes. This change makestau run statusthe normal Tau-owned inspection surface for Jobs and RayJobs.It adds per-container readiness, restart, exit-code, and reason details to the table output; preserves controller state separately from degraded container health; and introduces a normalized, versioned
-o jsonsnapshot. Resource reads are classified asobserved,notFound, orunavailable, so missing resources remain distinct from RBAC denial without widening permissions. RayJob execution state also takes precedence over submitter Job and teardown artifacts.AI tools assisted with implementation and review; the changes were validated with the repository test suites below.
Related issue
N/A
Validation
All commands passed.
Compatibility and operational impact
The default human-readable
tau run statusoutput gains a Containers section and degraded-health line.tau run status -o jsonintroduces av1alpha1normalized output contract; JSON is intentionally unsupported with--watchor--run-profile. No CRDs, dependencies, credentials, or RBAC grants change. Manager-only MultiKueue views continue to avoid worker-cluster access, and status explicitly reports unavailable terminal pod evidence after RayJob teardown.Checklist