Skip to content

Expose complete workload lifecycle status - #30

Open
Kevin Cho (chokevin) wants to merge 3 commits into
mainfrom
chokevin-expose-workload-status
Open

Kevin Cho (chokevin) wants to merge 3 commits into
mainfrom
chokevin-expose-workload-status

Conversation

@chokevin

Copy link
Copy Markdown
Contributor

Summary

Researchers currently need kubectl to verify Kueue admission, node placement, container restarts, and termination causes. This change makes tau run status the normal Tau-owned inspection surface for Jobs and RayJobs.

It adds per-container readiness, restart, exit-code, and reason details to the table output; preserves controller state separately from degraded container health; and introduces a normalized, versioned -o json snapshot. Resource reads are classified as observed, notFound, or unavailable, so missing resources remain distinct from RBAC denial without widening permissions. RayJob execution state also takes precedence over submitter Job and teardown artifacts.

AI tools assisted with implementation and review; the changes were validated with the repository test suites below.

Related issue

N/A

Validation

cd core && go test ./... -count=1
cd cli && go test ./internal/cli -count=1
git diff --check

All commands passed.

Compatibility and operational impact

The default human-readable tau run status output gains a Containers section and degraded-health line. tau run status -o json introduces a v1alpha1 normalized output contract; JSON is intentionally unsupported with --watch or --run-profile. No CRDs, dependencies, credentials, or RBAC grants change. Manager-only MultiKueue views continue to avoid worker-cluster access, and status explicitly reports unavailable terminal pod evidence after RayJob teardown.

Checklist

  • The change is focused and does not include unrelated edits.
  • Tests cover the changed behavior, or I explained why tests are not needed.
  • Documentation and examples are updated when public behavior changes.
  • Generated files are updated from their source and included in this pull request.
  • New dependencies are publicly available and their license and provenance were reviewed.
  • The diff contains no secrets, private endpoints, customer data, or Microsoft-internal information.
  • Fork-based CI can validate the change without repository secrets, or maintainer-only validation is clearly identified.

…oad-status

# Conflicts:
#	cli/internal/cli/run_lifecycle.go
#	cli/internal/cli/run_lifecycle_test.go
Comment thread core/status/status.go Outdated
Comment thread core/status/status.go Outdated
Comment thread core/status/output.go
out.Reason = issue.summary()
} else {
out.Reason = outputFailureReason(snapshot)
out.Degraded = out.Reason != "" || WorkloadFailed(snapshot)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 deriveRayJobState can return Failed from jobDeploymentStatus while rayJobStatusFailed returns false when a stale nonterminal jobStatus is present. This then emits state: Failed with degraded: false and no reason. Derive degradation from the normalized state, or make the failure helpers use the same field precedence.

Comment thread core/status/status.go
var out []containerItem
rayJob := snapshotRayJob(s)
for _, pod := range s.Pods {
teardown := podDisplayPhase(rayJob, pod) == "Teardown"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 A same-name Job/RayJob snapshot can contain pods from both workloads, but this applies the single RayJob state to every merged pod. A successful RayJob can mark a failed batch pod as Teardown and suppress its container failure; unrelated Ray pods can also degrade the batch Job. Preserve pod ownership when hydrating and classify/filter health against the workload that owns each pod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants