Round-trip multi-role JobSets + k8s helper consolidation - #21
Merged
Conversation
Kueue and YuniKorn emit multi-role jobs as a JobSet (one replicatedJob per role), but the parsers only understood a single batch/v1 Job, so those jobs could not round-trip back to SPLAT. Add JobSet parsing: - internal/jobset.ToTasks inverts the emitters' replicatedJobs into SPLAT tasks (execution incl. inlined-script detection, resources, placement, replicas) plus shared job volumes and the max BackoffLimit. - The Kueue and YuniKorn parsers detect kind: JobSet and delegate; YuniKorn additionally recovers gang scheduling from the task-groups annotation. Tech debt folded in while here: - Move the duplicated tolerations helper and the Volcano parser's volume recovery into k8senc (Tolerations, VolumesFromPod, SortVolumes); Volcano and the JobSet parser now share them. - Trigger the dry-run workflow on parser/** and k8senc/** changes too, not just emitters. Tests: JobSet parse for Kueue (container + inlined-script roles) and YuniKorn (gang from task-groups), plus an emit->parse round-trip guarding against emitter/parser drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PnTz6Zxqa4jHocK8kCbyx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Multi-role jobs round-trip now, plus some k8s-helper consolidation.
JobSet round-trip
Kueue and YuniKorn emit multi-role jobs as a JobSet (one
replicatedJobper role), but the parsers only understood a single
batch/v1Job — so amulti-role job could be emitted but never parsed back to SPLAT. Fixed:
internal/jobset.ToTasksinverts the emitters'replicatedJobsinto SPLATtasks — execution (including detecting an inlined
/bin/bash -cscript backto
execution.script), resources, placement, and replicas — plus the sharedjob-level volumes and the max
BackoffLimit.kind: JobSetand delegate. YuniKornadditionally recovers gang scheduling from the
task-groupsannotation(gang
minAvailable= sum of the groups'minMember).Tech debt (folded in)
tolerationshelper (Volcano + Armada) and the Volcanoparser's volume recovery into
k8senc:Tolerations,VolumesFromPod,SortVolumes. Volcano and the JobSet parser now share one implementation.internal/parser/**andinternal/k8senc/**changes, not just emitters.Testing
TestParse_JobSet(Kueue): a JobSet with a container role and aninlined-script role parses back to two tasks with correct replicas,
resources, GPU, node selector, and script recovery.
TestParse_JobSet_Gang(YuniKorn): queue + two roles + gang recovered fromthe task-groups annotation.
TestRoundTrip_MultiRoleJobSet: SPLAT → Kueue JobSet → SPLAT preserves roles,replicas, images, and GPU — guards against emitter/parser drift.
go vet, andgofmtall clean.🤖 Generated with Claude Code