docs(resources): DefaultTraining is the per-rank base; GPU pods carry it times their rank count - #650
Merged
Conversation
… it times their rank count The comment on DefaultTraining said the value was "quietly wrong above world_size 1" and that per-job-vs-per-rank was an open question. Since client-runtime#500 that is no longer what the runtime does with the number: a GPU pod's rank count is its nvidia.com/gpu limit, and the runtime stamps this per-rank base times that count, capped so the node's GPU pods still pack, refusing admission on a single-node edge when the scaled envelope can never be placed (client-runtime#483). A CPU pod runs a CPU-only torch, so its world_size is 1 by construction. Records the shipped behaviour where the number lives; changes no code. The package caveat about GPU pods being BestEffort is narrowed to the case where that is still true (single-GPU pods with the bound dark). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2f4933c. Configure here.
saqlainsyed007
approved these changes
Sep 9, 2026
saqlainsyed007
left a comment
Contributor
There was a problem hiding this comment.
Approve. Verified the documented arithmetic against the runtime, not just the prose:
- "GPU pod carries this per-rank base × its nvidia.com/gpu limit, capped so the node's GPU pods still pack" matches gpu_pod_envelope exactly: memory/cpu = per_rank × ranks, slots = node_gpus // ranks, result = max(floor, min(scaled, cap)) with cap = machine_ceiling // slots.
- "unconditional for limits above one; single-GPU pod bounded only behind the GPU_POD_ENVELOPE opt-in, BestEffort otherwise" matches the one-rank-vs-N-rank gating.
- "CPU quota then divided the other way (quota/world_size), so N×one-rank-cpu split N ways is one rank's cpu each" is consistent, and the header caveat that the reported value is the per-rank base a CPU run carries as-is (not the stamped multi-GPU total) is correct — DefaultTraining returns the contract floor.
The old "quietly wrong above world_size 1 / per-rank vs per-job open question" note is correctly retired now that the runtime does the multiplication. CI green, no open threads, public-repo hygiene clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comment-only change in
internal/resources/resources.go. The note onDefaultTrainingsaid the contract floor was "quietly wrong above world_size 1" and that per-job-vs-per-rank sizing was an open question. That no longer describes what the runtime does with the number: since client-runtime#500 a GPU pod's rank count is itsnvidia.com/gpulimit, andjobs_manager.pystamps this per-rank base times that count, capped so the node's GPU pods still pack, and refuses admission on a single-node edge when the scaled envelope can never be placed (client-runtime#483). A CPU pod runs the CPU-only torch wheel, so its world_size is 1 by construction and the question never binds there.The package caveat that "a GPU pod is BestEffort whatever this writes" is narrowed to the case where that is still true: single-GPU pods while the runtime's
GPU_POD_ENVELOPEbound is dark.Related
Follow-up to cli#585, which recorded the one-rank constraint before the runtime's multiplication existed.
Type of change
Test plan
gofmt -l internal/resources/clean,go build ./...,go vet ./internal/resources/go test ./internal/resources/— ok (the threeTestDefaultTraining_*drift/fit tests still pass; no code changed)bash scripts/check-style.sh— style + terminology cleanChecklist
go build ./...,go vet, and the Lint job's checks pass locallybash scripts/check-style.shpassesNote
Low Risk
Comment-only documentation; no executable code, config, or API surface changed.
Overview
Documentation-only update in
internal/resources/resources.goso comments match how client-runtime actually uses the contract floor—no code or CLI behavior changes.The package header and
DefaultTrainingnotes no longer say multi-rank replication makes the reported ceiling “quietly wrong” or leave per-job vs per-rank sizing as an open question. They now state that this value is the per-rank base: CPU pods use it as-is (world_size 1); GPU pods get per-rank base ×nvidia.com/gpulimit oncejobs_manager.pybounds the envelope (with packing caps and single-node admission rules per client-runtime#483/#500). Multi-GPU limits above one are bounded unconditionally; single-GPU pods stay BestEffort until the runtime’sGPU_POD_ENVELOPEopt-in applies.It also clarifies that in-pod CPU quota is still split across ranks (
quota/world_size), sotracebloc resources showreports the per-rank base—not the stamped total for a multi-GPU pod.Reviewed by Cursor Bugbot for commit 2f4933c. Bugbot is set up for automated code reviews on this repo. Configure here.