Skip to content

docs(resources): DefaultTraining is the per-rank base; GPU pods carry it times their rank count - #650

Merged
LukasWodka merged 1 commit into
developfrom
docs/default-training-per-rank-base
Sep 9, 2026
Merged

docs(resources): DefaultTraining is the per-rank base; GPU pods carry it times their rank count#650
LukasWodka merged 1 commit into
developfrom
docs/default-training-per-rank-base

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Comment-only change in internal/resources/resources.go. The note on DefaultTraining said the contract floor was "quietly wrong above world_size 1" and that per-job-vs-per-rank sizing was an open question. That no longer describes what the runtime does with the number: since client-runtime#500 a GPU pod's rank count is its nvidia.com/gpu limit, and jobs_manager.py stamps this per-rank base times that count, capped so the node's GPU pods still pack, and refuses admission on a single-node edge when the scaled envelope can never be placed (client-runtime#483). A CPU pod runs the CPU-only torch wheel, so its world_size is 1 by construction and the question never binds there.

The package caveat that "a GPU pod is BestEffort whatever this writes" is narrowed to the case where that is still true: single-GPU pods while the runtime's GPU_POD_ENVELOPE bound is dark.

Related

Follow-up to cli#585, which recorded the one-rank constraint before the runtime's multiplication existed.

Type of change

  • Docs

Test plan

  • gofmt -l internal/resources/ clean, go build ./..., go vet ./internal/resources/
  • go test ./internal/resources/ — ok (the three TestDefaultTraining_* drift/fit tests still pass; no code changed)
  • bash scripts/check-style.sh — style + terminology clean

Checklist

  • Tests added / updated and passing locally (no code change; existing tests pass)
  • Docs updated if behavior or config changed
  • No secrets / credentials in the diff
  • go build ./..., go vet, and the Lint job's checks pass locally
  • Terminal output follows STYLE.md — no output changed; bash scripts/check-style.sh passes

Note

Low Risk
Comment-only documentation; no executable code, config, or API surface changed.

Overview
Documentation-only update in internal/resources/resources.go so comments match how client-runtime actually uses the contract floor—no code or CLI behavior changes.

The package header and DefaultTraining notes no longer say multi-rank replication makes the reported ceiling “quietly wrong” or leave per-job vs per-rank sizing as an open question. They now state that this value is the per-rank base: CPU pods use it as-is (world_size 1); GPU pods get per-rank base × nvidia.com/gpu limit once jobs_manager.py bounds the envelope (with packing caps and single-node admission rules per client-runtime#483/#500). Multi-GPU limits above one are bounded unconditionally; single-GPU pods stay BestEffort until the runtime’s GPU_POD_ENVELOPE opt-in applies.

It also clarifies that in-pod CPU quota is still split across ranks (quota/world_size), so tracebloc resources show reports the per-rank base—not the stamped total for a multi-GPU pod.

Reviewed by Cursor Bugbot for commit 2f4933c. Bugbot is set up for automated code reviews on this repo. Configure here.

… it times their rank count

The comment on DefaultTraining said the value was "quietly wrong above
world_size 1" and that per-job-vs-per-rank was an open question. Since
client-runtime#500 that is no longer what the runtime does with the
number: a GPU pod's rank count is its nvidia.com/gpu limit, and the
runtime stamps this per-rank base times that count, capped so the
node's GPU pods still pack, refusing admission on a single-node edge
when the scaled envelope can never be placed (client-runtime#483).
A CPU pod runs a CPU-only torch, so its world_size is 1 by construction.

Records the shipped behaviour where the number lives; changes no code.
The package caveat about GPU pods being BestEffort is narrowed to the
case where that is still true (single-GPU pods with the bound dark).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LukasWodka LukasWodka self-assigned this Sep 9, 2026
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 2f4933c. Configure here.

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Verified the documented arithmetic against the runtime, not just the prose:

  • "GPU pod carries this per-rank base × its nvidia.com/gpu limit, capped so the node's GPU pods still pack" matches gpu_pod_envelope exactly: memory/cpu = per_rank × ranks, slots = node_gpus // ranks, result = max(floor, min(scaled, cap)) with cap = machine_ceiling // slots.
  • "unconditional for limits above one; single-GPU pod bounded only behind the GPU_POD_ENVELOPE opt-in, BestEffort otherwise" matches the one-rank-vs-N-rank gating.
  • "CPU quota then divided the other way (quota/world_size), so N×one-rank-cpu split N ways is one rank's cpu each" is consistent, and the header caveat that the reported value is the per-rank base a CPU run carries as-is (not the stamped multi-GPU total) is correct — DefaultTraining returns the contract floor.

The old "quietly wrong above world_size 1 / per-rank vs per-job open question" note is correctly retired now that the runtime does the multiplication. CI green, no open threads, public-repo hygiene clean.

@LukasWodka
LukasWodka merged commit e812c5a into develop Sep 9, 2026
37 checks passed
@LukasWodka
LukasWodka deleted the docs/default-training-per-rank-base branch September 9, 2026 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants