Skip to content

Enforce cumulative execution allowances for planner reports - #946

Merged
bradhilton merged 12 commits into
mainfrom
joshua/planner-retention-budget-20260923
Sep 24, 2026
Merged

bradhilton merged 12 commits into
mainfrom
joshua/planner-retention-budget-20260923

Conversation

@bradhilton

@bradhilton bradhilton commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on merged #945. Planner reports can now use an execution-assigned cumulative allowance. Charges persist before payload writes and never replenish after deletion, restart or ambiguous failure. Duplicate reports spend no additional quota; ordinary failed attempts increment bounded omission counters.

An explicit capture=False scope prevents report creation before replay/source work or filesystem I/O, including initialization before rank binding. Nested scopes cannot re-enable it. Duplicate delivery revalidates the original bytes and completes file and directory fsync before returning, including after an earlier flush failure.

The Caladan caller owns enrollment, aggregate reservation, reporting-thread context and authenticated custody acknowledgments. Its integrated PR is OpenPipe/caladan#588. Standalone behavior remains the default; this PR does not adopt an ART revision in Caladan.

Validation at a2e4b7a: all 30 affected retention controls and Ruff/format/type checks pass. Four causal retry failures were reproduced on the previous head. Fresh Astra/Fable reviews and same-session correction follow-ups are CLEAR. A broader local leaf run passed 60 tests but hit four explicit Torch-import exclusions; full-package validation is left to normal CI. Exact-head Prek 35900833278 and GPU 35900833317 passed. The original CI log confirms trainer-rank-gpu-35900833317-1 terminated at 12:31 PM Mountain on September 23. Prior results remain separately retained.

Limits: quotas cover retained payloads, with caller-reserved metadata; omission counts are failed attempts, not unique lost reports. Corrupt ledgers/process death can prevent counting. ContextVar propagation covers asyncio.to_thread, not arbitrary new threads. No new live W&B delivery or end-to-end GPU retention claim.

Integration with current main

Commit 7580ff59d includes main through a655c051 (#898 and #888). The #898 conflict combined its dataclasses.asdict import with retention's contextlib.nullcontext. All non-PR main files exactly match main, and the three non-reporter retention files are unchanged. The reporter preserves both grouped replay diagnostics and cumulative retention.

The preceding integration head 8448b791b passed fresh Astra/Fable reviews, 841 Megatron tests, 1,854 unit tests, and two-H200 CI. Its CI cluster terminated at 6:47 PM Mountain on September 23. The newer composition has two clean follow-up reviews, 1,157 Megatron tests and 1,875 unit tests passing, and green CPU/GPU CI: quality, GPU. The final CI GPU cluster terminated at 7:13 PM Mountain on September 23. No deployment or frozen experiment change.

@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 23, 2026 16:03 — with GitHub Actions Active
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 23, 2026 16:58 — with GitHub Actions Active
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 23, 2026 17:48 — with GitHub Actions Active
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 23, 2026 18:12 — with GitHub Actions Active
@bradhilton
bradhilton marked this pull request as ready for review September 24, 2026 00:00
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 24, 2026 00:00 — with GitHub Actions Active
@bradhilton
bradhilton changed the base branch from joshua/planner-report-fidelity-20260923 to main September 24, 2026 00:26
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 24, 2026 00:27 — with GitHub Actions Active
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation September 24, 2026 00:49 — with GitHub Actions Active
@bradhilton
bradhilton merged commit 7bdd26c into main Sep 24, 2026
7 checks passed

This branch was successfully deployed

1 active deployment
trainer-rank-gpu-validation — 7580ff59 Deployed Sep 24, 2026 by bradhilton via Run on 2x H200 #635
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant