Raise EP routed-row pricing to each checkpoint's observed routing

[bradhilton]

At EP>1, TrainerRank prices the rows each rank's experts receive at a fixed allowance over balanced routing (1.4x at EP2). A checkpoint whose routing is more skewed than that is under-priced every time, and nothing notices. This PR measures each checkpoint's routed share on the micro-batch wave path. When that share plus a 0.10 margin exceeds the allowance, the checkpoint's routed rows are raised to match. Observations never lower a charge. Reloading a checkpoint starts over from the allowance.

Scope: HybridEP with the EP group equal to the CP group and TP/ETP 1, the only path where routed rows are priced from the CP group.

What changes

• Observation. HybridEP keeps each MoE layer's received rows per local expert after combine. At the handoff of a forward_micro_batches wave with one group (with or without gradients), each rank reads the worst layer's received pairs, divided by top-k times the balanced rows that pricing starts from.
  • dp_rank_forward, split and multi-group waves record nothing. They are still priced from whatever the checkpoint has recorded.
• Exchange. At EP>1, the existing handoff MAX reduce carries the share and the checkpoint's route epoch as epoch, -epoch. A share is recorded only if every rank observed the same epoch. An empty, unsupported, failed, cancelled or different-checkpoint rank records nothing, so every rank keeps an identical table.
• Identity. A checkpoint gets a new route epoch after its load or snapshot commits on every rank. Epochs are never reused.
  • Loads, snapshots and discards now check that every rank targets the same name and epoch.
  • snapshot_checkpoint starts from its source's share, since it has the same weights. A snapshot loaded from disk starts cold.
  • Reloading a name forgets the old content's share, and discarding a snapshot forgets its share.
  • A failed load rolls back to the previous epoch.
  • Optimizer steps keep the epoch, so a training slot's share is a high-water mark.
• Pricing. _plan_group_routed_rows scales a group's balanced rows by (share + 0.10) / allowance when that exceeds 1. The scaled rows feed every consumer: plan cost, admission, splits and the checkpoint floor.
• Reporting. last_forward_telemetry()["routed_share"] gives each wave's observed share. Planner-miss reports include it, along with the share behind each group's routed rows.

Before/after

• Estimates are unchanged while observed shares stay below 1.3 at EP2. Every measured EP2 case so far is in that range: pretrained up to 1.22 and a trained policy up to 1.24 in the routing probe.
• A checkpoint observed at 1.35 (one production run was inferred at that level) would be priced at 1.45 on the waves after that observation.

Not in this PR: pricing below the allowance (the "warm discount" of Part B2). It waits on measurements that this PR makes possible.

Testing:

• 24 new CPU unit tests. They cover simulated peers, a two-process gloo exchange, real checkpoint load/snapshot/discard paths, and admission with raised rows. The trainer-rank unit suite passes.
• A local 2xH200 run: Qwen3.6-35B-A3B, 40 real layers, CP2/EP2, 12 real-data calls per arm.
  • Every call recorded a share.
  • With learning rate 0, shares stayed at 1.15-1.16 and estimates were unchanged.
  • Training at lr 1e-4 reached a share of 1.364, and B1 raised the following calls' charges.
  • No call was under-estimated.

Stacked on #978 (itself on #963); neither base is merged, so review after them.

🤖 Generated with Claude Code