Reuse checkpoint snapshot headers and shard indexes per block

[bradhilton]

Checkpoint finalization repeatedly opens and indexes the same snapshot for LoRA tensors, optimizer components and steps. Reuse each reader/header and local shard index for one block while preserving selective tensor reads, checkpoint bytes and failure cleanup. This draft is stacked on #984 at 5fc36d9579c8466ace362c9b7307c9f8db490c21.

A public full-finalizer CPU fixture with 41,440 keys across 40 blocks measured 17.75% less CPU time (median 9.6313s → 7.9218s), with identical output hashes and 201 → 41 snapshot opens including final validation. The measurement includes merging, serialization, consolidation, hashing, manifest validation and cleanup, using safetensors 0.8.0, one CPU thread, warm local files, 16 elements/key and two alternating-order paired samples. It measures metadata scale, not a full model/NFS save.

Readers and their mmaps now live for one block. Closing them adds one error collective per block so a rank-local close failure reaches its peers; an existing primary error is preserved and the close failure is attached as a note. The distributed latency cost remains unmeasured and can reduce the CPU benefit. No tensor dictionary or model/optimizer cache is retained.

Validation: 60 checkpoint tests and five metadata-selection cases pass locally on safetensors 0.8.0, with 0.7 compatibility also checked for the checkpoint group. Portable tests cover full checkpoint equality, four dtypes, selected/missing keys, tensor lifetime, real two-process Gloo sharding and duplicate CP owners, read/close/cancellation fanout, primary-error identity and bounded child/temporary-file cleanup. Ruff, format, lock and affected-test typing checks pass. No manual GPU benchmark was run.

At 33e3e8d5ab5e481ddbc5b811b36c77e95f271a91, quality CI passed: lint/format/type/lock checks, 1,264 Megatron tests and 2,086 unit tests (34 skipped, two deselected). Required GPU validation stopped before tests because cks-wb3 could not provision two H200s; its request was cancelled and cleanup reported no cluster. The PR remains a draft.