Add opt-in AutoEP Python GC policy

[yh0903]

Summary

Add an experimental, opt-in AutoEP policy for disabling automatic Python cyclic garbage collection during training:

{
  "expert_parallel": {
    "enabled": true,
    "python_gc_policy": "disable_during_training"
  }
}

The default is "default" and leaves process GC behavior unchanged.

Motivation

Fixed-routing Qwen3-30B-A3B EP16 profiling found repeated single-rank generation-2 GC pauses of 234-327 ms. The other ranks then waited for the paused rank in the next DeepEP dispatch, making communication look slow even though dispatch was only the synchronization point.

Across 20 measured steps, 27 generation-2 collections accumulated 7.60 s; all long collections occurred during forward. Generation-0/1 collections had maximum durations of only 1.04/2.31 ms.

Implementation

• Collect once after the concrete DeepSpeed engine has initialized, then disable automatic cyclic GC.
• Reference-count the process-wide policy across multiple engines.
• Restore the original GC state when the last managed engine is explicitly destroyed.
• Restore GC in forked data-loader workers and use generation tokens so inherited engines cannot release child-process leases.
• Guarantee lease release even if another engine teardown operation raises.
• Expose engine.collect_python_gc() for explicit collection at application-selected safe boundaries.

AutoEP-only performance

One 16xH100 allocation, fixed routing, dual warmup, default -> managed -> managed -> default, 20 measured L48 steps per arm:

Policy	Full measured-window mean	Median	Median p95	Steps >800 ms	Steps >1 s
Default	688.57 ms	535.13 ms	1072.52 ms	11/40	4/40
Managed GC	526.42 ms	522.89 ms	551.12 ms	0/40	0/40

• Paired measured-window savings: 166.37 ms, 157.92 ms
• Paired mean: 162.14 ms
• 95% CI: [108.45, 215.84] ms
• Both paired deltas favored the managed policy.
• Peak allocated/reserved memory was unchanged: 45,106,782,720 / 50,899,976,192 bytes.
• Observed routes were identical across all arms.
• Maximum paired measured-window loss differences were 0.00266 and 0.01439.

Also validated in a precision-aligned AutoEP-vs-Megatron E2E benchmark

This policy was re-tested outside the isolated AutoEP-only ABBA above, in a
separate cross-framework paired benchmark (AutoEP vs Megatron, same model/
routing, AutoEP running #8423's DeepEP cleanup plus fp32 gradient
accumulation and a fp32 gradient all-reduce to match Megatron's
grad_reduce_in_fp32=True). That combination is unrelated to this PR's own
scope, but it depends on this policy to produce a usable result:

• Without this policy (precision alignment alone): both AutoEP arms had
  25% of steps over 800 ms, the same generation-2 GC pause this PR targets.
  Median step time favored AutoEP by ~19-20%, but the tail pulled mean step
  time to ~18.8% slower than Megatron -- an artifact of this exact pause
  pattern, not a real regression.
• With this policy enabled, on the same precision-aligned setup: AutoEP
  tail dropped to 1/40 steps over 800 ms (Megatron: 0/40), and median/mean
  step time agreed in direction: AutoEP faster by 21.9% (median) / 10.3%
  (mean) over 2 paired blocks, 95% CI [121.0, 184.6] ms, not crossing zero.

This is external confirmation that the fix generalizes past the isolated
AutoEP-only benchmark above: without it, a mean-step-time comparison against
another framework is dominated by this pause, not by real per-step cost.

Testing Done

• Targeted config parsing and validation tests.
• GC manager lifecycle tests for multiple engines, pre-disabled GC, explicit collection, fork restoration, stale pre-fork leases, and reentrant finalizers.
• Changed files pass repository pre-commit hooks.
• AutoEP-only distributed ABBA benchmark passed routing, loss, GC-state, memory, and tail gates.

This is independent of the DeepEP local-preparation cleanup in #8423.