[Test] Add off-policy data-parallel learning suite

[vmoens]

What

Adds sota-implementations/offpolicy-dp/, a Hydra-configured distributed suite for DQN, SAC, DDPG, and TD3 using the merged data-parallel Trainer and flexible transport APIs.

The suite includes:

• Ray collectors writing directly to Ray-owned replay through Gloo tensor transport
• NCCL learner groups and direct policy-weight publication
• inner-collector rollout flattening from [B, T] to [B * T] before replay insertion
• transition-count, replay-shape, policy-version, optimizer-version, and finite-metric assertions
• W&B metric namespaces, asynchronous evaluation, and rendered Humanoid video
• persistent stochastic collection for every algorithm
• replay action dispersion and saturation metrics
• smoke profiles plus 100K-environment learning-scale profiles

DDPG now accepts an optional exploration_module in DDPGTrainer and DDPGTrainerConfig. Local and Ray execution publish weights with the same actor + exploration structure used by the collector, while the loss and evaluator continue to use the deterministic actor. The SOTA DDPG configuration applies fixed Gaussian action noise with standard deviation 0.2.

Experiment design

Random-only collection is disabled in these profiles. SAC is stochastic, TD3 and DDPG use persistent Gaussian action noise, and DQN begins with epsilon equal to one. Learner startup is separated from action selection: scale runs collect ten million stochastic-policy transitions before the first optimization step instead of requesting a distributed random-action warmup against the shared replay counter.

The scale defaults are a starting point for the next allocation:

• 100,000 environments across four GPU collectors
• four NCCL learner ranks
• 200 million collected transitions, or 2,000 decisions per environment
• 10 million stochastic-policy replay-prefill transitions
• 20 million-transition replay capacity
• 16,384 global batch size, or 4,096 samples per learner rank
• 25 optimizer steps per approximately 100,000 collected transitions, preserving a 4.096 sample update-to-data ratio
• 64 deterministic evaluation episodes of up to 1,000 steps every ten million transitions

A full 1,000-step prefill trajectory from every environment would require 100 million transitions before learning begins. The shorter stochastic prefill targets early terminations while exploration continues throughout the run. Humanoid retains frame_skip=1 for batched solver stability and therefore is not directly comparable to the standard five-substep benchmark.

Why

The distributed components need an end-to-end workload that exercises collection, replay transport, multi-rank learning, policy synchronization, numerical health, and shutdown together. DDPG collection must remain stochastic after replay prefill; publishing only its deterministic actor can optimize losses without improving behavior. Separating replay fill from random-only action selection also avoids per-collector warmup thresholds being compared with a shared replay write count.

Validation

• python3 -m py_compile torchrl/trainers/algorithms/ddpg.py torchrl/trainers/algorithms/configs/trainers.py sota-implementations/offpolicy-dp/train.py sota-implementations/offpolicy-dp/run_suite.py test/test_trainer.py test/test_configs.py
• pre-commit run --files <changed files>
• python -m pytest test/test_trainer.py test/test_configs.py -k 'TestDDPGTrainer or test_ddpg_trainer_config' -q — 3 passed
• Hydra composition checks for every smoke and scale profile
• Exploration-mode check: Gaussian collection action standard deviation 0.201; deterministic action standard deviation 0.000

The earlier runs below validate transport, replay accounting, flat sampled batches, policy publication, numerical finiteness, and shutdown. They predate the learning-scale redesign and should not be interpreted as successful learning curves. The redesigned 200-million-transition profiles have not been run because the original allocation has ended.

• DQN, CartPole, 100K frames
• SAC, Hopper, 100K environments, 20M frames
• DDPG, Hopper, 100K environments, 20M frames
• TD3, Hopper, 100K environments, 20M frames
• DDPG, Humanoid, 100K environments, 20M frames

Related stacks: #4005 and #3958.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4026

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit f0888fd with merge base f2a2337 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.10, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

I reviewed the current PR head f0888fd, including the latest DDPG exploration-policy publication changes. The two TestDDPGTrainer cases and test_ddpg_trainer_config pass locally, and both new SOTA scripts compile. I could not run Hydra composition or the Ray workload because Ray is unavailable in this environment. The required rebase is currently blocked: replaying f0888fd onto the freshly fetched origin/main produces a content conflict in torchrl/trainers/algorithms/configs/trainers.py, so no force-push was attempted and the PR remains conflicting. The DDPG additions themselves follow the existing TD3 execution-weight structure, but this cannot be considered ready until that config conflict is resolved and the distributed profiles are revalidated on the rebased tree.