[Feature] Add DreamerV3 slow-critic regularization

[vmoens]

Stack from ghstack (oldest at bottom):

• [Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark #4075
• [Feature] Complete DreamerV3 continuation training semantics #4074
• [BugFix] Match DreamerV3 categorical and KL semantics #4073
• [Feature] Add the DreamerV3 block RSSM core #4072
• -> [Feature] Add DreamerV3 slow-critic regularization #4068
• [Feature] Normalize DreamerV3 REINFORCE return scale #4067
• [BugFix] Decode DreamerV3 rewards and values end to end #4066
• [BugFix] Correct DreamerV3 symexp two-hot semantics #4065

Summary:

• add functional online and checkpointed target critic parameters to DreamerV3ValueLoss
• regularize categorical online logits toward decoded target-critic predictions
• keep actor lambda-return bootstrapping attached to the online critic
• update the target with SoftUpdate(tau=0.02) after each maintained-example critic step
• add eager and torch.compile DreamerV3 objective benchmarks

Rationale:
A slowly moving critic supplies stable auxiliary targets without changing the online critic used for imagined lambda returns. Functional target parameters isolate gradients, participate in state-dict checkpoints, and integrate with TorchRL target updaters.

Test plan:

• uv run pytest -q test/objectives/test_dreamer_v3.py
• run the maintained DreamerV3 SOTA smoke with 11-bin CI overrides
• smoke the DreamerV3 value benchmark in CPU eager and torch.compile modes
• verify decoded target supervision, online gradients, absent target gradients, exact tau=0.02 movement, online actor ownership, and target state-dict restoration
• flake8 --config=setup.cfg on the changed Python files
• ufmt check on the changed Python files
• git diff --check

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4068

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit cb37f72 with merge base d7659c7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Update after the stack-level ghstack rebase: The full eight-PR Dreamer stack was checked out from #4075, rebased onto current origin/main, linted with pre-commit run --all-files, and submitted successfully as vmoens. This PR is now at 78abb869c298. The core Dreamer suites pass (267 tests) and the two doctest modules pass (15 tests); the DMC-only smoke still requires dm_control, which is not installed locally. Any earlier conflict/head-status note below is superseded; the substantive code-review observations remain applicable.

Reviewed published head c606b9824cf9, including the complete layer diff, latest checks/commit, and the full PR discussion (no human review threads yet). The requested rebase onto fetched origin/main stopped on a content conflict in sota-implementations/dreamer_v3/dreamer_v3.py; the rebase remains in progress and no partial force-push was attempted.

The slow-critic path uses functional online parameters and detached target parameters, decodes target logits before constructing the two-hot regularization target, and the SOTA loop steps SoftUpdate after the critic optimizer. Both new focused tests (regularization/update and checkpoint/bootstrap) pass, and git diff --check is clean. I did not find another code defect specific to this layer.

Residual risk is integration with current main and the cumulative stack, which cannot be established until the conflict is resolved. The added benchmark exercises eager and compiled forwards, but not backward/update cost. The existing bot lint failure is an infrastructure/container failure rather than a code diagnostic. Please rerun the focused suite, benchmark, and SOTA smoke after producing the rebased head.