[BugFix] Match DreamerV3 categorical and KL semantics

[vmoens]

Stack from ghstack (oldest at bottom):

• [Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark #4075
• [Feature] Complete DreamerV3 continuation training semantics #4074
• -> [BugFix] Match DreamerV3 categorical and KL semantics #4073
• [Feature] Add the DreamerV3 block RSSM core #4072
• [Feature] Add DreamerV3 slow-critic regularization #4068
• [Feature] Normalize DreamerV3 REINFORCE return scale #4067
• [BugFix] Decode DreamerV3 rewards and values end to end #4066
• [BugFix] Correct DreamerV3 symexp two-hot semantics #4065

Summary:

• add configurable categorical unimix to the DreamerV3 prior and posterior
• add separate dynamics and representation KL terms with aggregated free nats
• configure the maintained example for 1% unimix and 1.0/0.1 KL weighting

Rationale:
The reference objective mixes categorical probabilities with a uniform
component, sums KL across stochastic variables before applying free nats, and
optimizes stopped-gradient dynamics and representation terms separately. The
previous balanced aggregate used different gradient weights and free-bit
semantics, which prevented numerical and training-curve parity.

Test plan:

• pytest test/objectives/test_dreamer_v3.py -q
• pytest test/modules/test_dreamer_components.py -k DreamerV3Components -q
• pytest --doctest-modules torchrl/objectives/dreamer_v3.py -q
• run the maintained DreamerV3 SOTA smoke with the worktree virtual environment
• verify frozen 1% unimix KL values, stopped-gradient routing, aggregated free
  nats, separate loss keys, and end-to-end optimization

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4073

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit f2a8357 with merge base 2a77e48 ():

NEW FAILURES - The following jobs have failed:

• Lint / lint-done (gh)
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t c2bec4cb3a192db2dfd76f0a71b33c9f422bac6a23b940b6c472f5b8529f79cb /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Unit-tests on Linux / tests-cpu (3.14, bulk) / linux-job (gh) (similar failure)
  test/test_configs.py::TestConfigClassParity::test_wrapped_class_kwargs_have_config_fields[LBFGSConfig]

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-cpu (3.10, bulk) / linux-job (gh) (trunk failure)
  test/test_configs.py::TestConfigClassParity::test_wrapped_class_kwargs_have_config_fields[LBFGSConfig]
• Unit-tests on Linux / tests-cpu (3.13, bulk) / linux-job (gh) (trunk failure)
  test/test_configs.py::TestConfigClassParity::test_wrapped_class_kwargs_have_config_fields[LBFGSConfig]

This comment was automatically generated by Dr. CI and updates every 15 minutes.