[BugFix] Correct DreamerV3 symexp two-hot semantics

[vmoens]

Stack from ghstack (oldest at bottom):

• [Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark #4075
• [Feature] Complete DreamerV3 continuation training semantics #4074
• [BugFix] Match DreamerV3 categorical and KL semantics #4073
• [Feature] Add the DreamerV3 block RSSM core #4072
• [Feature] Add DreamerV3 slow-critic regularization #4068
• [Feature] Normalize DreamerV3 REINFORCE return scale #4067
• [BugFix] Decode DreamerV3 rewards and values end to end #4066
• -> [BugFix] Correct DreamerV3 symexp two-hot semantics #4065

Summary:

• add the exported SymExpTwoHot representation and shared encode, decode, and cross-entropy helpers
• construct exact symmetric raw supports from symexp(linspace(-20, 20, bins))
• train reward and value logits against raw-space interpolated targets
• document the public API and cover official numerical fixtures, saturation, symmetry, state, and torch.compile

Rationale:
DreamerV3 categorical scalar predictions operate on raw symexp support values. The previous implementation interpolated symlog targets over a linear support, which changed both the training targets and decoded magnitudes. A single representation now defines and tests the complete contract.

Test plan:

• uv run pytest -q test/objectives/test_dreamer_v3.py
• ufmt check torchrl/modules/models/model_based_v3.py torchrl/objectives/dreamer_v3.py test/objectives/test_dreamer_v3.py
• git diff --check

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4065

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ceff4db with merge base d7659c7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Update after the stack-level ghstack rebase: The full eight-PR Dreamer stack was checked out from #4075, rebased onto current origin/main, linted with pre-commit run --all-files, and submitted successfully as vmoens. This PR is now at 487a458d101e. The core Dreamer suites pass (267 tests) and the two doctest modules pass (15 tests); the DMC-only smoke still requires dm_control, which is not installed locally. Any earlier conflict/head-status note below is superseded; the substantive code-review observations remain applicable.

Reviewed the freshly rebased head 071e813a862e, including the complete layer diff, the current DreamerV3 reference implementation, and the latest PR discussion. The raw symexp(linspace) support, raw-space interpolation, and symmetric paired decode are internally consistent and match the current reference structure.

One actionable test issue remains at test/objectives/test_dreamer_v3.py:366: the new compile-parity assertion is flaky. A focused run passed, but the subsequent full file run failed with an eager/compiled difference of 8.0 on an approximately 1.9e6 output (relative error 4.13e-6), exceeding the default assert_close tolerance. Because logits is unconstrained random data and the five-bin support reaches approximately 4.85e8, this can depend on the sample and reduction lowering. Please use a fixed moderate logits fixture and/or an explicit float32-appropriate tolerance so this test is stable while still checking compile parity.

Checks: the four new SymExpTwoHot-focused tests passed in isolation; the full test/objectives/test_dreamer_v3.py run was 44 passed / 1 failed at the assertion above; git diff --check is clean. CI has restarted for the rebased head.