[BugFix] [TQC] [1/3] Support list-valued critics in SoftUpdate

[gtnv]

Summary

Closes #4076

Initialize target updaters from an unlocked shallow source container, allowing list-backed functional parameters to be structurally selected. Extend the existing SAC list-critic regression through a real SoftUpdate.step().

Validation

Tested on an H100 80 GB host with PyTorch 2.11.0+cu130.

• Upstream reproduction fails in SoftUpdate(...) with RuntimeError: Cannot modify locked TensorDict.
• Patched reproduction moves both delayed critic leaves by exactly 0.5 after adding 1 to each source.
• Targeted updater/SAC tests: 12 passed, 196 deselected in 1.98 s.
• Full SAC suite: 1679 passed, 494 skipped in 27.16 s.
• All pre-commit hooks pass.

cc @vmoens

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 15 Awaiting Approval

As of commit ab1bdf7 with merge base d7659c7 ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build Aarch64 Linux Wheels (gh)
• Build Linux Wheels (gh)
• Build M1 Wheels (gh)
• Build Windows Wheels (gh)
• Continuous Benchmark (PR) (gh)
• Generate documentation (gh)
• Habitat Tests on Linux (gh)
• Libs Tests on Linux (gh)
• Lint (gh)
• LLM Tests on Linux (gh)
• Push Binary Nightly (gh)
• SOTA Tests on Linux (gh)
• Tutorials Tests on Linux (gh)
• Unit-tests on Linux (gh)
• Validate Test Partitioning (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Reviewed rebased head ab1bdf76ebf7577ee29fb4a8cba79f7ee7a82fc5 against current origin/main and the latest bot/check state. I checked that the shallow clone unlocks only the temporary TensorDict container needed by select, while preserving the underlying parameter objects and source/target mapping; the extended regression also exercises a real SoftUpdate.step() for both SAC versions.

Targeted result: pytest -q test/objectives/test_sac.py::TestSAC::test_sac_list_qvalue_networks -> 2 passed. I did not find a functional issue in this diff. Residual risk is the broader suite, which is still awaiting workflow approval on this fork PR.