[Feature] [LLM] Add TRL interoperability adapters

[coder-jayp]

Description

This PR introduces zero-overhead interoperability adapters for Hugging Face trl.

1. TorchRLBufferDataset (TorchRL -> TRL)

   • Wraps any ReplayBuffer as a torch.utils.data.IterableDataset.
   • Iteration yields flat python dicts perfectly compatible with HF's DataCollatorWithPadding and standard HF Trainer / trl pipelines.
   • Includes support for device placement, key filtering, and nested NestedKey serialization.

2. HFRewardModelWrapper (TRL -> TorchRL)

   • Wraps standard HF AutoModelForSequenceClassification models as a TensorDictModuleBase.
   • Normalizes HF model outputs (logits, rewards, or raw tensors) into a squeezed [B] shape reward tensor written back to the TensorDict in-place.
   • Supports strict gradient control via inference_mode.

3. Documentation & Validation

   • Both modules are optionally-dependency-guarded via lazy __getattr__ in torchrl.modules.llm.__init__.py to ensure trl/transformers remain strictly optional.
   • Added 17 tests in TestTRLInterop achieving complete coverage, including E2E round-trip data flow.
   • Added a comprehensive Sphinx tutorial (tutorials/sphinx-tutorials/trl_interop.py).

Motivation and Context

Currently, many teams use Hugging Face trl (e.g., PPOTrainer, GRPOTrainer) but face bottlenecks with rollout storage and sampling at scale. Conversely, teams using TorchRL's highly efficient collectors and recipes often want to utilize the vast ecosystem of Hugging Face Reward Models without writing tedious custom wrappers.

This PR implements Workstream 2 of the Post-training efforts RFC (#3948), allowing users to seamlessly mix and match TorchRL and TRL components without adopting a full stack at once, significantly lowering adoption friction.

close #4058

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• New feature (non-breaking change which adds core functionality)
• Documentation (update in the documentation)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4070

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit b5247d3 with merge base d7659c7 ():

NEW FAILURES - The following jobs have failed:

• Lint / lint-done (gh)
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 8299fc5c7c1a248be916436748f0f055dec67710caab1efbf90ea8a9c405517d /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: feat(llm): add TRL interoperability adapters

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[Algorithm]	new algo	[Algorithm] Add new RL objective
[BE]	BE	[BE] Improve error messages
[Benchmark] or [Benchmarks]	Benchmarks	[Benchmark] Add collector benchmark
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Example] or [Examples]	Examples	[Example] Add training script
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Trainer] or [Trainers]	Trainers	[Trainer] Add trainer config
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[LLM]	llm/	[LLM] Add reward model integration
[Minor]	small change	[Minor] Fix typo in error message
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Algorithm] or [Algorithms]	new algo	[Algorithm] Add new objective
[Quality]	Quality	[Quality] Fix typos and add codespell
[Versioning]	versioning	[Versioning] Bump release version
[WIP]	WIP	[WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[coder-jayp]

@vmoens could you take a look when you have a moment?

[vmoens]

Reviewed rebased head b5247d34d80bd51451090527634e64dc7718e267, including the two latest trainer-contract/test commits and the author ping.

One blocking issue: torchrl/modules/llm/trl_interop.py::_HFRewardModule.forward returns rewards.float() after creating rewards inside torch.inference_mode(). When the reward is already float32, that remains an inference tensor; using it in the advertised PPO/GRPO training path with a grad-tracked tensor raises RuntimeError: Inference tensors cannot be saved for backward. I reproduced this with (parameter * reward).sum().backward(). Please return a normal tensor (for example, clone outside the inference-mode context, or use no_grad if inference-tensor semantics are not required) and add a regression that consumes the wrapper output in a downstream backward pass.

Targeted result: pytest -q test/llm/test_wrapper.py -k "TRLInterop and not grpo_trainer" -> 23 passed, 2 skipped (optional datasets/transformers unavailable locally). The current lint failure is also real: CI reports µfmt changes needed in test/llm/test_wrapper.py and torchrl/modules/llm/trl_interop.py.