feat: Adds GenRM Response API Model with support for custom roles used in pairwise response comparison.

[ffrujeri]

What does this PR do?

Adds GenRM support via a dedicated Response API Model package (responses_api_models/genrm_model/). The package provides a single local variant: a GenRM model that uses a locally managed vLLM server (download model + start vLLM, e.g. via Ray).

The key design goal is to keep all GenRM-specific logic inside the model server, so that the resources server (and the base schema) can use standard OpenAI roles throughout. Role remapping to the GenRM chat-template roles (response_1, response_2, principle) happens as a final preprocessing step, just before the request is forwarded to vLLM.

Related to PR #523. Part of #516.

---

Architecture

Package layout (responses_api_models/genrm_model/)

genrm_model/
├── __init__.py
├── app.py
├── pyproject.toml
├── setup.py
├── README.md
├── configs/
│   └── genrm_model.yaml
└── tests/
    └── test_app.py      # config tests + preprocessing unit tests

Components

• GenRMModelConfig — extends LocalVLLMModelConfig with supports_principle_role.
• GenRMModelMixin — overrides two hooks on VLLMModel:
  • get_converter() — returns a plain VLLMConverter (no custom converter needed).
  • _preprocess_chat_completion_create_params() — reads the comparison payload from
    metadata and appends the GenRM chat-template roles immediately before the vLLM call:
    • metadata["response_1"] → appended as a "response_1" message
    • metadata["response_2"] → appended as a "response_2" message
    • metadata["principle"] → appended as a "principle" message (when supports_principle_role=True)
    • metadata is consumed here and not forwarded to vLLM.
• GenRMModel — GenRMModelMixin + LocalVLLMModel.

Changes to vllm_model/app.py

_preprocess_chat_completion_create_params(self, request, body_dict) -> Dict[str, Any] is extracted from chat_completions() as an overrideable hook. The base implementation covers replace_developer_role_with_system, model / chat_template_kwargs injection, token-ID augmentation, reasoning-parser handling, and extra_body merging. Subclasses override it to apply model-specific transformations before the vLLM call.

Schema (nemo_gym/openai_utils.py)

NeMoGymEasyInputMessage.role and NeMoGymMessage.role use standard OpenAI roles only (user, assistant, system, developer). The custom response_1 / response_2 / principle literals are not part of the request schema — they are an internal vLLM chat-template detail handled entirely within GenRMModelMixin.

---

Usage

Config key: genrm_model under responses_api_models, with entrypoint: app.py. See configs/genrm_model.yaml.

from responses_api_models.genrm_model.app import GenRMModel, GenRMModelConfig

How the resources server calls the GenRM model server:

The input field carries only the conversation history (standard OpenAI roles). The comparison
payload is passed via metadata so the request schema stays generic:

responses_create_params.input = conversation_history_messages   # user / assistant turns only
responses_create_params.metadata = {
    "response_1": response_1_text,
    "response_2": response_2_text,
    "principle":  principle_text,   # omit key when use_principle=False
}

GenRMModelMixin._preprocess_chat_completion_create_params reads metadata, appends the
custom-role messages to the conversation, and pops metadata before forwarding to vLLM.

---

Testing

Launching a multi-node config:

ng_run "+config_paths=[responses_api_models/genrm_model/configs/genrm_model.yaml]"

genrm_model:
  responses_api_models:
    genrm_model:
      entrypoint: app.py
      model: <MODEL_PATH>
      uses_reasoning_parser: true
      return_token_id_information: false
      supports_principle_role: true
      debug: true
      hf_home: null

      vllm_serve_env_vars:
        VLLM_RAY_DP_PACK_STRATEGY: strict

      vllm_serve_kwargs:
        tensor_parallel_size: 8
        data_parallel_size: 2
        data_parallel_size_local: 1
        pipeline_parallel_size: 1
        reasoning_parser: deepseek_r1
        gpu_memory_utilization: 0.85
        max_model_len: 60000
        model_loader_extra_config:
          enable_multithread_load: true
          num_threads: 108

We get a runnable server:

All 1 / 1 servers ready! Polling every 60s

####################################################################################################
#
# Server Instances
#
####################################################################################################

[1] genrm_model (responses_api_models/genrm_model)
{
    'config_path': 'genrm_model',
    'entrypoint': 'app.py',
    'host': '127.0.0.1',
    'name': 'genrm_model',
    'port': 11093,
    'server_type': 'responses_api_models',
    'url': 'http://127.0.0.1:11093',
}
####################################################################################################

And a successful response (input is conversation history only; responses and principle go in metadata):

curl -s -X POST http://127.0.0.1:11093/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "input": [
      {"role": "user", "content": "What is the capital of France?", "type": "message"}
    ],
    "metadata": {
      "principle":  "Judge which response is better.",
      "response_1": "The capital of France is Paris.",
      "response_2": "Paris is the capital city of France."
    },
    "temperature": 0.0,
    "max_output_tokens": 512
  }'

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.