[BUG] DeepSpeed kernel injection crashes on a DeepSeek configuration without rope_theta

[Redempt1onzzZZ]

Describe the bug
DeepSpeed kernel injection assumes that the loaded Llama configuration exposes LlamaConfig.rope_theta. DeepSeek-R1-Distill-Llama-8B uses a newer rotary-configuration schema without that attribute.

To Reproduce
Steps to reproduce the behavior:

1. Save the following as a shell command and run it in a shell on a machine with a free CUDA device.
2. Run the command in the target Python environment with the listed dependencies installed.
3. Observe the startup or generation result.

import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Llama-8B/snapshots/6a6f4aa4197940add57724a7707d069478df56b1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16
).to("cuda")
engine = deepspeed.init_inference(
    model,
    config={
        "triangular_masking": False,
        "set_empty_params": True,
        "replace_with_kernel_inject": True,
    },
)
print("DeepSpeed inference engine initialized")

Actual behavior

Startup fails during module injection with AttributeError: LlamaConfig object has no attribute rope_theta. The same normalized configuration passes on the independent Qwen3 control.

Expected behavior

The current configuration schema should be supported, or the incompatibility should be reported with a clear diagnostic before injection.

System info (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• GPU count and types: 1 x NVIDIA H100 80GB HBM3
• DeepSpeed version: 0.19.1
• DeepSpeed-MII: not used
• PyTorch: 2.9.1+cu128
• Transformers: 5.15.0
• CUDA reported by PyTorch: 12.8
• nvcc: 13.3
• Python version: 3.12.12
• Target runtime: /root/anaconda3/envs/deepspeed128

Docker context

No Docker image was used. The reproduction ran directly on Ubuntu 22.04.3 LTS in the target conda environment.

[alanhuangyoo]

Opened #8341 for this.

It is broader than the DeepSeek checkpoint — it reproduces with a default LlamaConfig on stock transformers 5.8.0. transformers 5.0 folded the rotary settings into config.rope_parameters and dropped the attribute, and both branches of the current lookup are dead:

self_attn has .config                 True
self_attn has .rope_theta             False
self_attn.config has rope_theta       False
self_attn.config has rope_parameters  True   -> {"rope_theta": 10000.0, "rope_type": "default"}

LlamaAttention still carries .config, so the first branch is taken and raises exactly the AttributeError you saw.

The PR reads rope_parameters["rope_theta"] when the attribute is gone, keeping the old spellings first so pre-5.0 installs are untouched. Same drift #7443 adapted to; the num_heads accessor it fixed alongside still resolves on 5.8, so only this one moved again.