Read rope_theta from rope_parameters across Inference V2

[alanhuangyoo]

Follow-up to #8341, which covers the same drift in the v1 kernel-injection policy. Separate engine, separate change; #8341 noted this exposure but did not touch it.

What breaks

transformers 5.0 folded the rotary settings into config.rope_parameters and dropped the rope_theta attribute. Eight V2 models still read the attribute.

Seven of them read it directly and raise; exaone4 reads it through a getattr default and quietly uses that default instead:

V2 model	on master (transformers 5.8.0)	with this PR
llama_v2	AttributeError: 'LlamaConfig' object has no attribute 'rope_theta'	10000.0
mistral	AttributeError: 'MistralConfig' …	10000.0
mixtral	AttributeError: 'MixtralConfig' …	1000000.0
phi	AttributeError: 'PhiConfig' …	10000.0
phi3	AttributeError: 'Phi3Config' …	10000.0
qwen_v2	AttributeError: 'Qwen2Config' …	10000.0
qwen_v2_moe	AttributeError: 'Qwen2MoeConfig' …	10000.0
exaone4	silently 1000000.0	10000.0

exaone4 is the worse of the two. Exaone4Config carries rope_parameters['rope_theta'] == 10000.0, so the getattr default puts a 100x rotary base into RotateHalfConfig with nothing raised — wrong frequencies rather than a startup failure.

requirements-inf.txt asks for transformers>=4.32.1 with no upper bound, so 5.x is in range.

The fix

exaone4_5 already reads both spellings — added with the model in #8121:

theta = rope_parameters.get("rope_theta", getattr(config, "rope_theta", None))

Hoist the same lookup onto DSTransformerModelBase as a rope_theta property. DSMoETransformerModelBase extends it, so all eight models are covered by one place and each call site becomes theta_base=self.rope_theta.

The attribute is tried first, so pre-5.0 installs take exactly the path they take today. When neither spelling carries a base it raises instead of guessing one — that is the only behaviour change beyond the fix, and it replaces exaone4's silent 1e6.

exaone4_5 is left alone; its own helper also handles the nested per-layer sliding_attention dict, which is specific to that model.

Test

tests/unit/inference/v2/model_implementations/test_rope_theta.py — the three layouts, the precedence between them, the raise, plus a parametrization over the eight real configs from the installed transformers:

13 passed

tests/unit/inference/v2/model_implementations/   19 passed
yapf --diff / flake8                             clean

I exercised the property and the configs, not a full V2 engine run against downloaded checkpoints.

[ebarkhordar]

The docstring says this is the same lookup exaone4_5 already does, and the two differ once RoPE is set per layer type. _get_rope_parameters (exaone4_5/model.py:21-31) unwraps a nested sliding_attention dict before reading the theta; this property only reads the top level.

Measured at 7535905 in a clean container on transformers 5.16.1, with an Exaone4Config.from_dict that sets RoPE per layer type:

rope_parameters: {'sliding_attention': {'rope_theta': 1000000.0, 'rope_type': 'default'},
                  'full_attention': {'rope_theta': 16000000.0, 'rope_type': 'default'},
                  'rope_theta': 10000.0, 'rope_type': 'default'}
this property    -> 10000.0
exaone4_5 helper -> 1000000.0

The 10000.0 is the class default that standardize_rope_params sets at the top level, not a value the checkpoint asked for, so this shape resolves to a wrong base quietly rather than raising.

I could not get a released EXAONE-4 config into that shape: 4.0-32B and 4.0-1.2B both carry a flat rope_parameters with rope_theta 1000000, and your property reads them correctly. So this is about the per-layer shape only, and unwrapping the nested dict the way exaone4_5 does would cover both.

One other thing not in the PR body: exaone4 used to fall back to 1000000.0 when nothing carried the value, and now raises instead. Deliberate seems right, it is just a behaviour change worth naming.

[alanhuangyoo]

You are right, and I reproduced it — Exaone4Config.from_dict on transformers 5.16.1, exactly the shape you gave:

rope_parameters: {'sliding_attention': {'rope_theta': 1000000.0, 'rope_type': 'default'},
                  'full_attention':    {'rope_theta': 16000000.0, 'rope_type': 'default'},
                  'rope_theta': 10000.0, 'rope_type': 'default'}
old property -> 10000.0

Fixed in ef14c37, though not quite the way you suggested, and I want to be explicit about why.

Unwrapping sliding_attention the way exaone4_5 does would return 1000000.0 here — right for the sliding layers, wrong by 16x for the full ones. exaone4_5 can make that pick because it is one model choosing for itself. This property cannot: every caller of it does RotateHalfConfig(theta_base=self.rope_theta), one base for the whole model, and there is no per-layer plumbing in V2 to pick differently. So in a config whose layer types genuinely disagree, both answers are wrong for half the layers.

What it does now:

• collects rope_theta from every nested per-layer entry, keyed by whatever the layer type is called rather than hardcoding sliding_attention
• one distinct value → return it
• more than one → raise, naming the values, because V2 cannot represent that config
• flat rope_parameters unchanged

Against real Exaone4Config objects:

config shape	old	new
per-layer, 1e6 / 1.6e7	10000.0	ValueError: ... sets a different rope_theta per layer type ({'sliding_attention': 1000000.0, 'full_attention': 16000000.0})
per-layer, 1e6 / 1e6	10000.0	1000000.0
flat (released 4.0-32B, 4.0-1.2B)	correct	unchanged

Note the middle row: the top level is wrong even when the layer types agree, so this is not only about the disagreeing case.

Two tests added for the nested shape, 15 passing in the file. The docstring no longer claims to be the same lookup as exaone4_5 — it is not, and now says what it does instead.

On the behaviour change: agreed it was worth naming, and it was under-described. exaone4 used to do getattr(self._config, "rope_theta", 1000000.0), so a config carrying nothing got 1e6 silently; it now raises. The PR body called that out for the flat case only, and I have extended it to cover the per-layer case as well. Thanks for reading it this closely — the nested shape is not something I would have found on my own.

[ebarkhordar]

That is a better call than the one I suggested. Unwrapping sliding_attention would have handed the full-attention layers 1e6 when their config says 1.6e7, and a single theta_base cannot carry both, so raising and naming the values is the honest answer where V2 has no per-layer plumbing.

The middle row is the part I had not seen: the top level is wrong even when the layer types agree, so the flat-vs-nested split matters more than the disagreement does.

[alanhuangyoo]

@tohtana — this is the inference/v2 companion to #8341, which you reviewed. Flagging it since it has had no reviewer on it for a week and the two are related.

#8341 is the injection-policy path for Llama; this one is the same rope_theta / rope_parameters move across the eight inference/v2 model implementations that read it. It is independent of #8341 and does not depend on how that one resolves — but if the conclusion there is that scaled rope types have to be refused rather than served, the same question applies here and I would rather hear it once than fix it twice.

[alanhuangyoo]

New information rather than a ping: #8341 merged this morning, and it is the same bug one layer up.

rope_theta moved into config.rope_parameters in transformers 5.x, and getattr(config, "rope_theta", <default>) now silently returns the default for every checkpoint — the value in config.json never reaches the kernel. #8341 fixed that in the AutoTP injection policy (module_inject/containers/llama.py) and @tohtana merged it, so the reasoning has been accepted; this PR is the same fix at a different layer, with no file overlap:

PR	layer
#8341 (merged)	module_inject/containers/llama.py — injection policy
#8345	inference/v2/model_implementations/* — 9 model implementations
#8373	ops/transformer/inference/op_binding/ — the rotary op binding

Merged current master in just now and re-ran on 1×H20; nothing has gone stale.

tests/unit/inference/v2/model_implementations/test_rope_theta.py: 15 passed.