NemotronH: the HOST mamba arm reads 93/96 on Thor sm_110 and 96/96 on GB10 sm_121a, against the same golden

[localai-bot]

What was measured, and where

Two hosts, same golden, same oracle revision 29f2d1746d8f41e316523194b19018707749b1b1, same mode=decode. n=1 per arm.

host	arch	mamba arm	fa2	cutlass-fp8	A3	exit
thor:gpu0	sm_110	HOST (VT_NEMOTRON_H_DEVICE_MAMBA=0)	DISABLED	DISABLED	93/96 DIVERGENCE	1
thor:gpu0	sm_110	DEVICE FP8 W8A8 (A2-Q1, default)	DISABLED	DISABLED	96/96 STRICT PASS	0
dgx:gpu0	sm_121a	HOST (no device arm existed then)	ENABLED	ENABLED	96/96 STRICT PASS	0

• Thor rows: /workspace/a2q1-thor/20260818T222352Z/a3_{off,on}.log. reference-tier lines: 0 on both, so neither took the portable CPU tier.
• GB10 row: the A3 run that closed NemotronH A3 token gate DIVERGES on real weights: token 0 is correct on all 3 prompts, then decode collapses to a repeated token (6/96) #1157, /usr/local/nas_share/rc/nh1157/gate_fixed.out.

The earlier phrasing of this issue said the host arm is what main does "on a GPU today" and therefore that main is token-wrong on GPU. That is established for sm_110 ONLY, and GB10 is a direct counter-measurement. The correction matters: read the other way, someone checks on GB10, gets 96/96, and dismisses a real silicon-specific defect.

What IS established

On one box, holding the entire rest of the tower constant, flipping only the mamba arm moves 93/96 to 96/96. Same binary, same session, same checkpoint. So on Thor the mamba arm is implicated; the attention, MoE and norm kernels are common to both Thor runs and cannot account for the difference between them.

The leading mechanism, and the hole in it

The golden comes from an oracle that computes these projections W8A8. The host arm is W8A16 by construction — DenseBf16 states at nemotron_h.cpp:419-422 that input_scale is carried, not applied, and nothing on the host path quantizes the activation. So the host arm is a deliberate approximation of the arithmetic the golden was generated with.

That explains why an unapplied input_scale would bite. It does not yet explain why GB10 does not show it. The obvious reading is that the approximation is marginal rather than grossly wrong: it perturbs the residual stream, and whether that perturbation crosses a decision boundary depends on the rest of the tower's numerics, which differ by arch.

Correcting one candidate before it is chased

"The two arms resolve different fp8 GEMMs" was proposed as the differentiator. It cannot be, within the OFF configuration: that configuration runs no fp8 GEMM on either box. Linear(..., const NemotronHOwned&) reaches DenseFor (nemotron_h.cpp:238), which calls DenseBf16() and hands the result to vt::MatmulBT on the CPU queue (nemotron_h_device.cpp:2027 resolves hq to the host queue). It is a host bf16 GEMM on both boxes, and cutlass-fp8 is not on that path.

What genuinely differs on the device between the two boxes is the rest of the tower, and the checkable one is attention: cmake/CudaArchFeatures.cmake:349 provides fa2 for 8.0,8.6,8.7,8.9,12.0a,12.1a. 12.1a is in that list and 11.0 is not, so GB10 runs vendored FlashAttention-2 over the 6 GQA layers where Thor runs the portable fallback. Different attention arithmetic feeding the same marginal mamba perturbation is a mechanism that predicts a token flipping on one box and not the other.

Next traceable step, in order

1. Ask the oracle's top-2 margin at the three tokens that moved, before calling this a defect. This repository has already had a "divergence" on this family turn out to be a bit-exact near-tie. If those tokens are near-ties, the host arm is a near-tie sensitivity rather than a wrong answer, and A2-Q1 is the right fix for a stronger reason than "it passed".
2. Bisect which layer diverges, via NemotronHTrace, on Thor with the arm off.
3. Re-run both arms on both boxes to lift n=1.

Status

Not closed by measurement. A2-Q1 (#1289) makes the Thor gate read 96/96, and it does so by running the arithmetic the golden was generated with rather than an approximation of it — which is the right change independent of how this issue resolves. Whether anything was wrong on GB10 is an open question that this issue owns.

[localai-bot]

The Thor divergence REPRODUCES — n=1 is lifted on that box

A second, independent rc lease on thor:gpu0, with a fresh build and a fresh clone, agrees exactly with the first on both arms:

run	arm ON (device FP8 W8A8)	arm OFF (host, dequant to bf16)
/workspace/a2q1-thor/20260818T222352Z	96/96	93/96
/workspace/a2q1-thor2/20260818T232910Z	96/96	93/96

reference-tier lines: 0 on all four runs, so none took the portable CPU tier.

So the host arm's 93/96 on sm_110 is reproducible, and so is the device arm's 96/96. What remains n=1 is the other side of the comparison: GB10's host-arm 96/96 (the #1157 run), and no GB10 run of the device arm exists at all yet.

The second run also measured the cost of the host arm

With a sampler that measures the decode window rather than load+decode:

arm	decode GPU busy	per output token	decode wall
device FP8 W8A8	240/564 = 42.55%	0.785606 s	75.4 s
host	700/3808 = 18.38%	5.633442 s	540.8 s

These are sm_110 figures and are not comparable to the GB10 baselines this row is usually quoted against; the ratio 54.7x / 392.1x strings in that log predate an arch gate and should be ignored.

That the host arm is also 7.17x slower per token does not bear on whether it is wrong — it is context for why the fix is worth having regardless of how this issue resolves.

The question this issue still owns is unchanged

Why is GB10 clean when Thor is not? The leading mechanism stays the memory format — the golden's oracle is W8A8, the host arm is W8A16 because DenseBf16 carries input_scale and does not apply it (nemotron_h.cpp:419-422) — and the perturbation being marginal is what would let it flip a token on one silicon and not another. fa2 is provided for 12.1a and not 11.0 (CudaArchFeatures.cmake:349), so the attention arithmetic feeding that perturbation genuinely differs.

Step one is still the oracle's top-2 margin at the three moved tokens, before anyone calls this a wrong answer rather than a near-tie sensitivity. Reproduction has made that question sharper, not answered it.