Support official nvidia/Qwen3.8-Flash-Next-NVFP4 model

[lonestriker]

Before you start

• I have checked the supported models and this checkpoint is not listed there. A listed checkpoint that fails is a bug; use a Bug report instead.
• I have read the Roadmap and this model is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.
• This is not a GGUF checkpoint. GGUF support is on the Roadmap; please do not open an issue for it.

Hugging Face link

https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4

Is the model architecture already supported

Yes, but this checkpoint or quantization does not load

Is the quantization already supported

Yes, but this checkpoint's weight format does not load

What happens when you load it

FreeToken detects the checkpoint as `qwen4_exp`, but resolves its routed experts as unquantized. It consequently selects the BF16 expert-bank provider, whose generic reader finds none of the NVFP4 expert tensors and reports every expert layer as missing.

Command:


uv run ft serve \
  --model /home/ubuntu/models/nvidia/Qwen3.8-Flash-Next-NVFP4 \
  --port 8181 \
  --host 0.0.0.0 \
  --kv-reserve-tokens 262144


Relevant log:


[core|rank=0] INFO Auto-selected attention backend: qsa_sparse
[core|rank=0] INFO Auto-selected MoE backend: offload
[core|rank=0] INFO Resolved config: moe_backend='offload', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
Loading weights: 100%|██████████| 11/11 [00:02<00:00, 3.77it/s]
[core|rank=0] WARNING parallel reader unavailable (freetoken.models.qwen4_exp provides no iter_weights_parallel); falling back to serial build
Process freetoken-TP0-scheduler:
[FrontendAPI] ERROR Backend supervisor: ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], 'down': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}

Traceback (most recent call last):
  File "/home/ubuntu/git/FreeToken/python/freetoken/models/loader.py", line 419, in _load
    raise ValueError(f"Missing MoE expert source layers: {missing}")
ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, 2, ..., 47], 'down': [0, 1, 2, ..., 47]}


Environment:

- FreeToken 0.1.2, source commit `af71ba4`
- Ubuntu 24.04.3 LTS, kernel 7.0.0-29-generic
- NVIDIA GeForce RTX 5090, 32607 MiB VRAM, driver 595.84
- AMD Ryzen 9 9950X3D, 249 GiB system RAM

Anything else

Root cause and suggested fix

The two checkpoints use compatible expert tensor names, but describe their quantization differently:

• RadixArk uses top-level quant_algo: NVFP4.
• NVIDIA uses top-level quant_algo: MIXED_PRECISION. Its config_groups.group_0 describes 4-bit float weights with group size 16 and targets model.language_model.layers.*.mlp.experts.
• NVIDIA's other groups contain FP8 MTP experts and the FP8 PLE table.

freetoken.models.qwen4_exp.config.parse_config() currently checks only whether the top-level algorithm contains fp4. It therefore sets expert_quant="none" for the NVIDIA checkpoint.

The parser should recognize a ModelOpt MIXED_PRECISION group as routed-expert NVFP4 when its weights have num_bits=4, type=float, group_size=16, and its targets are the main language model's .mlp.experts. For this checkpoint the resulting flags should be:

expert_quant=nvfp4
dense_quant=none
attn_quant=none
lm_head_quant=none

I tested this locally by adding mixed-precision group detection plus a regression test shaped like NVIDIA's config. The Qwen3.8 config test suite passes (9 passed), and both the NVIDIA and RadixArk configs resolve to the flags above. With that change, the NVIDIA checkpoint proceeds through expert-bank loading successfully.

I searched existing issues and found related Qwen3.8/NVFP4 reports, including #335, but none with this MIXED_PRECISION metadata detection failure.

[gdevenyi]

This checkpoint runs here. It has been our production model since 2026-09-05 on 2 x RTX 6000 Ada (sm_89, 48 GB each, PCIe 4 x16, no NVLink), TP=2, offload MoE backend, snapshot fab0aecb.

Why it fails on main

The symptom you describe — experts resolved as unquantized, the bf16 expert-bank provider selected, every expert layer reported missing — comes from detect_expert_quant in models/config.py. NVIDIA's build declares:

quant_algo    : MIXED_PRECISION
quant_method  : modelopt
quantized_layers:
  model.language_model.layers.<N>.mlp.experts -> {"quant_algo": "NVFP4", "group_size": 16}
  model.language_model.layers.1.ple.ple_embedding.ngram_embedding -> {"quant_algo": "FP8"}
ignore: 292 entries (lm_head, embed_tokens, hyper_connection_mixer*, ...*attn_hyper_connection*, ...)

detect_expert_quant tests "fp4" in algo (MIXED_PRECISION has no "fp4"), then a top-level format key — which this config does not have at all; the real algo sits per module in quantized_layers. Both tests miss, so the function falls through to return str(algo).lower() = "mixed_precision", which is not a recognised expert format, and the loader picks the bf16 provider. There is nothing wrong with the checkpoint.

The format: mixed-precision branch already in detect_expert_quant is the llm-compressor spelling (config groups). NVIDIA's is the modelopt spelling (quantized_layers), so #343 does not cover it either.

What makes it load

Reading quantized_layers for a .mlp.experts entry with quant_algo: NVFP4 — which is what #320 adds:

elif algo == "mixed_precision":
    quantized = get("quantized_layers") or {}
    experts_nvfp4 = any(
        ".mlp.experts" in str(module)
        and str((spec or {}).get("quant_algo", "")).upper() == "NVFP4"
        for module, spec in quantized.items()
    )
    expert_quant = "nvfp4" if experts_nvfp4 else "none"

Caveat on my evidence: I carry that branch on a deploy branch alongside #385 (TP), #386 (vision) and #389 (FP8 dense), so I have not tested #320 by itself against a clean main. The detection above is the part that decides bf16-provider vs NVFP4-provider, and it is what #320 contributes.

Note the 292-entry ignore list: on this build only the routed experts are NVFP4. Attention, GDN, the shared expert, the hyper-connections, PLE and lm_head are bf16 in the file, so a loader that assumes "NVFP4 checkpoint ⇒ NVFP4 dense" will also go wrong here; deriving each family's dtype from ignore is what works.

Serving numbers, production flags (--moe-backend offload --ple-backend pinned --num-tokens 2097152 --kv-cache-dtype fp8 --memory-ratio 0.94 --moe-prefill-hit-d2d --max-running-requests 8 --cuda-graph-max-bs 8 --tp-size 2), settled server, single-stream median of three 256-token generations:

single-stream	105.6 tok/s
8 concurrent	352.3 tok/s
TTFT, ~1k prompt	0.74 s
expert residency	71.7% (17,619 slots)
GSM8K-300, greedy, thinking off	97.0%
100k-token recall	intact

The 47.7 GiB FP8 PLE n-gram table loads as a pinned host bank (--ple-backend pinned, worth +6% single / +14% aggregate over the disk backend here).

Happy to run anything specific on this checkpoint if it would help — it is a live server, so most probes are cheap.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt