convert : add --fuse-qkv flag to fuse Q/K/V into QKV during HF-to-GGUF conversion

[JoursBleu]

Overview

#21245 already introduced the model-loading and graph-building paths for fused QKV computation. This PR further adds an opt-in --fuse-qkv flag to convert_hf_to_gguf.py: during HF-to-GGUF conversion, it concatenates the separate Q / K / V weight tensors into a single fused attn_qkv tensor.

• convert_hf_to_gguf.py: add the --fuse-qkv CLI flag. Buffer ATTN_Q / ATTN_K / ATTN_V per layer; once all three have arrived, concatenate them with torch.cat into a single fused ATTN_QKV. Without the flag, conversion is identical to master.
• gguf-py/gguf/constants.py: register MODEL_TENSOR.ATTN_QKV for every arch that already declares ATTN_Q + ATTN_K + ATTN_V (85 entries).
• src/llama-model.cpp: in the fused branch of create_tensor_qkv, if wqkv is found but wqkv_b is absent, fall back to loading the separate wq_b / wk_b / wv_b. --fuse-qkv only fuses weights, not biases, so this fallback is required.
• src/llama-graph.cpp: in the fused branch of build_qkv, if wqkv_b is absent but wq_b / wk_b / wv_b exist, use ggml_concat to concatenate the three bias segments and add them to the result of the fused matmul.

Reason for the changes in src/llama-model.cpp and src/llama-graph.cpp: on architectures with attention bias (qwen2, phi2, starcoder2, stablelm, etc.) the bias would otherwise be silently dropped, producing garbage output; after the fix the output of these archs on a --fuse-qkv GGUF is bit-for-bit identical to the nofuse GGUF.

Diffstat: 4 files, +138 −2.

Test

test-llama-archs: all OK, 0 FAIL.

Additional information

Following the two-step split discussed in #20628 with @am17an and @ngxson:

1. model : refactor QKV into common build_qkv and create_tensor_qkv helpers #21245 (merged) — pure refactor extracting the build_qkv / create_tensor_qkv helpers.
2. This PR — on top of the helpers from model : refactor QKV into common build_qkv and create_tensor_qkv helpers #21245, adds the converter-side Q/K/V weight fusion flag.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — used as a translation tool for translating the PR description

[am17an]

Thanks for the PR, do you have any performance analysis for this change?

[JoursBleu]

Hi @am17an ,

Too many models to cover, so I only benchmarked Q8_0 on a subset:

The speedup is not universal, but since the flag is opt-in, users can choose whether to enable it per model.

Hardware: AMD Radeon AI PRO R9700
Bench: llama-bench -p 512 -n 128 -r 5.

Q8_0 pp512 (t/s)

Arch	Model	nofuse	fuse	Δ
bloom *	bloom-7b1	1489.10	1488.10	~0%
gemma	gemma-7b	1145.86	1138.45	-0.6%
gemma2	gemma-2-9b-it	464.98	477.75	+2.7%
gemma3	gemma-3-4b-it	520.37	545.13	+4.8%
glm4	glm-4-9b-chat	459.73	449.42	-2.2%
internlm2	internlm2.5-7b	616.20	601.46	-2.4%
llama	Llama-3.1-8B	586.12	546.95	-6.7%
mistral3	Mistral-Small-3.1-24B	261.52	275.38	+5.3%
nemotron	Nemotron-Mini-4B	621.92	648.71	+4.3%
openai-moe	Mixtral-8x7B	309.50	334.66	+8.1%
phi2	phi-2	2678.32	2846.56	+6.3%
phi3 *	Phi-3.5-mini	2260.09	2263.78	~0%
qwen2	Qwen2.5-7B	593.65	671.34	+13.1%
qwen2moe	Qwen1.5-MoE-A2.7B	1952.52	1951.36	~0%
qwen2vl	Qwen2-VL-7B	601.64	618.33	+2.8%
qwen3	Qwen3-8B	579.11	593.76	+2.5%
qwen3moe	Qwen3-30B-A3B	404.64	386.03	-4.6%

Q8_0 tg128 (t/s)

Arch	Model	nofuse	fuse	Δ
bloom *	bloom-7b1	59.10	59.04	~0%
gemma	gemma-7b	51.60	52.27	+1.3%
gemma2	gemma-2-9b-it	42.42	43.26	+2.0%
gemma3	gemma-3-4b-it	79.16	79.81	+0.8%
glm4	glm-4-9b-chat	46.42	47.85	+3.1%
internlm2	internlm2.5-7b	55.96	57.17	+2.2%
llama	Llama-3.1-8B	55.01	56.25	+2.3%
mistral3	Mistral-Small-3.1-24B	20.30	20.75	+2.2%
nemotron	Nemotron-Mini-4B	97.55	102.14	+4.7%
openai-moe	Mixtral-8x7B	34.91	35.36	+1.3%
phi2	phi-2	112.82	114.23	+1.2%
phi3 *	Phi-3.5-mini	98.39	98.44	~0%
qwen2	Qwen2.5-7B	60.22	61.19	+1.6%
qwen2moe	Qwen1.5-MoE-A2.7B	116.67	116.96	+0.2%
qwen2vl	Qwen2-VL-7B	56.71	59.84	+5.5%
qwen3	Qwen3-8B	52.64	53.98	+2.5%
qwen3moe	Qwen3-30B-A3B	78.83	81.15	+2.9%

* = natively fused QKV

[am17an]

I think for PP your results might have a lot of noise, I will post some results on other hardware. According to me this should be +ve performance for everything

[am17an]

Looks like gemma 4 couldn't be converted, can you check?

[JoursBleu]

hi @am17an,

Gemma4 added. Greedy decode bit-identical vs nofuse. gemma-4-31B-it:

R9700:

quant	GPUs	pp512 nofuse → fuse	tg128 nofuse → fuse
Q4_K_M	1	890.35 → 854.15 (−4.1%)	24.13 → 24.06 (−0.3%)
Q8_0	2	314.13 → 312.10 (−0.6%)	15.31 → 15.50 (+1.2%)

Halo:

quant	pp512 nofuse → fuse	tg128 nofuse → fuse
Q4_K_M	290.20 → 299.87 (+3.3%)	9.76 → 9.96 (+2.0%)
Q8_0	298.76 → 304.02 (+1.8%)	6.35 → 6.42 (+1.1%)

[CISC]

Fun side quest, make sure this works for NVFP4 too.

[am17an]

shouldn't this be using the function in llama_graph?

[am17an]

Performance uplift mostly everywhere, but I guess larger ubatch will have larger benefit due to sharing the activation

Without fusion (2x 4090)

model	size	params	backend	ngl	n_ubatch	fa	test	t/s
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	1	1	pp512	28.27 ± 0.00
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	2	1	pp512	55.29 ± 0.00
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	4	1	pp512	109.55 ± 0.02
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	8	1	pp512	202.16 ± 0.04
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	16	1	pp512	386.55 ± 0.19
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	32	1	pp512	759.59 ± 0.05
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	512	1	pp512	2756.28 ± 21.14

With fusion

model	size	params	backend	ngl	n_ubatch	fa	test	t/s
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	1	1	pp512	28.45 ± 0.00
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	2	1	pp512	55.72 ± 0.01
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	4	1	pp512	110.37 ± 0.02
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	8	1	pp512	205.69 ± 0.05
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	16	1	pp512	394.31 ± 0.09
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	32	1	pp512	774.52 ± 0.10
gemma4 31B Q8_0	30.38 GiB	30.70 B	CUDA	99	512	1	pp512	2779.28 ± 19.25

[JoursBleu]

@am17an Thanks for the test! I reproduced the results:

2× 4090, gemma4-31B Q8_0, -ngl 99 -fa 1 -p 512 -n 0 -r 3:

n_ubatch	nofuse t/s	fuse t/s	Δ
1	28.95	29.14	+0.7%
8	206.24	210.00	+1.8%
16	396.04	403.81	+2.0%
32	779.30	794.65	+2.0%
512	2777.38	2801.18	+0.9%

ub=16~32 peaks at +2%, then ub=512 drops back to +0.9%.

We captured a profile trace, nsys at ub=32:

                        instances  total      avg/inst
mul_mat_q<Q8_0> nofuse  2442       224.6 ms   91.98 µs
mul_mat_q<Q8_0> fuse    1842       223.1 ms  121.14 µs
stream_k_fixup  nofuse  2382         8.85 ms   3.71 µs
stream_k_fixup  fuse    1482         6.24 ms   4.21 µs
quantize_mmq_q8_1 nf    2442         4.12 ms   1.69 µs
quantize_mmq_q8_1 fu    1842         3.26 ms   1.77 µs

Fusion did reduce 600 mul_mat_q + 900 fixup + 600 quantize launches, saving ~5 ms of launch time.
On the other hand, fusion lets Q/K/V share the same activation input, so the activation read from HBM drops from 3 times to 1 time;
as the input length grows, the arithmetic intensity also grows, so the benefit from reduced memory access shrinks;
at the same time, since launch time is fixed, that benefit also shrinks, which is why the gain drops back when the input length grows to 512.

We also tested on spark. Since spark's FLOPS/bandwidth ratio is much higher than 4090, the roofline knee is relatively further back (it takes a larger ubatch to enter the compute-bound region), so the BW-bound region is wider, and fusion can keep eating benefit even at large ub.

GB10 / Spark, gemma4-31B Q8_0:

n_ubatch	nofuse t/s	fuse t/s	Δ
1	7.16	7.17	+0.1%
8	52.95	53.36	+0.8%
32	179.66	182.18	+1.4%
128	564.89	577.81	+2.3%
512	691.13	757.33	+9.6%

In short: "larger ubatch ⇒ larger benefit" holds in the launch-bound region, but on 31B / 4090 the per-layer GEMM is already BW-bound, so the curve tops out at ~2% and drops again once ub=512 enters compute-bound. On spark the gain is noticeably larger.

[JoursBleu]

Fixes the Qwen-3.6-35B-A3B --fuse-qkv crash reported on #22710:

Arch	model	PATCHED+FUSED	PATCHED+NOFUSE	master+FUSED
qwen35	Qwen3.5-0.8B	ok	ok (identical)	crash
qwen35moe	Qwen-3.6-35B-A3B	ok	ok (identical)	crash
olmo2	OLMo-2-1B	ok	ok (identical)	crash
gemma3n	Gemma-3n-E4B	ok	ok (identical)	crash
olmoe	OLMoE-1B-7B	ok	ok (identical)	crash
falcon-h1	Falcon-H1-Tiny-90M	ok	ok (identical)	ok
lfm2	LFM2.5-1.2B	ok	ok (identical)	ok
jamba	Jamba-3B	ok	ok (identical)	ok
granite-hybrid	Granite-4.1-3B	ok	ok (identical)	ok
nemotron-h	Nemotron-H-8B	ok	ok (identical)	ok

[ynankani]

Fun side quest, make sure this works for NVFP4 too.

How will you fuse global scale for NVFP4? One option is to first dequantize fuse QKV and then re-quantize to NVFP4.

[ynankani]

This seems to be a merge issue --fuse_qkv is added in same parser as --target-model-dir

[ynankani]

Suggested change

	suffix = 'bias' if is_bias else 'weight'
	suffix = '.bias' if is_bias else '.weight'

suffix should have .bias and .weight otherwise it will default to '.weight' and the bias is never appended. See definition format_tensor_name

[ynankani]

Suggested change

	const int64_t n_embd_q = n_embd_head * n_head;
	const int64_t n_embd_qkv = n_embd_q + n_embd_k + n_embd_v;
	layer.wqkv = create_tensor(tn(LLM_TENSOR_ATTN_QKV, "weight", i),{n_embd, n_embd_qkv},TENSOR_NOT_REQUIRED);
	if (layer.wqkv) {
	layer.wqkv_b = create_tensor(tn(LLM_TENSOR_ATTN_QKV, "bias", i),{n_embd_qkv},TENSOR_NOT_REQUIRED);
	} else {

layer.wqkv is checked in graph but load never assigns it so it is still separate wq, wk, wv. I think For Gemma4 without this fix the PR should behave neutral?

[ynankani]

@BLSharda Can you please review this, i tried to repro the perf improvement on DGX spark with this PR, i see good perf improvement.
Params: -p 512 -n 0 -r 40 and varying -ub

gemma4-31B Q8_0

ub	nofuse	fuse	Δ%
1	6.79±0.02	6.80±0.01	0.15%
8	50.33±0.02	50.71±0.01	0.76%
32	163.02±0.07	164.32±0.09	0.80%
64	304.60±0.25	306.84±0.28	0.74%
128	522.85±0.90	533.15±0.76	1.97%
256	595.38±1.08	586.29±1.26	−1.53%
512	652.91±2.78	700.80±3.37	7.33%
1024	652.64±1.88	701.43±3.51	7.48%

Llama-3.1-8B Q8_0

ub	nofuse	fuse	Δ%
1	29.91±0.15	30.25±0.00	1.14%
8	224.78±0.13	227.37±0.12	1.15%
32	737.33±0.81	750.64±0.81	1.81%
64	1306.27±2.55	1335.80±2.67	2.26%
128	2118.53±9.33	2185.60±8.36	3.17%
256	2576.05±11.79	2686.80±12.58	4.30%
512	2904.81±23.21	3076.40±28.52	5.91%
1024	2884.06±27.16	3075.54±19.71	6.64%

[am17an]

@CISC can we merge after rebase?

[ynankani]

@CISC can we merge after rebase?

I think current state of this PR is broken, need some fixes before merge

[am17an]

@ynankani what is broken exactly? I can make the changes and push

[ynankani]

@ynankani what is broken exactly? I can make the changes and push

#22780 (comment) (rebase issue, two arg in same parser)
#22780 (comment) (suffix issue, will drop bias always)
#22780 (comment) (wqkv create tensor missing but checked in graph for Gemma)

Above 3 issues i feel need to be fixed before merge

[am17an]

@JoursBleu are you willing to complete this PR soon? Please reply.

[ynankani]

I tried this PR after latest commits and it works without any additional patch.
There are still some places where it needs some cleanup and handling like fused qkv with mtp .
@am17an maybe if you can trigger AI review for this PR or what next step you suggest to move forward with this PR?

[am17an]

/bot review

[ggml-gh-bot]

Automated code review

I've completed a static review of PR #22780. Below are my findings grouped by severity.

Summary

The PR adds an opt-in --fuse-qkv converter flag that concatenates separate Q/K/V weights (and biases) into a single ATTN_QKV tensor during HF-to-GGUF conversion, plus C++ load/graph fallbacks for fused weights with separate biases. I checked the converter logic, the C++ helpers (create_tensor_qkv/build_qkv), every manually-updated model graph, and cross-referenced the 110 arches that got ATTN_QKV registered against their loaders.

The design is sound: arches whose loaders/graphs cannot consume a fused wqkv (bitnet, dflash, eagle3, plm, gemma4-assistant, bailingmoe3, kimi-k3, deepseek2/32/4, glm-dsa, minicpm3, ...) were correctly not given ATTN_QKV registration, so prepare_qkv_fusion returns early for them and --fuse-qkv is a no-op (safe). I did not find a blocking correctness bug.

Will slow the review

(point 1) PR description is materially inaccurate. It states "Diffstat: 4 files, +138 -2" but the diff touches 19 files, and it says "--fuse-qkv only fuses weights, not biases, so this fallback is required" - but conversion/base.py clearly does fuse biases too (_q_bias_buffer/_k_bias_buffer/_v_bias_buffer, the .bias branch in modify_tensors, and _fusable_qkv_bias_layers). The description also omits the substantial model-graph changes in deepseek2.cpp, gemma3n.cpp, gemma4.cpp, jais2.cpp, kimi-linear.cpp, llada.cpp, minimax-m2.cpp, olmo2.cpp, olmoe.cpp, qwen35.cpp, qwen35moe.cpp, qwen3next.cpp, step35.cpp, deepseek2ocr.cpp. The contributor should rewrite the description to match the actual scope and the actual (weights and biases) behavior.

(point 2) The C++ separate-bias fallback in build_qkv (src/llama-graph.cpp:1635) is effectively dead for converter-produced GGUFs. Because the converter fuses biases whenever it fuses weights, a --fuse-qkv GGUF always has wqkv_b when it has biases, so the else if (layer.wq_b && layer.wk_b && layer.wv_b) branch never fires from the converter. It only matters for hand-crafted/legacy GGUFs. Either drop the converter bias-fusion (if weight-only was the intent) or update the description to explain the fallback is defensive. Right now the description and the code disagree on intent, which will confuse reviewers.

(point 3) Manual fused paths don't add bias; build_qkv paths do. The arches wired with manual ggml_view_2d splits (olmo2.cpp, olmoe.cpp, minimax-m2.cpp, step35.cpp, qwen3next.cpp, qwen35.cpp, qwen35moe.cpp) compute qkv = build_lora_mm(wqkv, cur) and never add wqkv_b, whereas build_qkv (src/llama-graph.cpp:1633) does. If any of these arches ship a model variant with Q/K/V attention bias, --fuse-qkv would fuse the bias into wqkv_b and the graph would silently drop it. The non-fused path already drops it too (so output is still bit-identical to nofuse, as claimed), but this is a latent inconsistency. Please confirm these arches have no Q/K/V bias, and if any could, route them through build_qkv (or add wqkv_b) rather than hand-rolling the split.

(point 4) No converter test for --fuse-qkv. The "test-llama-archs: all OK" line in the description does not exercise the converter. A minimal round-trip/shape test (e.g., convert a small model with and without the flag and assert the QKV tensor shape = q_dim+k_dim+v_dim and that load+decode matches) would catch regressions in the buffering/concat logic, which is order-dependent.

(point 5) "bit-for-bit identical" should be qualified. A fused wqkv @ cur is one matmul; separate wq/k/v @ cur are three. On CPU the per-element dot products are identical, but on CUDA/Metal the larger matmul can accumulate in a different order, so tiny numerical differences are possible on accelerators. The claim is CPU-centric; worth noting in the description.

Nits

(point 6) src/llama-model.cpp:3107 comment // fallback: when --fuse-qkv only fused weights, biases may still be stored separately references the CLI flag name. Per the project's comment guidance, prefer to describe the invariant (fused weight present without fused bias) rather than tie the comment to a converter flag.

(point 7) conversion/base.py prepare_qkv_fusion extracts bid from the mapped gguf name (new_name.split(".")), while the load loop and modify_tensors use bid from the original HF name. For standard model.layers.N. naming these match, but for unusual naming they could diverge and silently skip fusion (safe - no crash, just no fusion). Consider extracting bid from one consistent source.

(point 8) prepare_qkv_fusion blocks weight fusion for any layer whose biases are only partially present (weight_types == qkv_types and (not bias_types or bias_types == qkv_types)). A layer with all 3 weights but only 2 biases won't fuse at all. Unlikely in practice, but worth a comment if intentional.

(point 9) jais2.cpp inlines the create_tensor_qkv logic instead of calling the helper. It appears justified (jais2 requires biases with shape {n_embd}, stricter than the helper's NOT_REQUIRED), but a one-line comment explaining why it can't reuse create_tensor_qkv would help future readers.

This review was generated automatically by pi coding agent using zai-org/GLM-5.2. It may contain mistakes. Maintainers make the final call.

[ynankani]

#22780 (comment) Similar to this can we reuse or extend the llm_graph_context::build_qkv?
Rest of the PR looks good

[ynankani]

#22780 (comment) Similar to this can we reuse or extend the llm_graph_context::build_qkv?
Rest of the PR looks good

[am17an]

@CISC can you take a look at merge this please? it's been way too long

[CISC]

Can someone please verify old (and new if possible) conversions of touched models?

[ynankani]

Can someone please verify old (and new if possible) conversions of touched models?

Arch	Checkpoint	fused QKV before	fused QKV after	nofuse PPL (before/after)	fuse PPL
olmo2	OLMo-2-0425-1B	0 (Q/K/V=16/16/16)	16 (0/0/0)	9.8437 / 9.8437	9.8433
deepseek2ocr	DeepSeek-OCR	0 (12/12/12)	12 (0/0/0)	12.3186 / 12.3186	12.3186
olmoe	OLMoE-1B-7B	0 (16/16/16)	16 (0/0/0)	7.8886 / 7.8886	7.8848
gemma3n	gemma-3n-E2B-it	0 (30/30/30)	30 (0/0/0)	35.5655 / 35.5655	35.6041
llada	LLaDA-8B	0 (32/32/32)	32 (0/0/0)	847522 / 847522	842233
qwen35	Qwen3.5-0.8B-Base	18 (7/7/7)	25 (0/0/0)	13.8897 / 13.8897	13.8897
gemma4	gemma-4-E2B	0 (35/35/35)	35 (0/0/0)	8.3825 / 8.3825	8.3835
jais2	Jais-2-8B-Chat	0 (64/64/64)	64 (0/0/0)	19.5475 / 19.5475	19.5503
qwen35moe	Qwen3.5-35B-A3B	30 (11/11/11)	41 (0/0/0)	5.7610 / 5.7610	5.7665
kimi-linear	Kimi-Linear-48B	0 (Q/K/V=27/20/20)	20 (Q=7 remain)	4.8309 / 4.8309	4.8437

I did not test for step35, qwen3next, minimax due to size and mtp path