ggml-metal: fix int32 overflow in kernel_mul_mm batched dst offsets (deep-context corruption) - #28210
Closed
feni6 wants to merge 63 commits into
Closed
ggml-metal: fix int32 overflow in kernel_mul_mm batched dst offsets (deep-context corruption)#28210feni6 wants to merge 63 commits into
feni6 wants to merge 63 commits into
Conversation
Metadata and tensor loading only. The graph entry point throws, as qwen4exp did at the same stage. kda.gate_lower_bound is read as required: kimi-k3 selects the softplus branch when it is absent, which is a different function rather than a missing clamp. The absorbed MLA projections are 3D, so glm5next joins bailingmoe3 in the MXFP4 carve-out that would otherwise quantize them as expert tensors.
glm5next's mHC is DeepSeek-V4's hyper-connection block: same wide residual, same 24-row mixer split, same two activations, same Sinkhorn. Only the final collapse differs, so the graph derives from llama_model_deepseek4::graph and reuses build_hc_pre / build_hc_post / build_hc_sinkhorn rather than restating them, as graph_dsv4 already does in dflash.cpp. dsv4_hc_mean becomes a static member so both archs can reach it; the body and both deepseek4 call sites are otherwise untouched. The generated code for deepseek4 is unchanged apart from the endbr64 landing pad the helper now needs as a global symbol. The four streams start as exact copies of the token embedding and collapse to an unweighted mean after the last layer: this checkpoint has no hc_head. KDA, DSA and the MoE land in later commits, so the two sublayers throw. The mHC wiring around them is final.
Copy-adapts kimi-k3's KDA layer rather than kimi-linear's or bailingmoe3's: it already matches on the recurrence ordering, the bounded-sigmoid decay gate and its branch selection, dt_bias added per channel before the reshape, per-head A broadcast, SiLU after the conv, f/g/beta read from the pre-convolution hidden states, and the gated output RMSNorm with a plain weight. Three differences from kimi-k3. The output gate is low rank, g_b(g_a(x)) as in kimi-linear, which is what PR 1's converter emits. The q/k L2 eps is a literal 1e-6, the reference's own constant, not f_norm_rms_eps; ggml_l2_norm implements max(sqrt(sum), eps) rather than sqrt(sum + eps), which at head_dim 128 differs by about eps/(2*sum) and never trips the clamp, so it is close but not bit-exact. And the cross-layer residual, latent MoE, situ activation and MLA output gate have no counterpart here. The conv follows the reference and convolves q|k|v as one depthwise kernel, which keeps the conv state a single contiguous block so build_conv_state can snapshot it. That plus build_recurrent_attn is what makes the layer safe under recurrent-state rollback, so the arch joins llm_arch_supports_rs_rollback; without that entry the guard in llama_context silently clamps n_rs_seq to 0. build_delta_net_autoregressive reshaped a per-channel KDA gate onto ne1, but ne0 is the key axis everywhere else in that function, so it decayed along the value axis. Invisible for GDN, where the gate is scalar and both spellings produce the same [1, 1, H_v, n_seqs], and invisible to the shape checks because S_k == S_v. Fixed rather than asserted around, since glm5next reaches that path on any backend without the fused operator. llama_model_deepseek4::graph now derives from llm_build_delta_net_base so glm5next, which derives from it for the mHC residual, can reach build_delta_net. The base is a method-only mixin over llm_graph_context with no data members and no virtuals beyond the destructor llm_graph_context already has; deepseek4.cpp, dflash.cpp and kimi-k3.cpp compile to byte-identical instructions across the change. graph_max_nodes moves the arch to kimi-k3's tier. Measured on the Tiny fixture with the chunked fallback: 182 nodes plus 15/16 per token for each KDA layer and 46 per layer for the mHC mixers, so the 45-layer model needs 8.3k + 31.9 per token before DSA or the MoE are counted, which overruns the n_tokens*40 budget. test-llama-archs synthesised no MLA, hyper-connection, kpool or expert-weight keys for glm5next, so PR 1's required get_key calls threw out of the sweep and truncated it at 75 of 143 architectures. The fixture is complete now and the row is skipped explicitly while the DSA and feed-forward sublayers still throw.
The routing is DeepSeek-V3 noaux_tc exactly as build_moe_ffn already implements it: sigmoid scores, exp_probs_b added for the top-k SELECTION only, weights gathered from the unbiased scores, normalised, then scaled by routed_scaling_factor. n_group and topk_group are both 1, so the group-limited stage is degenerate and build_moe_ffn's n_expert_groups > 1 guard skips it; no group keys are written and none are needed. The clamp is the one thing that needed a change outside this arch. glm5next clamps the gate max-only and the up symmetrically, both BEFORE the SiLU, which is what the branch behind the DEEPSEEK4/DFLASH arch gate already does; the else branch clamps after the SiLU and is a different function. Adding the arch to both gates reuses it rather than restating it. The two conditions are separate because the dense path and the MoE path read different hparams arrays. The leading dense layers clamp too. The reference builds them from the same Glm5NextTextMLP as the shared expert, so swiglu_limit is not MoE-only, and the converter already writes swiglu_clamp_shexp for every layer rather than only the sparse ones. The shared expert is added unscaled.
nope-only MLA in the absorbed form, over every cached position. below index_topk + index_kpool - 1 resident tokens the indexer selects all of them, so this is exactly what the sparse path degenerates to, and it is a reference the sparse commit can be checked against. the attention half of the hybrid memory becomes the K-only variant: after absorption the cache holds the kv_lora_rank latent and V is a view of K.
both are required keys for glm5next, so a model saved without them cannot be loaded back. this is what stops test-llama-archs from round-tripping the arch.
the DSA sublayer no longer throws, so the arch can construct and run. it needs the MLA head shape as well: with n_head_kv taken from the per-layer array it would size the K cache row n_head times wider than the latent the graph writes.
index_topk + index_kpool - 1 is the number of positions the indexer keeps, and it is what makes the dense attention this branch builds exactly equal to the sparse path below that many cached tokens. an off-by-one in it is invisible to every output comparison measured so far, on both a dense and a sparse fixture, so it is checked against a second spelling of the same arithmetic instead.
The DSA layers of this model score pools of index_kpool consecutive positions
rather than single keys, and the pooled key cannot be rebuilt from the MLA
latents. llama_memory_hybrid therefore gains an optional third cache holding one
indexer key and one compressor gate per token, so the hybrid carries the KDA
conv+recurrent state, the MLA latents and the indexer keys at once.
Absent unless filter_idx is given, which defaults to null, so every existing
architecture gets exactly what it got before, state file layout included.
Two heads per cell, not one. GLM's compressor is not a mean pool: it is a
per-channel softmax over the kpool slots with logits gate + ape, where the gate
is a second projection of the hidden state of width indexer_head_size. Caching
it beside the key is the only way a pool survives its member tokens leaving the
batch. Architectures with indexer_kpool == 0 still get one head.
The indexer cache is handed the attention cache's slot layout rather than
finding its own, so the two agree cell for cell, and apply() asserts they do.
It also keeps its own dtype: -ctk q8_0 would otherwise quantise the gates, which
feed a softmax.
llama-kv-cache-kpool.{h,cpp} builds the pool <-> cell map host side. Pools are
defined on positions and cells are whatever find_slot handed out, so the
correspondence cannot be derived in the graph. Nothing here emits a negative
index: ggml_set_rows asserts i1 >= 0, so unpopulated entries are clamped into
range and neutralised by an additive -INFINITY instead.
Two things the map does that the qwen4exp shape it is ported from does not:
- the top-k budget is indexer_top_k exactly, with the always-selected tail
biased to -INFINITY so it spends none of it, and forced back in through a
host-built base mask for the scatter. indexer_top_k is a whole number of
pools, so the cut lands on a pool boundary; the reference's own output width
of indexer_top_k + kpool - 1 does not, and ggml_top_k is unordered among
equals on both CPU and CUDA.
- one map per ubatch, shared by every indexer layer, since nothing in it
depends on the layer. Measured on a 16 Ki cell cache with 512 tokens:
~4 ms once against ~4 ms x n_layers.
A unified cache with more than one sequence would let two sequences at the same
position pool each other's keys, so create_memory refuses it up front rather
than aborting mid-run.
tests/test-glm5next-memory.cpp: 74 checks, 0 failures, on both the full and the
trunk-only fixture. test-llama-archs is byte identical to the same build without
this commit at a fixed seed: 452 rows, 0 FAIL. Session state files for
qwen3next, falcon-h1, minimax-01, qwen35moe and a real Falcon-H1-0.5B are byte
identical too, across write, reload and rewrite.
Builds the pooled lightning indexer and gives the DSA layers a sparse attention path driven by it. Top-k runs over the POOL axis at select_k = index_topk/index_kpool, and the selected pools are expanded to their member cells through pool_cells. That is the reference's own two-step (modular_glm5_next.py, Glm5NextTextIndexer.forward: topk over the pool axis, then selected_indices = pool_indices[batch_idx, selected]), and it is not interchangeable with a single top-k of width index_topk over member cells. The argument for the cell-level form - a pool's members carry its score bit-exactly, so the cut must land on a pool boundary - assumes tie groups never span pools. They do: ReLU drives most pool scores to exactly 0.0, and ggml_top_k is explicitly unordered among equals, so the cut falls inside an inter-pool tie group and splits a pool. Measured on TinySparse at 512 tokens, the cell-level form leaves a partial pool on 7.51% of query rows at layer 3 and 5.93% at layer 7; this form leaves none. The indexer key and gate STORE is unconditional; only the SCORING is gated, on n_ctx > index_topk + index_kpool - 1. Gating the store the same way would leave every cell written below n_select with no indexer state, and the first ubatch to cross n_select would pool cells that were never written. Nothing here changes any other architecture: test-llama-archs produces a table byte-identical to the parent's, 300 rows over 143 archs, 0 FAIL.
The tower is the GLM-OCR ViT with a clamped SwiGLU: the gate is bounded above only, the up projection on both sides, and both before the SiLU. ggml_swiglu_oai clamps the same way but then adds one to the up branch, which is a gpt-oss detail this model does not share, so this adds an FFN_SILU_CLAMP op rather than reusing it. The clamp sits at the per-block MLP and again at the merger. Both read hparams.ffn_op, so the graph body stays the GLM-4V one and the pair is covered together. It gets its own projector type rather than a flag on glm4v because the image token limits differ (16/8000 against 8/4096, per the GLM-5.3-Flash preprocessor) and those are hardcoded per projector, and because the clamp must stay off for GLM-4V and GLM-OCR. Also writes clip.vision.spatial_merge_size. No GLM4V-family mmproj has ever carried it: Glm4VVisionModel skips the Qwen3VL parameters, which is where it is written, so clip.cpp's hardcoded 2 has been carrying it. Images only. glm5next spells video with its own token pair and distinct start/end spans, and that is not handled here.
the vision tower shipped with the shared dynamic-size preprocessor, which is a qwen-style smart_resize. the 2026-08-26 GLM-5-Next adaptation resizes differently: both edges are aligned up by ceil rather than round, an over-budget image is fitted by binary searching the content height for the largest aligned canvas still within max_pixels, and the resized content is pasted into the top-left of that canvas rather than centred and stretched to fill it. an image already at or above min_pixels is never upscaled. min_pixels/max_pixels stay in tokens. the reference scales them by temporal_factor * factor**2 and compares against aligned_frames * area, and aligned_frames equals temporal_factor for a still image, so the two cancel and hparams.image_min_pixels / image_max_pixels (16 and 8000 tokens, 12544 and 6272000 pixels) are used directly. glm4v and glm-ocr keep the dynamic-size preprocessor. images only. video has its own token pair (154855, distinct from the image token 154854) with its own start/end spans, and is out of scope here. the resize arithmetic is covered in test-mtmd-impl against values taken from the reference processor, including the 16- and 8000-token boundaries, extreme aspect ratios, and inputs where the binary search and smart_resize disagree.
glm4 / chatglm-bpe tokenizer.json files set "ignore_merges": true, meaning a pre-token that is already a vocab entry is emitted directly and the merge loop never runs. llama.cpp implements this (llama-vocab.cpp, the get_ignore_merges() short-circuit) but only enables it for a hardcoded list of pre-tokenizer names, and glm4 was never added. Without it the merges are applied - correctly - and reach a different answer, because greedy BPE cannot always reconstruct a vocab entry from its bytes. " 王" (Ġçİĭ, id 102322) is the case that exposed it: from Ġ ç İ ĭ the only merges available are (Ġ,ç)=27944, (ç,İ)=76417 and (çİ,ĭ)=239209, so the lowest rank wins first and yields Ġç İ ĭ, at which point neither (Ġç,İ) nor (İ,ĭ) exists and it stops three tokens short. Reaching Ġçİĭ needs (Ġ,çİĭ) at 242943, which requires never taking (Ġ,ç) at 27944. The trigger is whitespace immediately before a CJK character, so pure Chinese prose is unaffected and mixed Chinese-English is not: pure Chinese prose 620 vs 620 tokens, already identical mixed Chinese-English 680 -> 600 tokens, now identical to HF (-13.3%) wikitext-2 (289569 tok) one divergence -> byte-identical Found while comparing GLM-5.3-Flash perplexity against transformers, vLLM and SGLang: the mismatch bounded how many scoring windows could be compared at long context, and reads exactly like a model-port defect rather than a tokenizer one.
The scripted resolution used for the rebase mangled four files: it spliced a
condition into the middle of graph_max_nodes' multi-line else-if, dropped the
mtmd_image_preprocessor_glm5next declaration, dropped llama-kv-cache-kpool.cpp
from src/CMakeLists.txt (undefined llama_kpool_* and the llm_graph_input_kpool
vtable at link time), and left an "} else {" immediately followed by an
"} else if" in test-llama-archs.
These files are byte-identical between this base and the tree the glm5next
work was verified on, so each is taken from there verbatim.
deepseek4 sets n_embd_out_impl to hc_mult*n_embd to size its MTP h input. glm5next inherited that, but our t_embd is build_norm(build_hc_mean(...)), which is [n_embd, n_tokens]. n_embd_out() therefore reported 4*n_embd while the tensor held n_embd, and llama-context read n_outputs*n_embd_out floats out of it, four times what is there. The assert at that site sizes the destination buffer, so nothing catches the short source. Only --embeddings and llama_get_embeddings* reach the path, which is why plain generation never showed it. Note for when the NextN graph starts consuming h: give MTP its own width rather than widening n_embd_out again.
The mHC residual mixers, the lightning indexer (selection gate, learned k-pool position table, and the three indexer projections) and the KDA recurrence gates are about 1 GiB in total on GLM-5.3-Flash, so the size cost is noise against a 100-240 GB quant. Quantizing them perturbs which pools the indexer selects and how much state each KDA step retains, and those errors compound along a sequence rather than averaging out. Both spellings are required. The compressor tensors arrived with the DeepSeek-V4 merge and use an underscore (indexer_compressor_ape / _gate), while the projections use a dot (indexer.proj / .attn_k / .attn_q_b), so a single "indexer." prefix test silently misses the compressor pair. attn_q_a, attn_kv_a_mqa, attn_k_b and attn_v_b are deliberately not listed. They are precision sensitive too, but the release recipe pins them to q8_0 via --tensor-type, and that is the configuration the shipped quants were measured in. Verified with llama-quantize --dry-run q4_k_m on the BF16: all 12 pinned families report 0 quantized (45 mHC, 12 indexer, 34 KDA each), while ffn_gate_exps 43/43, attn_q_a 12/12 and attn_output 46/46 still quantize.
The 101-line glm5next block was dropped from test-mtmd-impl.cpp when the vision work was rebased, even though the commit message still claimed the resize arithmetic was covered there. It holds the 36-case table over the 16- and 8000-token budget boundaries, including six cases annotated as ones where a naive smart_resize disagrees, so it is the guard against sliding back to stretch-to-fill instead of ceil-align plus zero pad. Restored from 29c096371. test-mtmd-impl now runs 216 assertions, of which glm5next_resize contributes 185.
add_vision_swiglu_limit was inserted directly above the next method with no blank line between them, which flake8 flags as E301. Caught by ggml-org CI.
llama-embedding turns -np 1 into kv_unified with n_seq_max 256, so every --embeddings run hit the refusal in create_memory and llama-embedding then dereferenced the null context. Two fixes. The pool map is now per SEQUENCE rather than per stream. A non-unified cache already gives one sequence per stream, so nothing changes there. A unified cache puts every sequence of the ubatch in stream 0, and the stream's pool table is cut into one contiguous run per sequence, each rebased on its own lowest resident pool. pool_bias is -INFINITY outside the query's own run, so a query never spends budget on a foreign pool, and cand_mask already kept foreign cells out of the attention mask. Packed runs, not one full-width table per sequence: the indexer scores every pool slot against every query, so a full-width table per sequence multiplies the score tensor by the sequence count and graph_reserve asked for 286 GB at n_seq_max 256. The table is n_kv/kpool shared plus 2 slots per sequence for rebasing, which is exact while the sequences' cells are disjoint. A prefix shared through seq_cp can oversubscribe it, and then a sequence keeps its newest pools -- the same cut a large hole in the cache already forces. The top-k stays over POOLS and pool_cells still holds whole pools, so pool integrity is untouched. examples/embedding also checked only the model for null, not the context.
test-glm5next-memory asserted that create_memory REFUSES -kvu with n_seq_max 2, which was the contract before the pool map became per sequence. Assert the new one: the cache is built, and one ubatch holding both sequences is driven through llama_kv_cache_set_input_kpool. Three checks replace the guard. llama_kpool_n_pools is n_kv/kpool plus 2 slots per sequence, so the table is a shared budget and not one full-width table each. Every pool a query may spend budget on holds only that query's own visible cells -- the invariant a shared cells array breaks if the map is keyed per stream. The two sequences get disjoint runs and neither run is empty. Both cell-level checks fail if the runs are made to overlap, so they are not tautologies. cell_pool is not requested here: it has one row per stream and a cell that two sequences share has nowhere to put its second pool.
sel_mask and cand_mask are KQ-mask shaped and hold only 0.0f and -INFINITY, both exact in f16, so storing them in half the bytes is lossless. At n_ctx = 1 Mi, n_ubatch = 512 that is 2 GiB saved per mask plus 1 GiB on the per-layer ggml_dup. ggml_add gives its result src0's type and f16 + f32 -> f16 is a supported bin_bcast on CUDA and on the CPU, so the f16 selection mask absorbs the f32 KQ mask that flash-attention-off builds, and ggml_soft_max_ext takes an f16 mask as readily as an f32 one. Under flash attention the KQ mask is already f16 and the per-layer ggml_cast disappears. llama_kv_cache_set_input_kpool now writes either width and asserts the two masks share a type instead of asserting f32.
Replaces the 7-node score chain (mul_mat, cont/permute, relu, mul, sum_rows, cont/permute, add) with one ggml_lightning_indexer, as glm-dsa, deepseek4, deepseek32 and dots3note already do. The op needs an f16 mask, so pool_bias is cast once per graph in build_inp_kpool rather than once per DSA layer. pool_k is left in f32 so the CUDA op takes its f32 vector path, not the f16 wmma path, which would undo the GGML_PREC_F32 on the head weights. The unfused chain stays behind cparams.fused_lid, plus a LLAMA_FUSED_LID_DISABLE escape hatch.
Deletes comments that restate the code, section banners, paragraph spacers, pointers, development narration and measured numbers that belong in the PR description. What survives is limited to correctness constraints, reference implementation citations, warnings that a tempting alternative is wrong, and explanations of real bugs - each stated in one or two lines. Comments only: verified by stripping every comment from each file and comparing the normalised source against the pre-pass baseline.
…2-4): dense build_attn mirror wiring, wrapper can_reuse width check, strict V parser + MLA_V_MIRROR ACTIVE marker, construction guards, stream-copy keeps mirror clean
…te v_mirror_upd in both consumers (grokk 036)
The env is parsed in the generic hybrid construction path; is_mla && v_trans alone does not prove K carries no rope-key half. Fail closed on any other arch and print arch/n_rot in the DISABLED diagnostic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
github-actions
Bot
added
ggml
This was referenced Sep 1, 2026
|
Hi @feni6, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
github-actions
Bot
removed
the
draft
…hes are exercised (grokk 054 s3) The chunked sweep never sends n_tokens==1, so fbl_gate never opened and packed-path comparisons were vacuous. Decode one token after the sweep and compare full logits CPU-vs-device; the ACTIVE marker grep now has a real decode behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…n oracles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…LM5_GATHERED_DSA Execute the selection instead of masking it: concat the indexer's physical top-k cells with the always-selected tail, gather the <=2051 latent K rows (dim-1 view, F32 return cast back, reshape to n_head_kv=1 for packed-KQ composition), gather the cand+causal cell mask and add a per-SLOT validity mask (expanded pool_bias of the selected pools; a gather, unlike the dense scatter, is not idempotent - padded picks name cell 0 and must die by slot). V is the same compact tensor; the V-mirror is still updated, not read. Decode-1, single-stream, strict env, GATHERED_DSA ACTIVE marker. Toy oracle: dense-vs-gathered Metal logits NMSE 5.9e-15, top-5 identical; duplicate-cell-0 adversarial (select_k > valid pools) matches dense. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
- compact mask path: gather cand and kq at ids separately, add 2051-wide;
the dense cand+kq add ran per DSA layer = eleven O(n_kv) ops per token
- one gathered_decode boolean owns tail-input creation, slot_valid request
and builder choice (the 058 crash class was producer/consumer gate
disagreement); graph_mtp passes false until MTP has its own oracle
- name the compute owners gathered_k_rows / gathered_k_cast /
gathered_{cand,kq}_rows for the per-node cut; MQA-only assert on the view
- fixture: four successive decode-1 steps (rolling tail 1->2->3->0, logits
compared and dumped per step) and --skip-1tok
Suite: dense-vs-gathered NMSE ~1e-15 all four steps, adversarial
duplicate-cell-0 rolling green, h9/h64, env-off regression clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…grokk 064) bc_inp = src0->ne[0] % 32 (ggml-metal-device.cpp:768): compact packed KQV's reduction dim is n_sel, and 2,051 is both %32!=0 and odd — the slow guarded path. Pad the tail-slot allocation to GGML_PAD(n_top + kpool-1, 32) with dead slots (cell 0 / -inf), the representation the duplicate-cell-0 adversarial proved exact. Marker renamed GATHERED_DSA GRAPH (it prints at construction, not execution) and reports n_sel_pad / n_finite_max. Toy: PAD(11,32)=32 with 21 dead slots live in every gather; rolling four tails dense-vs-gathered at e-15 (step 3 bit-identical), duplicate-cell-0 rolling MATCH x4, h9 and env-off clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…ton permutes (codex 043/051, grokk 045) All four packed-decode transforms swap a singleton axis with the head axis; the permuted view already satisfies ggml_is_contiguous (singleton strides are ignored), and ggml_cont always emitted a real Metal copy. Keep the logical permutes as zero-copy views and assert contiguity at graph build. cb names stay on the surviving tensors (051 s5). No !is_permuted assert: the kq unpack view is contiguous AND permuted by design (045 s1). Probe: kq packed_view 1.186 ms vs packed 1.203 vs packed_perm 2.167 at n_kv=108544 - the 0.96 ms copy is gone. Fixture: fixed-seed byte-identical outputs at heads 64/16/9/4 and default vs pre-C build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk (cherry picked from commit d76dcb0b0b27ba5329b33fb81bccccbb0ae9417a)
…ex 068/072, grokk 070/073) llm_graph_input_kpool had no can_reuse override, so the inherited false vetoed reuse of every served GLM graph (every token re-ran build_graph + sched_alloc). The override rebinds both memory contexts and compares recomputed shapes and policy: k_idxs width, idx/attn n_kv (stored as build-time metadata so the contract survives E0c's optional masks), n_pools/mask/pool tensor shapes, rebuild==get_kpool_dirty(), n_new_max, and the recomputed PAD32 tail width (never a cached 2080). LLAMA_HOST_TIMERS=<N>: decode-1 graph_rebuild_us and set_inputs_us as SEPARATE p50/p95 buckets (R0 shrinks only the first; E0c/E1 the second), printed every N samples with the reuse counter. Reuse debug log names the input class via typeid. Fixture: reused=48 with rebuild p50=0 in steady decode (G1 and dense); pad-256 boundary crossing = exactly one rebuild; logits bit-identical to LLAMA_GRAPH_REUSE_DISABLE=1 in all cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…e class compare Under LLAMA_GLM5_E0A the gathered builder exposes the composed mask and the slot-certificate mask as named outputs; llama-context reads both back and compares 0-vs--inf class on the host. First version computed the delta with a graph-side exp/abs/sum chain, which mis-executed on one backend and produced impossible values (delta=-inf) — per the seen-to-fire doctrine the instrument was rebuilt backend-proof before any conclusion was drawn. Result: zero class mismatches on rolling tails and the duplicate-cell-0 adversarial (finite counts track the tail exactly); codex 066's algebra holds on all fixture-reachable cases. hole/seq_rm/seq_cp premises ride the env on real-server canaries, which is the point of a live invariant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…77/079) - HOST_TIMERS: nonzero-only rebuild p50/p95/max + n_rebuild/n_reuse + burst-amortized + /256 serving estimate; p50 of a 255:1 zero series hid the one pad-256 rebuild entirely - perf_reset clears both host-timer vectors (instrument correctness must not depend on fresh processes) - kpool can_reuse: exhaustive optional-pair contract (sel<->cand paired + same shape; pool_reps/new_pool_cells/new_pool_reps shapes and pairedness; tail pair same shape) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
… s2, codex 082 s3) first_rebuild_us (cold graph after restore, never a pad cost) is separated from later events; rebuild_events=ordinal:us,... preserves chronology as the width-cadence oracle; percentiles over <=2 events removed. serving estimates move to the harness, which knows the horizon and the distance to the next pad boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
…mond PASS The slot certificate subsumes cand+causal on the served lane (proven by E0a). The gathered graph now builds ONLY slot_valid||tail_valid as its mask; the dense sel/cand and the ordinary KQ mask are never created on that graph (build_inp_mem_hybrid_k(maskless) computed before the hybrid input, per 069/082), the kpool host fill is nullable-mask + explicit n_kv, and both hybrid-k and attn-k can_reuse fall back to a stored build-time n_kv witness when the mask is absent. Lane fail-closed: causal + no-ALiBi + no-SWA + scalar positions asserted in the builder. Oracle: maskless-vs-dense rolling tails NMSE <=7e-11 argmax-identical; duplicate-cell-0 adversarial and h9 clean; set_inputs_us p50 25->19 us at toy 8K with the two mask fills deleted. U/L/R/C diamond PASS at same- partition S: U==L==R==C, max_dlogprob 0.0 on every edge — the rewind, FULL serialization, and determinism paths are all exonerated, so the canary's live/cold divergence was purely the partition near-tie (084). Landmines fixed en route: b_n_kv member in the wrong base class; three unguarded sel_mask derefs in the host fill (n_padq the last, an anchor miss). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk
feni6
force-pushed
the
metal-mul-mm-u64-dst-offsets
branch
from
September 2, 2026 15:23
9ddfd0b to
18dcfb3
Compare
github-actions
Bot
added
model
mihailescu2m
added a commit
to mihailescu2m/llama.cpp
that referenced
this pull request
Sep 2, 2026
The per-batch output offset is computed in int32. For a large f32 destination it wraps once a batch base reaches 2^31 elements, storing that batch ~8.59 GiB below the tensor and leaving its own region unwritten. Upstream: ggml-org#28210.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kernel_mul_mmcomputes its per-batch output offsets in 32-bit:dstBatch = dst + im * N * M(dst4 path)... + im*args.ne1*args.ne0(tensor path, two sites)For a large f32 destination the product overflows
int32once a batch's base offset reaches 2³¹ elements. The visible case is attention KQ = [n_kv, n_ubatch, n_head] at long context: headh's base wraps onceh·n_ubatch·n_kv ≥ 2³¹, i.e. first ath = ceil(2³¹/(n_ubatch·n_kv)). The wrapped heads are stored ~8.59 GiB below the tensor — silently corrupting whatever buffer happens to live there in GPU address space — while their own region is never written, so softmax ingests stale scratch.This PR promotes the offset arithmetic to
uint64_tat the three affected sites. Three lines, no functional change below 2³¹.Real-world impact
GLM-5.3-Flash (#27754, #27752) collapses to an infinite
@(token 31) from the first sampled token once a joint (depth,n_ubatch,n_ctx) threshold is crossed on Metal — e.g. a 108,710-token prompt at-c 131072 -ub 512. The model's dense-masked DSA attention keeps KQ at full[n_kv, 512, 64]during prefill, crossing the 13 GiB f32 mark deep into context. Whether the wrap is fatal depends on what lives below the compute buffer — on our hosts, the model's own K-cache (GPU-VA adjacent, gap 0.000 GiB), which explains the depth/n_ctxdependence of the failure boundary: full diagnosis in this comment.Any model/backend-path combination that drives
mul_mmwith ≥2³¹-element batched f32 output is exposed; deep-context MLA/DSA-style attention is simply the first to get there.Evidence
ceil(2³¹/(ub·n_kv))across five geometries and three machine classes: M4 Pro, M3 Ultra (ours), M3 Max (independent confirmation by @eauchs, including the negative control: casts removed → probe fails again).-c 524288) all pass on the patched build; byte-identical prompts on the unpatched build still collapse (battery summary).-s 1234),test_mtpdraft+reload OK on three backends, seven neighbouring archs FAIL=0.Notes
-ub 128is the workaround in the wild for affected models (keepsh·ub·n_kv < 2³¹longer); this fix removes the need for it.Fixes the Metal deep-context collapse reported in #27754 / #27752 (model PRs, not yet merged; the defect is in shared ggml code and reproduces on both).
🤖 Generated with Claude Code
https://claude.ai/code/session_01YUN8B4NPpyNhziETmLiNpk