spec : add adaptive MTP draft depth (draft-mtp-adaptive)

[stew675]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

Suggested configuration to use is: --spec-type draft-mtp-adaptive --spec-draft-n-max 12

Note: It is not recommended to set spec-draft-n-max to values lower than 7 with the Adaptive MTP algorithm. In this instance the algorithm isn't able to reach depths high enough to provide the benefits it aims to achieve. If you find yourself in a situation where insufficient VRAM prevents you from setting a depth of at least 7, or more preferably 8+, then it's generally going to be best to just stick with regular MTP.

The algorithm is a counting based state machine with a climb counter and a weighted drop-pressure accumulator. The depth climbs one step after a number consecutive verifies that accepted every drafted token; any miss adds (N - acceptance) to the drop pressure and the depth drops one step once it exceeds a certain amount. High depths fall quickly (a total miss adds N), low depths hold, and at the floor no pressure accumulates at all. The specific climb difficulty and drop pressure values were empirically determined over a wide range of tests.

--spec-draft-n-max bounds the upper adaptive range and both the floor and the cold-start depth default to 3. The default floor of 3 may be adjusted with --spec-draft-n-min-adaptive, although in testing 3 seems to be the best value to choose here.

Overview of the climb-cost algorithm

It was experimentally determined that a draft MTP depth of 2 or 3 is close to optimal for reasoning and prose.
More typically an MTP depth of 3 was found to be optimal, and so this was chosen as the baseline default.

When the depth is less than three the algorithm allows for an easy transition from a depth of 1 to 2.
A slightly higher cost from 2 to 3 is used to minimise the depth oscillating between two and three.

To climb past a depth of three requires passing a hardened barrier. The hardened barrier is in place to
minimise easily reaching a depth of four which is known to be harmful to performance for reasoning and prose.

Beyond a depth of four the barrier to climb higher is gradually reduced.
This allows for predictive runs to quickly ascend as the admission rates indicate that this is useful.

Latest Testing and Performance Results

(Updated 23 Aug 2026)

Latest performance results can be seen here: #27210 (comment)

Extended discussion of results can be seen here: #27210 (comment)

Test Setup

• Model: Qwen3.8-27B Q8_0 (Qwen3.8-27B-Q8_0.gguf
• Hardware: 2x AMD Radeon AI PRO R9700 (32.6 GiB each), tensor split, Ryzen 9
  9950X3D2, ROCm, HIP_VISIBLE_DEVICES=0,2, GGML_CUDA_DISABLE_GRAPHS=0
• Server: llama-server from branch adaptive-mtp (build 10459, 0c6426b,
  freshly rebuilt)
• Decode: greedy (temp 0.0, seed 675), top-k 20, top-p 0.95, min-p 0.001,
  ctx 262144, f16 KV, --parallel 1, mlock, reasoning on
  (--reasoning-budget 65536 --reasoning-preserve)
• 2 repeats per cell; tok/s = mean of timings.predicted_per_second
  (generation only, >= 4000 tokens per request)

Configurations Tested

ID	spec type	args
C0	none	-
C1	draft-mtp, fixed 3	--spec-draft-n-max 3
C2	draft-mtp, fixed 12, p-min	--spec-draft-n-max 12 --spec-draft-p-min 0.75
C3	draft-mtp-adaptive 3..12	--spec-draft-n-min-adaptive 3 --spec-draft-n-max 12
C4	adaptive 3..12 + p-min	C3 + --spec-draft-p-min 0.75
C5	ngram-mod	--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
C6	adaptive + ngram-mod	C3 args + C5 args
C7	fixed 3 + ngram-mod	C1 args + C5 args
C8	fixed 12 + p-min + ngram-mod	C2 args + C5 args

Results

config	reasoning	prose	code	recall
C0 baseline	30.03	30.08	30.03	29.93
C1 fixed 3	51.68	55.89	69.28	81.94
C2 fixed 12 + p-min 0.75	38.59	43.46	71.46	151.25
C3 adaptive 3..12	51.42	56.64	78.03	147.46
C4 adaptive + p-min 0.75	40.67	43.35	70.92	147.43
C5 ngram-mod	30.03	30.10	29.98	292.88
C6 adaptive + ngram-mod	51.48	56.61	73.06	321.25
C7 fixed 3 + ngram-mod	51.65	55.90	68.36	324.10
C8 fixed 12 + p-min + ngram-mod	41.34	43.73	64.39	325.68

Additional information

The total diff size is +451/-26

This PR touches on 10 files which seems large at first, but this mostly arises from integrating the new command line options. The main functional changes take place within a single file (speculative.cpp).

218 lines are for the unit test file. 92 lines are for the header file that contains the algorithm itself.

The rest of the line changes are the wiring of the algorithm into the llama.cpp code base, and the command line option handling.

I tried to keep the size of the functional changes to llama.cpp itself as small as possible, and I tried to keep this diff as independent as I could so that unless it's specifically activated, it won't interfere with normal llama.cpp operations.

"Horses for Courses"

While Adaptive MTP generally works well for most hardware and models, it can make performance worse on older hardware that cannot run the deep drafting and acceptance steps fast enough to realise the potential gains it offers. In these instances it is generally better to not enable Adaptive MTP and just stick with regular MTP with a low fixed depth.

Dense Models generally receive the most benefit, but gains on MoE models are generally weaker. For MoE models it's generally preferable to lower the maximum depth to no more than 8 and then assess. If the performance is no better than using a fixed MTP depth, then that's fine. Just use a fixed MTP depth instead.

Related Issues/PRs

I searched known issues and open PRs and did not find anything precisely mentioning this sort of feature at first.

@Us5rName pointed out in the comments below that they have a similar feature PR here: https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes. I designed the exact algorithm. AI chose the best place for it, and added it. I then went over it all, reviewed it, cleaned up and changed half of it to better reflect my intent, and then tested and refined the algorithm manually over the day.

[Us5rName]

Hi!

I've also created a fork with adaptive length for mtp and dflash with a rolling window based heuristic.

https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

and this PR #25726 with the groundwork.

Would you like to colab?

[theIvanR]

Excellent job! I would propose to merge this with my fix for a more robust datatype selection in MTP implementation for nvidia cards.

(bug)
#25713
(and solution)
#25680

[stew675]

Excellent job! I would propose to merge this with my fix for a more robust datatype selection in MTP implementation for nvidia cards.

(bug) #25713 (and solution) #25680

I corrected a different MTP issue in this commit to my personal fork's working branch here: stew675@b2655d3

I really should file a PR for that, but its exposure range is pretty narrow. My fix targets which custom kernel runs for F32-activation small batches. It fixes a corner case for the handling of BF16 KV caches though (most of the work in my branch there targets adding BF16 KV to the ROCm backend), and my code exposed that latent bug.

It appears there is a handful of inconsistencies in the mainline MTP code for certain corner cases and you have found another. I do see that your PR got closed. Did you ever work on a more targetted fix as per the reviewer's suggestion?

[stew675]

Hi!

I've also created a fork with adaptive length for mtp and dflash with a rolling window based heuristic.

https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

and this PR #25726 with the groundwork.

Would you like to colab?

Thank you for bringing your PR up. I searched for similar PR's and I don't know why yours didn't show up.

I've looked over your branch and groundwork PR. There's a number of conceptual similarities, both being counting based heuristics. Both are sliding window approach, with yours being explicit, and mine being implicit. Both employ a form of "depth stickiness". Yours has a lot more knobs where I made a conscious effort to keep the knobs to a minimum. I do like your Bias knob though. That could be a genuinely useful knob for users to tweak for certain models to adjust the level of depth stickiness. The falling back approaches do differ significantly in concept. I spent quite a bit of time trying difference approaches, and then dialing in the one I settled on.

Your personal branch seems to target DFlash too, right?

I'm fine with us combining efforts, but we'd need to settle on what the UX approach should be. I'm a big fan of KISS. Years ago I would've had knobs for everything, but nowadays I'm a firm believer of use as few knobs as possible.

Do you have any performance comparisons of your implementation and how it affects both prose and code? I tried pretty hard to ensure that prose performance was affected as little as possible, so I'm curious to see results on how you solved that issue.

At the end of the day though, we both need one of the maintainers to offer some guidance on what they'd like to see. That will ultimately set the direction on what the next steps will be here.

Edited to correct my earlier statement of both being implicit sliding windows. That was my mistake earlier.

[simongonzalezdc]

Numbers from our side supporting adaptive draft depth, measured on a Strix Halo APU (gfx1151, Ryzen AI Max+ 395) serving Qwen3.8-27B with draft-mtp + ngram-mod speculative decoding (datapost: #27154):

At --spec-draft-n-max 12, draft acceptance splits sharply by workload class:

• code / low-reasoning traffic: ~0.96 acceptance. Deep drafts pay off; going from n-max 3 to n-max 12 was the difference between ~40 and ~60 tok/s class generation on this chip.
• heavy thinking/reasoning traffic: ~0.5 acceptance. Deep drafts mostly miss, so most drafted tokens become wasted verification work that adds per-token cost.

The second lane replicates independently on a different Strix Halo system: acceptance "way below 50%" under xhigh-reasoning coding traffic, with n-max 12 regressing generation (peak ~26 t/s, dips into the ~5-6 t/s range) versus lower n-max values — that user is now moving down to n-max 2 (comment: #27154 (comment)).

So a fixed n-max serves one lane and starves the other: deep enough for the ~0.96-acceptance lane, or shallow enough to cap wasted verification in the ~0.5-acceptance lane, but not both. Acceptance-driven draft depth is the knob that spans both, which is exactly what this PR adds.

[stew675]

I went over all the check failures and it appears that they are all pre-existing/known flaky tests unrelated to my code changes.

Now, I do have another commit that's almost ready which actually solves the 3% performance regression on hard prose, and actually boosts normal prose. The question I have is shall I wait for this PR to go ahead and file a followup, or shall I attach that change to this same PR. I'm fine either way.

[mndodd]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

[stew675]

I went over all the check failures and it appears that they are all pre-existing/known flaky tests unrelated to my code changes.
Now, I do have another commit that's almost ready which actually solves the 3% performance regression on hard prose, and actually boosts normal prose. The question I have is shall I wait for this PR to go ahead and file a followup, or shall I attach that change to this same PR. I'm fine either way.

If this is working, then you don’t really have many options.

Yeah, I'll cover it in a followup PR. That will give me more time to refine it further.

[nomandormosh]

Getting this error with plain draft-mtp:

 invalid adaptive draft range: n_min_adaptive=3, n_max=2 (n_min_adaptive must be in [1, n_max])                                                                                                

It occurs whenever the effective n_max < 3 (e.g. --spec-draft-n-max 2). The range check added in
speculative.cpp:1401 runs unconditionally in the ctor, but the ctor is shared by both draft-mtp and draft-mtp-adaptive, and n_min_adaptive (default 3) is only used in adaptive mode. To fix it, I moved the check inside the if (adaptive) block.

[stew675]

Getting this error with plain draft-mtp:

 invalid adaptive draft range: n_min_adaptive=3, n_max=2 (n_min_adaptive must be in [1, n_max])                                                                                                

It occurs whenever the effective n_max < 3 (e.g. --spec-draft-n-max 2). The range check added in speculative.cpp:1401 runs unconditionally in the ctor, but the ctor is shared by both draft-mtp and draft-mtp-adaptive, and n_min_adaptive (default 3) is only used in adaptive mode. To fix it, I moved the check inside the if (adaptive) block.

Thank you for catching this. I'll be pushing a fix shortly.

[stew675]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

You've provided both a broad range (8-16) and an ambiguous range of p-min > 0.5

I'm guessing you meant --spec-draft-p-min, and not --min-p?

Just to be clear, what's your ask here? For me to test all depths from 8-16 and vaguely a number of --spec-draft-p-min values as a matrix? Wouldn't that be highly dependent on what exactly it is you're doing? It's kind of the point of this PR that you don't need to be fiddling about trying to dial in the perfect depth and p-value each time.

It's also been my experience that raising --spec-draft-p-min, whether it be normal or adaptive MTP, absolutely destroys prose performance. With the rise of deep thinking models (like our good friend Qwen3.8 here) there's likely to be significant amounts of time spent generating prose before code gets written.

This is why I've been focusing on preventing/minimising harm to prose performance. We don't want trash that thinking performance just to get some more t/s on code generation and end up slower overall.

[mndodd]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

You've provided both a broad range (8-16) and an ambiguous range of p-min > 0.5

I'm guessing you meant --spec-draft-p-min, and not --min-p?

Just to be clear, what's your ask here? For me to test all depths from 8-16 and vaguely a number of --spec-draft-p-min values as a matrix? Wouldn't that be highly dependent on what exactly it is you're doing? It's kind of the point of this PR that you don't need to be fiddling about trying to dial in the perfect depth and p-value each time.

No ask, just curious because I had your exact though this weekend "man, we need an adaptive n-max or something." and then found spec-draft-p-min and it gave me the sort of results I was looking for, with Qwen3.8.

It's also been my experience that raising --spec-draft-p-min, whether it be normal or adaptive MTP, absolutely destroys prose performance. With the rise of deep thinking models (like our good friend Qwen3.8 here) there's likely to be significant amounts of time spent generating prose before code gets written.

This is why I've been focusing on preventing/minimising harm to prose performance. We don't want trash that thinking performance just to get some more t/s on code generation and end up slower overall.

Your PR is on my list to test.

[bucknova]

Tested the fixed-depth equivalent of this on the target hardware/backend this PR doesn't cover yet — Intel Arc Pro B70 (BMG G31, 32GB), llama.cpp SYCL via LocalAI intel-sycl-f16-llama-cpp gallery build 2026-08-03, Qwen3.8-27B UD-Q4_K_XL, 256k ctx, q8_0 KV. Fixed draft-mtp depth sweep, 700-token decode, single stream:

depth	t/s
2	~29
3	32
4	31
6	24.2

Draft acceptance 75–90% (mean accepted length 2.5–4.7) across runs. Two observations that support the adaptive approach directly:

1. The optimum is sharp and shallow. Peak at 3, and 6 is a 25% loss — deeper drafting actively hurts on this backend. Your default floor of 3 is exactly where I landed by hand.

2. The penalty is dispatch-bound, so it's worse on SYCL than CUDA/ROCm. Each extra draft step pays the SYCL per-kernel dispatch cost (100–500µs vs ~5µs CUDA), so the "keep it at the floor unless acceptance earns it deeper" logic should pay off proportionally more here than on the platforms already benchmarked.

Happy to run the adaptive build against the same 700-token workload on this B70 once it's testable and post the side-by-side.

[Us5rName]

@stew675

Do you have any performance comparisons of your implementation and how it affects both prose and code? I tried pretty hard to ensure that prose performance was affected as little as possible, so I'm curious to see results on how you solved that issue.

I ran my heuristic against static length in speedbench with qwen 3.6 35B and mtp

roleplay category was used for prose
coding category was used for coding

*note: on my hardware (rtx 4080 super + rx 7900 xtx, both using vulkan backend), I noticed the maximum draft length that doesn't hurt my performance was 5, and the minimum draft length that didn't hurt it was 2.

draft length 2 was used for prose, and draft length 5 was used for coding.

got very similar results in both the non adaptive case and the adaptive case.

Speedbench, qwen 3.6 35B UD-Q4_K_XL, using mtp:

coding, draft length 5: 157.70 tps
coding, adaptive draft length 2-5: 163.19 tps

roleplay, draft length 2: 147.71 tps
roleplay, adaptive draft length 2-5: 149.08 tps

In my personal experience with my fork, the draft length of my heuristic usually stabilizes at 2 tokens for prose,
3-4 tokens in math,
and 4-5 tokens in code.

Tell me if there are additional files/logs you would like to see.

Your personal branch seems to target DFlash too, right?

Yes. By targeting Dflash I meant I copy pasted the code from my mtp implementation into Dflash - the heuristic works exactly the same in both cases.

In the code, Because the draft length update is checked on every accept(), which every draft spec implementation has, it can absolutely be ported to draft-simple and draft-eagle3, I don't use them so I didn't apply the heuristic to them.

I now also deduplicated code in my fork by using your approach of putting the heuristic in a header, and added a short explanation on the algorithm I used, so it should be easier to read and trivial to port to other draft implementations.

I looked at your code, and it seems like adding the same heuristic you implemented in mtp to Dflash (and other spec types) should be simple, and I think adding your heuristic to Dflash as well is a good experiment because even with Dflash generating all tokens at once, the target model can still try to verify "useless tokens" (In my experiments, my heuristic worked well on Dflash)

I'm fine with us combining efforts, but we'd need to settle on what the UX approach should be. I'm a big fan of KISS. Years ago I would've had knobs for everything, but nowadays I'm a firm believer of use as few knobs as possible.

I agree I that have too many knobs, and when I use my fork I always use --spec-adaptive-length-default (sets my knobs to params that worked well for me)

However, I still think that 1-3 knobs of configuration is still good to have, to let people experiment.

I think a good approach would be like with n-gram speculative decoding - configurable but includes a parameter that sets sane defaults (--spec-default).
What do you think?

[stew675]

@stew675 , small enhancement from my side:

diff --git a/common/speculative.cpp b/common/speculative.cpp
index 0c7fb38d9..459eea494 100644
--- a/common/speculative.cpp
+++ b/common/speculative.cpp
@@ -1457,6 +1457,12 @@ struct common_speculative_impl_draft_mtp : public common_speculative_impl {
             return;
         }

+        // new generation: the depth learned for the previous content is stale,
+        // so the controller starts from the floor again
+        if (adaptive) {
+            adaptive_ctrl[seq_id].reset(params.n_max, params.n_min_adaptive);
+        }
+
         auto * ctx_dft = this->params.ctx_dft;
         const llama_pos pos_max = llama_memory_seq_pos_max(llama_get_memory(ctx_dft), seq_id);

Its a fix for issue with adaptive which has depth from previous state... If you want just pick it up.... however, if you intend to use previous state and let adaptive decrease/increase N this will create unnecessary work

Ooh, good catch! Thank you for that. I see that another work-flow hasn't been approved yet, so now's a good time to merge that in.

[stew675]

@theIvanR

Thank you for the detailed response. You're drawing the correct distinction here and
the "effective speculative memory horizon" is more sharply focused than what I had
written.

To clarify: my "~50–200 tokens" was about the controller's own state. The depth and
the evidence behind it have a bounded lifetime defined by its implementation. After
acceptance drops, the current draft depth is capable of draining entirely within a
few tens of rounds. I was expressing how long the algorithm remembers its own decisions,
and not how the draft model conditions on context. I understand that when taken literally
my sentence can be perceived as an over simplification, and your examples prove that.

The data supports your conflation point. The same model and backend show a floor of 3
on prose and a sustained ceiling of 12 on verbatim recall, so the acceptance derived
"memory" tracks local predictability, not model structure. Recall is also the
where deeper drafting pays hugely, so the benefit is very much workload-dependent, even
within one model. The mechanism is structurally local, ie. bucket/depth state, and
lookahead is capped at 12. This means the algorithm's measurement never has a long
range channel to observe. In these scenarios it is wholly reliant upon the model's
prediction.

Regarding the Oracle: it's an offline perfect-foresight replay for ranking controller
algorithm variants, with its own biases with traces truncating at live depth and the
reasoning/prose headrooms are easily overstated by a factor of three. For coding it's
arguably even more over-stated because there was some variance here caused by identical
logit probabilities, and taking one path gives a higher prediction than the other path.

With regards to controller memory though it predicts an even shorter algorithmic optimum.
It found that just 1-3 tokens of memory was good enough for the controller to do an
effective job, but this obviously ignores the impact of hardware churn. That prediction
must be grounded against the hardware impact of adopting such an approach, which is
almost certainly going to be disastrous.

On the topic of robustness, I do agree. It's just one model, one backend, and one
sampling config with 2 repeats, and one prompt per lane. The experiment that I think
holds the most value here is your third suggestion of planting a long-range constraint,
and then compare acceptance on anchor-dependent tokens vs matched local controls.
That would settle whether the horizon is a workload property or a mechanism property.

So your framing is correct. Adaptive MTP learns an effective speculative memory horizon
under these workloads and nothing about the target model's distribution follows from it.

[stew675]

It looks like the 5 failed checks are all known flakes independent of this PR's changes.

I don't intend to work on this PR further. Barring blocking issues being found, if any followup work is needed they should be handled in follow-up PR's.

[stew675]

Forced pushed in order to rebase against current master tip and resolve a handful of merge conflicts due to drift.

[stew675]

@am17an sorry to bug you again as I know the upstream developers are all busy working on exciting new model support, but I'd like to know if this PR here of any interest to the llama.cpp maintainers? I'm reaching out to you since you were responsible for most of the excellent MTP work that this is building upon.

I'd just like to know if I should keep holding out hope for this PR, or if the maintainers have other plans instead? A good number of people have independently proven that this PR does exactly what it says on the tin, so to speak, across a number of architectures.

It just seems like a free 10% boost and an improved user configuration experience is a lot to sleep on.

[satoyon]

I've been testing this Adaptive MTP implementation for a week, and the results are excellent.
One major pain point in Japanese language modeling is that standard MTP doesn't always function efficiently, preventing us from increasing spec-draft-n-max values. This unfortunately limits performance even in high-potential contexts like programming.
I have verified that Adaptive MTP solves this by optimizing performance based on the context. It would be fantastic to see this merged into the main branch.

[47Hunter47]

Long-context data point: on a single RTX 3090 (Qwen3.6-27B UD-Q3_K_XL, GPU KV q4_0, FA on) we compared fixed draft depths across 20K-120K context on a diverse corpus. n-max=4 beat n-max=3 at every level above 55K (+19% at 65K, +23% at 75K) and only lost at short context.

So the drop-pressure side of the state machine should be conservative at long context: in our measurements the deeper draft keeps paying off as context grows.

[satoyon]

I merged PR #27210 into the latest master branch. Adaptive MTP works correctly on models like Gemma-4, but Qwen3.8-27B fails to load with an access violation crash. I'd like to report this issue.

Build steps

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout -b custom-adaptivemtp
git fetch origin pull/27210/head:pr-27210
git merge pr-27210
... build...

Configuration

[Qwen3.8-27B-Q8_0]
model = D:\Users\yoneda\Documents\my_models\ggml-org\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q8_0.gguf
mmproj = D:\Users\yoneda\Documents\my_models\ggml-org\Qwen3.8-27B-GGUF\mmproj-Qwen3.8-27B-BF16.gguf
md = D:\Users\yoneda\Documents\my_models\ggml-org\Qwen3.8-27B-GGUF\mtp-Qwen3.8-27B-Q8_0.gguf
ctx-size = 131072
spec-type = draft-mtp-adaptive
spec-draft-n-max = 12
spec-draft-n-min-adaptive = 2
backend-sampling = on
batch-size = 4096
ubatch-size = 4096
image-min-tokens = 1024 
temp = 0.7
repeat-penalty = 1.1
reasoning-effort = medium
cache-type-k = q8_0
cache-type-v = q8_0

Error log when loading Qwen3.8-27B

[56673] 0.00.069.387 I srv    load_model: loading model 'D:\Users\yoneda\Documents\my_models\ggml-org\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q8_0.gguf'
[56673] Exception Code: 0xC0000005
[56673] 0x00007FFF8CBA8266, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\ggml-base.dll(0x00007FFF8CBA0000) + 0x8266 byte(s), ggml_mul_mat() + 0x6 byte(s)
[56673] 0x00007FFF7B876413, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x56413 byte(s), llama_max_parallel_sequences() + 0x1C993 byte(s)
[56673] 0x00007FFF7BA4C477, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22C477 byte(s), llama_detokenize() + 0xE9877 byte(s)
[56673] 0x00007FFF7BA4BF98, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22BF98 byte(s), llama_detokenize() + 0xE9398 byte(s)
[56673] 0x00007FFF7BA4BD95, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22BD95 byte(s), llama_detokenize() + 0xE9195 byte(s)
[56673] 0x00007FFF7B9215E7, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x1015E7 byte(s), llama_model_rope_type() + 0xAEF7 byte(s)
[56673] 0x00007FFF7B849CA1, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x29CA1 byte(s), llama_get_memory() + 0x1EE1 byte(s)
[56673] 0x00007FFF7B8492BD, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x292BD byte(s), llama_get_memory() + 0x14FD byte(s)
[56673] 0x00007FFF7B849215, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x29215 byte(s), llama_get_memory() + 0x1455 byte(s)
[56673] 0x00007FFF7B848008, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x28008 byte(s), llama_get_memory() + 0x248 byte(s)
[56673] 0x00007FFF7B8465A5, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x265A5 byte(s), llama_chat_builtin_templates() + 0x3425 byte(s)
[56673] 0x00007FFF7B852EA8, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x32EA8 byte(s), llama_init_from_model() + 0x2C8 byte(s)
[56673] 0x00007FFF0883D631, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x1CD631 byte(s), ?common_get_device_memory_data@@YA?AV?$vector@Ucommon_device_memory_data@@V?$allocator@Ucommon_device_memory_data@@@std@@@std@@PEBDPEBUllama_model_params@@PEBUllama_context_params@@AEAV?$vector@PEAUggml_backend_device@@V?$allocator@PEAUggml_backend_device@@@std@@@2@AEAI44W4ggml_log_level@@@Z() + 0x321 byte(s)
[56673] 0x00007FFF088481B9, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x1D81B9 byte(s), ?common_fit_print@@YAXPEBDPEAUllama_model_params@@PEAUllama_context_params@@@Z() + 0x529 byte(s)
[56673] 0x00007FFF0883E10B, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x1CE10B byte(s), ?common_fit_params@@YA?AW4common_params_fit_status@@PEBDPEAUllama_model_params@@PEAUllama_context_params@@PEAMPEAUllama_model_tensor_buft_override@@PEA_KIPEBUcommon_fit_extra_model@@W4ggml_log_level@@@Z() + 0x53B byte(s)
[56673] 0x00007FFF0883DC4D, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x1CDC4D byte(s), ?common_fit_params@@YA?AW4common_params_fit_status@@PEBDPEAUllama_model_params@@PEAUllama_context_params@@PEAMPEAUllama_model_tensor_buft_override@@PEA_KIPEBUcommon_fit_extra_model@@W4ggml_log_level@@@Z() + 0x7D byte(s)
[56673] 0x00007FFF08805F4B, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x195F4B byte(s), ??0common_init_result@@QEAA@AEAUcommon_params@@_N@Z() + 0x40B byte(s)
[56673] 0x00007FFF08808379, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF08670000) + 0x198379 byte(s), ?common_init_from_params@@YA?AV?$unique_ptr@Ucommon_init_result@@U?$default_delete@Ucommon_init_result@@@std@@@std@@AEAUcommon_params@@_N@Z() + 0x49 byte(s)
[56673] 0x00007FFEB356AA46, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFEB34D0000) + 0x9AA46 byte(s), ??0?$_Hash@V?$_Umap_traits@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@V?$_Uhash_compare@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@U?$hash@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@U?$equal_to@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@@2@V?$allocator@U?$pair@$$CBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@@std@@@2@$0A@@std@@@std@@IEAA@$$QEAV01@@Z() + 0x1C8C6 byte(s)
[56673] 0x00007FFEB34D699E, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFEB34D0000) + 0x699E byte(s), ?llama_server@@YAHAEAUcommon_params@@HPEAPEAD@Z() + 0x57EE byte(s)
[56673] 0x00007FFEB34D115A, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFEB34D0000) + 0x115A byte(s), ?llama_server@@YAHHPEAPEAD@Z() + 0x9A byte(s)
[56673] 0x00007FF6027211F8, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server.exe(0x00007FF602720000) + 0x11F8 byte(s)
[56673] 0x00007FFFACFDCCB7, C:\windows\System32\KERNEL32.DLL(0x00007FFFACFB0000) + 0x2CCB7 byte(s), BaseThreadInitThunk() + 0x17 byte(s)
[56673] 0x00007FFFAE86AD6C, C:\windows\SYSTEM32\ntdll.dll(0x00007FFFAE7C0000) + 0xAAD6C byte(s), RtlUserThreadStart() + 0x2C byte(s)
0.16.392.416 I srv    operator(): instance name=Qwen3.8-27B-Q8_0 exited with status -1073741819

The crash occurs at ggml_mul_mat() + 0x6, which suggests a null or invalid tensor pointer is being passed — possibly the MTP tensor is not being resolved correctly during common_fit_params for this model architecture.

[stew675]

@satoyon

Does this still happen with --fit off?

The backtrace indicates that this is happening even before the model weights have loaded.

Another thing to try is to lower --spec-draft-n-max to, say, 5 and see if that fixes it. While the backtrace doesn't explicitly indicate an OOM error, a large draft-n-max does consume more memory on the GPU, and it's possible that a memory allocation check got skipped over somewhere, and something down the road is faulting trying to write into memory that isn't there. A lot of this pre-allocation stuff is rather fragile.

Basically the backtrace has failed at a point before the Adaptive MTP stuff even really kicks in. This is the pre-allocation part of the setup, so my immediate guess is a memory buffer overflow from a too large n-max value, or the good old: "You must use --fit off with MTP" issue.

If it still persists after these changes then let me know.

[satoyon]

Thanks for the quick response. I tried both --fit on and lowering --spec-draft-n-max to 5, but unfortunately the crash persists with the same backtrace. Here's the error log:

[49973] 0.00.072.045 I srv    load_model: loading model 'D:\Users\yoneda\Documents\my_models\ggml-org\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q8_0.gguf'
[49973] Exception Code: 0xC0000005
[49973] 0x00007FFF97D58266, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\ggml-base.dll(0x00007FFF97D50000) + 0x8266 byte(s), ggml_mul_mat() + 0x6 byte(s)
[49973] 0x00007FFF7B876413, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x56413 byte(s), llama_max_parallel_sequences() + 0x1C993 byte(s)
[49973] 0x00007FFF7BA4C477, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22C477 byte(s), llama_detokenize() + 0xE9877 byte(s)
[49973] 0x00007FFF7BA4BF98, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22BF98 byte(s), llama_detokenize() + 0xE9398 byte(s)
[49973] 0x00007FFF7BA4BD95, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x22BD95 byte(s), llama_detokenize() + 0xE9195 byte(s)
[49973] 0x00007FFF7B9215E7, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x1015E7 byte(s), llama_model_rope_type() + 0xAEF7 byte(s)
[49973] 0x00007FFF7B849CA1, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x29CA1 byte(s), llama_get_memory() + 0x1EE1 byte(s)
[49973] 0x00007FFF7B8492BD, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x292BD byte(s), llama_get_memory() + 0x14FD byte(s)
[49973] 0x00007FFF7B849215, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x29215 byte(s), llama_get_memory() + 0x1455 byte(s)
[49973] 0x00007FFF7B848008, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x28008 byte(s), llama_get_memory() + 0x248 byte(s)
[49973] 0x00007FFF7B8465A5, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x265A5 byte(s), llama_chat_builtin_templates() + 0x3425 byte(s)
[49973] 0x00007FFF7B852EA8, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama.dll(0x00007FFF7B820000) + 0x32EA8 byte(s), llama_init_from_model() + 0x2C8 byte(s)
[49973] 0x00007FFF6483D631, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x1CD631 byte(s), ?common_get_device_memory_data@@YA?AV?$vector@Ucommon_device_memory_data@@V?$allocator@Ucommon_device_memory_data@@@std@@@std@@PEBDPEBUllama_model_params@@PEBUllama_context_params@@AEAV?$vector@PEAUggml_backend_device@@V?$allocator@PEAUggml_backend_device@@@std@@@2@AEAI44W4ggml_log_level@@@Z() + 0x321 byte(s)
[49973] 0x00007FFF648481B9, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x1D81B9 byte(s), ?common_fit_print@@YAXPEBDPEAUllama_model_params@@PEAUllama_context_params@@@Z() + 0x529 byte(s)
[49973] 0x00007FFF6483E10B, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x1CE10B byte(s), ?common_fit_params@@YA?AW4common_params_fit_status@@PEBDPEAUllama_model_params@@PEAUllama_context_params@@PEAMPEAUllama_model_tensor_buft_override@@PEA_KIPEBUcommon_fit_extra_model@@W4ggml_log_level@@@Z() + 0x53B byte(s)
[49973] 0x00007FFF6483DC4D, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x1CDC4D byte(s), ?common_fit_params@@YA?AW4common_params_fit_status@@PEBDPEAUllama_model_params@@PEAUllama_context_params@@PEAMPEAUllama_model_tensor_buft_override@@PEA_KIPEBUcommon_fit_extra_model@@W4ggml_log_level@@@Z() + 0x7D byte(s)
[49973] 0x00007FFF64805F4B, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x195F4B byte(s), ??0common_init_result@@QEAA@AEAUcommon_params@@_N@Z() + 0x40B byte(s)
[49973] 0x00007FFF64808379, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-common.dll(0x00007FFF64670000) + 0x198379 byte(s), ?common_init_from_params@@YA?AV?$unique_ptr@Ucommon_init_result@@U?$default_delete@Ucommon_init_result@@@std@@@std@@AEAUcommon_params@@_N@Z() + 0x49 byte(s)
[49973] 0x00007FFF64D0AA46, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFF64C70000) + 0x9AA46 byte(s), ??0?$_Hash@V?$_Umap_traits@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@V?$_Uhash_compare@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@U?$hash@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@U?$equal_to@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@@2@V?$allocator@U?$pair@$$CBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@V12@@std@@@2@$0A@@std@@@std@@IEAA@$$QEAV01@@Z() + 0x1C8C6 byte(s)
[49973] 0x00007FFF64C7699E, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFF64C70000) + 0x699E byte(s), ?llama_server@@YAHAEAUcommon_params@@HPEAPEAD@Z() + 0x57EE byte(s)
[49973] 0x00007FFF64C7115A, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server-impl.dll(0x00007FFF64C70000) + 0x115A byte(s), ?llama_server@@YAHHPEAPEAD@Z() + 0x9A byte(s)
[49973] 0x00007FF6027211F8, D:\Users\yoneda\Documents\ROCm\llama-adaptive-mtp\llama-server.exe(0x00007FF602720000) + 0x11F8 byte(s)
[49973] 0x00007FFFACFDCCB7, C:\windows\System32\KERNEL32.DLL(0x00007FFFACFB0000) + 0x2CCB7 byte(s), BaseThreadInitThunk() + 0x17 byte(s)
[49973] 0x00007FFFAE86AD6C, C:\windows\SYSTEM32\ntdll.dll(0x00007FFFAE7C0000) + 0xAAD6C byte(s), RtlUserThreadStart() + 0x2C byte(s)
0.12.097.873 I srv    operator(): instance name=Qwen3.8-27B-Q8_0 exited with status -1073741819
2.

[satoyon]

Sorry, my mistake — I had --fit on when I tested earlier. With --fit off, the model loads without any issues.
Thanks for the help!

[stew675]

Sorry, my mistake — I had --fit on when I tested earlier. With --fit off, the model loads without any issues. Thanks for the help!

All good. I'm just glad we got you sorted out in the end.

[simongonzalezdc]

Follow-up from our August 18 datapost — same Strix Halo box, but this time at the PR head (45e3d26) with the embedded-MTP draft and the adaptive controller, not ngram-mod. Qwen3.8-27B UD-Q4_K_XL from unsloth, -fit off, 16K context, q4_0 KV, temp 0.2, 300 max tokens. Five code and five prose prompts, two reps each — n=10 per cell. Throughput is the median wall-clock tok/s over those ten; acceptance is the share of drafted tokens the target model accepted.

• adaptive [3..12] — code 18.61 tok/s (61.7% accepted), prose 14.88 (41.9%)
• fixed n=4 — code 17.65 (53.5%), prose 13.98 (35.4%)
• fixed n=12 — code 17.72 (24.5%), prose 11.79 (13.2%)

Code is a wash on this run — the edge adaptive shows there is smaller than the run-to-run spread we see with this setup, so I'm not claiming anything from it.

Prose is where it shows: adaptive is ~6% faster than fixed-4 and accepts 6.5 points more of what it drafts.

The fixed-12 row is the one I keep looking at. 13.2% acceptance means most of those deep drafts were verification work that went nowhere, and on this chip drafts share memory bandwidth with the target so that work isn't free. This is the lane I called out in the August post — deep drafting into low-acceptance traffic — showing up in one run.

We ran -fit off throughout on Linux with this quant — zero load failures.

Happy to share the raw per-request jsonl, or run whatever config anyone wants on this box.

[Hudendudel]

Been using --spec-type draft-mtp-adaptive --spec-draft-n-max 12 for about 2 weeks now with Qwen3.8-27B-UD-Q4_K_XL on the R9700 on Linux/ROCm. It's one of the few methods that speed up both coding and prose in a single setting without sacrificing either. Getting roughly 34 tok/s on prose, 72 tok/s on code. As a comparison, before that i used a static n-max of 3 as a middle ground which gave about also 34 tok/s for prose yet 53 tok/s for code. Would be great to see this land in main. @CISC @ggerganov

[simongonzalezdc]

Follow-up from our August 18 datapost — same Strix Halo box, but this time at the PR head (45e3d26) with the embedded-MTP draft and the adaptive controller, not ngram-mod. Qwen3.8-27B UD-Q4_K_XL from unsloth, -fit off, 16K context, q4_0 KV, temp 0.2, 300 max tokens. Five code and five prose prompts, two reps each — n=10 per cell. Throughput is the median wall-clock tok/s over those ten; acceptance is the share of drafted tokens the target model accepted.

• adaptive [3..12] — code 18.61 tok/s (61.7% accepted), prose 14.88 (41.9%)
• fixed n=4 — code 17.65 (53.5%), prose 13.98 (35.4%)
• fixed n=12 — code 17.72 (24.5%), prose 11.79 (13.2%)

Code is a wash on this run — the edge adaptive shows there is smaller than the run-to-run spread we see with this setup, so I'm not claiming anything from it.

Prose is where it shows: adaptive is ~6% faster than fixed-4 and accepts 6.5 points more of what it drafts.

The fixed-12 row is the one I keep looking at. 13.2% acceptance means most of those deep drafts were verification work that went nowhere, and on this chip drafts share memory bandwidth with the target so that work isn't free. This is the lane I called out in the August post — deep drafting into low-acceptance traffic — showing up in one run.

We ran -fit off throughout on Linux with this quant — zero load failures.

Happy to share the raw per-request jsonl, or run whatever config anyone wants on this box.

Small precision note on my post above. Throughput is the median wall-clock tok/s across the n=10 runs; acceptance is the aggregate share of drafted tokens the target model accepted across those runs. The adaptive-vs-fixed-4 prose throughput delta is +6.4%. The code comparison is descriptive of this single run only; I should not have characterized it beyond that. Finally, the hardware explanation in that post was not measured and should not be treated as a result.

[JCraigWasTaken]

I tried this on two AMD MI50 cards (gfx906) running Qwen3.8-27B at 8-bit, split across both cards, on the mxxm gfx906 fork of llama.cpp (https://github.com/mxxm-t/mx-llama.cpp) at its b10811 build. Sampling at temperature 0.8, reasoning off, compared against the fixed 3-token draft.

Letting the draft length move between 2 and 6: a file-rewrite prompt went from 75.7 to 79.8 tok/s, fresh code went from 68-71 down to 66.3, and fresh prose from 42.4 down to 41.3, with the length changing 36 times over the run. Between 2 and 4: 79.4 / 69.8 / 41.5 on the same three prompts.

I think the loss on code and prose comes from how this backend caches its GPU work: it keeps a separate recorded version per draft length, every new length costs a few slow rounds to set up, and lengths that are only used now and then get their recorded version dropped as unstable. Both showed up in the logs during these runs.

The other change in this PR, starting the recurrent-state copy at the first slot a rollback can reach, helped on its own: with the fixed 3-token draft, prose went from 41.3 to 42.4 tok/s with it. It might be worth splitting that part out so it can go in separately.

Happy to rerun with other settings if that helps.

[stew675]

@JCraigWasTaken

Limiting the depth range from 2-6 isn't really expected to make a huge difference from, say, setting a fixed MTP depth of 3. There are always short bursts, even for predictable content, where the acceptance rates collapse for short periods, and it's in those periods that the algorithm needs enough of a buffer to ride over those short collapses.

When the algorithm is prevented from scaling to the higher depths where the biggest speedups are found for predictable content, then those short periods of collapsed acceptance means that it will fall back to assuming that it's working with prose again far too quickly, and so it's not being given the chance to ride those sweet deep predictive highs for long enough for it to provide a real speedup.

It should be noted that for llama.cpp, adjusting the depth does come with a small temporary performance hit, and the algorithm aims to amortise that cost through sustaining the high depths long enough to hide it. When the peak depth is set too low, it cannot do that. In my belief therefore that the algorithm working given the constraints you've set for it to work under.

If, for memory reasons, that you'd like to reduce the upper range, then 8 would be about the lower end of where that should be. As the max-gain is dropped much below 8, then the algorithm essentially collapses back to not being a whole lot different to simply running a fixed depth of 2, 3, or 4.

In a car analogy sense, you've basically put a speed governor on a Ferrari that limits it from going faster than the regular speed limit, and are then wondering why the car can't reach its destination any faster than a Toyota.

I would guess, if anything, what probably needs to change here is to have the adaptive MTP code emit a warning at startup that if the maximum depth has been set to a value of 6 or lower that it's probably best to just stick with normal fixed MTP instead. Even a max depth of 7 is going to be kind of borderline. A maximum depth of 8 is the lowest I would set the maximum at to be sure of starting to see what the algorithm aims to deliver.

I'll add a note to the PR description to point this out.

[JCraigWasTaken]

Thanks, that is a fair point about the cap. I reran with the ranges you suggested, same setup as before: two AMD MI50 cards, Qwen3.8-27B at 8-bit split across both, the mxxm gfx906 fork at b10811, temperature 0.8, reasoning off. Three prompts (a file rewrite, fresh code, fresh prose), code and prose twice each. Decode speed in tokens per second; the fixed 3-token draft was run twice as a control.

	file rewrite	fresh code	fresh prose
fixed 3 (first run)	76.1	70.4	43.3
adaptive 2..8	64.3	50.3	42.7
adaptive 3..12	77.6	48.1	42.8
fixed 3 (second run)	75.8	70.3	41.7

The controller did what you describe. On code it climbed to depth 9 to 12 and stayed there for most of the run (58 depth changes, 41 of them between 9 and 12). Acceptance per position held up out to the deep positions, for example 0.93, 0.87, 0.79, 0.64, 0.51, 0.41, 0.36, 0.29, 0.21, 0.14, 0.09, 0.05 on one code prompt, and the mean accepted length per round went from 3.7 tokens to about 6.3.

It still lost 30 percent on code because each round got much more expensive. From the server's own timings, one round at fixed depth 3 took about 53 ms, of which 9 ms was drafting. At depth 12 a round took about 131 ms, of which 24 ms was drafting. So verifying 13 tokens cost about 2.5 times as much as verifying 4 on this hardware, while the round produced 1.7 times as many tokens. I since measured the forward-pass cost against batch size directly on this build: 31 ms at 1 token, 42 at 4, 69 at 8, then a jump to 107 at 9 and flat to 32, which is the backend's kernel switching from a narrow mat-vec to a 32-column tile. Details in mxxm-t#10. On a card where a 13-token batch is still bandwidth-bound that cost would not show, which would explain why your numbers and ours differ.

Prose was a wash at every setting. Acceptance at position 1 is under 50 percent on prose here, so the controller sat at the floor. The file rewrite gained 2 percent at 3..12 and lost 15 percent at 2..8, where the depth moved 45 times over the run.

I agree a startup warning for low caps makes sense. It may also be worth capping on measured round time rather than on depth alone, since on compute-limited cards the best depth is low even when acceptance is high. Raw logs are available if useful.

[stew675]

@JCraigWasTaken

At this stage it sounds like the algorithm just isn't a good fit for your hardware. The cost of the drafting at high depth is high enough that it offers no gains for you. In your scenario, it may just be best to stick with the normal MTP use case. It's interesting because yours is the first scenario reported where it hasn't provided the proposed gains. Can't win 'em all I guess, and this is EXACTLY why I made this feature an explicitly optional configuration.