[Feature]: Speculative Decoding

[aropb]

Background & Description

ggml-org/llama.cpp#22673

API & Usage

No response

How to implement

No response

[Kago-Ryu]

Looking forward to seeing this implemented in LlamaSharp. Thanks for the hard work!

[martindevans]

MTP (or any speculative decoding) is not going to be easy to add support for unfortunately.

It's not implemented as part of the core llama.h, which is what we wrap. Instead it's implemented in common/speculative.h, which doesn't look suitable for wrapping (it doesn't offer a simple c-api). We would have to reimplement a lot of it (and keep that implementation up to date as upstream changed).

Realistically for this to happen I think some sort of speculative interface will need to be moved into llama.h.