docs(llama-cpp): clarify multimodal speculative decoding

[aim9sour]

Description

Upstream llama.cpp removed the former general multimodal/speculative restriction in ggml-org/llama.cpp#19493, while ggml-org/llama.cpp#22673 later added MTP support and explicitly documented compatibility with vision input. This page still said LocalAI automatically disabled speculation whenever mmproj was active.

This PR:

• documents that current llama.cpp backends can use mmproj with speculative decoding, including MTP;
• keeps the backend-version and model-architecture compatibility caveats;
• tells operators to verify projector/speculative initialization and the draft acceptance counts;
• corrects the mmproj_use_gpu guidance so vision users do not remove the projector unnecessarily;
• adds a combined vision + MTP model configuration example; and
• removes the adjacent unsupported claim that n_draft can be overridden through each OpenAI request.

No issue was opened because this is a focused correction to one stale documentation section and does not require a code change.

Runtime evidence

Tested the official qwythos-9b-v2 Gallery configuration, which combines an embedded MTP head with mmproj, using a real base64 image request on an NVIDIA RTX PRO 2000 Blackwell (16,311 MiB):

• LocalAI v4.8.2-gpu-nvidia-cuda-12 (sha256:90b8eb9505b4ea5f60e6f1057621a96100cde559fb4d92dd743ed5091b6b4b9f) with its default installed llama.cpp backend.
• LocalAI master-gpu-nvidia-cuda-12, commit 38ba3fec636e684f81320e5d8ab497f5f1e985a0 (sha256:65487335cacd5395fd16c1bf78d3d23f90a41911bc7e98f663212352a4a84dd7).
• The same master image with the explicitly installed cuda12-llama-cpp-development backend (master-gpu-nvidia-cuda-12-llama-cpp, amd64 digest sha256:805c89619771eb12ac0261f9c1a60c2105d38f816e603a287d242c3c9f15ec3c).

All three runs logged both:

• creating MTP draft context against the target model
• loaded multimodal model, '...mmproj-Qwythos-9B-v2-BF16.gguf'

The image request completed successfully, and both recorded inference runs reported:

draft acceptance = 0.97143 (102 accepted / 105 generated)

Verification

• hugo v0.146.3+extended ... --minify — built 223 documentation pages successfully (matching the version pinned in Pages CI)
• go test ./core/config --count=1
• git diff --check upstream/master..HEAD
• Independent read-only review: no remaining Critical, Important, or Minor findings

Notes for Reviewers

The current LocalAI gRPC bridge forwards MMProj and the speculative settings independently. Upstream #19493 removed the old general multimodal/speculative exclusion; #22673 later added MTP with explicit vision-input compatibility.

AI assistance is disclosed in the commit's Assisted-by trailer in accordance with the repository policy.

Signed commits

• Yes, I signed my commits.
• Documentation updated (docs/content/) for user-facing changes, or not applicable

[aim9sour]

@localai-org-maint-bot please review this documentation correction and its runtime validation.

[localai-org-maint-bot]

Reviewed the documentation correction against the current LocalAI llama.cpp bridge and pin. The mmproj and speculative settings are forwarded independently, the pinned upstream revision contains the multimodal/speculative support change, and the mmproj_use_gpu guidance matches the option parser. The example and compatibility caveats look accurate, git diff --check passes, and DCO is green. Good to merge, @mudler.

[aim9sour]

Follow-up verification of the upstream attribution:

• llama.cpp #19493 removed the former general multimodal/speculative disable and rejection paths.
• llama.cpp #22673 later added MTP support and explicitly documented compatibility with vision input.

Updated the documentation and PR description in f9c2859 so each PR is cited only for its exact role. Fresh verification: Hugo 0.146.3 built all 223 pages, go test ./core/config passed, and git diff --check passed.

[aim9sour]

@localai-org-maint-bot please re-review the follow-up attribution clarification in f9c2859. It distinguishes #19493 (removal of the general multimodal/speculative restriction) from #22673 (MTP support with explicit vision-input compatibility).

[aim9sour]

@localai-org-maint-bot please submit an updated review for current head f9c2859. The follow-up precisely distinguishes #19493 (general multimodal/speculative restriction removal) from #22673 (MTP with explicit vision-input compatibility); no review threads or CI failures are open.

[localai-org-maint-bot]

Review pass. The LocalAI half of the claim checks out, with one caveat about the upstream half.

Verified: nothing in backend/cpp/llama-cpp/grpc-server.cpp gates speculative decoding on mmproj being set. grpc-server.cpp:494 just forwards params.mmproj.path, and :917 handles mmproj_use_gpu independently. So the old note at docs/content/advanced/model-configuration.md:392 was stating a restriction LocalAI does not implement, and removing it is right. The new known_usecases: [chat, vision] example is valid: GetUsecasesFromYAML (core/config/model_config.go:2035-2050) accepts the bare lowercase form via stringToFlag, and UsecaseVision = "vision" exists at core/config/backend_capabilities.go:16.

Caveat: I could not verify the upstream half. The doc now cites two upstream llama.cpp PR numbers as load-bearing evidence, including #19493. LLAMA_VERSION is pinned at d59d455fd8ea (backend/cpp/llama-cpp/Makefile:2) and there is no vendored checkout, so confirming that the pin actually contains #19493 needs a clone at that sha. If the pin predates it, the note becomes wrong in the other direction.

Either confirm the pin contains it, or soften the wording to "recent llama.cpp" so the claim cannot go stale on the next pin bump. Otherwise this is good.

Note that CI has never run on this PR: the workflow runs are action_required and expired.