From 68b327656c267f86f9983f91cb85bc24e654e35a Mon Sep 17 00:00:00 2001 From: "Jakub A. W" Date: Sun, 16 Aug 2026 20:12:59 +0200 Subject: [PATCH 1/3] docs(providers): add llama.cpp / LM Studio guide llama.cpp users have no setup guidance today (the embeddings-classification report in #689 came from one). Documents registering llama-server as a vLLM-type provider, the pooling requirement for /v1/embeddings, ID-based model classification, and what llama.cpp does not serve (rerank via passthrough only; no image-generation or OpenAI audio endpoints). Co-Authored-By: Claude Fable 5 --- docs/docs.json | 1 + docs/providers/llamacpp.mdx | 101 ++++++++++++++++++++++++++++++++++++ docs/providers/overview.mdx | 5 ++ 3 files changed, 107 insertions(+) create mode 100644 docs/providers/llamacpp.mdx diff --git a/docs/docs.json b/docs/docs.json index 3b12f3ede..bc847b7e7 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -177,6 +177,7 @@ "providers/opencode-go", "providers/sglang", "providers/vllm", + "providers/llamacpp", "providers/llmd", "providers/multiple-ollama", "providers/kimicode", diff --git a/docs/providers/llamacpp.mdx b/docs/providers/llamacpp.mdx new file mode 100644 index 000000000..3e864fd99 --- /dev/null +++ b/docs/providers/llamacpp.mdx @@ -0,0 +1,101 @@ +--- +title: "llama.cpp" +description: "Route OpenAI-compatible GoModel requests to a llama.cpp llama-server instance (the same recipe works for LM Studio)." +icon: "server" +keywords: ["llama.cpp", "llama-server", "LM Studio", "local models", "self-hosted", "OpenAI-compatible server"] +--- + +llama.cpp's `llama-server` speaks the OpenAI API directly, so GoModel needs no +dedicated provider type for it: register the server as a **vLLM-type provider**. +That type fits llama.cpp exactly — the API key is optional (llama-server +usually runs keyless), and slash-shaped model IDs work. Do **not** register it +as an +`ollama` provider: that type speaks Ollama's native API, which llama.cpp does +not implement. + +The same recipe applies to LM Studio and any other plain OpenAI-compatible +local server. + +Start llama-server first: + +```bash +llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8081 +# add --api-key token-abc123 if you want llama-server to require bearer auth +``` + +## Configure + +```bash +VLLM_BASE_URL=http://host.docker.internal:8081/v1 # include /v1 +# VLLM_API_KEY=token-abc123 # only if llama-server was started with --api-key +GOMODEL_MASTER_KEY=change-me +``` + + + These examples assume GoModel runs in Docker and llama-server is on the host + at `localhost:8081` — hence `host.docker.internal`. If GoModel runs on the + host directly, use `http://localhost:8081/v1`. Already using `VLLM_BASE_URL` + for a real vLLM server? Register llama.cpp under a suffixed instance instead: + `VLLM_LLAMACPP_BASE_URL=...` creates provider `vllm-llamacpp`. + + +## Verify + +```bash +curl -s http://localhost:8080/v1/chat/completions \ + -H "Authorization: Bearer change-me" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "vllm/gemma-3-4b-it", + "messages": [{"role": "user", "content": "Reply with exactly ok."}] + }' +``` + +`GET /v1/models` returns llama-server's model IDs prefixed by provider name. + +## Embeddings + +`/v1/embeddings` works through GoModel as long as llama-server can serve it: +the loaded model must use a pooling type other than `none`. Dedicated embedding +GGUFs usually declare pooling in their metadata; otherwise pass `--pooling mean` +(or `cls`/`last`). The `--embeddings` flag is optional — it restricts the +server to embeddings only: + +```bash +llama-server -hf nomic-ai/nomic-embed-text-v1.5-GGUF --embeddings --port 8081 +``` + +```bash +curl -s http://localhost:8080/v1/embeddings \ + -H "Authorization: Bearer change-me" \ + -H "Content-Type: application/json" \ + -d '{"model": "vllm/nomic-embed-text-v1.5", "input": "hello"}' +``` + +## Model classification + +llama-server's `/v1/models` listing carries no capability metadata, so GoModel +classifies its models by ID: names containing `embed` or matching well-known +embedding families (`bge`, `e5`, `gte`, `minilm`) are categorized as embedding +models, and names containing `rerank` as reranking models — namespaced IDs are +checked by their final path segment. For anything that stays unclassified, +declare `modes` (and context window or pricing) under the provider's model +metadata — see [Model metadata](/advanced/model-metadata). Categories only +affect dashboard grouping and failover suggestions; `/v1/embeddings` routes to +any model the provider serves regardless of category. + +## Beyond chat and embeddings + +- **Multimodal input** — image and audio *input* in chat messages works with + multimodal models when llama-server is started with a projector + (`--mmproj`, auto-loaded with `-hf` when available). These requests flow + through `/v1/chat/completions` normally. +- **Reranking** — llama-server serves `/v1/rerank` (start with `--rerank`, a + reranker model, and `--pooling rank`), but GoModel has no rerank endpoint; + reach it through [passthrough](/features/passthrough-api): + `POST /p/vllm/v1/rerank`. +- **Image generation and audio endpoints** — llama.cpp has neither: no + `/v1/images/generations`, `/v1/audio/speech`, or `/v1/audio/transcriptions` + (text-to-image lives in the separate stable-diffusion.cpp project, and + whisper.cpp's server is not OpenAI-compatible). Requests for those + capabilities must route to a provider that serves them. diff --git a/docs/providers/overview.mdx b/docs/providers/overview.mdx index 82c51547b..e6df90be3 100644 --- a/docs/providers/overview.mdx +++ b/docs/providers/overview.mdx @@ -65,6 +65,7 @@ support, not every individual model capability exposed by an upstream provider. | Ollama | `OLLAMA_BASE_URL` | `llama3.2` | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | [Ollama](/providers/multiple-ollama) | | SGLang | `SGLANG_BASE_URL` (`SGLANG_API_KEY` optional) | `Qwen/Qwen2.5-0.5B-Instruct` | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | [SGLang](/providers/sglang) | | vLLM | `VLLM_BASE_URL` (`VLLM_API_KEY` optional) | `meta-llama/Llama-3.1-8B-Instruct` | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | [vLLM](/providers/vllm) | +| llama.cpp / LM Studio (as vLLM type) | `VLLM_BASE_URL` (`VLLM_API_KEY` optional) | `gemma-3-4b-it` | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | [llama.cpp](/providers/llamacpp) | | llm-d | `LLMD_BASE_URL` (`LLMD_API_KEY` optional) | `Qwen/Qwen2.5-0.5B-Instruct` | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | [llm-d](/providers/llmd) | | Amazon Bedrock | `BEDROCK_BASE_URL` (region or endpoint) + AWS credentials | `anthropic.claude-3-5-haiku-20241022-v1:0` | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | [Amazon Bedrock](/providers/bedrock) | | Amazon Bedrock Mantle | `BEDROCK_MANTLE_API_KEY` or AWS credentials | `openai.gpt-5.6-sol` | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | [Bedrock Mantle](/providers/bedrock-mantle) | @@ -155,6 +156,10 @@ support, not every individual model capability exposed by an upstream provider. routes to any model the provider serves regardless of category. - **vLLM** — set `VLLM_API_KEY` only if the upstream server was started with `--api-key`. +- **llama.cpp / LM Studio** — no dedicated provider type; register the server + as a vLLM-type provider (`VLLM_BASE_URL` or a suffixed instance), not as + `ollama`, which speaks Ollama's native API. See + [llama.cpp](/providers/llamacpp). - **llm-d** — `LLMD_BASE_URL` is required. `LLMD_API_KEY` is optional and is sent only when the Gateway in front of the Router requires bearer auth. - **SGLang** — set `SGLANG_API_KEY` only if the upstream server was started From 60c7172de5f5681e0446d4a7aba298953276705e Mon Sep 17 00:00:00 2001 From: "Jakub A. W" Date: Sun, 16 Aug 2026 20:19:16 +0200 Subject: [PATCH 2/3] docs(providers): drop unsupported-capability list from llama.cpp guide Co-Authored-By: Claude Fable 5 --- docs/providers/llamacpp.mdx | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/providers/llamacpp.mdx b/docs/providers/llamacpp.mdx index 3e864fd99..7b51d290d 100644 --- a/docs/providers/llamacpp.mdx +++ b/docs/providers/llamacpp.mdx @@ -94,8 +94,3 @@ any model the provider serves regardless of category. reranker model, and `--pooling rank`), but GoModel has no rerank endpoint; reach it through [passthrough](/features/passthrough-api): `POST /p/vllm/v1/rerank`. -- **Image generation and audio endpoints** — llama.cpp has neither: no - `/v1/images/generations`, `/v1/audio/speech`, or `/v1/audio/transcriptions` - (text-to-image lives in the separate stable-diffusion.cpp project, and - whisper.cpp's server is not OpenAI-compatible). Requests for those - capabilities must route to a provider that serves them. From f5ee4cb771a795cef5630fa2741198def800aa8b Mon Sep 17 00:00:00 2001 From: "Jakub A. W" Date: Sun, 16 Aug 2026 20:22:15 +0200 Subject: [PATCH 3/3] docs(providers): fix llama.cpp model aliases and rerank flags Co-Authored-By: Claude Fable 5 --- docs/providers/llamacpp.mdx | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/providers/llamacpp.mdx b/docs/providers/llamacpp.mdx index 7b51d290d..90ec61c38 100644 --- a/docs/providers/llamacpp.mdx +++ b/docs/providers/llamacpp.mdx @@ -16,10 +16,11 @@ not implement. The same recipe applies to LM Studio and any other plain OpenAI-compatible local server. -Start llama-server first: +Start llama-server first. Without `--alias`, the model ID in `/v1/models` is +the model file's path — set an alias so requests can use a clean name: ```bash -llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8081 +llama-server -hf ggml-org/gemma-3-4b-it-GGUF --alias gemma-3-4b-it --port 8081 # add --api-key token-abc123 if you want llama-server to require bearer auth ``` @@ -62,7 +63,7 @@ GGUFs usually declare pooling in their metadata; otherwise pass `--pooling mean` server to embeddings only: ```bash -llama-server -hf nomic-ai/nomic-embed-text-v1.5-GGUF --embeddings --port 8081 +llama-server -hf nomic-ai/nomic-embed-text-v1.5-GGUF --embeddings --alias nomic-embed-text-v1.5 --port 8081 ``` ```bash @@ -90,7 +91,8 @@ any model the provider serves regardless of category. multimodal models when llama-server is started with a projector (`--mmproj`, auto-loaded with `-hf` when available). These requests flow through `/v1/chat/completions` normally. -- **Reranking** — llama-server serves `/v1/rerank` (start with `--rerank`, a - reranker model, and `--pooling rank`), but GoModel has no rerank endpoint; - reach it through [passthrough](/features/passthrough-api): - `POST /p/vllm/v1/rerank`. +- **Reranking** — llama-server serves `/v1/rerank` (start with a reranker + model and `--rerank --embedding --pooling rank`), but GoModel has no rerank + endpoint; reach it through [passthrough](/features/passthrough-api): + `POST /p/vllm/v1/rerank` (use the instance name, e.g. + `/p/vllm-llamacpp/v1/rerank`, for a suffixed setup).