From dbcca9fbbc7143f322bc8274c6d332afbf4176f1 Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Mon, 3 Aug 2026 14:05:24 +0800 Subject: [PATCH 1/4] docs: add LocalAI integration tutorial Signed-off-by: Ming Wen --- docs/en/latest/config.json | 1 + docs/en/latest/tutorials/proxy-localai.md | 260 ++++++++++++++++++++++ 2 files changed, 261 insertions(+) create mode 100644 docs/en/latest/tutorials/proxy-localai.md diff --git a/docs/en/latest/config.json b/docs/en/latest/config.json index 1619e8000b0d..452e765feefb 100644 --- a/docs/en/latest/config.json +++ b/docs/en/latest/config.json @@ -25,6 +25,7 @@ "label": "Tutorials", "items": [ "tutorials/expose-api", + "tutorials/proxy-localai", "tutorials/protect-api", { "type": "category", diff --git a/docs/en/latest/tutorials/proxy-localai.md b/docs/en/latest/tutorials/proxy-localai.md new file mode 100644 index 000000000000..3f93ad6ced4d --- /dev/null +++ b/docs/en/latest/tutorials/proxy-localai.md @@ -0,0 +1,260 @@ +--- +title: Proxy LocalAI APIs with Apache APISIX +keywords: + - Apache APISIX + - AI Gateway + - LocalAI + - OpenAI-compatible API +description: This tutorial shows how to proxy LocalAI's OpenAI-compatible APIs through Apache APISIX. +--- + + + +[LocalAI](https://localai.io/) exposes local models through OpenAI-compatible APIs. In this tutorial, you will place APISIX in front of LocalAI, proxy only the `/v1/*` API surface, and verify non-streaming and streaming chat completions with a real model. + +The base configuration in this tutorial provides routing, upstream authentication forwarding, streaming, and timeouts. It does not enable APISIX authentication, rate limiting, observability, model routing, or failover unless you configure the corresponding plugins. + +## Prerequisites + +- [Docker](https://docs.docker.com/get-docker/) +- [curl](https://curl.se/), [jq](https://jqlang.github.io/jq/), and [yq](https://github.com/mikefarah/yq) +- Apache APISIX 3.17.0 or later. The [`proxy-buffering`](../plugins/proxy-buffering.md) Plugin used for streaming was added in APISIX 3.17.0. + +This tutorial assumes that APISIX can reach LocalAI at `127.0.0.1:8080`. This address works when both processes run on the same host or APISIX uses Docker host networking, as in the APISIX [getting started guide](../getting-started/README.md). For other deployments, replace the upstream node with a reachable LocalAI address, such as `localai:8080` on a shared Docker network. + +## Start LocalAI with a Model + +Set a demonstration LocalAI API key. Use a secret value in production. + +```shell +export LOCALAI_API_KEY="localai-demo-key" +``` + +Start LocalAI 4.7.1 with the CPU-friendly `llama-3.2-1b-instruct:q4_k_m` model from the LocalAI model gallery: + +```shell +docker run --detach \ + --name localai \ + --publish 8080:8080 \ + --env LOCALAI_API_KEY="${LOCALAI_API_KEY}" \ + --env LOCALAI_BASE_URL="http://127.0.0.1:9080" \ + --volume localai-models:/models \ + --volume localai-backends:/backends \ + localai/localai:v4.7.1 \ + run llama-3.2-1b-instruct:q4_k_m +``` + +The first start downloads the model and its backend and can take several minutes. Check LocalAI until it is ready: + +```shell +curl --fail --retry 120 --retry-delay 5 --retry-max-time 900 \ + --retry-connrefused "http://127.0.0.1:8080/readyz" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" +``` + +The legacy `LOCALAI_API_KEY` grants full LocalAI administrator access. Use [LocalAI user authentication](https://localai.io/docs/features/authentication/) when you need users, roles, or per-user API keys. + +## Create a Route + +Retrieve the APISIX Admin API key from `config.yaml`: + +```shell +admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g') +``` + +Create a Route for LocalAI's OpenAI-compatible APIs: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "uri": "/v1/*", + "plugins": { + "proxy-buffering": { + "disable_proxy_buffering": true + }, + "proxy-rewrite": { + "headers": { + "set": { + "X-Forwarded-Host": "$host", + "X-Forwarded-Proto": "$scheme" + } + } + } + }, + "timeout": { + "connect": 60, + "send": 3600, + "read": 3600 + }, + "upstream": { + "type": "roundrobin", + "scheme": "http", + "pass_host": "pass", + "nodes": { + "127.0.0.1:8080": 1 + } + } + }' +``` + +The Route forwards the LocalAI `Authorization` header without changing it. The `proxy-buffering` Plugin lets clients receive streaming Server-Sent Events (SSE) without response buffering. The send and read timeouts apply to individual upstream I/O operations, not to the total request duration. + +Using `/v1/*` prevents this Route from exposing LocalAI's Web UI and management endpoints. Add separate, explicit Routes only when clients require other LocalAI APIs. + +## Verify the Route + +List the available models through APISIX: + +```shell +curl "http://127.0.0.1:9080/v1/models" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" | jq +``` + +Send a non-streaming chat completion request: + +```shell +curl "http://127.0.0.1:9080/v1/chat/completions" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "llama-3.2-1b-instruct:q4_k_m", + "messages": [ + {"role": "user", "content": "Reply with exactly: APISIX reaches LocalAI"} + ], + "temperature": 0, + "max_tokens": 32 + }' | jq +``` + +Send a streaming chat completion request: + +```shell +curl --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "llama-3.2-1b-instruct:q4_k_m", + "messages": [ + {"role": "user", "content": "Count from one to five."} + ], + "stream": true, + "temperature": 0, + "max_tokens": 32 + }' +``` + +You should receive multiple `data:` events followed by `data: [DONE]`. + +Confirm that a LocalAI management endpoint is not exposed by the Route: + +```shell +curl --output /dev/null --write-out "%{http_code}\n" \ + "http://127.0.0.1:9080/api/p2p/token" +``` + +APISIX should return `404`. + +## Add APISIX Client Authentication + +LocalAI authentication protects the upstream. To authenticate clients at the gateway as a separate layer, create an APISIX Consumer and `key-auth` Credential: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{"username": "localai-client"}' + +curl "http://127.0.0.1:9180/apisix/admin/consumers/localai-client/credentials" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "localai-client-key", + "plugins": { + "key-auth": { + "key": "gateway-client-key" + } + } + }' +``` + +Enable `key-auth` on the Route. The APISIX key uses the `apikey` header, while the LocalAI key remains in the `Authorization` header. `hide_credentials` prevents APISIX from forwarding its key upstream. + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PATCH \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "plugins": { + "key-auth": { + "header": "apikey", + "hide_credentials": true + } + } + }' +``` + +Send both independent credentials to access LocalAI through APISIX: + +```shell +curl "http://127.0.0.1:9080/v1/models" \ + -H "apikey: gateway-client-key" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" | jq +``` + +A request without the `apikey` header is rejected by APISIX. A request with the APISIX key but without the LocalAI bearer token is rejected by LocalAI. + +Verify both rejection cases. Each command should return `401`: + +```shell +curl --output /dev/null --write-out "%{http_code}\n" \ + "http://127.0.0.1:9080/v1/models" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" + +curl --output /dev/null --write-out "%{http_code}\n" \ + "http://127.0.0.1:9080/v1/models" \ + -H "apikey: gateway-client-key" +``` + +## Production Considerations + +- This tutorial configures a local HTTP Route. Configure [TLS](../certificate.md) before exposing the Route and set `LOCALAI_BASE_URL` to its public HTTPS URL. +- Keep the APISIX Admin API private and rotate its key. Never expose port `9180` publicly. +- Add APISIX authentication, rate limiting, and observability plugins according to your requirements. These capabilities are not enabled by the base Route. +- Restrict the Route further to specific methods and paths if clients need only a subset of `/v1/*`. + +## Clean Up + +Delete the APISIX Route: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X DELETE \ + -H "X-API-KEY: ${admin_key}" +``` + +If you added APISIX client authentication, delete the Consumer and its Credential: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/consumers/localai-client" -X DELETE \ + -H "X-API-KEY: ${admin_key}" +``` + +Remove the LocalAI container. The named model and backend volumes are retained for reuse. + +```shell +docker rm --force localai +``` From 63234f354030c426d0f4736752af8769424a2204 Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Mon, 3 Aug 2026 14:56:19 +0800 Subject: [PATCH 2/4] docs: restrict LocalAI integration access Signed-off-by: Ming Wen --- docs/en/latest/tutorials/proxy-localai.md | 87 +++++++++++++++-------- 1 file changed, 58 insertions(+), 29 deletions(-) diff --git a/docs/en/latest/tutorials/proxy-localai.md b/docs/en/latest/tutorials/proxy-localai.md index 3f93ad6ced4d..9a1e0f2e491b 100644 --- a/docs/en/latest/tutorials/proxy-localai.md +++ b/docs/en/latest/tutorials/proxy-localai.md @@ -27,14 +27,14 @@ description: This tutorial shows how to proxy LocalAI's OpenAI-compatible APIs t # --> -[LocalAI](https://localai.io/) exposes local models through OpenAI-compatible APIs. In this tutorial, you will place APISIX in front of LocalAI, proxy only the `/v1/*` API surface, and verify non-streaming and streaming chat completions with a real model. +[LocalAI](https://localai.io/) exposes local models through OpenAI-compatible APIs. In this tutorial, you will place APISIX in front of LocalAI, proxy only model discovery and chat completions, and verify non-streaming and streaming responses with a real model. The base configuration in this tutorial provides routing, upstream authentication forwarding, streaming, and timeouts. It does not enable APISIX authentication, rate limiting, observability, model routing, or failover unless you configure the corresponding plugins. ## Prerequisites - [Docker](https://docs.docker.com/get-docker/) -- [curl](https://curl.se/), [jq](https://jqlang.github.io/jq/), and [yq](https://github.com/mikefarah/yq) +- [curl](https://curl.se/) and [jq](https://jqlang.github.io/jq/) - Apache APISIX 3.17.0 or later. The [`proxy-buffering`](../plugins/proxy-buffering.md) Plugin used for streaming was added in APISIX 3.17.0. This tutorial assumes that APISIX can reach LocalAI at `127.0.0.1:8080`. This address works when both processes run on the same host or APISIX uses Docker host networking, as in the APISIX [getting started guide](../getting-started/README.md). For other deployments, replace the upstream node with a reachable LocalAI address, such as `localai:8080` on a shared Docker network. @@ -52,7 +52,7 @@ Start LocalAI 4.7.1 with the CPU-friendly `llama-3.2-1b-instruct:q4_k_m` model f ```shell docker run --detach \ --name localai \ - --publish 8080:8080 \ + --publish 127.0.0.1:8080:8080 \ --env LOCALAI_API_KEY="${LOCALAI_API_KEY}" \ --env LOCALAI_BASE_URL="http://127.0.0.1:9080" \ --volume localai-models:/models \ @@ -61,6 +61,8 @@ docker run --detach \ run llama-3.2-1b-instruct:q4_k_m ``` +Binding the port to `127.0.0.1` prevents remote clients from bypassing APISIX and accessing LocalAI directly. + The first start downloads the model and its backend and can take several minutes. Check LocalAI until it is ready: ```shell @@ -73,19 +75,25 @@ The legacy `LOCALAI_API_KEY` grants full LocalAI administrator access. Use [Loca ## Create a Route -Retrieve the APISIX Admin API key from `config.yaml`: +Set `admin_key` to the key configured in `deployment.admin.admin_key` for your APISIX installation: ```shell -admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g') +export admin_key="" ``` +The local APISIX quickstart disables Admin API authorization for testing. If you use it, omit the `X-API-KEY` headers below. Keep Admin API authorization enabled outside a local test environment. + Create a Route for LocalAI's OpenAI-compatible APIs: ```shell -curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PUT \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PUT \ -H "X-API-KEY: ${admin_key}" \ + -H "Content-Type: application/json" \ -d '{ - "uri": "/v1/*", + "uris": [ + "/v1/models", + "/v1/chat/completions" + ], "plugins": { "proxy-buffering": { "disable_proxy_buffering": true @@ -117,21 +125,22 @@ curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PUT \ The Route forwards the LocalAI `Authorization` header without changing it. The `proxy-buffering` Plugin lets clients receive streaming Server-Sent Events (SSE) without response buffering. The send and read timeouts apply to individual upstream I/O operations, not to the total request duration. -Using `/v1/*` prevents this Route from exposing LocalAI's Web UI and management endpoints. Add separate, explicit Routes only when clients require other LocalAI APIs. +Using exact paths prevents this Route from exposing LocalAI's Web UI and management endpoints, including management endpoints under `/v1/backend/*`. Add separate, explicit paths only when clients require other LocalAI APIs. ## Verify the Route List the available models through APISIX: ```shell -curl "http://127.0.0.1:9080/v1/models" \ - -H "Authorization: Bearer ${LOCALAI_API_KEY}" | jq +curl --fail-with-body "http://127.0.0.1:9080/v1/models" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" | \ + jq -e '.data[] | select(.id == "llama-3.2-1b-instruct:q4_k_m")' ``` Send a non-streaming chat completion request: ```shell -curl "http://127.0.0.1:9080/v1/chat/completions" \ +curl --fail-with-body "http://127.0.0.1:9080/v1/chat/completions" \ -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ @@ -141,13 +150,19 @@ curl "http://127.0.0.1:9080/v1/chat/completions" \ ], "temperature": 0, "max_tokens": 32 - }' | jq + }' | \ + jq -e ' + .object == "chat.completion" and + (.choices[0].message.content | type == "string" and length > 0) + ' ``` Send a streaming chat completion request: ```shell -curl --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ +stream_output=$(mktemp) + +curl --fail-with-body --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ @@ -158,7 +173,11 @@ curl --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ "stream": true, "temperature": 0, "max_tokens": 32 - }' + }' | tee "${stream_output}" + +test "$(grep -c '^data:' "${stream_output}")" -gt 1 +grep -q '^data: \[DONE\]$' "${stream_output}" +rm "${stream_output}" ``` You should receive multiple `data:` events followed by `data: [DONE]`. @@ -166,8 +185,9 @@ You should receive multiple `data:` events followed by `data: [DONE]`. Confirm that a LocalAI management endpoint is not exposed by the Route: ```shell -curl --output /dev/null --write-out "%{http_code}\n" \ - "http://127.0.0.1:9080/api/p2p/token" +test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ + "http://127.0.0.1:9080/v1/backend/monitor" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}")" = "404" ``` APISIX should return `404`. @@ -177,12 +197,14 @@ APISIX should return `404`. LocalAI authentication protects the upstream. To authenticate clients at the gateway as a separate layer, create an APISIX Consumer and `key-auth` Credential: ```shell -curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \ -H "X-API-KEY: ${admin_key}" \ + -H "Content-Type: application/json" \ -d '{"username": "localai-client"}' -curl "http://127.0.0.1:9180/apisix/admin/consumers/localai-client/credentials" -X PUT \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/consumers/localai-client/credentials" -X PUT \ -H "X-API-KEY: ${admin_key}" \ + -H "Content-Type: application/json" \ -d '{ "id": "localai-client-key", "plugins": { @@ -193,11 +215,14 @@ curl "http://127.0.0.1:9180/apisix/admin/consumers/localai-client/credentials" - }' ``` -Enable `key-auth` on the Route. The APISIX key uses the `apikey` header, while the LocalAI key remains in the `Authorization` header. `hide_credentials` prevents APISIX from forwarding its key upstream. +The key values in this tutorial are for local testing. Use separate, secret values in production. + +Enable `key-auth` on the Route. The APISIX key uses the `apikey` header, while the LocalAI key remains in the `Authorization` header. ```shell -curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PATCH \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PATCH \ -H "X-API-KEY: ${admin_key}" \ + -H "Content-Type: application/json" \ -d '{ "plugins": { "key-auth": { @@ -208,12 +233,15 @@ curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PATCH \ }' ``` +The requests below carry the APISIX key only in the `apikey` header. With `hide_credentials` enabled, APISIX removes that authenticated header before proxying. Do not duplicate the gateway key in an `apikey` query parameter. + Send both independent credentials to access LocalAI through APISIX: ```shell -curl "http://127.0.0.1:9080/v1/models" \ +curl --fail-with-body "http://127.0.0.1:9080/v1/models" \ -H "apikey: gateway-client-key" \ - -H "Authorization: Bearer ${LOCALAI_API_KEY}" | jq + -H "Authorization: Bearer ${LOCALAI_API_KEY}" | \ + jq -e '.data[] | select(.id == "llama-3.2-1b-instruct:q4_k_m")' ``` A request without the `apikey` header is rejected by APISIX. A request with the APISIX key but without the LocalAI bearer token is rejected by LocalAI. @@ -221,35 +249,36 @@ A request without the `apikey` header is rejected by APISIX. A request with the Verify both rejection cases. Each command should return `401`: ```shell -curl --output /dev/null --write-out "%{http_code}\n" \ +test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ "http://127.0.0.1:9080/v1/models" \ - -H "Authorization: Bearer ${LOCALAI_API_KEY}" + -H "Authorization: Bearer ${LOCALAI_API_KEY}")" = "401" -curl --output /dev/null --write-out "%{http_code}\n" \ +test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ "http://127.0.0.1:9080/v1/models" \ - -H "apikey: gateway-client-key" + -H "apikey: gateway-client-key")" = "401" ``` ## Production Considerations - This tutorial configures a local HTTP Route. Configure [TLS](../certificate.md) before exposing the Route and set `LOCALAI_BASE_URL` to its public HTTPS URL. +- Do not publish LocalAI on a public interface. Bind it to loopback when APISIX uses host networking, or place APISIX and LocalAI on a private Docker network without publishing LocalAI port `8080`. - Keep the APISIX Admin API private and rotate its key. Never expose port `9180` publicly. - Add APISIX authentication, rate limiting, and observability plugins according to your requirements. These capabilities are not enabled by the base Route. -- Restrict the Route further to specific methods and paths if clients need only a subset of `/v1/*`. +- Add only the exact LocalAI paths your clients require. Do not replace the path list with a broad wildcard. ## Clean Up Delete the APISIX Route: ```shell -curl "http://127.0.0.1:9180/apisix/admin/routes/localai" -X DELETE \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/routes/localai" -X DELETE \ -H "X-API-KEY: ${admin_key}" ``` If you added APISIX client authentication, delete the Consumer and its Credential: ```shell -curl "http://127.0.0.1:9180/apisix/admin/consumers/localai-client" -X DELETE \ +curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/consumers/localai-client" -X DELETE \ -H "X-API-KEY: ${admin_key}" ``` From 5c7dbdc623c3897e8865dc26b4a0257cabda59c5 Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Mon, 3 Aug 2026 15:10:34 +0800 Subject: [PATCH 3/4] docs: harden LocalAI verification commands Signed-off-by: Ming Wen --- docs/en/latest/tutorials/proxy-localai.md | 59 +++++++++++++---------- 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/docs/en/latest/tutorials/proxy-localai.md b/docs/en/latest/tutorials/proxy-localai.md index 9a1e0f2e491b..0213c91fe590 100644 --- a/docs/en/latest/tutorials/proxy-localai.md +++ b/docs/en/latest/tutorials/proxy-localai.md @@ -162,22 +162,28 @@ Send a streaming chat completion request: ```shell stream_output=$(mktemp) -curl --fail-with-body --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ - -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "llama-3.2-1b-instruct:q4_k_m", - "messages": [ - {"role": "user", "content": "Count from one to five."} - ], - "stream": true, - "temperature": 0, - "max_tokens": 32 - }' | tee "${stream_output}" - -test "$(grep -c '^data:' "${stream_output}")" -gt 1 -grep -q '^data: \[DONE\]$' "${stream_output}" -rm "${stream_output}" +( + set -o pipefail + + curl --fail-with-body --no-buffer "http://127.0.0.1:9080/v1/chat/completions" \ + -H "Authorization: Bearer ${LOCALAI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "llama-3.2-1b-instruct:q4_k_m", + "messages": [ + {"role": "user", "content": "Count from one to five."} + ], + "stream": true, + "temperature": 0, + "max_tokens": 32 + }' | tee "${stream_output}" +) && + test "$(grep -c '^data:' "${stream_output}")" -gt 1 && + grep -q '^data: \[DONE\]$' "${stream_output}" + +stream_status=$? +rm -f "${stream_output}" +test "${stream_status}" -eq 0 ``` You should receive multiple `data:` events followed by `data: [DONE]`. @@ -233,7 +239,7 @@ curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/routes/localai" -X PAT }' ``` -The requests below carry the APISIX key only in the `apikey` header. With `hide_credentials` enabled, APISIX removes that authenticated header before proxying. Do not duplicate the gateway key in an `apikey` query parameter. +The requests below carry the APISIX key only in the `apikey` header. Header lookup takes precedence over query lookup. If both forms are present, `hide_credentials` removes only the authenticated header, and the duplicate `apikey` query value is still forwarded upstream. Never put the gateway key in the query string or send both forms. Send both independent credentials to access LocalAI through APISIX: @@ -244,18 +250,17 @@ curl --fail-with-body "http://127.0.0.1:9080/v1/models" \ jq -e '.data[] | select(.id == "llama-3.2-1b-instruct:q4_k_m")' ``` -A request without the `apikey` header is rejected by APISIX. A request with the APISIX key but without the LocalAI bearer token is rejected by LocalAI. +A request without an APISIX gateway credential is rejected by APISIX. A request with the APISIX key but without the LocalAI bearer token is rejected by LocalAI. Verify both rejection cases. Each command should return `401`: ```shell test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ "http://127.0.0.1:9080/v1/models" \ - -H "Authorization: Bearer ${LOCALAI_API_KEY}")" = "401" - -test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ - "http://127.0.0.1:9080/v1/models" \ - -H "apikey: gateway-client-key")" = "401" + -H "Authorization: Bearer ${LOCALAI_API_KEY}")" = "401" && + test "$(curl --silent --output /dev/null --write-out "%{http_code}" \ + "http://127.0.0.1:9080/v1/models" \ + -H "apikey: gateway-client-key")" = "401" ``` ## Production Considerations @@ -282,8 +287,12 @@ curl --fail-with-body "http://127.0.0.1:9180/apisix/admin/consumers/localai-clie -H "X-API-KEY: ${admin_key}" ``` -Remove the LocalAI container. The named model and backend volumes are retained for reuse. +Remove the LocalAI container and named volumes: ```shell -docker rm --force localai +docker rm --force localai && + docker volume rm localai-models localai-backends && + unset LOCALAI_API_KEY admin_key ``` + +To keep the downloaded model and backend for reuse, omit the `docker volume rm` command. From 513ad3fa8faa4027000029a8459f04efa765f9a7 Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Mon, 3 Aug 2026 15:12:52 +0800 Subject: [PATCH 4/4] docs: require secure Docker port binding Signed-off-by: Ming Wen --- docs/en/latest/tutorials/proxy-localai.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/latest/tutorials/proxy-localai.md b/docs/en/latest/tutorials/proxy-localai.md index 0213c91fe590..e58a2edcefa7 100644 --- a/docs/en/latest/tutorials/proxy-localai.md +++ b/docs/en/latest/tutorials/proxy-localai.md @@ -33,7 +33,7 @@ The base configuration in this tutorial provides routing, upstream authenticatio ## Prerequisites -- [Docker](https://docs.docker.com/get-docker/) +- [Docker](https://docs.docker.com/get-docker/) with Docker Engine 28.0.0 or later - [curl](https://curl.se/) and [jq](https://jqlang.github.io/jq/) - Apache APISIX 3.17.0 or later. The [`proxy-buffering`](../plugins/proxy-buffering.md) Plugin used for streaming was added in APISIX 3.17.0. @@ -61,7 +61,7 @@ docker run --detach \ run llama-3.2-1b-instruct:q4_k_m ``` -Binding the port to `127.0.0.1` prevents remote clients from bypassing APISIX and accessing LocalAI directly. +With Docker Engine 28.0.0 or later, binding the published port to `127.0.0.1` limits access to the Docker host. On older Engine releases, upgrade or additionally block port `8080` with the host firewall. See [Docker port publishing](https://docs.docker.com/engine/network/port-publishing/) for details. The first start downloads the model and its backend and can take several minutes. Check LocalAI until it is ready: