diff --git a/docs/content/features/backends.md b/docs/content/features/backends.md index 4aebf97ac1bc..728ab52f6776 100644 --- a/docs/content/features/backends.md +++ b/docs/content/features/backends.md @@ -167,7 +167,7 @@ For getting started, see the available backends in LocalAI here: https://github. LocalAI supports various types of backends: -- **LLM Backends**: For running language models (e.g., llama.cpp, vLLM, vllm.cpp, SGLang, transformers, MLX) +- **LLM Backends**: For running language models (e.g., llama.cpp, vLLM, vllm.cpp, SGLang, transformers, MLX, and [RKLLM on Rockchip NPUs]({{% relref "features/rkllm" %}}) through the cloud-proxy backend) - **Speech-to-Text Backends**: For transcription, forced alignment and speaker diarization (e.g., whisper.cpp, parakeet.cpp, moss-transcribe.cpp, [NeMo-Speech.cpp]({{%relref "features/nemo-speech-cpp" %}}), faster-whisper, NeMo, [audio.cpp]({{%relref "features/audio-cpp" %}})) - **Text-to-Speech Backends**: For speech synthesis (e.g., piper, Kokoro, VibeVoice, Qwen3-TTS, [NeMo-Speech.cpp]({{%relref "features/nemo-speech-cpp" %}}), [audio.cpp]({{%relref "features/audio-cpp" %}})) - **Sound Generation Backends**: For music and audio generation (e.g., ACE-Step, [audio.cpp]({{%relref "features/audio-cpp" %}})) diff --git a/docs/content/features/rkllm.md b/docs/content/features/rkllm.md new file mode 100644 index 000000000000..bac3964bad4e --- /dev/null +++ b/docs/content/features/rkllm.md @@ -0,0 +1,113 @@ ++++ +title = "RKLLM on Rockchip NPUs" +weight = 37 +toc = true +url = "/features/rkllm/" +description = "Serve RKLLM models from Rockchip NPUs through LocalAI" +tags = ["LLM", "Rockchip", "NPU", "RKLLM"] +categories = ["Features"] ++++ + +[RKLLM](https://github.com/airockchip/rknn-llm) runs converted language +models on supported Rockchip NPUs. Its server already exposes an +OpenAI-compatible chat-completions API, so LocalAI can put its authentication, +routing, usage tracking, and web UI in front of the board with the +`cloud-proxy` backend. + +This integration does not install or convert RKLLM models. Model conversion is +a separate host-side step using RKLLM-Toolkit, and the resulting `.rkllm` file +must match the target SoC. + +## Requirements + +- A Linux Rockchip board supported by the installed RKLLM runtime. Upstream + RKLLM 1.3.0 accepts `rk3588`, `rk3576`, `rv1126b`, and `rk3562` in its server + demo. +- A model converted to `.rkllm` for that target. +- The upstream RKLLM Flask server running on the board and reachable from + LocalAI. + +RK3566 boards, including Quartz64 models with that SoC, are not in the current +upstream RKLLM server target list. LocalAI cannot add support for a SoC that the +RKLLM runtime does not support. + +## Start the RKLLM server + +Follow the upstream +[`rkllm_server_demo`](https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_server_demo) +instructions to deploy the runtime library, server, and converted model to the +board. For example, from an RKLLM checkout on the host: + +```bash +cd examples/rkllm_server_demo +./build_rkllm_server_flask.sh \ + --workshop /userdata/rkllm-server \ + --model_path /userdata/models/qwen3.rkllm \ + --platform rk3588 \ + --adb_device YOUR_DEVICE_SERIAL +``` + +The helper starts the server on port `8080`. Confirm it is reachable before +configuring LocalAI: + +```bash +curl http://ROCKCHIP_BOARD_IP:8080/v1/models +``` + +## Configure LocalAI + +Create `models/rkllm.yaml`: + +```yaml +name: rockchip-rkllm +backend: cloud-proxy + +proxy: + mode: passthrough + provider: openai + upstream_url: http://ROCKCHIP_BOARD_IP:8080/v1/chat/completions + upstream_model: rkllm + request_timeout_seconds: 600 + +# The upstream is on the local network, so cloud-egress PII filtering is not +# enabled by default in this example. Enable it if your deployment needs it. +pii: + enabled: false +``` + +No API key setting is required by the upstream demo server. If you expose the +board beyond a trusted network, put an authenticated reverse proxy in front of +it and configure the corresponding key through `api_key_env` or `api_key_file`. + +Start LocalAI, then use the local model name with any OpenAI-compatible client: + +```bash +curl http://localhost:8080/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "rockchip-rkllm", + "messages": [{"role": "user", "content": "Hello"}], + "stream": true + }' +``` + +LocalAI forwards the request and the streaming response without translating +the wire format. Sampling fields supported by the upstream server, including +`temperature`, `top_p`, `top_k`, `max_tokens`, `repeat_penalty`, and +`enable_thinking`, pass through unchanged. + +## Limitations + +- The upstream demo serves chat completions and model listing only. Embeddings, + image generation, audio APIs, and legacy text completions are not available. +- Model conversion and runtime installation remain upstream RKLLM operations. +- Upstream serializes inference for a loaded model and returns HTTP 503 while + it is busy. Scale with multiple boards and LocalAI routing if concurrent + inference is required. +- The upstream demo returns model-generated tool calls as markup in + `message.content`; passthrough mode does not convert that markup into OpenAI + `message.tool_calls`. Clients that use tools must parse the model-specific + format or put an adapter in front of the RKLLM server. + +See [Cloud passthrough proxy]({{% relref "operations/cloud-proxy" %}}) for the +full proxy configuration, authentication, routing, and PII options. diff --git a/docs/content/reference/compatibility-table.md b/docs/content/reference/compatibility-table.md index 28e8c2a1decb..643fbed66bc3 100644 --- a/docs/content/reference/compatibility-table.md +++ b/docs/content/reference/compatibility-table.md @@ -33,6 +33,7 @@ All backends listed here can be installed on demand from the [Backend Gallery]({ | [MLX-VLM](https://github.com/Blaizzy/mlx-vlm) | Vision-Language Models on Apple Silicon | Multimodal GPT, Functions | no | yes | CPU, CUDA 12/13, Metal, Jetson L4T | | [MLX Distributed](https://github.com/ml-explore/mlx-lm) | Distributed LLM inference across multiple Apple Silicon Macs | GPT | no | no | CPU, CUDA 12/13, Metal, Jetson L4T | | [tinygrad](https://github.com/tinygrad/tinygrad) | Minimalist deep-learning framework with zero runtime dependencies | GPT, Embeddings, Multimodal | yes | yes | CPU | +| [RKLLM](https://github.com/airockchip/rknn-llm) | Rockchip NPU runtime connected through its OpenAI-compatible server and LocalAI's [cloud-proxy]({{% relref "features/rkllm" %}}) configuration | GPT | no | yes | Rockchip NPU (RK3588, RK3576, RV1126B, RK3562) | ## Speech-to-Text