A high-performance Node.js/Express-based intelligent router for LLM requests. This system acts as an OpenAI-compatible gateway that dynamically routes incoming prompts to the most suitable "specialist" model based on the content of the request.
- Incoming Request: The router receives an OpenAI-compatible chat completion request (including multimodal/image data).
- Specialist Selection: A dedicated "Chooser" model analyzes the prompt and compares it against a list of defined specialist topics.
- Dynamic Routing: The request is forwarded to the selected specialist model, or a fallback model if no match is found.
- Streaming Support: Full support for Server-Sent Events (SSE) streaming and large payloads.
- OpenAI Compatible: Seamlessly integrates with any frontend or tool that supports OpenAI endpoints.
- Multimodal Support: Forwards images and other content blocks without modification.
- Zero Timeouts: Configured for long-running reasoning tasks and deep streams.
- Dynamic Specialists: Easily add or remove specialists via environment variables.
Copy the .env.example to .env and configure your models:
cp .env.example .envROUTER_VIRTUAL_MODEL_NAME: The model name users should specify in their client (e.g.,specialist-router).CHOOSER_MODEL_...: Configuration for the model that performs the routing logic.SPECIALIST_{N}_...: Define your specialists. Each needs aTOPIC,MODEL_ENDPOINT,MODEL_NAME, andMODEL_API_KEY.FALLBACK_MODEL_...: The "other" model used if the router cannot determine a specific specialist.
npm install
npm startPoint your AI client to:
http://localhost:3000/v1
Model Name: Use the ROUTER_VIRTUAL_MODEL_NAME you defined.
API Key: Use the ROUTER_API_KEY you defined (or leave empty if not set).
The router performs a "pre-flight" call to the Chooser model. For optimal performance, use a fast/lightweight model for the Chooser (e.g., GPT-4o-mini or a local small model).