mlx-whisper-server

A local, self-hosted Whisper speech-to-text server for Apple Silicon. Exposes an OpenAI-compatible API so any client that speaks the OpenAI Whisper API can use it as a drop-in replacement -- faster, free, and completely private.

Why this exists

If you use AI agents (Open Claw, Claude Code, or anything that accepts text input), typing long prompts gets tedious. The better workflow: record a voice memo and let Whisper transcribe it.

OpenAI hosts a Whisper API, but you already own hardware that can run the same model. This server puts that hardware to work:

• Faster -- runs on your local network, no round-trip to OpenAI's servers. Transcription typically completes in under a second for short voice memos.
• Free -- no per-minute API charges. The model runs on your Mac's Neural Engine and GPU.
• Private -- audio never leaves your machine or network.
• Compatible -- implements /v1/audio/transcriptions with the same request/response contract as OpenAI, so existing clients work without modification.

The only requirement is a Mac with Apple Silicon on your local network.

Features

• OpenAI Whisper API-compatible endpoint (/v1/audio/transcriptions)
• Runs Whisper inference locally via MLX on Apple Silicon
• Supports json, text, and verbose_json response formats
• Accepts wav, mp3, m4a, ogg, webm, and flac audio (up to 25 MB)
• Optional Bearer token authentication matching the OpenAI API contract
• Health check endpoint with model status and uptime
• Automatic model download from HuggingFace on first run
• JIT warmup on startup for fast first-request response
• Prevents macOS sleep via caffeinate while running
• Includes a launchd plist for auto-start on login

Requirements

• macOS with Apple Silicon (M1, M2, M3, M4, or later)
• Python 3.11+
• ffmpeg (for decoding mp3, m4a, ogg, and other compressed formats)

brew install ffmpeg

Quick start

# Clone or copy the project
cd ~/MLX\ Servers/Voice

# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the package and its dependencies
pip install -e .

# (Optional) Configure settings
cp .env.example .env
# Edit .env as needed

# Start the server
python run.py

On first launch, the Whisper model (~461 MB for whisper-small-mlx) is downloaded from HuggingFace and cached locally. Subsequent starts use the cached model. The server performs a warmup transcription at startup to JIT-compile the MLX graph, so the first real request is fast.

Once running, the server listens on http://0.0.0.0:8765 by default.

Configuration

Settings are loaded from environment variables or a .env file in the project root. Copy .env.example to .env to get started:

cp .env.example .env

Variable	Default	Description
PORT	8765	Server port
HOST	0.0.0.0	Bind address (0.0.0.0 = all interfaces, 127.0.0.1 = localhost only)
STT_MODEL	mlx-community/whisper-small-mlx	HuggingFace model repo ID (see Available models)
LOG_LEVEL	info	Logging level (debug, info, warning, error)
API_KEY	(none)	Optional API key for Bearer token authentication
OPENAI_API_KEY	(none)	Alternative name for the API key (same effect as API_KEY)

Authentication

Authentication is optional and disabled by default. When no API key is configured, all requests are accepted.

To enable authentication, set either API_KEY or OPENAI_API_KEY in your .env file:

API_KEY=your-secret-key-here

When enabled, all /v1/* endpoints require a Bearer token in the Authorization header:

curl http://localhost:8765/v1/audio/transcriptions \
  -H "Authorization: Bearer your-secret-key-here" \
  -F "file=@voice_memo.m4a" \
  -F "model=whisper-1"

The /health endpoint is always accessible without authentication.

This matches the OpenAI API authentication contract, so clients that already send an Authorization: Bearer sk-... header (like the official OpenAI SDKs, Open Claw, etc.) work without modification -- just point them at your local server and set the API key to match.

If the key is missing or incorrect, the server returns a 401 response with an error body matching the OpenAI error format:

{
  "error": {
    "message": "You didn't provide an API key.",
    "type": "invalid_request_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

API reference

Health check

GET /health

Returns server status, model name, load state, and uptime. No authentication required.

curl http://localhost:8765/health

{
  "status": "ok",
  "model": "mlx-community/whisper-small-mlx",
  "model_loaded": true,
  "uptime_seconds": 342.5
}

status is "loading" while the model is still being loaded at startup.

---

List models

GET /v1/models

Returns the available model list in the OpenAI format. Always returns a single model entry with the ID whisper-1.

curl http://localhost:8765/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "whisper-1",
      "object": "model",
      "created": 1706800000,
      "owned_by": "local"
    }
  ]
}

---

Transcribe audio

POST /v1/audio/transcriptions

Transcribes an audio file to text. This is the primary endpoint and is compatible with the OpenAI POST /v1/audio/transcriptions API.

Parameters (multipart form data):

Parameter	Type	Required	Default	Description
file	file	Yes	--	Audio file to transcribe
model	string	No	whisper-1	Model ID (accepted for compatibility; the server always uses the configured STT_MODEL)
language	string	No	(auto-detect)	ISO 639-1 language code (e.g., en, es, fr)
prompt	string	No	(none)	Optional text to guide the model's style or continue a previous segment
response_format	string	No	json	Output format: json, text, or verbose_json
temperature	float	No	0.0	Sampling temperature (0.0 = deterministic)

Supported audio formats: wav, mp3, m4a, ogg, webm, flac

Maximum file size: 25 MB

Examples

JSON response (default):

curl -X POST http://localhost:8765/v1/audio/transcriptions \
  -F "file=@voice_memo.m4a" \
  -F "model=whisper-1"

{"text": "Hey, I need you to refactor the auth module to use JWT tokens instead of session cookies."}

Plain text response:

curl -X POST http://localhost:8765/v1/audio/transcriptions \
  -F "file=@voice_memo.m4a" \
  -F "model=whisper-1" \
  -F "response_format=text"

Hey, I need you to refactor the auth module to use JWT tokens instead of session cookies.

Verbose JSON response (includes segments, language, and duration):

curl -X POST http://localhost:8765/v1/audio/transcriptions \
  -F "file=@voice_memo.m4a" \
  -F "model=whisper-1" \
  -F "response_format=verbose_json"

{
  "task": "transcribe",
  "language": "en",
  "duration": 4.8,
  "text": "Hey, I need you to refactor the auth module.",
  "segments": [...]
}

With language hint and prompt:

curl -X POST http://localhost:8765/v1/audio/transcriptions \
  -F "file=@memo.ogg" \
  -F "model=whisper-1" \
  -F "language=en" \
  -F "prompt=The speaker is discussing Kubernetes deployments." \
  -F "temperature=0.0"

Error responses

Status	Condition
400	Unsupported file type
401	Missing or incorrect API key (when auth is enabled)
413	File exceeds 25 MB
503	Model is still loading
500	Transcription failed

All errors follow the OpenAI error format: {"error": {"message": "...", "type": "..."}}.

Use with Open Claw and other OpenAI-compatible clients

Any client that targets the OpenAI Whisper API can use this server. Point the client's base URL at your server and, if you have authentication enabled, set the API key to match.

Open Claw / OpenAI SDK configuration:

Setting	Value
Base URL	http://<your-mac-ip>:8765/v1
API Key	Whatever you set as API_KEY (or leave blank if auth is disabled)
Model	whisper-1

For the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://192.168.1.100:8765/v1",
    api_key="your-key-here",  # or "not-needed" if auth is disabled
)

with open("voice_memo.m4a", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
    )
    print(transcript.text)

Replace 192.168.1.100 with your Mac's local IP address. You can find it with:

ipconfig getifaddr en0

Available models

The server uses MLX-optimized Whisper models from the mlx-community on HuggingFace. Change the model by setting STT_MODEL in your .env file.

Model	HuggingFace repo	Size	Notes
Tiny	mlx-community/whisper-tiny-mlx	~75 MB	Fastest, lowest accuracy
Base	mlx-community/whisper-base-mlx	~142 MB
Small	mlx-community/whisper-small-mlx	~461 MB	Default. Good balance of speed and accuracy
Medium	mlx-community/whisper-medium-mlx	~1.5 GB
Large v3	mlx-community/whisper-large-v3-mlx	~3.1 GB	Highest accuracy, slower
Large v3 Turbo	mlx-community/whisper-large-v3-turbo	~1.6 GB	Near-large accuracy, faster than large

Example:

# In .env
STT_MODEL=mlx-community/whisper-large-v3-turbo

The model is downloaded automatically on first run and cached in the HuggingFace cache directory (~/.cache/huggingface/).

Running as a macOS service

The project includes a launchd plist (com.local.voice-service.plist) that starts the server on login and restarts it if it crashes.

Install

# Create the log directory
mkdir -p ~/Library/Logs/voice-service

# Symlink the plist into LaunchAgents
ln -s "$PWD/com.local.voice-service.plist" ~/Library/LaunchAgents/

# Load the service
launchctl load ~/Library/LaunchAgents/com.local.voice-service.plist

# Verify it is running
launchctl list | grep voice-service

View logs

# stdout (server output)
tail -f ~/Library/Logs/voice-service/stdout.log

# stderr (errors)
tail -f ~/Library/Logs/voice-service/stderr.log

Stop / restart

# Stop the service
launchctl unload ~/Library/LaunchAgents/com.local.voice-service.plist

# Restart (unload + load)
launchctl unload ~/Library/LaunchAgents/com.local.voice-service.plist
launchctl load ~/Library/LaunchAgents/com.local.voice-service.plist

Uninstall

launchctl unload ~/Library/LaunchAgents/com.local.voice-service.plist
rm ~/Library/LaunchAgents/com.local.voice-service.plist

Note: The plist assumes the project lives at ~/MLX Servers/Voice with a virtual environment at .venv/. If your paths differ, edit the plist before installing.

Project structure

Voice/
  app/
    __init__.py
    config.py          # Settings loaded from env vars / .env
    main.py            # FastAPI app, endpoints, middleware
    stt_engine.py      # MLX Whisper engine with async wrapper
  run.py               # Entry point: starts uvicorn + caffeinate
  pyproject.toml       # Package metadata and dependencies
  .env.example         # Example configuration
  com.local.voice-service.plist  # launchd service definition

Troubleshooting

Server starts but /health returns {"status": "loading"}

The model is still being downloaded or the MLX warmup is in progress. The first start takes longer because it downloads the model from HuggingFace. Wait for the log message Model loaded in X.Xs (warmup complete) before sending requests.

ffmpeg errors when transcribing mp3/m4a/ogg files

Install ffmpeg: brew install ffmpeg. The wav and flac formats work without it, but compressed audio formats require ffmpeg for decoding.

Unsupported file type error

The server only accepts files with these extensions: .wav, .mp3, .m4a, .ogg, .webm, .flac. Make sure your audio file has the correct extension.

File too large error (413)

The maximum upload size is 25 MB. For longer recordings, split the audio or use a more compressed format (ogg or mp3 instead of wav).

Model uses too much memory

Switch to a smaller model. Set STT_MODEL=mlx-community/whisper-tiny-mlx or whisper-base-mlx in your .env file.

Cannot connect from another device on the network

Make sure HOST is set to 0.0.0.0 (the default) rather than 127.0.0.1. Check that your Mac's firewall allows incoming connections on the configured port. Find your Mac's IP with ipconfig getifaddr en0.

Server crashes or becomes unresponsive

If running via launchd, the service will restart automatically (the plist has KeepAlive set to true). Check the logs at ~/Library/Logs/voice-service/ for details.

Mac goes to sleep and the server becomes unreachable

The run.py entry point runs caffeinate -s to prevent system sleep while the server is active. If you are running the server a different way, wrap it with caffeinate -s manually.