[Feature] Custom base URL for the realtime provider + server-side VAD for the turn-based pipeline (factory firmware never sends listen:stop)

[woshiyig]

[Feature] Add openai_base_url and server-side VAD for the openai_compatible pipeline

Chinese abstract：出厂固件以 auto（服务端 VAD）模式工作，从不发送 listen:stop，导致 openai_compatible 回合制管线永远收不到音频、会话在 2–15 秒后以 WS 1006 异常关闭。请求：①给 openai 实时通道加 openai_base_url（可接 GLM-Realtime 等 OpenAI 兼容实时端点）；②给 openai_compatible 管线加服务端 VAD（修好后 Hermes / MiniMax / DeepSeek 等任意 LLM 均可用）。

---

Summary

Two related capability gaps prevent using self-hosted / third-party LLMs with the M5Stack StackChan factory firmware:

1. openai realtime provider has no configurable base URL. It is hard-wired to api.openai.com, so OpenAI-Realtime-compatible providers (e.g. Zhipu GLM-Realtime, which has a first-party realtime voice API) cannot be used.
2. openai_compatible (turn-based) pipeline never receives audio from the factory firmware, because the firmware never emits listen:stop. The pipeline therefore never triggers STT.

Net effect: the only working configuration today is provider=gemini (Gemini Live). Any text-only LLM the user already pays for (Hermes agent, MiniMax-M3, DeepSeek, …) is unreachable.

---

Environment

Item	Value
Server	StackChan AI Server macOS standalone preview 0.1.1 (source rev 2b5b09de6e38daa7417b31866ba7437609fe58a8), run directly from the app bundle binary
Host	macOS, STACKCHAN_LOCAL_HOST=192.168.1.36, device port 12800
Device	M5Stack StackChan (SKU:K151), CoreS3, factory firmware 1.5.1 (app_init: App version: 1.5.1, Launcher banner 2.3.3, build Jul 31 2026)
Device MAC	44:1b:f6:e5:5c:20
Onboarding	OTA URL injected into NVS (wifi namespace, ota_url = http://192.168.1.36:12800/xiaozhi/ota/) — device successfully fetches it and receives {"websocket":{"url":"ws://192.168.1.36:12800/xiaozhi/ws","version":3}}

---

What works

• Device discovers the local server via the injected OTA URL ✅
• WebSocket handshake succeeds every time: hello → hello OK (session id assigned) ✅
• provider=gemini (gemini-2.5-flash-native-audio-latest) — full voice conversation works ✅
• Custom STT/TTS adapters (local whisper + macOS say) both serve /v1/models, /v1/audio/transcriptions, /v1/audio/speech and are reachable from the server ✅

What fails

provider=openai_compatible, with LLM pointing at either a local Hermes agent (http://127.0.0.1:8642/v1, model hermes-agent) or MiniMax (https://api.minimaxi.com/v1, model MiniMax-M3), and STT/TTS pointing at the local adapters.

The LLM is never reached — in fact STT is never reached. Across 14 consecutive attempts the server log shows the same pattern:

[WS] device=44:1b:f6:e5:5c:20 realtime session ready provider=openai_compatible
[WS] device=44:1b:f6:e5:5c:20 recv type=hello
[WS] device=44:1b:f6:e5:5c:20 session=6d7810d4-… hello OK
[WS] device=44:1b:f6:e5:5c:20 recv type=listen state=start
[WS] device=44:1b:f6:e5:5c:20 listening started
      ↓ nothing else ↓
[WS] device=44:1b:f6:e5:5c:20 read error: websocket: close 1006 (abnormal closure): unexpected EOF
[WS] device=44:1b:f6:e5:5c:20 session closed

Event tally over all attempts (log level debug):

Event	Count
recv type=hello	12
recv type=listen state=start	14
recv type=listen state=stop	0
audio frames observed	0 (opus/audio/frame/vad keyword hits = 0)
STT adapter invocations	0
TTS adapter invocations	0
Server-side errors	none

Time from listening started to the 1006 close varied between ~2 s and ~14 s (also one clean server-side timeout: conversation idle; returning device to wake-word standby after ~45 s).

No error/warn/fail entries appear in the server log.

Not the cause (ruled out)

Hypothesis	Verdict
Network / reachability	❌ ruled out — same device, same network, Gemini path works
Power supply	❌ ruled out — the device holds normal conversations with the stock Xiaozhi cloud on the same PSU
Bad API key / upstream model	❌ ruled out — MiniMax-M3 returns 200 for the exact same credentials/base URL used in the settings; llm_api_key reads back at full length
Missing /v1/models on my adapters	❌ ruled out — added it (now both return 200); failure unchanged
Wake word / microphone	❌ ruled out — green LED lights, wake word is recognized

---

Root-cause analysis

Xiaozhi protocol v3 has two listening modes:

Mode	Who decides "user finished speaking"
manual	Device — sends {"type":"listen","state":"stop"}
auto	Server — server-side VAD detects end of speech

The factory firmware appears to run in auto mode: it opens with listen:start and then waits for the server to drive the turn. It never sends listen:stop.

But the openai_compatible pipeline is turn-based — per the README it is a STT → Chat Completions → TTS path, and it waits for the device's listen:stop before invoking STT.

Both sides wait for the other → deadlock → the device eventually tears down the socket (1006).

This is consistent with the fact that provider=gemini (and presumably provider=openai) work: those are realtime sessions with server-side VAD, so they never need listen:stop.

Note this is independent of which LLM is configured — swapping MiniMax-M3 for Hermes, DeepSeek, or anything else changes nothing, because no audio is ever handed to STT.

---

Requests

1. Add openai_base_url to the openai (realtime) provider

Currently:

$ curl -X PUT .../api/settings -d '{"openai_base_url":"https://open.bigmodel.cn/api/paas/v4"}'
HTTP 400  {"error":"unsupported setting"}   # (exact body wording may differ)

The realtime provider is hard-wired to api.openai.com. Making the base URL configurable would immediately unlock any OpenAI-Realtime-protocol-compatible realtime endpoint — for example Zhipu GLM-Realtime (https://docs.bigmodel.cn/cn/guide/models/sound-and-video/glm-realtime, SDK: https://github.com/MetaGLM/glm-realtime-sdk), which is a genuine realtime voice model and is substantially cheaper than OpenAI Realtime for CN users.

This is a small, low-risk change: one settings field, one base-URL constant.

2. Add server-side VAD to the openai_compatible pipeline

If the compatible pipeline performed (or accepted) server-side VAD — i.e. treated "N ms of silence after detected speech" as end-of-turn instead of requiring listen:stop — then any text LLM would work: local Hermes agents, MiniMax-M3, DeepSeek, OpenRouter models, etc.

This greatly widens the project's usefulness: users can keep their existing (already paid for) LLM subscriptions instead of buying realtime-audio tokens.

Possible implementation notes:

• Reuse whatever VAD already exists for the openai realtime frontend.
• If the device advertises its mode in hello (features / listen mode), branch on it: auto → server VAD; manual → current behaviour.
• Expose an option such as compatible_server_vad: true / compatible_vad_silence_ms: 800 so users can opt in.
• Consider logging received audio frame counts at debug level — right now it is impossible to tell from the log whether audio is arriving at all. (This alone cost me hours; see "Debugging note" below.)

---

Debugging note (suggestion)

Please consider adding a debug log line for binary/incoming audio frames (e.g. count + bytes per second). The current debug output shows only JSON control messages, so a "no audio at all" failure is indistinguishable from "audio arriving but never triggering a turn". A single counter line would have made this report much shorter.

---

Reproduce

# 1. Device points at the local server (NVS: wifi.ota_url)
curl -s http://192.168.1.36:12800/xiaozhi/ota/
# → {"websocket":{"token":"","url":"ws://192.168.1.36:12800/xiaozhi/ws","version":3}}

# 2. Configure the compatible pipeline
curl -X PUT http://127.0.0.1:8099/api/settings \
  -H "Authorization: Bearer $SETTINGS_TOKEN" -H 'Content-Type: application/json' \
  -d '{"provider":"openai_compatible",
       "llm_base_url":"https://api.minimaxi.com/v1","llm_api_key":"...","llm_model":"MiniMax-M3",
       "stt_base_url":"http://127.0.0.1:8801/v1","stt_api_key":"sk-local","stt_model":"whisper-1",
       "tts_base_url":"http://127.0.0.1:8802/v1","tts_api_key":"sk-local","tts_model":"tts-1","tts_voice":"Tingting"}'

# 3. Wake the device ("Hi, StackChan") and speak
# 4. Observe: listen state=start, then WS 1006 within 2–15 s; STT adapter never called
tail -f "~/Library/Application Support/StackChan AI Server/logs/$(date +%F).log"

Impact

Without either change, M5Stack StackChan factory-firmware users are limited to provider=gemini (or OpenAI Realtime). Every already-paid-for text LLM subscription — including locally hosted agents — is unusable with this server, which is arguably the project's main value proposition.

[rudyll]

Thanks — this is an unusually well-instrumented report, and your diagnosis is correct. I have fixed it; the changes are committed and will ship in the next beta.

中文摘要：问题确认，已修复。出厂固件以 auto 模式监听、不发送 listen:stop，而回合制管线在等这条消息，双方互等导致永远不触发 STT。现在服务端自己做静音判定（compatible_server_vad，默认开启，静音窗口 compatible_vad_silence_ms 默认 800ms），设备若先发来 listen:stop 仍以设备为准。同时修掉了一个你接下来必然会撞上的 Base URL 缺陷，并把 realtime 地址做成可配置（openai_base_url）。

What was actually wrong

The turn-based pipeline committed audio only on listen:stop — that was the single code path that handed the buffer to STT. The listen:start handler parsed the mode field but never branched on auto/realtime, so with firmware that leaves end-of-turn to the server, both sides waited. gemini and openai were unaffected because the VAD runs on the provider's side. This was a gap in my server, not a firmware problem.

Two corrections to the report, for the record:

• Audio was arriving. The binary-frame branch had no log line at all, so those zero keyword hits were a logging blind spot rather than an absence of audio — the same decode path feeds Gemini successfully on your device. Your "Debugging note" identified the real cause of that confusion, and it is fixed below.
• The 2–15 s 1006 teardown is device-side and I have no evidence about what triggers it, so I am not treating it as a separate defect. If it still happens once turns complete, please reopen with a log.

A second bug you would have hit next

Base-URL handling was inconsistent. Model discovery appended /models to the configured base URL (so .../v1 was correct — that is why your /v1/models probe returned 200), but the runtime client appended /v1/audio/transcriptions, /v1/chat/completions and /v1/audio/speech. With the https://.../v1 form my own GUI placeholder suggests, every runtime call would have gone to /v1/v1/... and answered 404. The shipped OpenRouter default had the same defect, meaning provider=openrouter could not have worked either. Both spellings are now accepted and discovery uses the same rule, so you will not have to re-test that separately.

What changed

1. Server-side end-of-turn detection for the turn-based pipeline, at the protocol layer. It ends a turn after a configurable trailing silence, requires at least 300 ms of speech first (so a door slam does not fire an empty transcription), and does not cut a turn at a mid-sentence pause. listen:stop still works: whichever signal arrives first ends the turn, the other becomes a no-op. New options compatible_server_vad (default on) and compatible_vad_silence_ms (default 800 ms, range 200–5000; standalone: STACKCHAN_COMPATIBLE_SERVER_VAD, STACKCHAN_COMPATIBLE_VAD_SILENCE_MS).
2. Base-URL normalisation for all three stages, as described above.
3. openai_base_url for the realtime provider. https is converted to wss and a missing /realtime suffix is added, so both https://host/api/paas/v4 and wss://host/v1/realtime work (standalone: STACKCHAN_OPENAI_BASE_URL).
4. Audio-path logging, as you asked: frame and byte counters every 10 seconds, separate counters for frames that arrived while the session was not listening, and the listening mode the device announced on listen:start. A stalled pipeline can now be told apart from missing audio from the log alone. A turn that produces no audio (empty transcript, upstream error) also reports its end now, so the device is not left waiting.

One honest caveat on the realtime base URL

Making the URL configurable was easy; whether GLM-Realtime works through it is a different question. It implements something OpenAI-Realtime-like, and event names, the session.update shape, audio formats and auth may all differ. I am shipping the setting documented as untested against third-party realtime endpoints — a host that only resembles the protocol will connect and then fail on session events. I would rather not claim support I have not verified. If you try it, a log of the first few events after connect would show how close it is, and I will look at what it would take.

What I am asking for

I have no device with factory firmware 1.5.1 here, so the auto-mode path is covered by unit tests only. When you run the next beta, could you post the listening started mode=… line and the audio in frames=… lines from the log? Two things I would like to learn from them: the exact mode string your firmware announces, and whether it re-announces listen:start after each reply. The server handles both cases, but knowing which one you have lets me tune the defaults properly.

[rudyll]

Released, and my earlier version of this comment was wrong about how you would get it — corrected below.

Two separate artifacts, because the macOS app is versioned independently of the add-on:

• macOS (what you are running): macos-v0.1.2 — universal DMG, the download you need. My first reply told you to "pull main and restart", which does not apply to an app-bundle install; sorry for the noise. Quit the running stackchan-server process in Activity Monitor first, then replace the app in Applications. Your settings and history in ~/Library/Application Support/StackChan AI Server are kept, so the STT/LLM/TTS values you already entered stay as they are. Verify the download first with shasum -a 256 -c StackChan-AI-Server-0.1.2-macos-universal.dmg.sha256.
• HA add-on / Docker: v2.8.0-beta.4.

The two new options work at their defaults (compatible_server_vad on, compatible_vad_silence_ms 800), so nothing needs configuring. Your existing stt_*/llm_*/tts_* values with the trailing /v1 need no edit either — that spelling now resolves correctly instead of producing /v1/v1/....

Note that macos-v0.1.2 is the first AGPL-3.0-only build of the app; the release carries the matching corresponding-source archive and checksum. The older macos-v0.1.1 download keeps its original MIT license and was not rebuilt.

I am leaving this open until you confirm it works on your device, since the auto-mode path is covered by unit tests only on my side. The two log lines I would like are listening started mode=… and audio in frames=….