Add qwentts-serve HTTP server, WebUI, device selection and configurable talker KV context

[soymh]

Hi,

I was working on a server binary and some other stuff. This PR was opened by me but written and done by my agent, so I want to be transparent about the process from the start.

We were not aware of the project's AI submission rules when we started this work. Because of limited time, we opened the PR in this way. If that is not acceptable, we apologize, and you are free to close it.

This branch is based on the latest upstream master (0 commits behind) and only adds our changes on top. Here is a detailed description of each change.

qwentts-serve: a new OpenAI-compatible HTTP server

A separate server binary (qwentts-serve) following the llama-server pattern, built on the same qwen-core library as the existing tools. It shares the vendored HTTP and JSON libraries (cpp-httplib, yyjson) already used by tts-server.

Endpoints:

• GET /health - liveness check.
• GET /v1/models - reports the loaded model (with an optional --alias).
• GET /v1/voices - lists the named CustomVoice speakers of the loaded model.
• POST /v1/audio/speech - OpenAI-compatible synthesis. Supports response_format=wav (one-shot RIFF) and response_format=pcm (real-time streaming of s16le 24 kHz chunks as they are generated). Accepts voice, instruct, and standard sampling overrides per request.
• POST /v1/audio/speech/cancel - cancels the in-flight synthesis.

The PCM streaming path runs synthesis on its own thread and pushes bytes into a queue that the connection thread drains into a chunked response, so a slow client cannot stall the batched worker. A client disconnect aborts generation and frees the GPU.

While merging, we also ported qwentts-serve to the new qt_init_params API (the codec chunk configuration moved from the per-request TTS params into qt_init_params), so it stays compatible with the current upstream internals.

WebUI (tools/webui/)

A browser frontend (FastAPI backend + plain HTML/JS/EventSource page, no build step) that proxies to the HTTP server.

• Streaming mode: the page splits input into sentences and synthesizes each one as it arrives, playing them back-to-back as a continuous stream.
• One-shot mode (--one-shot, or per-request oneshot): the whole input is synthesized as a single utterance so the model sees the complete context.
• WAV download: every generated chunk is collected and stitched into one WAV file (keeps the first RIFF header, concatenates the PCM payloads, fixes the size fields), downloadable via a button.
• Automatic fallback: if one-shot fails because the prompt overflows the talker KV cache, the page notices and transparently re-synthesizes per sentence, emitting a notice event. This keeps long text working on small GPUs.
• split_sentences now protects ellipsis (... and ... unicode) so they no longer split a sentence.

Backend device selection

New --device <name> and --list-devices flags (also exposed in the library API as qt_init_params.device and qt_list_devices()). The backend can be pinned explicitly to a device such as cuda0, vulkan, or cpu instead of relying only on the GGML_BACKEND environment variable and auto-selection.

Configurable talker KV cache size

New --talker-ctx <n> flag (and qt_init_params.talker_max_ctx) that sizes the talker KV cache in positions instead of the hardcoded 4096. Default remains 4096 (0 selects it). This is useful for one-shot synthesis of long prompts where the VRAM allows a larger context.

models.sh fix

The download script pointed at a stale repository (Serveurperso/qwentts.cpp-GGUF) with outdated file names. It now points at the current Serveurperso/Qwen3-TTS-GGUF repository and downloads qwen-tokenizer-12hz-Q8_0.gguf, qwen-talker-1.7b-customvoice-Q8_0.gguf, and qwen-talker-1.7b-customvoice-Q4_K_M.gguf.

Verification

Built with CUDA (SM86) and verified end to end on an RTX 3050 Laptop with 4 GB VRAM:

• test-abi-c (the C99 ABI lock-in test) passes.
• --list-devices reports CUDA0 and CPU.
• CLI synthesis with the 1.7B CustomVoice model works (--speaker vivian, --lang English).
• Both servers (tts-server and qwentts-serve) served valid 24 kHz WAV and streaming PCM audio; the WebUI synthesized and streamed audio over SSE.
• The 1.7B model at Q4_K_M fits on 4 GB VRAM (peaks ~3.5 GB); Q8_0 does not fit, so the webui one-shot fallback is what makes long text practical on this GPU.

We understand this work may not be merged, and that is fine. Our main goal was to share the changes in case they are useful to the project. We are happy to adjust, split, or rework anything the maintainers prefer.

Thank you for the project.