--vad inflates the latency it is measuring (pyannote blocks the event loop)

[jkroll-deepgram]

--vad runs pyannote via loop.run_in_executor(None, run_vad, filename), in the same process as the streaming coroutines. Torch inference holds the GIL for long stretches, which stalls the asyncio event loop and delays the timestamping of inbound messages. The result is that enabling --vad corrupts the measurement it exists to improve.

Same file, same URL, hosted nova-3, interim latency by position in the stream:

position	with --vad p50	max	without --vad p50
0-25 s	572 ms	1468 ms	~112 ms
25-50 s	432 ms	2100 ms	~112 ms
50-100 s	436 ms	3360 ms	~112 ms
100-150 s	104 ms	132 ms	~112 ms
150-202 s	100 ms	136 ms	~112 ms

Pyannote finishes roughly 100 s into a 202 s file and latency immediately returns to normal. Session summary for that run reported Message Latency p95=1.568s p99=3.300s and EOT Latency p95=2.472s p99=4.324s, versus p95=0.140s for the identical config without --vad.

Pacing itself stays correct (the absolute-time anchor means the sender catches up, and max audio_cursor still reached 201.9 s), so the damage is confined to receive-side timestamps — which is exactly what latency is derived from.

Since VAD analyses the file offline and does not depend on the stream at all, it does not need to run concurrently. Options:

1. Run VAD to completion before opening the websocket. Simplest, costs wall-clock time up front, and makes the numbers trustworthy.
2. Run it in a separate process (ProcessPoolExecutor / subprocess) so the GIL is not shared.
3. Cache VAD results keyed on the audio file so repeat runs against the same file skip inference entirely. Useful when sweeping request parameters against one file, which is the common latency-testing workflow.

Option 1 or 3 would have been enough in my case — I ended up reusing the VADResult block from the --vad run and applying it to a separate clean run, since the segments are purely a function of the audio file.

Worth noting the EOT Latency figure is the headline number for anyone evaluating turn-taking latency, and it is currently the most affected.

Separately, run_vad defaults to min_duration_off=0.15, so pyannote absorbs silences under 150 ms into speech and its speech offsets are biased somewhat late. That is a reasonable default for VAD but it does propagate into EOT as a systematic bias, so it may be worth surfacing as a CLI flag.