Skip to content

perf: one speaker-embedding pass per chunk, not per speaker (#133) - #136

Draft
raphaelm22 wants to merge 2 commits into
ObscureAintSecure:masterfrom
raphaelm22:feat/faster-diarization
Draft

raphaelm22 wants to merge 2 commits into
ObscureAintSecure:masterfrom
raphaelm22:feat/faster-diarization

Conversation

@raphaelm22

Copy link
Copy Markdown

Closes #133.

Summary

pyannote's SpeakerDiarization.get_embeddings runs the WeSpeaker ResNet once per (chunk, local speaker). community-1 has three local speakers per 10 s chunk, so every chunk goes through the network three times. Only the final statistics pooling depends on the speaker, and the model already pools (batch, speakers, frames) weights from one pass.

This PR swaps the pipeline's get_embeddings for a version that runs the network once per chunk and pools per speaker. The embedding step was 96% of diarization time.

  • app/transcription/chunk_embeddings.py (new). Mirrors pyannote's method line for line: overlap-free mask selection, NaN handling, batch size and progress hook. The only change is the (batch, speakers, frames) weights in place of one row per speaker.
  • diarizer._get_pipeline installs it right after from_pretrained. Not installed when:
    • the embedding model is another backend (SpeechBrain, NeMo, ONNX), which takes one mask per waveform;
    • the pipeline is training; that path also keeps pyannote's own method.

Measurements

5-minute excerpt of a real 31-minute meeting, Core Ultra 7 265U, CPU, 12 threads:

Diarization DER vs original Speaker centroids vs original
pyannote as is 267 s (385 s on a busier run) — —
One pass per chunk 63–100 s (~3x) 0.0000 cosine 1.0 for all 6

Inside the app's process (PyQt6 loaded), pyannote's own method and this one agree exactly: DER 0.0000 and identical centroids. Both differ by DER 0.0002 from a run without PyQt6 loaded. That difference comes from the process, not from this change.

Diarization of that 31-minute meeting took 34 minutes, so this brings it to roughly 11. Because the output is unchanged, saved voice profiles (#128) and everything downstream keep working as before.

Tests

tests/test_chunk_embeddings.py, 13 new tests. The suite cannot load torch, so the tests restate pyannote 4.0.7's get_embeddings loop in numpy, with a stand-in model that pools the way StatsPool does. They check that:

  • per-chunk output equals that loop, with and without exclude_overlap, for batch sizes 1, 2, 3, 4, 7 and 32;
  • inactive speakers are pooled with zero weights, as pyannote does;
  • the model runs once per batch of chunks rather than once per speaker;
  • the progress hook reports every batch;
  • the pipeline falls back to pyannote's method for other embedding models and for training;
  • _get_pipeline installs the per-chunk method.

Full suite: 421 passed.

When pyannote.audio is upgraded, _pyannote_loop in the test needs re-checking against the new get_embeddings. This is noted in the pipeline rules.

🤖 Generated with Claude Code

…intSecure#133)

pyannote's get_embeddings runs the WeSpeaker ResNet once per (chunk, local speaker): three passes over the same 10 s of audio, though only the statistics pooling depends on the speaker. The model already pools (batch, speakers, frames) weights in one pass, so run it once per chunk.

Embeddings were 96% of diarization time. On a 5-minute excerpt (Core Ultra 7 265U, CPU): 267 s -> 63-100 s, DER 0.0 against the original in the same process, identical speaker centroids.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Diarization runs the speaker-embedding network three times per chunk (~3x slower than needed)

1 participant