Repository navigation
perf: one speaker-embedding pass per chunk, not per speaker (#133) - #136
Draft
raphaelm22 wants to merge 2 commits into
Draft
raphaelm22 wants to merge 2 commits into
raphaelm22 wants to merge 2 commits into
Conversation
…intSecure#133) pyannote's get_embeddings runs the WeSpeaker ResNet once per (chunk, local speaker): three passes over the same 10 s of audio, though only the statistics pooling depends on the speaker. The model already pools (batch, speakers, frames) weights in one pass, so run it once per chunk. Embeddings were 96% of diarization time. On a 5-minute excerpt (Core Ultra 7 265U, CPU): 267 s -> 63-100 s, DER 0.0 against the original in the same process, identical speaker centroids.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #133.
Summary
pyannote's
SpeakerDiarization.get_embeddingsruns the WeSpeaker ResNet once per (chunk, local speaker). community-1 has three local speakers per 10 s chunk, so every chunk goes through the network three times. Only the final statistics pooling depends on the speaker, and the model already pools(batch, speakers, frames)weights from one pass.This PR swaps the pipeline's
get_embeddingsfor a version that runs the network once per chunk and pools per speaker. The embedding step was 96% of diarization time.app/transcription/chunk_embeddings.py(new). Mirrors pyannote's method line for line: overlap-free mask selection, NaN handling, batch size and progress hook. The only change is the(batch, speakers, frames)weights in place of one row per speaker.diarizer._get_pipelineinstalls it right afterfrom_pretrained. Not installed when:Measurements
5-minute excerpt of a real 31-minute meeting, Core Ultra 7 265U, CPU, 12 threads:
Inside the app's process (PyQt6 loaded), pyannote's own method and this one agree exactly: DER 0.0000 and identical centroids. Both differ by DER 0.0002 from a run without PyQt6 loaded. That difference comes from the process, not from this change.
Diarization of that 31-minute meeting took 34 minutes, so this brings it to roughly 11. Because the output is unchanged, saved voice profiles (#128) and everything downstream keep working as before.
Tests
tests/test_chunk_embeddings.py, 13 new tests. The suite cannot load torch, so the tests restate pyannote 4.0.7'sget_embeddingsloop in numpy, with a stand-in model that pools the wayStatsPooldoes. They check that:exclude_overlap, for batch sizes 1, 2, 3, 4, 7 and 32;_get_pipelineinstalls the per-chunk method.Full suite: 421 passed.
When pyannote.audio is upgraded,
_pyannote_loopin the test needs re-checking against the newget_embeddings. This is noted in the pipeline rules.🤖 Generated with Claude Code