Skip to content

README.md

Latest commit

 

History

History
651 lines (495 loc) · 20.1 KB

README.md

File metadata and controls

651 lines (495 loc) · 20.1 KB
Raw

ffvoice - Python Bindings

High-performance offline speech recognition library for Python, powered by C++ ffvoice-engine.

Features

⚡ High Performance

  • 3-10x faster than pure Python solutions
  • C++ core with Python ease of use
  • ~272MB memory footprint (vs 1-2GB for pure Python)

🔒 Privacy First

  • 100% offline operation
  • No data uploaded to cloud
  • GDPR/HIPAA compliant

🎙️ Complete Audio Pipeline

  • Real-time audio capture
  • AI-powered noise reduction (RNNoise)
  • Voice Activity Detection (VAD)
  • Offline speech recognition (Whisper)
  • Intelligent audio segmentation

🛠️ Easy to Use

  • Simple Python API
  • One-line installation: pip install ffvoice
  • Comprehensive examples

Installation

Platform Compatibility

Platform PyPI Wheel Installation Method Status
🍎 Apple Silicon (M1/M2/M3) ✅ ARM64 pip install ffvoice ✅ Fully Supported
🍎 Intel Mac ❌ Not Compatible Build from source ⚠️ Manual Build Required
🐧 Linux x86_64 ✅ x86_64 pip install ffvoice ✅ Fully Supported
🪟 Windows x86_64 ✅ x86_64 pip install ffvoice ✅ Fully Supported

Quick Install (Recommended Platforms)

Apple Silicon (M1/M2/M3), Linux x86_64 & Windows x86_64:

pip install ffvoice

Platform-Specific Notes

Apple Silicon Users 🍎

  • ✅ One-line install works out of the box
  • ✅ Native ARM64 performance
  • ✅ No Rosetta 2 required
  • Important: Make sure Python is running in ARM64 mode: 
    python -c "import platform; print(platform.machine())"
# Should output: arm64

Intel Mac Users 🍎

  • ⚠️ PyPI wheel is not compatible with Intel Macs
  • Solution: Build from source (see instructions below)
  • The ARM64 wheel cannot be used even with Rosetta 2 translation

Rosetta 2 Users ⚠️

  • If you see ImportError: mach-o file, but is an incompatible architecture, you're running x86_64 Python
  • Fix: Use native ARM64 Python instead: 
    # Force ARM64 architecture
arch -arm64 python3 -m pip install ffvoice

# Or permanently switch to ARM64 Python
arch -arm64 /bin/bash
python3 -m pip install ffvoice

Linux Users 🐧

  • ✅ Direct install from PyPI works
  • Tested on Ubuntu 20.04+, Debian 11+
  • Requires system libraries (see Prerequisites below)

Windows Users 🪟

  • ✅ One-line install works with prebuilt x86_64 wheels
  • ✅ Supports Python 3.9-3.12
  • ✅ Bundles required dependencies (no manual FFmpeg install needed)
  • Note: RNNoise noise reduction is disabled on Windows (MSVC lacks VLA support); all other features work

Prerequisites

System Dependencies:

macOS:

brew install ffmpeg portaudio flac cmake

Linux (Ubuntu/Debian):

sudo apt-get install libavcodec-dev libavformat-dev libavutil-dev \
                     libswresample-dev portaudio19-dev libflac-dev cmake

Python Requirements:

  • Python 3.7 or later
  • pip

Install from Source

# Clone repository
git clone https://github.com/chicogong/ffvoice-engine.git
cd ffvoice-engine

# Install Python package
pip install .

# Or install in development mode
pip install -e .

Build Options

The installation automatically enables:

  • RNNoise noise suppression
  • Whisper ASR

To customize the build:

# Set environment variables before pip install
export CMAKE_ARGS="-DENABLE_RNNOISE=ON -DENABLE_WHISPER=ON"
pip install .

Quick Start

Basic Transcription

import ffvoice

# Configure Whisper ASR
config = ffvoice.WhisperConfig()
config.model_type = ffvoice.WhisperModelType.TINY
config.language = "auto"  # Auto-detect language

# Initialize ASR
asr = ffvoice.WhisperASR(config)
asr.initialize()

# Transcribe audio file
segments = asr.transcribe_file("audio.wav")

# Print results
for segment in segments:
    print(f"[{segment.start_ms}ms -> {segment.end_ms}ms]")
    print(f"  {segment.text} (confidence: {segment.confidence:.2f})")

NumPy Array Support

import ffvoice
import numpy as np

# Load audio as NumPy array
audio = np.zeros(48000, dtype=np.int16)  # 1 second at 48kHz

# Transcribe from NumPy array
config = ffvoice.WhisperConfig()
config.model_type = ffvoice.WhisperModelType.TINY
asr = ffvoice.WhisperASR(config)
asr.initialize()

segments = asr.transcribe_buffer(audio)
for segment in segments:
    print(segment.text)

Real-time Audio Capture with Callback

import ffvoice
import numpy as np

# Initialize audio capture
ffvoice.AudioCapture.initialize()

# List available devices
devices = ffvoice.AudioCapture.get_devices()
for device in devices:
    print(f"{device.id}: {device.name} (channels: {device.max_input_channels})")

# Create capture instance
capture = ffvoice.AudioCapture()
capture.open(sample_rate=48000, channels=1, frames_per_buffer=256)

# Define callback to process audio
def audio_callback(audio_array):
    """Receives NumPy array with audio samples"""
    print(f"Received {len(audio_array)} samples")
    # Process audio here...

# Start capture with callback
capture.start(audio_callback)

# ... capture runs in background ...

# Stop capture
capture.stop()
capture.close()
ffvoice.AudioCapture.terminate()

Noise Reduction with NumPy

import ffvoice
import numpy as np

# Configure RNNoise
config = ffvoice.RNNoiseConfig()
config.enable_vad = True

# Initialize noise reduction
rnnoise = ffvoice.RNNoise(config)
rnnoise.initialize(sample_rate=48000, channels=1)

# Process audio from NumPy array (in-place modification)
audio = np.random.randint(-1000, 1000, 256, dtype=np.int16)
rnnoise.process(audio)  # Audio array is modified in-place

# Get VAD probability
vad_prob = rnnoise.get_vad_probability()
print(f"Voice activity: {vad_prob:.2%}")

Voice Activity Detection with Callbacks

import ffvoice
import numpy as np

# Create VAD config with preset
config = ffvoice.VADConfig.from_preset(ffvoice.VADSensitivity.BALANCED)

# Initialize VAD segmenter — constructor takes only config, no sample_rate arg
vad = ffvoice.VADSegmenter(config)

# Define callback for complete segments
def segment_callback(segment_array):
    """Called when a complete speech segment is detected"""
    print(f"Speech segment: {len(segment_array)} samples")
    # Process or save the segment...

# Process audio frames with VAD
audio_frame = np.zeros(256, dtype=np.int16)
vad_prob = 0.8  # From RNNoise

vad.process_frame(audio_frame, vad_prob, segment_callback)

# Flush remaining audio at the end
vad.flush(segment_callback)

# get_statistics() returns a tuple (avg_vad_prob, speech_ratio), NOT a dict
avg_vad_prob, speech_ratio = vad.get_statistics()
print(f"Average VAD: {avg_vad_prob:.2f}")
print(f"Speech ratio: {speech_ratio:.2%}")
print(f"Is in speech: {vad.is_in_speech()}")

Writing Audio Files from NumPy

import ffvoice
import numpy as np

# Create audio data
sample_rate = 48000
audio = np.random.randint(-1000, 1000, 48000, dtype=np.int16)

# Write WAV file
wav_writer = ffvoice.WAVWriter()
wav_writer.open("output.wav", sample_rate, channels=1)
samples_written = wav_writer.write_samples_array(audio)
wav_writer.close()
print(f"Wrote {samples_written} samples to WAV")

# Write FLAC file (with compression)
flac_writer = ffvoice.FLACWriter()
flac_writer.open("output.flac", sample_rate, channels=1, bits_per_sample=16, compression_level=5)
samples_written = flac_writer.write_samples_array(audio)
compression_ratio = flac_writer.get_compression_ratio()
flac_writer.close()
print(f"Wrote {samples_written} samples to FLAC (compression: {compression_ratio:.2f}x)")

Word-Level Timestamps

import ffvoice

# Enable per-word timestamps in the Whisper config
config = ffvoice.WhisperConfig()
config.model_type = ffvoice.WhisperModelType.TINY
config.word_timestamps = True  # Populate `words` for every segment

asr = ffvoice.WhisperASR(config)
asr.initialize()

segments = asr.transcribe_file("audio.wav")
for segment in segments:
    print(f"[{segment.start_ms}ms -> {segment.end_ms}ms] {segment.text}")
    # Each segment now carries a list of `Word` objects
    for word in segment.words:
        print(f"  {word.start_ms}-{word.end_ms}ms  "
              f"'{word.text}'  (p={word.probability:.2f})")

When transcribing a NumPy buffer whose sample rate is not 48000 Hz, set config.input_sample_rate so the audio is resampled correctly:

config = ffvoice.WhisperConfig()
config.input_sample_rate = 16000  # Sample rate of the buffer passed to transcribe_buffer()

Multi-Track Audio Mixing

import ffvoice
import numpy as np

# Create and initialize a mixer (channels must be 1 or 2)
mixer = ffvoice.AudioMixer()
mixer.initialize(sample_rate=48000, channels=2)

# Add tracks; add_track() returns the new track id
voice = mixer.add_track(gain=1.0, pan=0.0)    # centered
music = mixer.add_track(gain=0.5, pan=-0.3)   # quieter, slightly left

# Adjust tracks at any time
mixer.set_gain(music, 0.4)
mixer.set_pan(voice, 0.2)
mixer.set_mute(music, False)
mixer.set_master_gain(0.9)

# Mix a block: {track_id: int16 ndarray}; all arrays must be the same length
voice_block = np.random.randint(-2000, 2000, 480, dtype=np.int16)
music_block = np.random.randint(-2000, 2000, 480, dtype=np.int16)
mixed = mixer.mix_block({voice: voice_block, music: music_block})
print(f"Mixed {len(mixed)} samples across {mixer.get_track_count()} tracks")

Lock-Free Ring Buffer

import ffvoice
import numpy as np

# SPSC ring buffer holding up to `capacity` int16 samples
buf = ffvoice.RingBuffer(capacity=4096)

# Bulk transfer with NumPy (e.g. audio thread -> processing thread)
samples = np.random.randint(-1000, 1000, 1024, dtype=np.int16)
pushed = buf.push_bulk(samples)          # number actually pushed
print(f"pushed={pushed}, size={buf.size()}, full={buf.full()}")

chunk = buf.pop_bulk(512)                # 1-D int16 ndarray (may be shorter)
print(f"popped {len(chunk)} samples, remaining={buf.size()}")

# Single-element access
buf.push(42)
value = buf.pop()                        # int, or None if empty
buf.clear()

API Reference

Core Classes

WhisperASR

Offline speech recognition using Whisper models.

Methods:

  • initialize() - Load Whisper model
  • transcribe_file(filename) - Transcribe audio file, returns list of TranscriptionSegment
  • transcribe_buffer(audio_array) - Transcribe from NumPy array (int16, 1D), returns list of TranscriptionSegment
  • get_last_error() - Get last error message (string)
  • get_last_inference_time_ms() - Get inference time in milliseconds (int)
  • is_initialized() - Check if model is loaded (bool)

AudioCapture

Real-time audio capture from microphone with callback support.

Methods:

  • open(device_id=-1, sample_rate=48000, channels=1, frames_per_buffer=256) - Open audio device. Note: the parameter is device_id (not device_index).
  • start(callback) - Start capture with Python callback receiving NumPy arrays (int16, 1D)
  • stop() - Stop audio capture
  • close() - Close audio device
  • is_open() - Check if device is open (bool)
  • is_capturing() - Check if currently capturing (bool)
  • get_sample_rate() - Get sample rate (int)
  • get_channels() - Get number of channels (int)

Static Methods:

  • initialize() - Initialize PortAudio system
  • terminate() - Terminate PortAudio system
  • get_devices() - Get list of AudioDeviceInfo objects
  • get_default_input_device() - Get default input device ID (int)

RNNoise

AI-powered noise reduction with NumPy support.

Methods:

  • initialize(sample_rate, channels) - Initialize processor
  • process(audio_array) - Process NumPy array in-place (modifies input)
  • reset() - Reset internal state
  • get_vad_probability() - Get VAD probability 0.0-1.0 (float)

VADSegmenter

Voice activity detection and intelligent segmentation with callbacks.

Constructor: VADSegmenter(config) — takes only a VADConfig; there is no sample_rate argument.

Methods:

  • process_frame(audio_array, vad_prob, callback) - Process audio frame with callback for complete segments
  • flush(callback) - Flush remaining audio with callback
  • reset() - Reset segmenter state
  • is_in_speech() - Check if currently in speech (bool)
  • get_buffer_size() - Get current buffer size in samples (int)
  • get_current_threshold() - Get adaptive threshold (float)
  • get_statistics() - Get VAD statistics as a tuple (avg_vad_prob, speech_ratio); unpack with avg_vad_prob, speech_ratio = vad.get_statistics() — it is not a dict.

WAVWriter

Write audio to WAV files with NumPy support.

Methods:

  • open(filename, sample_rate, channels, bits_per_sample=16) - Open WAV file for writing
  • write_samples_array(audio_array) - Write NumPy array (int16, 1D) to file, returns samples written (int)
  • close() - Close file and finalize headers
  • is_open() - Check if file is open (bool)
  • get_total_samples() - Get total samples written (int)

FLACWriter

Write audio to FLAC files with compression and NumPy support.

Methods:

  • open(filename, sample_rate, channels, bits_per_sample=16, compression_level=5) - Open FLAC file
    • compression_level: 0 (fastest) to 8 (best compression), default 5
  • write_samples_array(audio_array) - Write NumPy array (int16, 1D) to file, returns samples written (int)
  • close() - Close file and finalize
  • get_compression_ratio() - Get compression ratio (float)
  • is_open() - Check if file is open (bool)
  • get_total_samples() - Get total samples written (int)

AudioMixer

Multi-track mixer that combines several int16 audio tracks into one output.

Methods:

  • initialize(sample_rate, channels) - Initialize the mixer (channels must be 1 or 2)
  • is_initialized() - Check if the mixer is initialized (bool)
  • get_sample_rate() / get_channels() - Get the mixer configuration (int)
  • add_track(gain=1.0, pan=0.0) - Add a track, returns its track id (int; -1 if not initialized)
  • remove_track(track_id) - Remove a track, returns True if it existed (bool)
  • has_track(track_id) - Check whether a track exists (bool)
  • get_track_count() - Number of registered tracks (int)
  • set_gain(track_id, gain) / get_gain(track_id) - Track linear gain (clamped to [0, 8])
  • set_pan(track_id, pan) / get_pan(track_id) - Track stereo pan (-1 = left, 0 = center, +1 = right)
  • set_mute(track_id, muted) / is_muted(track_id) - Mute or unmute a track
  • set_master_gain(gain) / get_master_gain() - Master gain applied to the mix (clamped to [0, 8])
  • mix_block(tracks) - Mix {track_id: int16 ndarray} into a single mixed int16 ndarray; all arrays must have the same length
  • reset() - Remove all tracks and reset the master gain

RingBuffer

Lock-free single-producer/single-consumer (SPSC) ring buffer for int16 audio samples.

Constructor:

  • RingBuffer(capacity) - Create a buffer holding up to capacity int16 samples

Methods:

  • push(value) - Push one value, returns False if the buffer is full (bool)
  • pop() - Pop one value, returns None if the buffer is empty (int or None)
  • push_bulk(data) - Push values from a 1-D int16 ndarray, returns the number actually pushed (int)
  • pop_bulk(count) - Pop up to count values, returns them as a 1-D int16 ndarray
  • size() / capacity() - Current element count and maximum capacity (int)
  • empty() / full() - Check buffer state (bool)
  • available_read() - Number of elements available to pop (int)
  • available_write() - Number of elements that can still be pushed (int)
  • clear() - Reset the buffer to empty

Data Classes

TranscriptionSegment

Speech recognition result with timestamp.

Fields:

  • start_ms - Start time in milliseconds (int)
  • end_ms - End time in milliseconds (int)
  • text - Transcribed text (string)
  • confidence - Confidence score 0.0-1.0 (float)
  • words - List of Word objects (empty unless WhisperConfig.word_timestamps was enabled)

Word

Per-word timestamp within a TranscriptionSegment.

Fields:

  • start_ms - Word start time in milliseconds (int)
  • end_ms - Word end time in milliseconds (int)
  • text - Word text (string)
  • probability - Mean token probability for this word, 0.0-1.0 (float)

AudioDeviceInfo

Audio device information.

Fields:

  • id - Device ID (int)
  • name - Device name (string)
  • max_input_channels - Maximum input channels (int)
  • max_output_channels - Maximum output channels (int)
  • supported_sample_rates - List of supported sample rates (list of int)
  • is_default - Is default device (bool)

Configuration Classes

WhisperConfig

  • model_path - Path to Whisper model file
  • language - Language code ('en', 'zh', 'auto')
  • model_type - Model size (TINY, BASE, SMALL, MEDIUM, LARGE)
  • n_threads - Number of CPU threads
  • enable_performance_metrics - Enable timing metrics
  • word_timestamps - Populate per-word timestamps in each TranscriptionSegment.words (bool, default False)
  • input_sample_rate - Sample rate (Hz) of audio passed to transcribe_buffer() (int, default 48000)

AudioCapture.open() parameters

There is no separate AudioCaptureConfig class. Pass these arguments directly to AudioCapture.open():

  • device_id - Audio device ID (-1 for system default)
  • sample_rate - Sample rate in Hz (default: 48000)
  • channels - Number of channels (default: 1)
  • frames_per_buffer - Buffer size in frames (default: 256)

VADConfig

  • speech_threshold - VAD probability threshold
  • min_speech_frames - Min frames to start speech
  • min_silence_frames - Min frames to end speech
  • enable_adaptive_threshold - Enable adaptive adjustment
  • from_preset(sensitivity) (static) - Create from preset

Enums

WhisperModelType

  • TINY - Fastest (~39MB, ~10x realtime)
  • BASE - Balanced (~74MB, ~7x realtime)
  • SMALL - Better accuracy (~244MB, ~3x realtime)
  • MEDIUM - High accuracy (~769MB, ~1x realtime)
  • LARGE - Best accuracy (~1550MB, <1x realtime)

VADSensitivity

  • VERY_SENSITIVE - Detect very quiet speech (threshold=0.3)
  • SENSITIVE - Detect quiet speech (threshold=0.4)
  • BALANCED - Balanced detection (threshold=0.5)
  • CONSERVATIVE - Avoid false positives (threshold=0.6)
  • VERY_CONSERVATIVE - Only clear speech (threshold=0.7)

Examples

See the examples directory for complete working examples:

  • basic_transcription.py - Simple file transcription
  • realtime_transcription.py - Real-time audio processing
  • complete_realtime_pipeline.py - End-to-end capture, denoise, VAD and recognition pipeline

Performance

Benchmark on Apple M1 Pro (8 cores):

Model Speed (RTF) Memory Use Case
TINY ~10x ~272MB Real-time, fastest
BASE ~7x ~345MB Balanced
SMALL ~3x ~512MB Better accuracy
MEDIUM ~1x ~1.2GB High quality
LARGE <1x ~2.1GB Best quality

RTF = Real-time Factor (higher is faster)

Troubleshooting

Import Error

If you get ImportError: Failed to import ffvoice native module:

  1. Ensure all system dependencies are installed
  2. Rebuild the package: pip install --force-reinstall .
  3. Check CMake build logs for errors

Audio Capture Issues

If audio capture fails:

  1. List available devices: ffvoice.AudioCapture.get_devices()
  2. Check device permissions (microphone access)
  3. Try a different device_id in AudioCapture.open()

Model Loading Issues

If Whisper model fails to load:

  1. Check model file exists at the specified path
  2. Ensure you have enough disk space
  3. Verify model file is not corrupted

Development

Running Tests

# Install dev dependencies
pip install -e .[dev]

# Run tests
pytest python/tests -v

Code Formatting

# Format code
black python/

# Check style
flake8 python/

# Type checking
mypy python/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Links