DeepFilterNet-mlx

Standalone Swift/MLX implementation of DeepFilterNet and CEVA DPDFNet real-time speech enhancement for Apple Silicon.

Supports offline batch processing and stateful per-hop streaming through MLX, Accelerate, and Core ML. Core ML models are downloaded automatically from public, versioned Hugging Face repositories or can be bundled with an app.

Performance

Benchmarked on Apple Silicon with the 52.128-second Extract-short.wav fixture. The stereo 44.1 kHz source is equal-channel downmixed and resampled to each model's native sample rate by the public audio loader.

Offline

Configuration	Time	Real-Time Factor	Notes
Hybrid (recommended)	0.47s	112x	Compiled graphs + Accelerate GRU
MLX GPU (throughput)	0.55s	95x	Metal fused kernels + Accelerate GRU
MLX GPU (pre-optimization)	1.55s	33.5x	Baseline before Accelerate GRU
DeepFilterNet3 Core ML streaming graph	1.49s	34.9x	Stateful 10 ms FP16 graph; computeUnits = .all

Streaming

Configuration	Per-Hop Latency	Real-Time Factor
MLX GPU / Hybrid	~2.0ms	4.9x
CPU + Accelerate	~8.0ms	1.2x
DeepFilterNet3 Core ML	0.264ms unpaced steady state	34.9x

The stateful Core ML graph emits one enhanced hop per prediction after the model's fixed lookahead. A Mac live-clock run is recorded in docs/live-device-benchmark.md; the older iPad MLX/Accelerate figures remain historical until the new Core ML graph is rerun on that device.

Output Quality

Comparison	Correlation
MLX vs PyTorch reference	0.9999984
Hybrid vs MLX	1.0000000 (bit-identical)
vs mlx-audio-swift	1.0000000 (bit-identical)

The Hybrid configuration is the fastest for offline processing. It uses compile() for conv blocks and the Accelerate-optimized GRU (batch GPU input projection + CPU vDSP_mmul hidden projection), eliminating ~15,000 sequential GPU dispatches across 3 GRU layers. For streaming, MLX GPU and Hybrid perform identically at ~2ms/hop.

See docs/performance-optimizations.md for a detailed breakdown of all optimizations, and docs/benchmark-report.md for the full 4-engine benchmark comparison.

Published models

Profile	Hugging Face repository	Recommended runtime
DeepFilterNet3	iky1e/DeepFilterNet3-Streaming-CoreML	Core ML streaming
DPDFNet-4	iky1e/DPDFNet4-CoreML	Core ML explicit FP32 state
DPDFNet-8 48 kHz HR	iky1e/DPDFNet8-48kHz-HR-CoreML	Core ML explicit FP32 state

Each Core ML repository is self-contained and includes matching Swift model configuration/weights, required state files, licenses, validation metadata, and SHA-256 checksums.

What is implemented

• Full Swift MLX DeepFilterNet runtime (DeepFilterNet, DeepFilterNet2, DeepFilterNet3 checkpoints)
• DPDFNet-4 and DPDFNet-8 48 kHz HR offline and stateful streaming runtimes, validated against the official ONNX models
• Offline enhancement (enhance(_:)) with Accelerate-optimized GRU
• Stateful streaming enhancement (DeepFilterNetStreamer) with ~2ms/hop latency
• Local/Hugging Face model loading (config.json + .safetensors)
• Standalone DSP layer (STFT/ISTFT) without mlx-audio-swift dependency
• Custom fused Metal kernels via MLXFast:
  • GRU gate fusion (sigmoid + tanh + update in one dispatch)
  • ERB mask multiply and ERB inverse mask apply
  • Streaming and offline deep-filter complex multiply-accumulate
• Accelerate-optimized GRU for offline (batch GPU matmul + CPU vDSP_mmul)
• Compiled graph caching for analysis, synthesis, and decoder subgraphs
• Pre-computed weight caches (conv OHWI, batch norm affine, GRU transpose)
• NHWC memory layout for streaming encoder (eliminates intermediate transposes)
• Tensor ring buffers for zero-allocation frame history

Repo layout

Path	Description
Sources/DeepFilterNetMLX	Core library (model, streaming, DSP, kernels)
Sources/DeepFilterNetCoreML	Production Core ML API, model loader, and stateful streaming runtimes
Sources/deepfilternet-mlx-cli	CLI tool for file enhancement
Sources/DeepFilterNetBenchmark	Benchmark framework and adapters for MLX, Accelerate, hybrid, and Core ML engines
Sources/deepfilternet-benchmark-cli	Benchmark CLI tool
BenchmarkApp	XcodeGen-based macOS/iOS live-device benchmark app
Examples	External application integration samples
Scripts/Conversion	PyTorch/ONNX to MLX and Core ML conversion tools
Scripts/Benchmarking	Fidelity, throughput, and live-latency benchmarks
Scripts/Visualization	Report and chart generation
Tests/DeepFilterNetMLXTests	Smoke tests
docs/	Performance reports and optimization documentation

Build

swift build -c release

Swift Package Manager

Add https://github.com/kylehowells/DeepFilterNet-mlx as a package dependency, then link DeepFilterNetMLX, DeepFilterNetCoreML, or both. The package supports macOS 14+, iOS 17+, and Apple Silicon.

CLI usage

# Offline enhancement (recommended for files)
swift run -c release deepfilternet-mlx /path/to/noisy.wav \
  --output /path/to/enhanced.wav

# Streaming mode (10ms hops)
swift run -c release deepfilternet-mlx /path/to/noisy.wav \
  --stream \
  --output /path/to/enhanced.wav

# Download model from HuggingFace automatically
swift run -c release deepfilternet-mlx /path/to/noisy.wav \
  --model iky1e/DeepFilterNet3-MLX

# Performance presets
--performance throughput   # (default) all optimizations enabled
--performance safe         # disable fused kernels for debugging

# DPDFNet streaming recurrent backend
--dpdfnet-backend accelerate  # default; MLX conv/decoder + Accelerate DPRNN
--dpdfnet-backend mlx         # fully fused GPU DPRNN implementation

Library usage

Core ML streaming

The default configuration selects DeepFilterNet3, permits all Core ML compute units, and downloads the validated model from iky1e/DeepFilterNet3-Streaming-CoreML:

import DeepFilterNetCoreML

let enhancer = try await DeepFilterNetCoreMLStreamer.load()

// Exactly one 10 ms mono hop: 480 samples for this 48 kHz model.
let output = try enhancer.processHop(inputHop)
let tail = try enhancer.flush()

Select DPDFNet explicitly when its rate/quality profile is required:

let realtime16k = try await DeepFilterNetCoreMLStreamer.load(
    configuration: .init(variant: .dpdfNet4)
)

let highResolution48k = try await DeepFilterNetCoreMLStreamer.load(
    configuration: .init(variant: .dpdfNet8HighResolution)
)

Models can instead be loaded from .local(URL) or .bundle(Bundle, subdirectory:) sources. See docs/coreml.md for model selection, caching, fixed delay, and real-time integration. A complete wrapper is available in Examples/CoreMLStreaming.swift.

MLX

import DeepFilterNetMLX
import MLX

// Load model (downloads from HuggingFace if needed)
let model = try await DeepFilterNetModel.fromPretrained("iky1e/DeepFilterNet3-MLX")

// Offline enhancement (fastest for file processing)
model.configurePerformance(.throughput)
let enhanced = try model.enhance(inputAudio)

// Streaming enhancement (for real-time processing)
let streamer = model.createStreamer(config: DeepFilterNetStreamingConfig(
    compensateDelay: true,
    materializeEveryHops: 512
))
for chunk in audioChunks {
    let output = try streamer.processChunk(chunk)
    // output is enhanced audio, one hop at a time
}
let tail = try streamer.flush()

Or file-based helper:

try model.enhanceFile(
    inputURL: inputURL,
    outputURL: outputURL,
    useStreaming: false  // use offline for best speed
)

Benchmarking

The benchmark tool compares MLX, Accelerate, hybrid, and Core ML streaming implementations:

Engine	Description
MLX GPU	Production engine with Metal kernels + Accelerate GRU
CPU + Accelerate	Full neural net reimplemented in Accelerate/vDSP/BLAS
Hybrid	MLX with compile() for conv blocks, Accelerate GRU
DeepFilterNet3 Core ML	Stateful one-hop Core ML graph with explicit GRU state

# Build benchmark tool
swift build -c release --product deepfilternet-benchmark

# Run all engines
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines all --runs 3 --save-output

# Simulate 10 seconds of live 48kHz microphone callbacks. This reports
# processing jitter, deadline misses, queue delay, and capture-to-output latency.
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines all --skip-offline --skip-streaming --live --live-duration 10

# Measure queued-hop amortization and recovery from one 15 ms scheduler stall.
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines mlx --skip-offline --stream-batch-sizes 1,8 \
  --live --live-duration 5 --catch-up --max-catch-up-hops 8 \
  --inject-stall-hop 100 --inject-stall-ms 15

# Run specific engines
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines mlx,hybrid --runs 3

Model conversion

Pre-converted models are available on HuggingFace (iky1e/DeepFilterNet3-MLX). To convert from PyTorch:

python Scripts/Conversion/convert_deepfilternet.py \
  --input /path/to/DeepFilterNet/checkpoint_dir \
  --output /path/to/DeepFilterNet3 \
  --name DeepFilterNet3

Export the stateful Core ML 10 ms graph from the official PyTorch checkpoint:

PYTHONPATH=/path/to/DeepFilterNet/DeepFilterNet \
python Scripts/Conversion/convert_deepfilternet_to_coreml.py \
  --model-dir /path/to/DeepFilterNet3 \
  --output /path/to/DeepFilterNet3-Streaming.mlpackage

DPDFNet checkpoints use a separate converter:

python Scripts/Conversion/convert_dpdfnet.py \
  --checkpoint /path/to/dpdfnet4.pth \
  --profile dpdfnet4 \
  --output /path/to/output/dpdfnet4

# High-resolution 48 kHz profile
python Scripts/Conversion/convert_dpdfnet.py \
  --checkpoint /path/to/dpdfnet8_48khz_hr.pth \
  --profile dpdfnet8_hr \
  --output /path/to/output/dpdfnet8_48khz_hr

# Export both DPDFNet profiles as deployable Core ML packages and compiled models.
# This writes FP32 explicit-state (production), FP16 explicit-state, and FP16
# MLState variants under --model-root/coreml.
python Scripts/Conversion/convert_dpdfnet_to_coreml.py \
  --model-root /path/to/dpdfnet-assets --profile all --force

Run the official ONNX fidelity gate and three-run Swift benchmark on Extract-short.wav with:

python Scripts/Benchmarking/benchmark_dpdfnet.py --input /path/to/test.wav \
  --variants swift_offline swift_stream_accelerate swift_stream_coreml_fp32 \
  --repeats 3 \
  --live-runs 3 \
  --live-duration 10 \
  --live-backends mlx accelerate coreml-fp32

# Repeat the same fidelity, throughput, and live-latency pipeline for HR.
python Scripts/Benchmarking/benchmark_dpdfnet.py --input /path/to/test.wav \
  --profile dpdfnet8_hr \
  --variants swift_offline swift_stream_accelerate swift_stream_coreml_fp32 \
  --repeats 3 \
  --live-runs 3 \
  --live-duration 10 \
  --live-backends mlx accelerate coreml-fp32

The canonical JSON, Markdown, WAV, and PNG results are written directly to outputs/compare and outputs/benchmark. The benchmark reports the input duration, total throughput, steady 10 ms callback latency, capture-to-output audio age, deadline misses, and official ONNX fidelity. See docs/dpdfnet.md for architecture, model status, and current measurements.

Run the cross-version one-minute throughput, live recovery, and official reference fidelity matrix with:

python3 Scripts/Benchmarking/benchmark_catchup_matrix.py \
  --input /path/to/test.wav --quick

Catch-up batching preserves every individual recurrent state transition. The DPDFNet pure-MLX backend uses paired fused DPRNN kernels that execute the ERB and DF branches concurrently. The quality-qualified Core ML FP32 backend now has the lowest measured per-hop latency for both DPDFNet profiles; MLX and MLX+Accelerate remain available for applications that cannot deploy Core ML model assets.

Notes

• Targets macOS 14+ and iOS 17+
• Always build with -c release for Metal library support and performance
• DeepFilterNet v1-v3 streaming keeps the MLX recurrent path. DPDFNet-4 and DPDFNet-8 HR can use the explicit accelerate or mlx DPRNN backend.

License

DeepFilterNet-mlx is dual-licensed under either:

• Apache License, Version 2.0
• MIT License

at your option. Copyright 2026 Kyle Howells. See NOTICE.

The DeepFilterNet model and original implementation are separate works, copyright their respective authors and contributors, and are distributed under the terms provided by the original DeepFilterNet project.