VoxLock

The best performing voice isolator there is. VoxLock is an open-source voice isolation model: it keeps the caller's voice and removes the other voices in the room before speech-to-text hears them. Made for voice agents and phone calls: one input, the call audio, and it picks out the caller by itself. It streams on a CPU with 24 ms of algorithmic latency.

Write-up · Code · pip · Try it in your browser

Results

Model	Access	Word error rate %	Background words leaked / min
VoxLock	open	4.95	0.11
Krisp VI 2.5 Default	paid	5.30	1.52
Krisp VI 2.5 Balanced	paid	5.32	2.02
Krisp VI 2.5 HD	paid	5.79	2.74
Krisp VI 2.5 Lite	paid	6.21	4.52
ai-coustics Voice Focus 2.2 L	paid	6.84	0.66
ai-coustics Voice Focus 2.2 S	paid	6.91	1.00
Sanas Voice Isolation SE 1.0	paid	9.38	13.76
REAL-TSE causal (VoxLock's starting checkpoint)	open	9.81	13.73
Hush	open	12.09	19.58
DeepFilterNet3	open	13.30	22.77
TIGER	open	14.72	37.24
No processing		23.21	66.48
SAM-Audio small (Meta)	open	33.77	21.15
ElevenLabs Voice Isolator**	paid	43.79**	144.6**

265 real recordings from the Krisp Voice Isolation Benchmark, transcribed with Parakeet TDT 0.6B v2 and scored with the benchmark's published method. Background words leaked counts words transcribed while only the background was talking, per minute of that audio.

** ElevenLabs Voice Isolator, run on the 7 of these recordings with the most background speech, not all 265. On the same 7, no processing scores 45.82% and 154.3 leaked words per minute, and VoxLock 2.98% and 0.0.

On real AMI meetings not used in training, with the headset wearer as the caller:

Model	Access	Word error rate %	Background words leaked / min
VoxLock	open	10.12	2.12
ai-coustics Voice Focus 2.2 S	paid	11.85	1.06
ai-coustics Voice Focus 2.2 L	paid	12.13	1.72
Sanas Voice Isolation SE 1.0	paid	13.30	5.30
TIGER	open	22.84	36.85
DeepFilterNet3	open	26.78	38.84
Hush	open	33.00	68.93
No processing		33.03	78.34

Caller words missed, deletions, insertions and 95% intervals: bench/results.

Use

pip install voxlock
voxlock call.wav -o caller.wav

Any sample rate, mono or stereo. The model downloads from Hugging Face on first use: 74 MB, the 8-bit version (see Quantized models).

import voxlock

vl = voxlock.VoxLock()                     # precision="fp32" for the benchmarked model
call = voxlock.load_audio("call.wav")      # 16 kHz mono float32
caller = vl.isolate(call)                  # same length as the input

VoxLock keeps the voice in a 3-second sample of the caller. isolate takes the loudest 3 seconds in the first 15 seconds of the call, which is how it was benchmarked. Pass sample= to choose it yourself.

Streaming

Push the call in as it arrives, as 16 kHz float32 chunks of any size. Each push returns the isolated audio that is ready, about 24 ms behind the input.

vl = voxlock.VoxLock(threads=1)
stream = vl.live()                         # finds the caller by itself
for chunk in call_audio:                   # e.g. 40 ms = 640 samples
    send(stream.push(chunk))
send(stream.flush())                       # at the end of the call

live() passes the audio through unchanged until it has heard 3 seconds of speech, takes that speech as the voice sample and isolates from then on. On our test calls it switched over a median of 3.6 seconds in. At the switch it first runs the model over the call so far (up to 30 seconds), so the model never starts cold in the middle of a sentence; that is a one-time 2 to 5 seconds of work on one CPU thread. On the 265 Krisp recordings live() scores 5.15% word error rate with the fp32 model, against 4.95% on the whole file.

If you already have a sample, from earlier in the call or from a previous call, isolation starts with the first chunk:

stream = vl.stream(sample)                 # about 3 s of the caller, 16 kHz mono

Streaming gives the same output as isolate with the same sample, to float rounding with fp32; int8 differs slightly, since its 8-bit scaling depends on the chunk size. On one CPU thread it keeps up with chunks of 40 ms or more, and with 20 ms chunks on two threads (see Performance). Larger chunks use less CPU and add their length to the delay.

Pipecat

pip install "voxlock[pipecat]"

from pipecat.transports.base_transport import TransportParams
from voxlock.pipecat import VoxLockFilter

params = TransportParams(audio_in_enabled=True, audio_in_filter=VoxLockFilter())

The filter cleans the caller's audio before VAD and speech-to-text, at any transport sample rate (Twilio's 8 kHz included). Like live(), it passes audio through until it has heard 3 seconds of the caller; pass voice_sample= to isolate from the start. It runs the int8 model in 80 ms chunks on one thread by default (chunk_ms=, threads=, precision=), which takes 61% of one CPU core, so a single vCPU such as Pipecat Cloud's agent-2x runs it. Half a vCPU (agent-1x) is too little. A full voice agent: examples/pipecat_bot.py. Tested with Pipecat 1.12.0.

sherpa-onnx

VoxLock is built into the speech denoisers of sherpa-onnx from version 2.0.1, in C++ with no Python, for whole files and for streams. Download the models with sherpa-onnx's meta data from this release, then:

sherpa-onnx-offline-denoiser \
  --speech-denoiser-voxlock-enroll-model=voxlock_enroll.onnx \
  --speech-denoiser-voxlock-chunk-model=voxlock_chunk_int8.onnx \
  --input-wav=call.wav --output-wav=caller.wav

sherpa-onnx-online-denoiser takes the same models and streams. Both take the voice sample from the audio as the voxlock package does; --speech-denoiser-voxlock-voice-sample=caller.wav gives it instead.

PyTorch

git clone https://github.com/arbazsiddiqui/voxlock && cd voxlock
pip install -r requirements.txt
huggingface-cli download arbazsiddiqui/voxlock model.safetensors config.yaml --local-dir ckpt
python isolate.py --ckpt ckpt --in call.wav --out caller.wav

Performance

One live call through VoxLock.stream with the default int8 model on one CPU thread: onnxruntime 1.31.1 on a Google Cloud g2-standard-8 (Intel Xeon @ 2.20 GHz), 60 seconds of AMI meeting audio.

Chunk	CPU per call	Real-time factor
20 ms	110% of a core	1.10
40 ms	77% of a core	0.77
80 ms	61% of a core	0.61

Below 1 keeps up with the call. 20 ms chunks keep up on two threads (0.86). The model has 27M parameters and 24 ms of algorithmic latency, plus the chunk. On the same machine's NVIDIA L4 GPU (fp32, CUDA), a whole recording runs 40 times faster than real time.

Quantized models

The ONNX model comes in four precisions, all in onnx/v2/. Choose with VoxLock(precision=...), voxlock --precision ... or VoxLockFilter(precision=...).

Precision	Quality	Files	Download	Word error rate %	Background words leaked / min	CPU per call, 20 ms chunks
int8 (default)	99.8%	enroll.onnx, chunk_int8.onnx	74 MB	5.11	0.12	110%
int8-full	99.1%	enroll.onnx, chunk_int8_full.onnx	71 MB	5.85	0.04	110%
fp16	100%*	enroll_fp16.onnx, chunk_fp16.onnx	47 MB	4.95*	0.11*	163%
fp32	100%	enroll.onnx, chunk.onnx	94 MB	4.95	0.11	165%

Quality: the share of fp32's correctly transcribed words each precision keeps, on the 265 Krisp recordings. CPU per call: one live call on one thread of the Xeon in Performance.

* fp16 is the float32 model with its weights rounded to float16. Its output matches float32 to over 70 dB, so it carries the float32 scores. The browser demo uses it.

enroll runs once on the voice sample, chunk runs on each chunk of the call and carries the LSTM state.

PyTorch: model.safetensors and config.yaml, 104 MB.

Training

A 27M causal band-split RNN target-speaker extractor, fine-tuned from the REAL-TSE challenge baseline on a single NVIDIA L4:

REAL-TSE tfmap_context_causal_100
  -> simulated calls        4,000 steps   LibriSpeech callers + background talkers in simulated rooms
  -> real AMI crosstalk     4,000 steps   headset recordings from 40 meetings
  -> long calls (30 s)        800 steps   callers who talk throughout
  -> rooms and distances     400 steps   callers near or far from the mic, background voices dry or in the room, room noise

Acknowledgements

Built on the REAL-TSE challenge baseline and WeSep by Shuai Wang, Ke Zhang and co-authors, with WeSpeaker's ECAPA model. Trained on LibriSpeech and the AMI Meeting Corpus.

License

Apache-2.0.