VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step
AuthorsArbaz Siddiqui
AffiliationsIndependent researcher
VoxLock filters a live call to keep the caller’s speech and suppress nearby voices, without requiring a separate enrollment recording.
Key results
Size of the REAL-TSE model adapted for VoxLock.
Word error rate on 265 real recordings.
Word error rate without audio processing.
Reduced from 42.95% on 400 meeting windows.
Measured for the 8-bit runtime with 40 ms chunks.
Milliseconds of model latency.
What the paper found
VoxLock adapts the 27M-parameter causal REAL-TSE band-split recurrent model to isolate a caller without a separate enrollment recording: it extracts a 3-second voice sample from the call, then uses spectrogram attention and ECAPA-TDNN speaker features to guide separation. Fine-tuned in stages on one NVIDIA L4, it was evaluated through Parakeet TDT 0.6B transcripts. On Krisp’s 265-recording benchmark, VoxLock achieved 4.95% word error rate, compared with 23.21% on unprocessed audio, and let just 9 background words into transcripts across 85.3 minutes of background-only speech. Its WER was statistically similar to Krisp Default, while lower than the other tested commercial and open systems. On a blind AMI test of 400 meeting windows, WER fell from 42.95% to 10.63%. The 8-bit runtime uses 77% of one CPU thread with 24 ms of algorithmic latency. VoxLock assumes the caller is the loudest early speaker, so a dominant background voice can lead it to isolate the wrong person; it is also limited to English and a single caller.
Original abstract
A voice agent's speech recognizer transcribes everyone near the microphone, including people other than the caller. I fine-tuned a public causal target-speaker extraction model (REAL-TSE, 27M parameters) on one GPU to work from a voice sample cut from the call itself, and released it as VoxLock. I score it by what a recognizer writes on real recordings. On Krisp's 265-recording benchmark it reaches 4.95% WER against 23.21% unprocessed and lets 9 background words into the transcript over 85 minutes of background-only speech. Its WER is not significantly different from Krisp's best filters and is significantly lower than the other commercial and open systems I tested. On 400 AMI meeting windows built and frozen before scoring, it cuts WER from 42.95% to 10.63%. The 8-bit model runs a live call on 77% of one CPU thread with 24 ms of algorithmic latency. Model, code and per-file scores are open.
Read the original paperMore in Speech AI
Browse all 39 papers →CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Yingzhuo Deng, Chong Li, Bingxin Liu, Yi Shu, Tianyu Peng, Can Xie, Siyao Song, Shuo Ren, Jiajun Zhang
CoSTA helps spoken dialogue systems generate speech faster by using compressed tokens internally and expanding them before synthesis.
Lipi-Ghor: A Large-Scale, Diarized, Multi-Domain Bengali Speech Corpus and a Scalable Pipeline for Building It
Sanjid Hasan, A H M Fuad, Risalat Labib, Bayazid Hasan Simon
The paper introduces a large, multi-domain Bengali speech corpus with speaker diarization and a scalable recipe for building similar resources.
AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
Yunjia Zhang, Junan Zhang, Jing Yang, Xueyao Zhang, Fan Fan, Zhizheng Wu
AnyEdit edits speech and singing while aiming to preserve the acoustic environment surrounding the original voice.