Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
AuthorsBagzhan Karl, Alizhan Karl
AffiliationsManifestro
Kezek turns a mixed two-person recording into speaker-specific tracks while preserving overlaps and short replies for full-duplex dialogue data.
Key results
Kezek’s share of short replies retained on the correct track; the pyannote-to-DialogueSidon baseline scores 0.687.
Share of recordings with no channel swaps; the baseline scores 0.793.
Share of input duration returned; the baseline scores 0.735.
Hours of audio per GPU-hour on Open Yap; the baseline processes 51.
Share of replies over the other speaker’s speech retained on the correct track in unseen telephone recordings.
Kezek’s retention for replies more than 10 dB below the other voice; an ideal ratio mask scores 0.743.
What the paper found
Kezek turns a mono recording of exactly two people into synchronized, per-speaker tracks, targeting the two-channel training data needed by full-duplex dialogue systems such as Moshi. Its two-stage design assigns speaker identities and activity once for the whole recording, then uses a FiLM-conditioned Conv-TasNet-style separator to carry out that assignment; a mixture-consistency constraint ensures the tracks sum exactly to the input. Word-gated activity labels help prevent microphone crosstalk from causing channel swaps, while phase-robust speaker fingerprints support identity tracking. Against the pyannote-to-DialogueSidon pipeline on Open Yap, Kezek retains 0.830 of short replies, avoids swaps in 0.934 of recordings, and returns 1.000 of the audio; it processes 231 hours of audio per GPU-hour. On unseen CallHome telephone speech, it retains 0.796 of short replies over the other speaker’s speech. A key remaining weakness is quiet backchannels: when replies are more than 10 dB below the other voice, Kezek keeps 0.171 under the paper’s stricter audibility test, compared with 0.743 for an ideal ratio mask.
Original abstract
Full-duplex spoken dialogue models learn turn-taking, overlaps and backchannels from conversations recorded with one channel per speaker, but most available conversational audio is a single mono mix. We present Kezek, a model that turns a mono recording of exactly two people into one track per person with the original length and timing; overlaps and short replies ("yeah", "mm-hm") stay on the track of the person who said them. Kezek decides who speaks when and who is who once for the whole recording (stage 1: a two-voice activity model guided by a speaker fingerprint trained to be insensitive to phase), and a conditioned time-domain separator only carries out that decision (stage 2), with outputs that always sum to the input. Standard separation metrics miss the failures that matter for dialogue data, so we report short-reply retention, leakage onto the other person's solo, channel swaps and coverage. Against the open pipeline pyannote -> DialogueSidon on the same recordings, Kezek keeps more short replies (0.83 vs. 0.69 on Open Yap test; 0.80 vs. 0.71 over the other's speech on unseen telephone speech from CallHome English), mostly because it returns the whole recording while the external pipeline drops part of it (on the audio both return, reply retention is about the same: 0.859 vs. 0.865 on Open Yap); it also leaks less, swaps channels less often, and runs 4-5 times faster (about 230 hours of audio per GPU-hour). We also report negative results and an ideal-mask ceiling that locate the remaining gap: quiet replies under a louder voice. Code and weights are released under permissive licences.
Read the original paperMore in Speech AI
Browse all 48 papers →Marimba: A System One Model for Speech
Solano Amadori
Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.
VoxMel TTS Technical Report
Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen
VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.
Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription
Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan
Confucius4-R2T2 transcribes speech in real time while keeping previously emitted words fixed.