Marimba: A System One Model for Speech
AuthorsSolano Amadori
Affiliationscharcot.ai
Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.
Key results
Frozen speech model used as Marimba’s encoder and decoder.
Marimba’s score, compared with 0.791 for Qwen3-Omni and 0.682 for Qwen2-Audio.
Marimba on Smart Turn’s test set; Smart Turn v3.2 scored 0.982.
Marimba’s result, above the ECAPA baseline of 0.962.
Private PII pipeline applied to Marimba’s greedy transcripts.
Seconds per question, including greedy and four-beam decoding.
What the paper found
Marimba is a typed speech-decision system built on a frozen Qwen3-ASR 0.6B model. It encodes each 8-second audio window once, then routes lexical questions to exact-token scoring across greedy and four-beam transcripts, calibrated with Platt scaling; other decisions use a question-conditioned head, a layer-18 turn-end readout, Qwen3-ASR’s language scores, or layer-12 speaker readouts. On FLEURS binary term detection, Marimba reached AUROC 0.901, ahead of Qwen3-Omni at 0.791 and Qwen2-Audio at 0.682. Turn-end AUROC was 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn’s test set, while results on ICSI were close: 0.716 versus 0.712. On VoxConverse, speaker-change AUROC reached 0.980, above the ECAPA baseline at 0.962. A private text pipeline applied to transcripts achieved item-level F1 0.904 on synthetic Brazilian Portuguese personal-information clips, but errors were largely inherited from speech recognition. The main operational tradeoff is latency: term detection takes 2.35 seconds at p95 per question, compared with 13–16 milliseconds for turn-end and voice-activity decisions. The system offers a shared typed interface, but lexical accuracy remains vulnerable to missed or confused words, and its results do not establish clinical deployment readiness.
Original abstract
Speech applications require decisions about term presence, turn completion, and language within an audio window. We present Marimba, an interface for typed speech decisions built on frozen Qwen3-ASR 0.6B features. Lexical questions use zero-shot scoring of ASR transcripts; non-lexical questions use a question-conditioned head and a separate layer-18 turn-end readout. Language identification uses the ASR model's own language scores. On private conversational test sets, decoder AUROC is 0.80-0.92; on public read speech it is 0.92-0.97. Speech Commands four-way accuracy is 0.93; binary lexical ECE is at most 0.08. Personal-information detection, which passes the greedy transcript to a private, curated PII pipeline, reaches item-level F1 0.904 on synthetic Brazilian Portuguese speech (1.000 on reference transcripts). Layer-12 speaker readouts detect speaker change with AUROC 0.966-0.980 on AMI and VoxConverse, above an ECAPA baseline, and count speakers (0, 1, 2, or 3+) more accurately than pyannote 3.1. The turn-end readout reaches AUROC 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn's own test set. On a subset of ICSI meetings labeled by next speaker, AUROC is 0.716 versus 0.712; the paired confidence interval includes zero.
Read the original paperMore in Speech AI
Browse all 48 papers →Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
Bagzhan Karl, Alizhan Karl
Kezek turns a mixed two-person recording into speaker-specific tracks while preserving overlaps and short replies for full-duplex dialogue data.
VoxMel TTS Technical Report
Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen
VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.
Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription
Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan
Confucius4-R2T2 transcribes speech in real time while keeping previously emitted words fixed.