NTH

Marimba: A System One Model for Speech

AuthorsSolano Amadori

Affiliationscharcot.ai

November 21, 2026 2 min read
Watch on YouTube
The one-line take

Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.

Key results

0.6B
Qwen3-ASR model size

Frozen speech model used as Marimba’s encoder and decoder.

0.901
FLEURS binary term-detection AUROC

Marimba’s score, compared with 0.791 for Qwen3-Omni and 0.682 for Qwen2-Audio.

0.930
Turn-end AUROC

Marimba on Smart Turn’s test set; Smart Turn v3.2 scored 0.982.

0.980
VoxConverse speaker-change AUROC

Marimba’s result, above the ECAPA baseline of 0.962.

0.904
Synthetic Portuguese PII item-level F1

Private PII pipeline applied to Marimba’s greedy transcripts.

2.35
Term-detection p95 latency

Seconds per question, including greedy and four-beam decoding.

What the paper found

Marimba is a typed speech-decision system built on a frozen Qwen3-ASR 0.6B model. It encodes each 8-second audio window once, then routes lexical questions to exact-token scoring across greedy and four-beam transcripts, calibrated with Platt scaling; other decisions use a question-conditioned head, a layer-18 turn-end readout, Qwen3-ASR’s language scores, or layer-12 speaker readouts. On FLEURS binary term detection, Marimba reached AUROC 0.901, ahead of Qwen3-Omni at 0.791 and Qwen2-Audio at 0.682. Turn-end AUROC was 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn’s test set, while results on ICSI were close: 0.716 versus 0.712. On VoxConverse, speaker-change AUROC reached 0.980, above the ECAPA baseline at 0.962. A private text pipeline applied to transcripts achieved item-level F1 0.904 on synthetic Brazilian Portuguese personal-information clips, but errors were largely inherited from speech recognition. The main operational tradeoff is latency: term detection takes 2.35 seconds at p95 per question, compared with 13–16 milliseconds for turn-end and voice-activity decisions. The system offers a shared typed interface, but lexical accuracy remains vulnerable to missed or confused words, and its results do not establish clinical deployment readiness.

Original abstract

Speech applications require decisions about term presence, turn completion, and language within an audio window. We present Marimba, an interface for typed speech decisions built on frozen Qwen3-ASR 0.6B features. Lexical questions use zero-shot scoring of ASR transcripts; non-lexical questions use a question-conditioned head and a separate layer-18 turn-end readout. Language identification uses the ASR model's own language scores. On private conversational test sets, decoder AUROC is 0.80-0.92; on public read speech it is 0.92-0.97. Speech Commands four-way accuracy is 0.93; binary lexical ECE is at most 0.08. Personal-information detection, which passes the greedy transcript to a private, curated PII pipeline, reaches item-level F1 0.904 on synthetic Brazilian Portuguese speech (1.000 on reference transcripts). Layer-12 speaker readouts detect speaker change with AUROC 0.966-0.980 on AMI and VoxConverse, above an ECAPA baseline, and count speakers (0, 1, 2, or 3+) more accurately than pyannote 3.1. The turn-end readout reaches AUROC 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn's own test set. On a subset of ICSI meetings labeled by next speaker, AUROC is 0.716 versus 0.712; the paired confidence interval includes zero.

Read the original paper

More in Speech AI

Browse all 48 papers →
02Speech

VoxMel TTS Technical Report

Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen

VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.

Read analysis