NTH

Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models

AuthorsYutong Bian, Yanglin Zhang, Yuxiang Wang, Luoyuan Zhang, Rongshen He, Zihao Wang, Junwen Qiu, Jiaqi Li, Zhizheng Wu

AffiliationsThe Chinese University of Hong Kong, Shenzhen, China · Amphion Technology Co., Ltd

November 15, 2026 2 min read
Watch on YouTube
The one-line take

Speculative Listening drafts speech-model responses during quiet moments, then verifies them when the next audio arrives to make conversations faster.

Key results

95.3%
Silence-draft match rate

Share of chunks where the silence-hypothesis output matches the real-audio output.

87.7%
Mismatch at first token

Share of mismatched chunks that diverge at the first output token.

15,883
TickTock-Bench chunks

Total chunk count in the benchmark.

23.41
Maximum tool-call speedup

Speedup over autoregressive decoding for tool-call chunks.

73%
Response-onset delay reduction

Average reduction across the four evaluated models.

94.4%
Maximum real-time lag reduction

Average reduction in maximum lag behind incoming audio.

What the paper found

Chunk-interleaved speech models often have spare compute while listening, then must generate speech or tool calls before the next chunk deadline. Speculative Listening uses that idle time to draft the next output assuming silence, then verifies the draft against real audio; a runner-up first-token branch and a cross-chunk hypothesis pool help recover when the draft diverges. This training-free method preserves the target model’s output while reducing serial decoding. Across four systems, silence-based drafts match real-audio outputs in 95.3% of chunks, and 87.7% of mismatches begin at the first output token. On TickTock-Bench, which contains 15,883 chunks, the method speeds up speak chunks by 2.6–2.8× on average and reaches 23.41× for tool-call chunks. It reduces response-onset decoding delay by 73% and maximum lag behind real time by 94.4%. The evaluation includes Streaming SLM, built from Qwen2.5 Omni-7B, and Gander, Realtime-Venus, and LWS, which use MiniCPM-o. The replay harness runs on an NVIDIA A100; DeepSeek-V4.1-Flash judges the response-quality evaluation.

Original abstract

Chunk-interleaved full-duplex speech language models listen and speak in fixed time chunks, and each chunk must be processed before the next one arrives. Their workload is asymmetric across chunks: listen chunks leave compute idle while awaiting audio, and speak and tool-call chunks require serial generation, which delays responses and tool calls. Using the idle compute for the next chunk requires decoding it before its audio arrives, under an assumed input. Taking silence as the assumed input and comparing the output with the output under real audio on four models, we observe two properties. First, the next audio chunk does not change the output in 95.3% of chunks. Second, 88% of changes begin at the first output token. Building on these properties, we propose Speculative Listening, a training-free, lossless framework for input-time self-speculative decoding. During idle time, the model drafts the next chunk under silence and, for changes at the first token, a second branch from the runner-up first token. It keeps all drafts in a cross-chunk hypothesis pool and verifies them when the audio arrives. On our eight-task benchmark TickTock-Bench, Speculative Listening speeds up speak chunks by 2.6-2.8x and tool-call chunks by up to 23.4x. On average, it reduces the decoding delay before a response starts by 73%, to 74-241 ms, close to the 200 ms gap between human turns, and the maximum lag behind real time by 94.4%.

Read the original paper

More in Speech AI

Browse all 48 papers →
02Speech

Marimba: A System One Model for Speech

Solano Amadori

Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.

Read analysis
03Speech

VoxMel TTS Technical Report

Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen

VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.

Read analysis