NTH

Emilia2: A Large-Scale Dataset for Controllable Long-Form Speech Generation

AuthorsQinke Ni, Dekun Chen, Junwen Qiu, Huan Liao, Xu Li, Xingwen Han, Wan Lin, Kunyu Feng, Yuxiang Wang, Yusheng Mu, Chaoren Wang, Yingda Shen, Mingjie Chen, Haolong Zheng, Xulin Fan, Mark A. Hasegawa-Johnson, Zhizheng Wu

AffiliationsKunyu Feng1 Yuxiang Wang1 Yusheng Mu1 Chaoren Wang1 Yingda Shen1 Mingjie Chen1 Haolong Zheng2 Xulin Fan2 Mark A. Hasegawa-Johnson2 Zhizheng Wu1 · The Chinese University of Hong Kong, Shenzhen · University of Illinois at Urbana-Champaign

November 15, 2026 2 min read
Watch on YouTube
The one-line take

Emilia2 is a large speech dataset and benchmark aimed at helping models generate expressive, consistent speech over long recordings and conversations.

Key results

229,388
Emilia2 dataset hours

Deduplicated dataset duration across all languages.

41
Languages

Languages represented in the dataset.

4,766,441
Instruction-annotated examples

Examples labeled with natural-language style instructions.

933,006
Nonverbal-vocalization examples

Examples containing nonverbal vocalization tags.

23.3
Emilia2-testset hours

Bilingual long-form benchmark duration.

5.27%
TTSD-Eval Chinese CER

Emilia2-NAR character error rate.

What the paper found

Emilia2 introduces a multilingual dataset and processing pipeline for speech generation that stays coherent across long recordings, speaker changes, and expressive delivery. It contains 229,388 hours across 41 languages, organized as short utterances, single-speaker long-form speech, and multi-speaker dialogue while preserving original pauses and timing. It adds 4,766,441 examples with natural-language style instructions and 933,006 with nonverbal vocalization tags, including laughter and sighs. Its 23.3-hour evaluation set includes bilingual recordings up to 30 minutes long, with as many as four speakers and reference audio for each. The researchers train autoregressive and non-autoregressive systems with a 0.6B backbone: the latter predicts masked acoustic tokens in parallel, while the former predicts speech tokens and uses flow matching for acoustic reconstruction. On TTSD-Eval, the non-autoregressive model records a Chinese character error rate of 5.27% and English word error rate of 9.93%, outperforming larger compared systems on those measures, including the 8B MOSS-TTSD. The autoregressive model’s instruction-following scores are comparable to Qwen3-TTS-12Hz-VD. Qwen3 components support transcription, instruction generation, and model initialization; DeepSeek helps prepare nonverbal-vocalization evaluation scripts, and Gemini-2.5-Pro serves as an instruction-following judge.

Original abstract

Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Emilia2, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Emilia2 constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Emilia2 provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Emilia2-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Emilia2. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Emilia2 as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://charlesnii.github.io/Emilia2-demo/.

Read the original paper

More in Speech AI

Browse all 48 papers →
02Speech

Marimba: A System One Model for Speech

Solano Amadori

Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.

Read analysis
03Speech

VoxMel TTS Technical Report

Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen

VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.

Read analysis