Emilia2: A Large-Scale Dataset for Controllable Long-Form Speech Generation
AuthorsQinke Ni, Dekun Chen, Junwen Qiu, Huan Liao, Xu Li, Xingwen Han, Wan Lin, Kunyu Feng, Yuxiang Wang, Yusheng Mu, Chaoren Wang, Yingda Shen, Mingjie Chen, Haolong Zheng, Xulin Fan, Mark A. Hasegawa-Johnson, Zhizheng Wu
AffiliationsKunyu Feng1 Yuxiang Wang1 Yusheng Mu1 Chaoren Wang1 Yingda Shen1 Mingjie Chen1 Haolong Zheng2 Xulin Fan2 Mark A. Hasegawa-Johnson2 Zhizheng Wu1 · The Chinese University of Hong Kong, Shenzhen · University of Illinois at Urbana-Champaign
Emilia2 is a large speech dataset and benchmark aimed at helping models generate expressive, consistent speech over long recordings and conversations.
Key results
Deduplicated dataset duration across all languages.
Languages represented in the dataset.
Examples labeled with natural-language style instructions.
Examples containing nonverbal vocalization tags.
Bilingual long-form benchmark duration.
Emilia2-NAR character error rate.
What the paper found
Emilia2 introduces a multilingual dataset and processing pipeline for speech generation that stays coherent across long recordings, speaker changes, and expressive delivery. It contains 229,388 hours across 41 languages, organized as short utterances, single-speaker long-form speech, and multi-speaker dialogue while preserving original pauses and timing. It adds 4,766,441 examples with natural-language style instructions and 933,006 with nonverbal vocalization tags, including laughter and sighs. Its 23.3-hour evaluation set includes bilingual recordings up to 30 minutes long, with as many as four speakers and reference audio for each. The researchers train autoregressive and non-autoregressive systems with a 0.6B backbone: the latter predicts masked acoustic tokens in parallel, while the former predicts speech tokens and uses flow matching for acoustic reconstruction. On TTSD-Eval, the non-autoregressive model records a Chinese character error rate of 5.27% and English word error rate of 9.93%, outperforming larger compared systems on those measures, including the 8B MOSS-TTSD. The autoregressive model’s instruction-following scores are comparable to Qwen3-TTS-12Hz-VD. Qwen3 components support transcription, instruction generation, and model initialization; DeepSeek helps prepare nonverbal-vocalization evaluation scripts, and Gemini-2.5-Pro serves as an instruction-following judge.
Original abstract
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Emilia2, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Emilia2 constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Emilia2 provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Emilia2-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Emilia2. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Emilia2 as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://charlesnii.github.io/Emilia2-demo/.
Read the original paperMore in Speech AI
Browse all 48 papers →Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
Bagzhan Karl, Alizhan Karl
Kezek turns a mixed two-person recording into speaker-specific tracks while preserving overlaps and short replies for full-duplex dialogue data.
Marimba: A System One Model for Speech
Solano Amadori
Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.
VoxMel TTS Technical Report
Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen
VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.