VoxMel TTS Technical Report
AuthorsWenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen
AffiliationsShanghai Innovation Institute · Xiamen University · Fudan University · Zhejiang University · Shanghai Jiao Tong University · Kyoto University
VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.
Key results
Hours of speech used to train VoxMel.
Seed-TTS-eval English word error rate.
Seed-TTS-eval English speaker similarity score.
Seed-TTS-eval Chinese character error rate.
Fold reduction in Coarse-Mel head evaluations per frame, from 96 to 4.
Seed-TTS-eval English word error rate after distillation.
What the paper found
VoxMel makes autoregressive text-to-speech work directly with mel spectrograms, without a learned acoustic tokenizer. It applies a fixed, exactly invertible two-axis Haar transform: a Qwen2.5-0.5B-initialized language-model backbone autoregressively generates the compact Coarse-Mel sequence with a flow-matching head, while separate non-autoregressive tiers predict temporal and frequency residuals. Keeping those residuals out of the autoregressive history preserves detail while limiting error feedback. Trained on 100K hours of Emilia, the 0.6B model, evaluated alongside systems including Qwen3-TTS, achieves 1.51% WER and 0.738 speaker similarity on Seed-TTS-eval test-en; frozen-head GRPO improves these to 1.15% and 0.753. On test-zh, GRPO reaches 1.17% CER and 0.776 speaker similarity. The team also distills guided flow sampling: a four-step student cuts Coarse-Mel head evaluations from 96 to 4 per frame, a 24-fold reduction, while recording 1.49% English WER and 1.20% Chinese CER. The central contribution is a tokenizer-free, information-preserving acoustic representation that separates coarse autoregressive structure from residual detail.
Original abstract
Continuous autoregressive TTS relies on an acoustic target that enables stable next-latent modeling while preserving fine-grained acoustic details. While existing approaches employ learned compressed representations (e.g., continuous VAE latents) to render prediction tractable, they rely on complex neural encoders that often cap generation fidelity. Conversely, training-free targets like mel-spectrograms eliminate encoder overhead, yet their potential for autoregressive generation remains largely underexplored. In this work, we propose VoxMel, a tokenizer-free framework for autoregressive TTS that makes stable mel modeling tractable through a fixed, exactly invertible two-axis Haar transform. The transform first decomposes each mel spectrogram along frequency into local averages and a frequency residual, then splits the averaged component along time into Coarse-Mel and a temporal residual. A language-model backbone with a flow-matching head autoregressively generates Coarse-Mel. Non-autoregressive residual tiers then predict the temporal residual from Coarse-Mel, followed by the frequency residual conditioned on the reconstructed Frequency-Averaged Mel. Neither residual is fed back into the autoregressive history. Trained on 100K hours of Emilia, the supervised 0.6B model obtains 1.51\% WER and 0.738 speaker similarity (SIM) on test-en, and 1.29\% CER and 0.755 SIM on test-zh. GRPO improves these results to 1.15\% WER / 0.753 SIM and 1.17\% CER / 0.776 SIM, respectively. We also distill the guided flow head into two- and four-step students, requiring 2 or 4 head evaluations per coarse frame. The four-step student retains competitive SIM and nearly matches the supervised model in WER/CER.
Read the original paperMore in Speech AI
Browse all 48 papers →Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
Bagzhan Karl, Alizhan Karl
Kezek turns a mixed two-person recording into speaker-specific tracks while preserving overlaps and short replies for full-duplex dialogue data.
Marimba: A System One Model for Speech
Solano Amadori
Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.
Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription
Daniel Huang, Qi He, Shaoxiong Duan, Yingjie Li, Qiao Cheng, Yitao Duan
Confucius4-R2T2 transcribes speech in real time while keeping previously emitted words fixed.