NTH

VoxMel TTS Technical Report

AuthorsWenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen

AffiliationsShanghai Innovation Institute · Xiamen University · Fudan University · Zhejiang University · Shanghai Jiao Tong University · Kyoto University

November 21, 2026 2 min read
Watch on YouTube
The one-line take

VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.

Key results

100K
Emilia training data

Hours of speech used to train VoxMel.

1.15%
GRPO test-en WER

Seed-TTS-eval English word error rate.

0.753
GRPO test-en speaker similarity

Seed-TTS-eval English speaker similarity score.

1.17%
GRPO test-zh CER

Seed-TTS-eval Chinese character error rate.

24
Four-step distillation compute reduction

Fold reduction in Coarse-Mel head evaluations per frame, from 96 to 4.

1.49%
Four-step student test-en WER

Seed-TTS-eval English word error rate after distillation.

What the paper found

VoxMel makes autoregressive text-to-speech work directly with mel spectrograms, without a learned acoustic tokenizer. It applies a fixed, exactly invertible two-axis Haar transform: a Qwen2.5-0.5B-initialized language-model backbone autoregressively generates the compact Coarse-Mel sequence with a flow-matching head, while separate non-autoregressive tiers predict temporal and frequency residuals. Keeping those residuals out of the autoregressive history preserves detail while limiting error feedback. Trained on 100K hours of Emilia, the 0.6B model, evaluated alongside systems including Qwen3-TTS, achieves 1.51% WER and 0.738 speaker similarity on Seed-TTS-eval test-en; frozen-head GRPO improves these to 1.15% and 0.753. On test-zh, GRPO reaches 1.17% CER and 0.776 speaker similarity. The team also distills guided flow sampling: a four-step student cuts Coarse-Mel head evaluations from 96 to 4 per frame, a 24-fold reduction, while recording 1.49% English WER and 1.20% Chinese CER. The central contribution is a tokenizer-free, information-preserving acoustic representation that separates coarse autoregressive structure from residual detail.

Original abstract

Continuous autoregressive TTS relies on an acoustic target that enables stable next-latent modeling while preserving fine-grained acoustic details. While existing approaches employ learned compressed representations (e.g., continuous VAE latents) to render prediction tractable, they rely on complex neural encoders that often cap generation fidelity. Conversely, training-free targets like mel-spectrograms eliminate encoder overhead, yet their potential for autoregressive generation remains largely underexplored. In this work, we propose VoxMel, a tokenizer-free framework for autoregressive TTS that makes stable mel modeling tractable through a fixed, exactly invertible two-axis Haar transform. The transform first decomposes each mel spectrogram along frequency into local averages and a frequency residual, then splits the averaged component along time into Coarse-Mel and a temporal residual. A language-model backbone with a flow-matching head autoregressively generates Coarse-Mel. Non-autoregressive residual tiers then predict the temporal residual from Coarse-Mel, followed by the frequency residual conditioned on the reconstructed Frequency-Averaged Mel. Neither residual is fed back into the autoregressive history. Trained on 100K hours of Emilia, the supervised 0.6B model obtains 1.51\% WER and 0.738 speaker similarity (SIM) on test-en, and 1.29\% CER and 0.755 SIM on test-zh. GRPO improves these results to 1.15\% WER / 0.753 SIM and 1.17\% CER / 0.776 SIM, respectively. We also distill the guided flow head into two- and four-step students, requiring 2 or 4 head evaluations per coarse frame. The four-step student retains competitive SIM and nearly matches the supervised model in WER/CER.

Read the original paper

More in Speech AI

Browse all 48 papers →
02Speech

Marimba: A System One Model for Speech

Solano Amadori

Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.

Read analysis