WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
AuthorsSang-Hoon Lee, Ha-Yeong Choi
AffiliationsDepartment of Artificial Intelligence, Ajou University, Suwon, Korea · Frontier AI Lab, KT Corp., Seoul, Korea
WaveGen aims to generate speech directly from text with one model, learning the path from language to waveform without separate audio or alignment tools.
Key results
WaveGen word error rate on the Seed-en benchmark.
WaveGen speaker-similarity score on the Seed-en benchmark.
Parameter count for the reported WaveGen model.
Approximate hours of public Emilia speech used in the large-scale comparison.
Mel-spectrogram bins used for Spectral Forcing in the baseline configuration.
What the paper found
WaveGen targets a longstanding weakness in text-to-speech: systems often generate an intermediate representation and rely on a separate vocoder or decoder. Instead, it uses one diffusion Transformer to generate raw speech directly from text, without codecs, vocoders, duration predictors, pretrained language representations, or adversarial training. Its key technique, Spectral Forcing, jointly learns an internal Mel-spectral trajectory and the waveform trajectory; a 64-bin spectral path guides text–speech alignment but is not decoded as an external intermediate. Shared Transformer blocks learn alignment, while separate heads refine each representation, and training combines clean-signal prediction with velocity-space losses and multi-scale spectral supervision. On Seed-en, the 680M-parameter WaveGen reports a word error rate of 1.47 and speaker similarity of 0.75, compared with 1.50 and 0.71 for Qwen3-TTS at 1.7B parameters. The system was trained on roughly 100K hours of public Emilia speech, and OpenAI’s Whisper was used to measure word error rate. The results suggest direct waveform generation can compete with more complex multi-stage systems while removing their external components.
Original abstract
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral $v$-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
Read the original paperMore in Speech AI
Browse all 48 papers →Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data
Bagzhan Karl, Alizhan Karl
Kezek turns a mixed two-person recording into speaker-specific tracks while preserving overlaps and short replies for full-duplex dialogue data.
Marimba: A System One Model for Speech
Solano Amadori
Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.
VoxMel TTS Technical Report
Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen
VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.