NTH

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

AuthorsSang-Hoon Lee, Ha-Yeong Choi

AffiliationsDepartment of Artificial Intelligence, Ajou University, Suwon, Korea · Frontier AI Lab, KT Corp., Seoul, Korea

November 15, 2026 2 min read
Watch on YouTube
The one-line take

WaveGen aims to generate speech directly from text with one model, learning the path from language to waveform without separate audio or alignment tools.

Key results

1.47
Seed-en WER

WaveGen word error rate on the Seed-en benchmark.

0.75
Seed-en speaker similarity

WaveGen speaker-similarity score on the Seed-en benchmark.

680M
WaveGen model size

Parameter count for the reported WaveGen model.

100K
Emilia training data

Approximate hours of public Emilia speech used in the large-scale comparison.

64
Internal Mel bins

Mel-spectrogram bins used for Spectral Forcing in the baseline configuration.

What the paper found

WaveGen targets a longstanding weakness in text-to-speech: systems often generate an intermediate representation and rely on a separate vocoder or decoder. Instead, it uses one diffusion Transformer to generate raw speech directly from text, without codecs, vocoders, duration predictors, pretrained language representations, or adversarial training. Its key technique, Spectral Forcing, jointly learns an internal Mel-spectral trajectory and the waveform trajectory; a 64-bin spectral path guides text–speech alignment but is not decoded as an external intermediate. Shared Transformer blocks learn alignment, while separate heads refine each representation, and training combines clean-signal prediction with velocity-space losses and multi-scale spectral supervision. On Seed-en, the 680M-parameter WaveGen reports a word error rate of 1.47 and speaker similarity of 0.75, compared with 1.50 and 0.71 for Qwen3-TTS at 1.7B parameters. The system was trained on roughly 100K hours of public Emilia speech, and OpenAI’s Whisper was used to measure word error rate. The results suggest direct waveform generation can compete with more complex multi-stage systems while removing their external components.

Original abstract

This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral $v$-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen

Read the original paper

More in Speech AI

Browse all 48 papers →
02Speech

Marimba: A System One Model for Speech

Solano Amadori

Marimba reuses a speech recognition model’s features to answer questions about words, speakers, language, and turn endings.

Read analysis
03Speech

VoxMel TTS Technical Report

Wenhao Guan, Jiaxin Ye, Haitao Li, Yifan Duan, Junxi Liu, Wangjin Zhou, Lin Li, Qingyang Hong, Xie Chen

VoxMel generates speech by autoregressively modeling transformed mel-spectrograms, avoiding a learned tokenizer while aiming for high fidelity and faster inference.

Read analysis