NTH

A Tutorial on Generative Models for Music Audio

AuthorsMateusz Modrzejewski, Joanna Kurkowska, Dorien Herremans

AffiliationsWarsaw University of Technology, Institute of Computer Science · ZAiKS Lab · Singapore University of Technology and Design

November 9, 2026 2 min read
Watch on YouTube
The one-line take

A practical guide to how modern generative models create, edit, and perform music audio—and how to evaluate their results.

Key results

200
MusicGen token rate

Tokens generated per second of audio with the described EnCodec setup.

105
Compared systems

Systems covered in the review’s comparison tables.

43
Systems with opaque data reporting

Rows reporting only undisclosed, in-house, or web-collected training data, out of 105.

11
Song systems using preference or RL post-training

Systems out of 25 song-generation systems reporting one of these stages.

5M
Qwen-Music training corpus

More than this many hours of multilingual audio are reported for training.

What the paper found

This tutorial organizes music-audio generation around four design choices: the representation, the generation process, how conditions enter the model, and the sampling procedure. It traces two main approaches—autoregressive models over codec tokens, exemplified by MusicGen, and diffusion or flow-matching models over continuous latents—and explains how planner-renderer systems combine long-range structure with detailed audio synthesis. The applications span instrumental tracks, vocal songs, editing and stem generation, and live performance; techniques such as masked decoding, distillation, preference optimization, and time-varying controls address different costs and creative requirements. For scale, MusicGen’s EnCodec setup generates 200 tokens per second, while the comparison covers 105 systems. The review also finds that 43 of those 105 systems report only undisclosed, in-house, or web-collected training data, and 11 of 25 song systems report preference or reinforcement-learning post-training. Qwen-Music reports training on more than 5M hours of multilingual audio. The tutorial cautions that Fréchet Audio Distance can disagree with listener preferences, recommends human listening tests and clearly reported evaluation protocols, and separates training-data provenance from memorization and output detection. It places products such as Stability AI’s Stable Audio, Suno, and Udio in this landscape, notes Meta’s AudioBox Aesthetics as a learned evaluator, and discusses the legal record involving OpenAI language models and music-generation disputes. Its central conclusion is that reported gains cannot be cleanly attributed to a model family when data, scale, and evaluation differ.

Original abstract

Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.

Read the original paper

More in Generative Models

Browse all 87 papers →
01Generative Model

dNaViT: Discrete Codes are General-Purpose Visual Representations

Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai

dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.

Read analysis