A Tutorial on Generative Models for Music Audio
AuthorsMateusz Modrzejewski, Joanna Kurkowska, Dorien Herremans
AffiliationsWarsaw University of Technology, Institute of Computer Science · ZAiKS Lab · Singapore University of Technology and Design
A practical guide to how modern generative models create, edit, and perform music audio—and how to evaluate their results.
Key results
Tokens generated per second of audio with the described EnCodec setup.
Systems covered in the review’s comparison tables.
Rows reporting only undisclosed, in-house, or web-collected training data, out of 105.
Systems out of 25 song-generation systems reporting one of these stages.
More than this many hours of multilingual audio are reported for training.
What the paper found
This tutorial organizes music-audio generation around four design choices: the representation, the generation process, how conditions enter the model, and the sampling procedure. It traces two main approaches—autoregressive models over codec tokens, exemplified by MusicGen, and diffusion or flow-matching models over continuous latents—and explains how planner-renderer systems combine long-range structure with detailed audio synthesis. The applications span instrumental tracks, vocal songs, editing and stem generation, and live performance; techniques such as masked decoding, distillation, preference optimization, and time-varying controls address different costs and creative requirements. For scale, MusicGen’s EnCodec setup generates 200 tokens per second, while the comparison covers 105 systems. The review also finds that 43 of those 105 systems report only undisclosed, in-house, or web-collected training data, and 11 of 25 song systems report preference or reinforcement-learning post-training. Qwen-Music reports training on more than 5M hours of multilingual audio. The tutorial cautions that Fréchet Audio Distance can disagree with listener preferences, recommends human listening tests and clearly reported evaluation protocols, and separates training-data provenance from memorization and output detection. It places products such as Stability AI’s Stable Audio, Suno, and Udio in this landscape, notes Meta’s AudioBox Aesthetics as a learned evaluator, and discusses the legal record involving OpenAI language models and music-generation disputes. Its central conclusion is that reported gains cannot be cleanly attributed to a model family when data, scale, and evaluation differ.
Original abstract
Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.
Read the original paperMore in Generative Models
Browse all 87 papers →dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai
dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.
Lipschitz-Controlled Flow Matching for Robust Image Restoration
Sedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner, Sebastian Peitz
LipsFlower aims to make flow-matching image restorations less sensitive to small input changes without sacrificing perceptual quality.
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
Yuxuan Chen, Xiaojie Li, Hao Chen, Juncheng Ma, Kewei Zhang, Zhongjing Du, Guoqiang Wei, Fangfang Wang, Zhen Dong, Zhiyang Dou, Liming Zhao, Yan Zeng, Jianwei Zhang, Daquan Zhou
PhysLoop uses LLMs’ physics knowledge to help video generators create and repair more physically consistent videos.