Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes

Dream-Cubed, from New York University and Sakana AI, is a block-level generative modeling paper that turns Minecraft into a controllable 3D diffusion benchmark by training on tens of billions of cubes. The core contribution is a curated dataset of 2,026,543 chunks at 32×32×32 resolution, combining 1,667,781 procedurally generated terrain chunks across 15 biome labels with 358,762 human-authored chunks from six professional maps, then using that corpus to compare a 280M-parameter 3D Diffusion Transformer under two formulations: discrete masked diffusion (MD4) and continuous DDPM in embedding space. The models perform almost identically on adjusted render-based FID, 59.26 for MD4 and 59.29 for DDPM, but the discrete formulation uniquely enables hard-constraint inpainting, outpainting, and user-authored block prompting without architectural changes. A human preference study with 19 Minecraft-experienced participants found generated chunks were preferred over real validation chunks for MD4 patch 2 at 67.1%, MD4 patch 4 at 57.1%, and DDPM patch 2 at 55.2%, while MD4 patch 2 narrowly beat DDPM patch 2 in direct comparison at 49.4%, showing that patch size matters more than diffusion type. The paper also shows that sampling quality degrades sharply for MD4 when reducing steps from 1,000 to 10, with average FID rising from 59.26 to 174.53, and that targeted data composition improves structured biomes such as villages and caves relative to naive frequency-based sampling.