JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation
AuthorsXiangjun Tang, Jianqi Chen, Peter Wonka
AffiliationsKing Abdullah University of Science and Technology (KAUST) · xiangjun.tang@outlook.com windvchen@gmail.com pwonka@gmail.com
JoVox generates detailed voxel shapes more efficiently by letting fine-scale structure help guide and correct coarse decisions.
Key results
Lower voxel Chamfer distance on image-conditioned Toys4k generation.
Higher throughput than TRELLIS.2 with the same backbone.
JoVox result on image-conditioned Toys4k generation; TRELLIS.2 scores 0.4017.
GB used on one NVIDIA A100, compared with 47.3 GB for TRELLIS.2.
H100 GPUs used for JoVox training, versus 32 for TRELLIS.2.
What the paper found
JoVox tackles a weakness in hierarchical voxel generation: early coarse cells can be plausible but wrong, and conventional methods never revisit them. Its discrete flow-matching model learns the joint distribution across all levels, allowing finer cells to inform coarse-cell predictions. Two-level prediction generates occupancy codes two levels down at once, halving the number of levels processed, while an unmasking strategy revisits the oldest predictions as more detail becomes available. On image-conditioned Toys4k generation, JoVox reaches a voxel IoU of 0.4158 versus 0.4017 for TRELLIS.2 and an 11% lower Chamfer distance. It also delivers 8× higher training throughput; on one NVIDIA A100 at batch size 32, it uses 29.0 GB of memory versus 47.3 GB for TRELLIS.2. The model was trained on 4 H100 GPUs, compared with 32 for TRELLIS.2, and uses DINOv3 features for image conditioning. JoVox also outperforms baselines on unconditional ShapeNet generation, while avoiding an autoencoder and processing only cells within occupied regions.
Original abstract
Generating high-fidelity voxels while keeping their lightweight nature remains an open problem in computer vision. Current generators are either high-fidelity but computationally heavy, or lightweight but of lower fidelity. Hierarchical methods keep the lightweight nature of voxels. However, their coarse levels, which decide where all finer cells can appear, are generated with the least information about the shape and are never revisited, so they are often plausible but wrong. Our key insight is to let finer cells take part in deciding coarse cells. To this end, we propose JoVox, a discrete flow matching framework that learns the joint distribution of all levels of the voxel hierarchy, instead of a separate distribution for each level, so that coarse cells can be predicted with finer cells already generated nearby. Within this framework, we further introduce a two-level prediction, which outputs the cells two levels finer at once instead of one level finer, halving the number of levels that the network processes and yielding voxels of higher fidelity. We also introduce an unmasking strategy that revisits early decisions once finer cells are generated and corrects inconsistencies in masked diffusion models. Our framework outperforms the baselines on both unconditional and image-conditioned voxel generation. Compared with the current state-of-the-art TRELLIS.2, we achieve 8x higher training throughput, 1/15 of the FLOPs in a forward pass, train on 4 vs. 32 GPUs, and obtain higher quality with an 11% lower Chamfer distance.
Read the original paperMore in Generative Models
Browse all 87 papers →dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai
dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.
Lipschitz-Controlled Flow Matching for Robust Image Restoration
Sedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner, Sebastian Peitz
LipsFlower aims to make flow-matching image restorations less sensitive to small input changes without sacrificing perceptual quality.
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
Yuxuan Chen, Xiaojie Li, Hao Chen, Juncheng Ma, Kewei Zhang, Zhongjing Du, Guoqiang Wei, Fangfang Wang, Zhen Dong, Zhiyang Dou, Liming Zhao, Yan Zeng, Jianwei Zhang, Daquan Zhou
PhysLoop uses LLMs’ physics knowledge to help video generators create and repair more physically consistent videos.