SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling
AuthorsXiaokun Sun, Jian Yang, Zhenyu Zhang
AffiliationsNanjing University
SeMo3D uses selective 3D diffusion to create or modify chosen regions while keeping the rest of an asset intact.
Key results
Generation perceptual distance; lower is better.
Generation image similarity; higher is better.
Editing consistency score outside the target region.
Completion result with sparse point guidance.
What the paper found
SeMo3D unifies 3D generation, editing, and completion by synthesizing only a selected region of a structured 3D asset while reusing clean context tokens, so unchanged geometry is preserved by construction. Its region-selective denoising trains on ordinary 3D assets without paired editing or completion data, and Omni3DiT combines image guidance with optional text and sparse 3D points across structured-latent diffusion stages. On Toys4K, SeMo3D achieves an LPIPS score of 0.079 and DINO similarity of 0.926; on Edit3D-Bench, its PSNR is 37.39; and on Omni-Comp, its completion F-score reaches 0.541 with point guidance. The system also supports interactive creation: an agent using OpenAI’s GPT-6 Astra, Codex, and GPT-Image can prepare inputs, identify edit regions, and invoke SeMo3D for successive changes.
Original abstract
3D generation, editing, and completion are typically treated as regenerating the complete 3D asset under task-specific constraints, even when only a local region needs modification. This leads to task isolation, inconsistency in content that should remain unchanged, and dependence on task-specific paired data. We address these issues with a key observation: the three tasks all synthesize a designated region while keeping the rest intact---the full region for generation, the target region for editing, and the missing region for completion. We term this paradigm Selective Modeling and build SeMo3D upon it; it naturally unifies the three tasks, preserves unchanged content by construction, and enables training without paired editing or completion data, since any 3D asset can be partitioned into a selected region and its context. For training, we propose region-selective denoising, which randomly selects regions of varying sizes on standard 3D assets and applies noise and loss only to them, jointly learning all three tasks. For architecture, we propose Omni3DiT, a multi-condition 3D diffusion transformer that distinguishes regions to create from those to retain and fuses image, text, and point guidance for fine-grained control. Experiments across three benchmarks show SeMo3D surpasses state-of-the-art methods on all tasks, delivering multimodal controllability in generation and strong consistency and accuracy in editing and completion. SeMo3D further serves as a unified modeling tool for agents, enabling efficient 3D creation through natural-language interaction. Project page: https://xiaokunsun.github.io/SeMo3D.github.io/.
Read the original paperMore in Diffusion Models
Browse all 82 papers →FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.
JVT: Just Video Transformer makes pixel-space video diffusion simple
Jianjun Zhou, Hua Geng, Haoyu Guo
JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.
Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
The paper restores variety in fast distilled image and video generators by finding and correcting a collapse in their internal activation directions.