NTH

SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

AuthorsXiaokun Sun, Jian Yang, Zhenyu Zhang

AffiliationsNanjing University

November 11, 2026 2 min read
Watch on YouTube
The one-line take

SeMo3D uses selective 3D diffusion to create or modify chosen regions while keeping the rest of an asset intact.

Key results

0.079
Toys4K LPIPS

Generation perceptual distance; lower is better.

0.926
Toys4K DINO similarity

Generation image similarity; higher is better.

37.39
Edit3D-Bench PSNR

Editing consistency score outside the target region.

0.541
Omni-Comp F-score

Completion result with sparse point guidance.

What the paper found

SeMo3D unifies 3D generation, editing, and completion by synthesizing only a selected region of a structured 3D asset while reusing clean context tokens, so unchanged geometry is preserved by construction. Its region-selective denoising trains on ordinary 3D assets without paired editing or completion data, and Omni3DiT combines image guidance with optional text and sparse 3D points across structured-latent diffusion stages. On Toys4K, SeMo3D achieves an LPIPS score of 0.079 and DINO similarity of 0.926; on Edit3D-Bench, its PSNR is 37.39; and on Omni-Comp, its completion F-score reaches 0.541 with point guidance. The system also supports interactive creation: an agent using OpenAI’s GPT-6 Astra, Codex, and GPT-Image can prepare inputs, identify edit regions, and invoke SeMo3D for successive changes.

Original abstract

3D generation, editing, and completion are typically treated as regenerating the complete 3D asset under task-specific constraints, even when only a local region needs modification. This leads to task isolation, inconsistency in content that should remain unchanged, and dependence on task-specific paired data. We address these issues with a key observation: the three tasks all synthesize a designated region while keeping the rest intact---the full region for generation, the target region for editing, and the missing region for completion. We term this paradigm Selective Modeling and build SeMo3D upon it; it naturally unifies the three tasks, preserves unchanged content by construction, and enables training without paired editing or completion data, since any 3D asset can be partitioned into a selected region and its context. For training, we propose region-selective denoising, which randomly selects regions of varying sizes on standard 3D assets and applies noise and loss only to them, jointly learning all three tasks. For architecture, we propose Omni3DiT, a multi-condition 3D diffusion transformer that distinguishes regions to create from those to retain and fuses image, text, and point guidance for fine-grained control. Experiments across three benchmarks show SeMo3D surpasses state-of-the-art methods on all tasks, delivering multimodal controllability in generation and strong consistency and accuracy in editing and completion. SeMo3D further serves as a unified modeling tool for agents, enabling efficient 3D creation through natural-language interaction. Project page: https://xiaokunsun.github.io/SeMo3D.github.io/.

Read the original paper

More in Diffusion Models

Browse all 82 papers →