SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement
AuthorsYiqi Liang, Xiaodong Cun
AffiliationsGVC Lab, Great Bay University
SketchDirector turns text prompts into complex editable sketches by planning scene layouts with an agent and refining the results with diffusion.
Key results
Number of scene sketches in the dataset.
Score for SketchDirector with detail refinement.
Score for SketchDirector with detail refinement.
What the paper found
SketchDirector turns text prompts into editable multi-object scene sketches by treating composition as a sequence of explicit layout decisions rather than a single image-generation step. It uses OpenAI GPT-5.5 to decompose a scene into object-level SVG drafts built from cubic Bézier curves, then applies a Plan–Execute–Verify–Correct loop with move, scale, rotate, layer, delete, and modify operations. Geometric checks and multimodal visual-semantic feedback guide revisions, while short-term failure records and reusable long-term layout experiences help the agent improve its plans. For detail refinement, curve-contour resampling preserves the draft geometry, and progressive local-plus-global optimization uses Rectified-Flow Distillation Sampling to add detail without disrupting scene structure. The MCSketch8K dataset contains 1,000 scenes spanning 800 object categories. On multi-model evaluation, the refined system scores 0.923 for semantic alignment and 0.788 for spatial composition, compared with 0.911 and 0.730 for OpenAI GPT-5.5 alone. The evaluators include Claude Opus 4.7 from Anthropic, Gemini 3.1 Pro from Google DeepMind, and Qwen3.7-Plus. The central contribution is a controllable, editable workflow that combines agentic scene layout reasoning with structure-preserving vector refinement.
Original abstract
Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the \textit{MCSketch8K} dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement. The project page is: \href{https://yiki77.github.io/SketchDirector/}{https://yiki77.github.io/SketchDirector/}.
Read the original paperMore in Generative Models
Browse all 87 papers →dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai
dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.
Lipschitz-Controlled Flow Matching for Robust Image Restoration
Sedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner, Sebastian Peitz
LipsFlower aims to make flow-matching image restorations less sensitive to small input changes without sacrificing perceptual quality.
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
Yuxuan Chen, Xiaojie Li, Hao Chen, Juncheng Ma, Kewei Zhang, Zhongjing Du, Guoqiang Wei, Fangfang Wang, Zhen Dong, Zhiyang Dou, Liming Zhao, Yan Zeng, Jianwei Zhang, Daquan Zhou
PhysLoop uses LLMs’ physics knowledge to help video generators create and repair more physically consistent videos.