NTH

SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement

AuthorsYiqi Liang, Xiaodong Cun

AffiliationsGVC Lab, Great Bay University

November 10, 2026 2 min read
Watch on YouTube
The one-line take

SketchDirector turns text prompts into complex editable sketches by planning scene layouts with an agent and refining the results with diffusion.

Key results

1,000
MCSketch8K scenes

Number of scene sketches in the dataset.

0.923
Semantic alignment

Score for SketchDirector with detail refinement.

0.788
Spatial composition

Score for SketchDirector with detail refinement.

What the paper found

SketchDirector turns text prompts into editable multi-object scene sketches by treating composition as a sequence of explicit layout decisions rather than a single image-generation step. It uses OpenAI GPT-5.5 to decompose a scene into object-level SVG drafts built from cubic Bézier curves, then applies a Plan–Execute–Verify–Correct loop with move, scale, rotate, layer, delete, and modify operations. Geometric checks and multimodal visual-semantic feedback guide revisions, while short-term failure records and reusable long-term layout experiences help the agent improve its plans. For detail refinement, curve-contour resampling preserves the draft geometry, and progressive local-plus-global optimization uses Rectified-Flow Distillation Sampling to add detail without disrupting scene structure. The MCSketch8K dataset contains 1,000 scenes spanning 800 object categories. On multi-model evaluation, the refined system scores 0.923 for semantic alignment and 0.788 for spatial composition, compared with 0.911 and 0.730 for OpenAI GPT-5.5 alone. The evaluators include Claude Opus 4.7 from Anthropic, Gemini 3.1 Pro from Google DeepMind, and Qwen3.7-Plus. The central contribution is a controllable, editable workflow that combines agentic scene layout reasoning with structure-preserving vector refinement.

Original abstract

Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the \textit{MCSketch8K} dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement. The project page is: \href{https://yiki77.github.io/SketchDirector/}{https://yiki77.github.io/SketchDirector/}.

Read the original paper

More in Generative Models

Browse all 87 papers →
01Generative Model

dNaViT: Discrete Codes are General-Purpose Visual Representations

Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai

dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.

Read analysis