Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
AuthorsHongyang Du, Lan Yan, Christian Flores, Asim Kadav
AffiliationsAdobe · Brown University Corresponding to hongyang
Resources
A frozen AI designer gets better over time by turning user interactions and successes into reusable, continually refined design procedures.
Key results
The frozen graphic-design agent operates through more than 230 tools.
The procedural memory grew from 76 documentation-derived skills to 139.
Claude-Sonnet-4 improved from 72.7% to 99.3% on GenEval2.
Generation quality increased by 11.99 points on Claude-Sonnet-4.
Claude-Opus-4.6 won 67.6% of pairwise comparisons against the no-skill agent.
The combined update mechanisms reached a 58.5% win rate, compared with 49.4% for widening alone and 48.6% for deepening alone.
What the paper found
This paper presents a continual-adaptation system for agentic graphic design in which a frozen language model operates Adobe Photoshop-, Illustrator-, and InDesign-like software through more than 230 tools, while an external procedural memory stores reusable natural-language workflows. The memory evolves by widening—minting skills for recurring uncovered subtasks—and deepening—rewriting failure-prone skills using successful and failed executions. A matched replay gate compares candidate and incumbent behaviors under identical contexts, admitting changes only when they improve at least one case without detected regressions. Across five rounds and 1,406 user-traffic briefs, the skill bank expanded from 76 to 139 procedures, producing 1,869 automatically graded trajectories without weight updates or human reward labels. On Claude-Sonnet-4, GenEval2 execution success increased from 72.7% to 99.3%, while generation quality improved by 11.99 points; the system also achieved overall pairwise win rates of 61.8% against the no-skill agent on Claude-Sonnet-4 and 67.6% on Claude-Opus-4.6 across OpenCOLE, GraphicBench, CreatiDesign, and BannerRequest400, with comparisons judged by OpenAI GPT-5.4. Ablations show that widening or deepening alone reached 49.4% and 48.6% win rates, but their combination reached 58.5% with p = 0.025. The results position procedural memory as a practical test-time learning mechanism for noisy, weakly verifiable design feedback, while exposing limits in geometric precision, long-procedure fidelity, and preference enforcement.
Original abstract
Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.
Read the original paperMore in Continual Learning
Browse all 24 papers →ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
Haodong Lu, Dong Gong
ASCENT lets deployed LLM agents learn from verified successes on the fly by converting hindsight about their own trajectories into lasting weight updates.
From Knowledge Access to Source Learning: Developing Source-Specific Competence
Lucheng Fu, Kejing Xia, Yiyang Wang, Yiqiao Jin, Jinjin He, Xiyuan Yang, Haoxin Liu, Ye Yu, Haibo Jin, Yijia Xiao, Wenke Lee, B. Aditya Prakash, Haohan Wang
SourceLearn helps LLM agents progressively build reusable expertise about trusted information sources instead of repeatedly starting from scratch.
Local Support Learning
Assaf Ben-Kish, Akarsh Kumar, James Glass, Raja Giryes
Local Support Learning helps large language models learn new skills without overwriting what they already know by activating updates only where they are locally needed.