Agent as Policy for Robotic Manipulation

Agent as Policy introduces AGP, a framework in which a general-purpose multimodal coding agent acts as the robot policy rather than selecting among separately trained skills. Given video, image, or language instructions and a documented robot interface, the agent requests camera and depth observations, writes Python programs for perception and geometry, computes gripper poses or joint targets, sends motion commands, and revises its programs after physical feedback, with model parameters fixed throughout execution. Using OpenAI’s GPT-6 Astra through Codex and the Mink inverse-kinematics library, AGP controlled real robots across AutoMate-based four-pair assembly, image-guided block construction, dice flipping, targeted throwing, and bimanual towel folding. It achieved at least 80% success in seven of eight main task configurations, including 100% on dice flipping, while simultaneous towel folding reached only 60%, exposing the difficulty of deformable-object coordination. Persistent files containing measurements, corrections, and executable procedures reduced two-pair assembly task time by 29.3% and reasoning-and-programming latency by 47.0% between the first and fifth executions. Experience transfer also raised a weaker GPT-5.6 Terra agent’s success rate from 20% to 80% and reduced successful-trial completion time by 34.3%. A comparison involving Anthropic’s Claude Code and Claude Opus 5 shows that AGP is not tied to OpenAI, but execution latency, inference cost, and reliability remain substantial deployment barriers.