AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
AuthorsQiming Shi, Yikun Li, Beidi Luan, Rui Sun, Sinuo Wang, Delin Wang, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
AffiliationsStepFun · Zhejiang University · Xi’an Jiaotong-Liverpool University · University of Adelaide · Renmin University of China · FinStep
AutoVend tests whether LLM agents can run vending-machine businesses profitably over six months—and finds that most struggle to beat a simple inventory baseline.
Key results
Each agent operates a 20-machine network over 180 simulated days.
Units sold in the six-month commercial dataset used to ground demand.
Mean terminal net assets in thousands of CNY; the model was the only one to beat the baseline.
Net-asset growth in CNY per USD of estimated API expenditure, the highest among evaluated models.
What the paper found
AutoVend tests whether language-model agents can run a physical retail business over time, not just complete individual transactions. Its data-grounded simulator has one agent manage 20 vending machines for 180 simulated days, choosing products and prices, ordering stock, and scheduling paid restocks across a shared warehouse. It models delivery lead times, product expiration, storage costs, and customer demand calibrated from six months of sales records containing 220,128 units sold. The benchmark evaluates OpenAI’s GPT-6.1 Sol, Anthropic’s Claude Sonnet 5.5, DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3-Flash against a rule-based inventory-control strategy. Four of the five models finish below that baseline; GPT-6.1 Sol is the exception, averaging 309.35 thousand CNY in terminal net assets, compared with the baseline’s reported 210.97 thousand CNY. But rankings change when inference costs count: DeepSeek V4.1 Flash leads in net-asset growth per API dollar, at 31,010 CNY, despite lower total returns than GPT. Operational analysis shows why profits alone are incomplete: agents struggle to coordinate products across sites, verify that stock is available before acting, avoid costly or fragmented restocking visits, and turn feedback into lasting policy changes. AutoVend’s central finding is that isolated gains in pricing or inventory metrics do not reliably produce stronger long-term economic performance.
Original abstract
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
Read the original paperMore in AI Benchmarks
Browse all 60 papers →Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
DrawScore checks generated CAD models through engineering-style views, making small but manufacturing-critical mistakes harder to overlook.
PReMind: When Personal Memory Meets Visual Hazards
Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.