NTH

AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

AuthorsQiming Shi, Yikun Li, Beidi Luan, Rui Sun, Sinuo Wang, Delin Wang, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai

AffiliationsStepFun · Zhejiang University · Xi’an Jiaotong-Liverpool University · University of Adelaide · Renmin University of China · FinStep

November 11, 2026 2 min read
Watch on YouTube
The one-line take

AutoVend tests whether LLM agents can run vending-machine businesses profitably over six months—and finds that most struggle to beat a simple inventory baseline.

Key results

20
Vending machines managed

Each agent operates a 20-machine network over 180 simulated days.

220,128
Historical sales volume

Units sold in the six-month commercial dataset used to ground demand.

309.35
GPT-6.1 Sol terminal net assets

Mean terminal net assets in thousands of CNY; the model was the only one to beat the baseline.

31,010
DeepSeek V4.1 Flash API efficiency

Net-asset growth in CNY per USD of estimated API expenditure, the highest among evaluated models.

What the paper found

AutoVend tests whether language-model agents can run a physical retail business over time, not just complete individual transactions. Its data-grounded simulator has one agent manage 20 vending machines for 180 simulated days, choosing products and prices, ordering stock, and scheduling paid restocks across a shared warehouse. It models delivery lead times, product expiration, storage costs, and customer demand calibrated from six months of sales records containing 220,128 units sold. The benchmark evaluates OpenAI’s GPT-6.1 Sol, Anthropic’s Claude Sonnet 5.5, DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3-Flash against a rule-based inventory-control strategy. Four of the five models finish below that baseline; GPT-6.1 Sol is the exception, averaging 309.35 thousand CNY in terminal net assets, compared with the baseline’s reported 210.97 thousand CNY. But rankings change when inference costs count: DeepSeek V4.1 Flash leads in net-asset growth per API dollar, at 31,010 CNY, despite lower total returns than GPT. Operational analysis shows why profits alone are incomplete: agents struggle to coordinate products across sites, verify that stock is available before acting, avoid costly or fragmented restocking visits, and turn feedback into lasting policy changes. AutoVend’s central finding is that isolated gains in pricing or inventory metrics do not reliably produce stronger long-term economic performance.

Original abstract

Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.

Read the original paper

More in AI Benchmarks

Browse all 60 papers →
03Benchmark

PReMind: When Personal Memory Meets Visual Hazards

Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon

PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.

Read analysis