VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents
AuthorsZhengXian Wu, Qianhao Nie, Zhuohong Chen, Junjie Gao, Yuyao Zhang, Yunyao Yu, Hangrui Xu, Cheng Cheng, Guiyu Ma, Jin Zhang, Xingzhong Xu, Kai Yang, Haoqian Wang
Affiliations[
VisSearch-Bench tests whether multimodal agents can keep track of visual clues and use them effectively across multi-step searches.
Key results
Human-written long-horizon visual search tasks.
Benchmark spans seven content domains and 25 subdomains.
Total steps across the benchmark’s reference solutions.
Highest complete-task accuracy reported.
Highest reported score for intermediate findings and subquestions.
What the paper found
VisSearch-Bench tests whether multimodal agents can keep track of visual evidence across long web searches—not just recognize an image, but connect objects, visual details, and external facts to guide later searches and build complete answers. The benchmark contains 204 human-written tasks across 7 domains and 25 subdomains, organized into five search patterns, from checking a set of objects to following spatial clues in an image; its reference solutions contain 3,593 steps. It scores both strict end-to-end Accuracy and a Process Score for correct intermediate findings. Across 16 models, even agents with search tools struggled: ReAct with Kimi-K3 achieved the highest Accuracy at 26.47%, while ReAct with GPT-5.5 led the Process Score at 49.52%. The evaluation also includes Gemini 3.5 Flash, Claude Opus 4.8, DeepSeek-V4.1-Flash, and Qwen models. The gap between partial progress and complete task success shows the central challenge: agents may retrieve relevant evidence yet still miss candidates, confuse similar objects, or fail to carry visual context consistently from one search step to the next.
Original abstract
Multimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.
Read the original paperMore in AI Benchmarks
Browse all 60 papers →Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
DrawScore checks generated CAD models through engineering-style views, making small but manufacturing-critical mistakes harder to overlook.
PReMind: When Personal Memory Meets Visual Hazards
Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.