The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Researchers at Shanghai Artificial Intelligence Laboratory and Shanghai Jiao Tong University present a causal audit of OpenAI’s “thinking-with-images” paradigm, testing whether crop-and-zoom observations genuinely influence multimodal model answers. Across 6 open models—including DeepEyes, Mini-o3, Qwen3-VL-4B/8B, Pixel Reasoner, and Thyme—and 5 fine-grained perception benchmarks, they compare tool-enabled inference with direct inference, dynamically replace every returned crop with a random crop, and perform step-level counterfactual interventions. Their key metric, Visual Evidence Gain, estimates the observation-mediated effect while cancelling action-induced shortcuts: a model may call a tool and change its answer simply because a call occurred, without using the image content. The results expose two calibration failures: “Calling Without Looking,” where observations have near-zero causal influence, and “Looking Without Planning,” where useful evidence exists but calls continue after confidence is already high or repeatedly target irrelevant regions. Aggregate gains can therefore be misleading: Mini-o3 improves VisualProbe by 21.3 percentage points, yet random-crop corruption drops its V* accuracy by 64.2 percentage points, partly through repair loops and budget exhaustion. On V*, only 20.9% of Qwen3-VL-8B trajectories are classified as Calibrated, but that minority contributes +7.6 percentage points to the overall gain. The paper argues that visual tool use should be evaluated by causal evidence use and stopping behavior, not call frequency, and hypothesizes that outcome-only reinforcement learning rewards tool-call rituals and poor credit assignment. The authors caution that these findings cover open models and crop-and-zoom, not closed systems such as OpenAI o3 or o4-mini.