PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs

PGT, short for Procedurally Generated Tasks, is a data-centric method from Mila, Universite de Montreal, and Meta FAIR that improves visual grounding in multimodal large language models by overlaying unambiguous geometric primitives onto training images and turning them into dense, verifiable supervision signals. The paper targets three failure modes in MLLMs: spatial relations, abstract counting, and relative distance or depth reasoning. Instead of adding human annotations or changing model architecture, PGT procedurally generates questions about colored boxes, circles, and labeled points directly on existing datasets such as LLaVA-Instruct-1.5 and Cambrian-7M, preserving the original semantic content while forcing the model to attend to image geometry rather than language priors. Across four backbones in instruction tuning—Vicuna-1.5-7B, Llama-3-8B, Qwen-2.5-7B, and Qwen-2.5-14B—PGT raises What’s Up by up to 20.0 points and CV-Bench-2D by up to 13.3 points, with consistent gains on CV-Bench-3D, BLINK, and TallyQA and little or no regression on general benchmarks like GQA, POPE, and MMSTAR. When fine-tuning state-of-the-art models such as Qwen-2.5-VL and InternVL3 with only 5k PGT samples, the method still improves spatial and depth benchmarks, sometimes outperforming specialized real-data mixes and RL-based baselines like ViGoRL and SpatialLadder. Ablations show that removing the relative-distance task sharply degrades 3D/depth performance, indicating that 2D geometric comparison may be the core transferable primitive behind emergent depth understanding.