Video-Index: A Curated Meta-Benchmark for Video Understanding

Video-Index argues that many video-understanding scores measure shortcuts rather than video comprehension. Its attack pyramid tests five increasingly informed adversaries: answer options, question text, previously evaluated items, a single frame or captions, and shuffled or truncated video. Auditing 115 multiple-choice benchmarks found that 35 can be solved nearly at full-video accuracy without seeing a frame, while on 51 benchmarks shuffled frames retain a median 96% of the original accuracy; near-duplicate questions constitute at least half of the items in 63 benchmarks. The authors screen 505,518 question–answer pairs with Qwen3-VL-8B and Qwen3-VL-2B, then use deterministic, coverage-aware selection, agent-generated specifications, Claude Opus 5 verification, and a red-team gate to release Video-Index: 840 verified hard items, divided into four capability groups with 210 items each and drawn from 76 sources. Under fixed inputs, Anthropic’s Claude Opus 5 reaches 56.8%, compared with 19.0% for the strongest open model, Molmo2-8B, while tool use raises matched-model performance by 19.8%. Agentic systems perform substantially better: OpenAI’s GPT-6-Astra reaches 79.3%, ahead of Claude Fable 5.1 at 71.5%, but performance still degrades on long videos, showing that evidence selection and efficient temporal inspection remain unresolved.