Verification Without Sufficiency: Per-Chunk Filtering Fails on Multi-Hop RAG, and Decomposition Repairs It

This paper shows that a standard RAG safeguard—scoring and dropping passages one chunk at a time—systematically fails on multi-hop questions because it assumes each passage is sufficient to answer the original query. On HotpotQA, 2WikiMultihopQA, and MuSiQue, the deployable entailment verifier reaches only 0.643, 0.523, and 0.560 AUC, while the same nli-deberta-v3-base pipeline reaches 0.951 on single-hop SQuAD. The verifier preferentially keeps the paragraph named by the question, usually the first-hop bridge, and rejects the unmentioned paragraph containing the final answer; this bias worsens with hop count and persists across BAAI/bge-small-en-v1.5, intfloat/e5-base-v2, and thenlper/gte-base. End to end, per-chunk gating is the worst selector across all three benchmarks, reducing Exact Match by 13.4 points on HotpotQA relative to unfiltered retrieval, with the penalty growing as generators become stronger. The repair is to decompose the question and verify each later hop against a sub-question: on MuSiQue, gold decomposition raises later-hop AUC from 0.546 to 0.840, a paired lift of +0.355. An off-the-shelf Qwen2.5-7B-Instruct decomposer conditioned on the question and top retrieved paragraph reaches 0.637 AUC, capturing 31% of the ceiling. The authors conclude that systems such as Self-Ask and IRCoT already generate the structure needed for verification, but discard it before filtering; verification should follow the decomposition rather than the original question.