ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

ModaLens introduces a paired image-swap audit for testing whether report-conditioned medical vision-language models actually use radiographs. In MIMIC-CXR, the researchers evaluated 3,199 cases from 293 patients across 14 questions per case, holding the question and radiology report fixed while replacing each image with another study, usually from the same patient. For MedGemma-27B, the generated answer changed with the substituted image on 4.26% of 44,786 trials when the report was present, versus 20.94% when it was removed, a paired increase of 16.7 percentage points under an explicit yes-or-no instruction. Continuous answer margins showed the same pattern even when the binary answer did not flip: mean absolute margin change rose from 0.690 with the report to 2.307 without it. Attention knockouts suggested that blocking direct access to report tokens from early decoder layers, especially layers 0 through 20, partially restored image sensitivity, while the study cautions that this does not localize all downstream report use. The direction replicated in MedGemma-4B, Qwen3.5-9B, Qwen3.5-27B, and LLaVA-NeXT on Mistral-7B, with experiments run on NVIDIA H100 or H200 GPUs. However, because labels were derived from report text rather than independent image annotations, ModaLens measures image sensitivity and report anchoring—not visual correctness.