Invisible Shortcuts: Why Vision Encoders Know Your Camera

Researchers from Czech Technical University in Prague and the University of Osaka show that vision encoders exploit invisible shortcuts left by cameras and image-processing pipelines. JPEG quality, resizing, sharpening, focal length, aperture, ISO, and camera model leave pixel-level traces that become predictive when correlated with semantic supervision. In ImageNet1k and ImageNet21k, JPEG–class correlations measured by Cramér’s V are 0.047 and 0.067, while models trained on the more correlated ImageNet21k encode metadata more strongly and suffer greater semantic distraction. Controlled ResNet50 experiments confirm a causal relationship: increasing JPEG–label correlation raises metadata prediction and harms semantic generalization, and the effect transfers to unseen metadata attributes. The same pattern appears in caption-based vision-language training: the authors analyze 40M Exif-tagged images from Re-LAION-2B, organize captions into 6238 topics, and train CLIP-loss models on 6.4M-image subsets whose topic–camera correlations reach 0.396, compared with 0.255 and 0.166 for baseline and weaker settings. Mitigation works during and after pretraining: DINOv2-style color jitter, blur, and grayscale reduce sensitivity, while a post-hoc adversarial linear layer suppresses JPEG traces and generalizes to other metadata without sacrificing semantic utility. The trade-off is revealing: metadata-sensitive models are better generated-image detectors, with average accuracy rising from 57.3 for the uncorrelated model to 60.4 at maximal correlation, but removing these shortcuts improves out-of-distribution robustness, including ImageNet-Sketch accuracy from 26.4 to 30.0. The findings also suggest Stable Diffusion images can inherit semantic metadata traces from their training data.