A Theory of Contrastive Learning with Natural Images

Antonio Torralba and Yair Weiss show that contrastive learning on natural images can be understood analytically through frequency-domain statistics rather than heuristic invariances. By replacing InfoNCE with a Gaussian-equivalent objective, they prove that the optimum is a “partial whitening” representation: it selects a subset of Fourier modes and rescales them inversely to expected power, so the embedding covariance becomes white. For simple augmentations such as circular crop, brightness/contrast jitter, and ideal blur, the globally optimal representation is just squared DFT magnitudes at K non-DC frequencies, normalized to unit norm; this can be computed by a one-hidden-layer CNN with sinusoidal filters, pointwise nonlinearity, global average pooling, and a linear projection. For harder augmentations including crop-plus-noise, arbitrary blur, and linear jitter, the optimal last-layer weights are generalized eigenvectors of the alignment matrix B and covariance Σ, with the selected frequencies found by a waterfilling algorithm. The theory predicts, and experiments confirm, that SGD-trained CNNs on CIFAR10, CIFAR100, ImageNet, fractal noise, dead leaves, and CelebA learn sinusoidal first-layer filters and frequency sensitivities shaped like rings or diamonds in Fourier space. In their CIFAR10 KNN study, training on non-CIFAR synthetic noise yields the best transfer when its expected power spectrum matches CIFAR10, reaching 45.1%, while simple partial whitening in PCA space gives 47.1% and close to the learned representation’s performance.