Functional Attention: From Pairwise Affinities to Functional Correspondences

Functional Attention From Pairwise Affinities to Functional Correspondences, by Jiefang Xiao, Maolin Gao, Simon Weber, Guandao Yang, and Daniel Cremers, reframes transformer attention for operator learning as a linear map between learned function spaces rather than a softmax over discrete tokens. The method, called F UNC ATTN, borrows from the functional maps framework in geometry processing and replaces pairwise affinities with a compact k × k spectral transport operator estimated by Tikhonov-regularized least squares, using learned adaptive bases instead of fixed Fourier modes. This makes the attention layer resolution-invariant, globally structured, and theoretically stable; the paper proves local Lipschitz continuity with the regularization parameter λ controlling the bound. Empirically, on six PDE benchmarks including Darcy, Airfoil, Pipe, Navier-Stokes, Elasticity, and Plasticity, F UNC ATTN reaches state-of-the-art relative L2 errors on five tasks and improves over Transolver by 6% to 26.3%, for example dropping Darcy error from 0.57 to 0.42 and Navier-Stokes from 9.44 to 8.00. It also improves 3D RNA segmentation accuracy to 89.0% on xyz coordinates, beats prior methods on a difficult triangular-domain Darcy notch problem with 0.64% relative L2 error, and generalizes best in out-of-distribution AirfRANS airfoil tests, reaching 23.4% error under unseen Reynolds numbers and 13.3% under unseen angles of attack. The key novelty is that attention is computed as an optimal operator in a learned spectral space, which yields linear scaling in sequence length and strong robustness across discretizations and geometries.