A Mechanistic Study of Tabular Foundation Models

This paper reverse-engineers three tabular foundation models, TabPFNv2, TabICLv2, and Mitra, on 49 classification and 10 regression datasets and shows that similar benchmark accuracy hides fundamentally different inference algorithms. TabPFNv2 and Mitra realize a late similarity-based readout: at layer 9, TabPFNv2’s query-row attention becomes sharply peaked, with mean maximum attention weight 0.925, and an attention-weighted label vote matches native probabilities with Pearson r=0.89; forcing uniform attention drops accuracy from 0.874 to 0.488. Mitra follows the same vote family with r=0.89 but is not identical, because its shared per-feature embedding and label slot make column permutation exact by construction. TabICLv2 is mechanistically different: most class structure is already readable at the column-embedding output, and a parameter-free class-prototype decoder on the final representation reaches 0.854 versus 0.864 native accuracy, while kNN on the final representation reaches 0.856. Cross-backbone readout transplantation fails catastrophically, losing 33 to 40 percentage points, showing the readout is jointly tuned to each backbone. The authors also identify cheap symmetry fixes: zeroing TabPFNv2’s positional matrix W or removing RoPE from TabICLv2 makes column invariance exact with no accuracy loss, and a One-vs-All wrapper yields exact class-order invariance. Finally, mechanism-grounded attacks confirm the inferred circuits: hub poisoning cuts accuracy by 3.3 to 3.9 points on TabPFNv2/TabICLv2, rank warp costs 8.0 to 10.1 points on both but barely affects Mitra, and SVD burial causes up to 8.3-point drops, directly linking model failures to their learned similarity or prototype geometry.