Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

This paper introduces Feature-Effect Geometry Analysis, or FEGA, an unsupervised method that evaluates SAE interventions by propagating feature ablations through the frozen model tail and analyzing their downstream logit-effect clouds rather than relying on local decoder directions. Using Google’s Gemma-2-2B with 65k-width post-layer-12 SAEs—ReLU, TopK, and Matryoshka Batch TopK—the authors compare value-like features identified by RAVEL city-country attribute editing with pointer-like features from Literal Sequence Copying, Word Content, PrOntoQA, and Token Translation. Pointer-like features recur across tasks, with TopK reaching an intersection-over-union of 0.40 between Literal Sequence Copying and Word Content, compared with a maximum of 0.13 for ReLU, and they are causally important: ablating 9 TopK Word Content features reduces accuracy to 18.4%, versus 93.6% ± 4.0% for matched random ablations. Yet their downstream effects are rarely stable steering directions. Among 239 pointer-like candidates, 159 have undefined geometry; of the 80 mapped cases, 74 are unresolved high-dimensional or diffuse, while only 3 form directed rays and 3 occupy global low-dimensional subspaces. Value-like features are more structured: low-dimensional categories cover 14.0% of eligible ReLU, 36.3% of TopK, and 30.3% of Matryoshka Batch TopK features, but directed rays remain rare. The central conclusion is that an SAE feature can encode a coherent concept or function and contribute causally without corresponding to a reusable one-dimensional logit-space control vector.