Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm

This University of Bath study examines whether Sharpness-Aware Minimization should measure parameter perturbations with matrix-aware spectral geometry rather than a flattened Euclidean norm. The authors introduce SpecSAM, which applies independent layerwise spectral-norm perturbations to weight matrices, approximating orthogonalization with Newton–Schulz iterations, and pairs this inner step with either AdamW, SGDW, or Muon—the momentum-plus-orthogonalization optimizer increasingly used in frontier language-model systems such as DeepSeek. On ImageNet-1K, SpecSAM-Muon achieves the strongest validation results for both ViT-Small/16 and ResNet-50: 80.23 Top-1 accuracy for the transformer, a 5.28% improvement over the non-SAM Muon baseline at 74.95, and 78.55 for ResNet-50, a 1.50-point gain over Muon at 77.05. The same method reaches 42.18 accuracy on the out-of-distribution ImageNet-R benchmark, although the three-seed differences there are not statistically resolvable. The central finding is an interaction between inner and outer geometry: Euclidean SAM adds roughly similar gains to different optimizers, while spectral perturbations selectively amplify Muon. A ResNet-50 ablation shows that layerwise budgeting alone is insufficient; replacing spectral norms with Frobenius norms removes the Muon advantage. CIFAR-100 diagnostics further indicate that Muon preserves higher effective weight and feature ranks across perturbation radii, potentially retaining representational capacity under stronger regularization. The evidence remains limited to image classification, three-seed main experiments, and methods that require two backward passes plus Newton–Schulz overhead.