OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

OmniOpt, developed by researchers from Shanghai AI Laboratory, Shanghai University, Westlake University, Shanghai Jiao Tong University, UCAS, Zhejiang University, and SUSTech, reframes optimizer design as a mechanism-aware system rather than a flat leaderboard. The paper unifies 108 optimizers with a five-stage meta-pipeline, an LMO-driven four-axis geometry, and a dual taxonomy that separates method families from effect objectives. In benchmarking 24 representative optimizers, it shows that no single method dominates: on C4 short-context pretraining, APOLLO reaches the best 1B perplexity at 13.53, while Muon and MARS-Shampoo are close behind at 13.72; yet the cheapest runtime comes from Lion, and the lowest optimizer-state memory comes from AdaFactor. Across FineWeb-Edu 32k long-context runs, SOAP is the most stable cross-scenario optimizer, holding the top perplexity position in 7 of 8 scenarios, while MARS-AdamW is the most stable AdamW-style enhancement. APOLLO is the sharpest cautionary case: it is best at 256 tokens with 13.53 perplexity but collapses to 35.40 at 32k, showing that aggressive state compression is rank-bounded and can fail as context length grows. The Muon ablation isolates Newton–Schulz orthogonalization as the core gain, recovering perplexity from 70.74 to 16.86 after removing AdamW-style second-moment scaling, and then improving to 13.58 when symmetric learning-rate scaling and post-orthogonalization Nesterov are combined at 1B. On CIFAR100, optimizer choice is backbone-dependent: AdaBelief leads ResNet50 with 80.53%, Muon leads DeiT-S with 77.38%, and Adan leads CAFormer-S12 with 84.89%, reinforcing the paper’s central claim that optimizer geometry must match model topology and training constraints.