Research on algorithms that train machine learning models. Compare convergence, stability, memory use, and computational cost.
36 papers · Latest edition September 30, 2026
Three of the latest briefs in this collection. Read the evidence and the original papers alongside them.
This work proves that stochastic bilevel optimization fundamentally requires up to epsilon^{-6} oracle queries, showing existing methods are asymptotically optimal.
A pair of self-improving coding agents evolves new learnable optimization algorithms from a simple template, reducing the need for handcrafted optimizer design.
A new multiscale matrix-weights algorithm learns hidden linear preferences online with provably optimal dimension-dependent regret.
Newest editions first.
Zhihao Gu, Qilong Wu, Junchi Yang
This work proves that stochastic bilevel optimization fundamentally requires up to epsilon^{-6} oracle queries, showing existing methods are asymptotically optimal.
Zipei Yu, Yue-Jiao Gong, Zeyuan Ma, Yuncheng Jiang, Zhiguang Cao
A pair of self-improving coding agents evolves new learnable optimization algorithms from a simple template, reducing the need for handcrafted optimizer design.
Shinsaku Sakaue
A new multiscale matrix-weights algorithm learns hidden linear preferences online with provably optimal dimension-dependent regret.
Alexander Ryabchenko, Jian Qian, Wenlong Mou
A new zeroth-order optimizer adaptively uses unreliable directional hints to approach first-order performance without needing to know how good those hints are.
Fei Yuan, Changjiang Gao, Yilei Tu, Yifeng Liu, Shujian Huang, Yu Qiao
This work argues that successful fine-tuning depends less on how far a model moves than on choosing the right direction while keeping behavioral drift within a fixed budget.
C. Emre Koksal, Deniz Sargun
This work treats probability distributions like physical systems, finding efficient paths between them using information geometry and a least-action principle.
Louis Abraham, Tuan-Anh Nguyen, Nicolas Devatine
This paper treats live configuration tuning for models and agents as a bandit problem, enabling systems to learn better inference-time settings directly from noisy production feedback.
Lihao Liu, Peng Tang, Kunwar Yashraj Singh, Shabnam Ghadar
ESPO makes evolutionary prompt optimization more accurate, shorter, and more reliable by diagnosing errors, diversifying candidate prompts, and selecting them with bootstrap stability.
Shuta Kobayashi, Andreas Dechant
This paper develops an information-theoretic speed limit for measuring how quickly SGD learns different hidden aspects of data.
Xianlei Zhou, Xiangdi Meng, Yu He, Tianyu Qi, Shuyan Guan, Xianli Zhang, Jian Zhang, Xin Li, Qika Lin, Jun Liu
ERPO regularizes the queries an LLM trains on instead of directly constraining its responses, aiming for more stable learning without sacrificing exploration.
Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, Lei Wu
The paper argues that language-model training dynamics are governed less by raw learning rate than by its ratio to parameter norm, revealing a common effective-learning-rate scale.
Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
This work argues that Muon wins by allocating step sizes across spectral directions and uses that insight to make training even more token-efficient.
Ben Anson, Conor Houghton, Edward Milsom
sMuon adapts the Muon optimizer to low-rank fine-tuning, offering a simpler and sometimes more effective alternative for LoRA-based training.
Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim
A compute-saving framework predicts the right learning rate for enormous MoE models by scaling up results from much smaller proxy runs.
Yushun Zhang
This work explains why Adam works so well for modern neural networks and proposes a slimmer version that retains its performance.
Heechang Kim, Ernest K. Ryu, Shuvomoy Das Gupta
AutoOPT combines numerical search, LLM reasoning, and formal proof assistants to automatically discover and verify new optimization algorithms.
Ali Janati, Kaoutar El Maghraoui, Andrei Kanavalau, Anass Belfatmi
The study shows that Muon can make transformers grok quickly but later destabilize their representation-to-output interface, causing learned solutions to collapse.
Devender Singh
The paper argues that Adam’s coordinate-wise behavior breaks a hidden symmetry that lets gradient descent favor low-rank solutions, showing that optimizer geometry—not just the loss—determines which answer a model learns.
Chuyan Chen, Peng Sun, Kun Yuan
CMuon makes diffusion-transformer training faster and more stable by orthogonalizing fused weight components independently instead of coupling them together.
Wenzhi Zhong, Edward Milsom, Michael Murray
This work combines spectral-norm sharpness control with the Muon optimizer to improve ImageNet training performance on both ViTs and ResNets.
Maria Smirnova, Alexey Kravatskiy
This paper shows that compressing Muon to signs can break its theory, yet the seemingly unsafe version often works best in real-world training.
Jiangwang Chen, Zixin Song, Junlin Liu, Shuaiyu Zhou, Haiyan Wu, Haihan Shi, Chenxi Zhou, Hanqing Li, Xiao Yang, Da Zhu, Guanjun Jiang, Hai Wan, Xibin Zhao
DecoEvo helps language models improve by evolving both their problem-solving instructions and the rubrics used to expose overlooked weaknesses.
Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, Bryan Dai
Hyperball optimizers may not win because of better update directions after all—their apparent advantage largely depends on how their effective learning rate evolves over training.
Takuya Yoshioka, Keita Sasada, Riku Usuki, Yuichiro Nakano, Keisuke Fujii
A qubit-efficient quantum optimization method tackles power-demand portfolios at surprisingly large scales, with promising results in both simulation and hardware.
Hanqing Zhu, Wenyan Cong, Zhizhou Sha, Sagnik Mukherjee, Xinyuan Song, David González-Martínez, Xiaoxia Wu, Yuandong Tian, Shiwei Liu, David Z. Pan, Zhangyang "Atlas" Wang
ISO speeds up RLVR training by preserving a model’s singular-value spectrum while optimizing the input and output directions that encode new reasoning behaviors.
Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun
Muon may substantially improve agentic RL training, but stronger evidence across seeds, tasks, and optimization settings is still needed.
Nuemaan Malik
SkewAdam cuts MoE optimizer memory dramatically by giving different parameter groups only the optimizer state they actually need, without sacrificing accuracy.
Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan
OmniOpt organizes the chaotic world of modern optimizers into a unified taxonomy and benchmark so researchers can compare methods more systematically.
Venkatesh Vadde, Roman Ovcharov, Victor H. González, Roman Khymyn, Artem Litvinenko, Johan Åkerman
This paper builds a 2,048-spin acoustic Ising machine that uses sound waves in solid-state delay lines to tackle hard optimization problems like MAX-CUT and Sudoku.
Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, Zhuoran Yang
This paper explains why the Muon optimizer can beat Adam in large language model training by showing it makes smaller curvature-induced mistakes, especially in imbalanced and high-curvature settings.
Youngjae Park, Jaemin Kim, Junghwa Hong
This paper shows that a smarter preconditioned optimizer can make physics-informed neural networks far more stable when solving tightly coupled multiphysics problems.
Yunwen Lei, Zimeng Wang, Xiaoming Yuan
This paper explains why the popular SVRG optimizer generalizes well, not just why it converges, and extends the same theory to related variance-reduced methods.
Aratrika Mustafi, Soumya Mukherjee, Bharath K. Sriperumbudur
This paper gives Muon optimizer a physics-style mathematical makeover, showing it as a damped Hamiltonian flow and deriving convergence and mean-field guarantees.
Nikhil Nayak, Julia White, Urchade Zaratiana, Kelton Zhang, Henrijs Princis, Dhruv Atreja, Henry Fawcett, Matthew Thomas, George Hurn-Maloney, Ash Lewis
This paper shows that popular optimizer updates for language models are subtly biased on small batches, and it introduces a practical correction that can make AdamW, Sophia, and Shampoo train a bit better.
Igor Ignashin, Anna Radovskaya, Andrew Semenov, Egor Lopatin, Stanislav Potapov, Aleksandr Kovalenko, Andrey Veprikov, Aleksandr Shestakov, Andrey Leonidov, Aleksandr Beznosikov
This paper argues that SGD should be viewed less like Brownian motion and more like motion in a randomly changing landscape, revealing when training trajectories diffuse versus stay trapped.
Xingyu Lyu, Qianqian Xu, Zhiyong Yang, Peisong Wen, Qingming Huang
This paper argues that GLU layers work better because they make the training dynamics better conditioned, helping models learn faster rather than necessarily generalizing better.