NTH
Optimization

Hyperball May Not Be a Free Lunch

AuthorsYihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, Bryan Dai

August 1, 2026 2 min read
Watch on YouTube

Resources

Read the original paper on arXiv GitHub repository
The one-line take

Hyperball optimizers may not win because of better update directions after all—their apparent advantage largely depends on how their effective learning rate evolves over training.

Key results

3.28
Target validation loss

Validation-loss threshold used to compare MuonH schedules.

3175
MuonH-Z steps

Steps required to reach the 3.28 validation-loss target.

What the paper found

In “Hyperball May Not Be a Free Lunch,” researchers from IQuest Research, Peking University, Shenzhen University of Advanced Technology, and Shanghai University of Finance and Economics investigate why Hyperball-style optimizers, especially MuonH, can initially lag behind MuonWD yet later overtake it. They derive an angular effective learning rate based on parameter-update angle, parameter norm, update norm, and learning rate, showing that the familiar norm-only formula is valid only when the update is orthogonal to the parameter. Controlled analyses find that suppressing radial updates has limited direct influence on one-step angular motion; instead, Hyperball’s fixed parameter norm and normalized updates mainly create a different, state-dependent effective learning-rate schedule. A learning-rate alignment experiment shows that much of the apparent optimizer advantage can be reproduced by changing only the schedule, rather than the update direction. In dense language-model pretraining, more aggressive decay accelerates MuonH early but can damage later convergence. For a validation-loss target of 3.28, the MuonH-Z schedule required 3175 steps, while the authors’ power-0.4 MuonH-Ours schedule reached it in 3150 steps. The central conclusion is that constant angular velocity is not a free optimization benefit: Hyperball does not remove learning-rate scheduling complexity and may make schedule design more consequential.

Original abstract

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Read the original paper

More in Optimization

Browse all 36 papers →