When Does Muon Help Agentic Reinforcement Learning?

This study from researchers at Renmin University of China, the Chinese Academy of Sciences, Duke University, and Zhejiang University tests whether Muon, a matrix-aware optimizer associated with large-scale training and supported in NVIDIA NeMo RL, transfers to agentic reinforcement learning. Using Qwen2.5-0.5B-Instruct on sparse-reward ALFWorld with 200 updates, the authors apply Muon only to hidden weight matrices and retain AdamW for embeddings and normalization parameters, comparing it with AdamW across GRPO, GiGPO, and GraphGPO, including GRPO’s DeepSeekMath lineage. Under GiGPO, Muon raises late-window validation success from 0.290 to 0.546, an 88% relative improvement, while matched high-rate AdamW controls collapse to zero post-update success. The effect is estimator- and learning-rate-dependent: under GRPO at 3e-5, Muon reaches 0.268 versus AdamW’s 0.161, whereas GraphGPO at 1e-5 reaches 0.901 and increases normalized validation AUC from 0.399 to 0.556, crossing 0.5 and 0.75 success 30 and 60 updates earlier. A GiGPO ablation shows Muon’s advantage persists without step-level credit, while adding step-level credit improves both optimizers. The proposed explanation is that Muon’s Newton–Schulz spectral whitening amplifies weak gradient directions only when their signs are reliable; finer-grained credit assignment may improve that signal-to-noise ratio in long-horizon tasks, unlike some negative single-turn RLVR results. The authors stress that this is exploratory evidence from one 0.5B model, one environment, and limited seeds.