LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

NVIDIA’s LoGRA reduces the memory burden of reinforcement-learning post-training by accumulating low-rank gradient sketches directly during backpropagation, rather than storing full gradients. For each selected attention or MLP matrix, it stores a compact projected gradient and applies the resulting low-rank update; the same factors also synchronize the updated policy with the rollout engine. A predicted-KL controller estimates how much an update will change next-token probabilities and scales the step to respect a chosen policy-change budget. On Qwen models evaluated with DAPO-Math-7.5K and MATH-500, LoGRA cuts average training memory by 21.8% at 1.5B and 45.7% at 7B, while retaining competitive reasoning performance. It also makes 27B-model RL training practical: Qwen3.8-27B trained for over 1100 steps on a single eight-GPU node, where dense Adam ran out of memory, and its Reasoning-Gym Hard held-out score rose from 39.69% to a peak of 65.52%. The key contribution is combining compressed gradients, compact policy synchronization, and KL-based step control to make larger-scale RL training feasible without relying on adapter-only updates.