InfraRSI: Harnessing Recursive Self-Improvement for Training Infrastructure
AuthorsInfraRSI Team, Yuchuan Tian, Haocheng Mei, Ye Yuan, Chao Xu, Boxun Li, Guohao Dai, Kai Han, Allen Chen, Yunhe Wang, Yu Wang
InfraRSI uses an agentic search process to make distributed LLM training faster and more memory-efficient.
Key results
Across four dense and MoE training workloads.
Mean Model FLOPs Utilization before optimization.
Mean Model FLOPs Utilization after optimization.
Gemma-4-26B-A4B MoE on 8×H200.
Qwen3.5-9B with cross-hardware experience transfer.
Trial at which transfer reached the 90% peak log-speedup threshold.
What the paper found
InfraRSI turns natural-language requests into verified distributed-training optimizations through a five-stage workflow: define fixed workload and timing rules, profile the system, search, report, and apply validated changes. Its central innovation is compositional search across both launch settings and source-code patches, guided by a global Bayesian execution-time model and a censored Tobit memory model that learns from OOM failures. This lets the system pair memory-intensive changes with memory-saving settings, while a 16-invariant verifier checks workload, numerical fidelity, timing, and reproducibility. Across four workloads using Qwen3.5, Qwen3.8, and Gemma-4, InfraRSI achieved a 1.83× geometric-mean speedup and raised mean Model FLOPs Utilization from 17.4% to 30.0%; Gemma-4-26B-A4B MoE reached 2.75× by combining torch.compile, FlexAttention, and memory-saving changes. A second self-improvement loop stores verified, hardware-aware experience for later searches: cross-SKU transfer on Qwen3.5-9B reached 1.82× and crossed its convergence threshold by Trial 9. The results show how online calibration, recovery from memory and compiler failures, and reusable cross-hardware evidence can improve training throughput without changing the defined training task.
Original abstract
While Recursive Self-Improvement (RSI) accelerates isolated algorithm and kernel synthesis, distributed LLM training depends on tightly coupled launch settings, operator kernels, and communication schedules, where single edits often hit out-of-memory (OOM) boundaries, compiler conflicts, measurement jitter, or silent numerical drift. We present InfraRSI, the agentic harness that turns open-ended human commands into verified training optimizations via a five-stage workflow: init locks editable scope and a fixed task contract over workload, loss, and timing invariants to prevent reward hacking, doctor profiles baseline phases and calibrates cluster noise, run co-optimizes launch knobs and code patches before re-verifying top candidates against the baseline and logging findings to a cross-hardware experience bank, and report and apply audit source paths, track failed-trial costs, and commit promoted changes. Rather than stepping greedily from a single incumbent, InfraRSI initializes from diverse experience-seeded configurations and fits global execution-time and censored peak-memory models across all trials, including OOMs. Using these models, it prices per-setting memory costs, revises miscalibrated LLM priors from live runs, and pairs memory-hungry optimizations with memory savers and verified code patches. Across four single- and multi-node dense and Mixture-of-Experts (MoE) LLM workloads, InfraRSI achieves a speedup of up to $2.75\times$ and lifts mean Model FLOPs Utilization (MFU) from $17.4\%$ to $30.0\%$, outperforming existing tuners and coding agents while transferring experience across GPUs. As it improves training infrastructure efficiency, InfraRSI forms a two-level Recursive Self-Improvement loop---revising priors online within runs and distilling experience across clusters---to push foundation model scaling.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.