NTH

InfraRSI: Harnessing Recursive Self-Improvement for Training Infrastructure

AuthorsInfraRSI Team, Yuchuan Tian, Haocheng Mei, Ye Yuan, Chao Xu, Boxun Li, Guohao Dai, Kai Han, Allen Chen, Yunhe Wang, Yu Wang

November 12, 2026 2 min read
Watch on YouTube
The one-line take

InfraRSI uses an agentic search process to make distributed LLM training faster and more memory-efficient.

Key results

1.83
Geometric-mean speedup

Across four dense and MoE training workloads.

17.4%
Baseline mean MFU

Mean Model FLOPs Utilization before optimization.

30.0%
Optimized mean MFU

Mean Model FLOPs Utilization after optimization.

2.75
Peak workload speedup

Gemma-4-26B-A4B MoE on 8×H200.

1.82
Cross-SKU final speedup

Qwen3.5-9B with cross-hardware experience transfer.

9
Cross-SKU convergence trial

Trial at which transfer reached the 90% peak log-speedup threshold.

What the paper found

InfraRSI turns natural-language requests into verified distributed-training optimizations through a five-stage workflow: define fixed workload and timing rules, profile the system, search, report, and apply validated changes. Its central innovation is compositional search across both launch settings and source-code patches, guided by a global Bayesian execution-time model and a censored Tobit memory model that learns from OOM failures. This lets the system pair memory-intensive changes with memory-saving settings, while a 16-invariant verifier checks workload, numerical fidelity, timing, and reproducibility. Across four workloads using Qwen3.5, Qwen3.8, and Gemma-4, InfraRSI achieved a 1.83× geometric-mean speedup and raised mean Model FLOPs Utilization from 17.4% to 30.0%; Gemma-4-26B-A4B MoE reached 2.75× by combining torch.compile, FlexAttention, and memory-saving changes. A second self-improvement loop stores verified, hardware-aware experience for later searches: cross-SKU transfer on Qwen3.5-9B reached 1.82× and crossed its convergence threshold by Trial 9. The results show how online calibration, recovery from memory and compiler failures, and reusable cross-hardware evidence can improve training throughput without changing the defined training task.

Original abstract

While Recursive Self-Improvement (RSI) accelerates isolated algorithm and kernel synthesis, distributed LLM training depends on tightly coupled launch settings, operator kernels, and communication schedules, where single edits often hit out-of-memory (OOM) boundaries, compiler conflicts, measurement jitter, or silent numerical drift. We present InfraRSI, the agentic harness that turns open-ended human commands into verified training optimizations via a five-stage workflow: init locks editable scope and a fixed task contract over workload, loss, and timing invariants to prevent reward hacking, doctor profiles baseline phases and calibrates cluster noise, run co-optimizes launch knobs and code patches before re-verifying top candidates against the baseline and logging findings to a cross-hardware experience bank, and report and apply audit source paths, track failed-trial costs, and commit promoted changes. Rather than stepping greedily from a single incumbent, InfraRSI initializes from diverse experience-seeded configurations and fits global execution-time and censored peak-memory models across all trials, including OOMs. Using these models, it prices per-setting memory costs, revises miscalibrated LLM priors from live runs, and pairs memory-hungry optimizations with memory savers and verified code patches. Across four single- and multi-node dense and Mixture-of-Experts (MoE) LLM workloads, InfraRSI achieves a speedup of up to $2.75\times$ and lifts mean Model FLOPs Utilization (MFU) from $17.4\%$ to $30.0\%$, outperforming existing tuners and coding agents while transferring experience across GPUs. As it improves training infrastructure efficiency, InfraRSI forms a two-level Recursive Self-Improvement loop---revising priors online within runs and distilling experience across clusters---to push foundation model scaling.

Read the original paper

More in Efficient AI

Browse all 79 papers →
01Efficiency

Latent Feedback Is a Compute and Data Amplifier

Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu

Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.

Read analysis