From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

This paper, from LARK at HKUST(GZ) and the University of Cambridge, reframes RL training as a closed-loop environment-design problem: after each stage, the current Qwen3-4B checkpoint becomes an “LLM-as-Environment-Engineer” that reads structured failure breakdowns, validation history, and training context, then rewrites the next environment generator instead of selecting examples. The authors introduce MAPF-FrozenLake, a controllable multi-agent path-finding testbed with 3×3 to 10×10 grids and generator knobs for data_ratio, hole_ratio, and wait_ratio, trained with GRPO under an adaptive reward that combines strict path-validity checks and a length penalty. On 3-, 4-, and 5-agent generalization benchmarks, the Qwen3-4B + GRPO + Ours system achieves the strongest aggregate performance, reaching 51.67 valid rate and 31.67 optimal rate on 3-agent tasks, 33.14/21.33 on 4-agent tasks, and 18.67/11.00 on 5-agent tasks, while outperforming larger proprietary LLMs including GPT-5.4, Grok-4.2, Gemini-3.1-Pro, and Kimi-K2.5 as environment designers. The analysis shows the key mechanism is evidence-driven adaptation: successful designs preserve what already works, target specific failure modes, and concentrate training near the competence frontier rather than monotonically increasing difficulty. An ablation further shows that “bookkeeping-only” training details outperform full RL-detail prompts, and that the current RL checkpoint is a better environment engineer than the untrained base model, indicating that policy learning improves self-diagnosis.