Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

This paper, from researchers at the University of Waterloo, the University of British Columbia, NVIDIA, Verdent AI, and the Vector Institute, proposes function-aware fill-in-the-middle, or FIM, as a dedicated mid-training stage for coding agents. The key idea is that a function call mirrors an agent step: context leads to an action, an externally computed return, and a continuation. The method parses Python files into program dependency graphs, selects single functions or connected function groups using complexity and inferability scores, and trains models to reconstruct the masked code together with a reasoning trace. Using Gemini-3-Flash to generate and filter rationales, the team built a decontaminated corpus from 968 GitHub repositories containing 400K FIM samples and 2.6B tokens. They mid-trained Qwen2.5-Coder-Instruct 7B and 14B, plus Qwen3-8B, before applying R2E-Gym, SWE-Smith, or SWE-Lego post-training. On Qwen3-8B with SWE-Lego, FIM mid-training raised SWE-Bench-Verified by 3.2 points and SWE-Bench-Lite by 5.4 points; similar gains appeared across the Qwen2.5-Coder models and post-training pipelines. The intervention also countered capability erosion: on the 14B model, LiveCodeBench recovered 11.1 points after agentic post-training, while τ-bench and BFCL improved by 3.9 and 2.4 points. Behavioral analysis suggests the gain comes from better recovery after negative tool observations and more iterative editing, especially on patches modifying multiple functions within one file. The main limitations are Python-only data, reliance on Gemini-3-Flash for the strongest rationales, and limited validation beyond Qwen model families.