💬 NLP

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

This paper introduces function-aware fill-in-the-middle (FIM) mid-training, a self-supervised approach that leverages the structural similarity between coding agent loops and function calls to significantly boost coding agent performance on benchmarks like SWE-Bench while simultaneously preserving general coding and tool-use capabilities often degraded by standard post-training.

Original authors: Yubo Wang, Jiarong Liang, Yuxuan Zhang, Xuye Liu, Cong Wei, Yuyu Zhang, Ping Nie, Wenhu Chen

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yubo Wang, Jiarong Liang, Yuxuan Zhang, Xuye Liu, Cong Wei, Yuyu Zhang, Ping Nie, Wenhu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're teaching a robot to fix bugs in a giant, messy library of code. Usually, we teach these robots by letting them read books from start to finish, left to right. But there's a problem: when a robot actually goes to fix a bug, it doesn't just read forward. It takes a step, looks at what happened (maybe a tool gave an error), and then decides what to do next. It's a loop: Action → Observation → Next Step.

The authors of this paper noticed something cool: this robot loop looks exactly like a function call in normal computer code.

The Robot's Action is like a function calling another function.
The Robot's Observation is like that function returning a value.
The Robot's Next Step is like the rest of the code using that value.

The big idea? Instead of just reading code forward, the authors taught their robot models to play a "fill-in-the-blank" game with these function calls. They call this Function-Aware Fill-in-the-Middle (FIM) Mid-Training.

The Game: "Guess the Missing Piece"

Imagine you have a story where a character (the "caller") asks a friend (the "callee") to do a task, but the friend's actual work is hidden. The robot has to guess what the friend did and explain why they did it, based only on the context before and after.

To make this game effective, the authors didn't just pick random words to hide. They used a special map of the code (a "Program Dependency Graph") to find the perfect puzzles. They looked for functions that were:

Complex enough to be a good challenge (not too easy, not impossible).
Predictable enough that the robot could actually figure them out from the surrounding clues.

They also made the robot write a "reasoning note" (a Chain-of-Thought) before writing the missing code, forcing it to think before it acts, just like a human programmer would.

The Results: Does It Work?

The authors tested this on three different robot brains (models): two from the Qwen2.5-Coder family (7 billion and 14 billion parameters) and one from the newer Qwen3 family (8 billion parameters). They trained these robots on a massive dataset of 2.6 billion tokens (words and symbols) taken from 968 real-world Python code repositories.

Here is what they found:

Better at Fixing Bugs: When tested on a famous benchmark called SWE-Bench-Verified (which simulates real-world software issues), the robots improved significantly.
- The 7B model improved by +2.8%.
- The 14B model improved by +3.0%.
- The 8B model improved by +3.2%.
Even Better on Easier Tasks: On a lighter version of the test (SWE-Bench-Lite), the gains were even higher, with the 8B model jumping +5.4%.
It Works Everywhere: These improvements happened even when the robots were trained using different methods afterward. This suggests the "fill-in-the-middle" training gave them a solid foundation that stuck, no matter how they were polished later.

The Surprise: It Didn't Break Other Skills

Usually, when you train a robot to be a super-specialist (like a bug-fixing agent), it forgets how to do other things, like writing simple code or using tools. The authors were worried this would happen.

But, the "mid-training" actually saved the robots' other skills!

Without this extra training, the robots got worse at writing code on their own (dropping on benchmarks like LiveCodeBench).
With the FIM training, they didn't just stay the same; they got better (+11.1% on LiveCodeBench).
Even though the training data was only Python code, the robots got better at using tools in completely different contexts (like τ-bench and BFCL), suggesting they learned a general "how to think about cause-and-effect" skill that transferred across tasks.

What They Explicitly Ruled Out

The authors were careful to say what this method is not:

It's not just about copying a smart teacher: They tested if the robot was just memorizing the answers from a powerful AI (Gemini-3-Flash) that helped generate the training data. They found that even when the robot generated its own reasoning notes, it still improved. So, the magic isn't just "copying the teacher"; it's the structure of the game itself.
It's not a magic bullet for everything: The method works best when the code is modular (broken into neat functions). If the code is a giant, messy script where everything is mixed together, the method doesn't work as well because it can't find clear "function call" puzzles to solve.
It's not proven for all languages yet: The training data was only Python. While the robots got better at other tasks, the authors admit they haven't proven this works for Java, C++, or Rust yet.

How Sure Are They?

The authors are quite confident in their numbers because they ran the tests three times with different random seeds to make sure the results weren't just luck. They measured the improvements precisely (e.g., +2.8%, +3.0%) and showed that the robots consistently solved more problems and made fewer mistakes when they got stuck.

They suggest that this "mid-training" stage is a crucial missing link. It's like giving the robot a specific type of gym workout before sending it into the real world, ensuring it has the right muscles to handle the complex loop of "act, observe, and continue" without losing its other abilities.

Technical Summary: Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Problem Statement

Coding agents designed to resolve real-world software engineering issues rely on a specific conditioning structure: maintaining a history ( $h_t$ ), sampling an action ( $a_t$ ), receiving an external observation ( $o_{t+1}$ ), and continuing reasoning based on the full trace. While standard code pretraining (left-to-right next-token prediction) exposes models to code, it rarely optimizes for this specific "action $\to$ observation $\to$ continuation" loop. Consequently, a gap exists between the base model's pretraining and the demands of agentic post-training pipelines (e.g., SWE-Gym, R2E-Gym).

Furthermore, standard Fill-in-the-Middle (FIM) objectives used in code pretraining are misaligned with agent reasoning for three reasons:

Arbitrary Boundaries: Random-span masking often cuts through expressions, failing to capture function-level dependencies.
Lack of Reasoning: Standard FIM requires direct generation without intermediate reasoning steps (Chain-of-Thought), unlike the "think-then-act" pattern of agents.
Amortized Signal: When FIM is dissolved into massive pretraining, the structural prior is diluted across trillions of unrelated tokens before agentic post-training begins.

Methodology

The authors propose Function-Aware FIM Mid-Training, a dedicated self-supervised stage inserted between pretraining and agentic post-training. This approach leverages the structural isomorphism between a coding agent's step and a function call site in ordinary code:

Context/History: Pre-call code establishing intent.
Action/Call: The function call itself.
Observation/Return: The value computed by the callee (outside the immediate scope).
Continuation: Downstream code consuming the return value.

Core Components

Function-Aware Target Selection:
Instead of random spans, the method selects masking targets at the function granularity using Program Dependency Graph (PDG) analysis.
- Complexity Score ( $\hat{H}$ ): Measures intrinsic difficulty based on Lines of Code (LoC), McCabe Cyclomatic Complexity, and nesting depth.
- Inferability Score ( $\hat{I}$ ): Measures how recoverable a function is from its context, aggregating signals from caller arguments, callee dependencies, type signatures, docstrings, and class siblings.
- Selection Criterion: Targets are selected via a harmonic-mean-like combination of $\hat{H}$ and $\hat{I}$ , penalizing targets that are either too trivial or unlearnable even with full context.
Chain-of-Thought (CoT) Augmentation:
For selected targets, the model is trained to generate a step-by-step rationale before the function body.
- Generation: A frontier model (Gemini-3-Flash) generates the rationale and a candidate body based only on the masked file context (no ground truth access).
- Filtering: A judge model filters pairs based on feasibility and quality.
- Training Format: The training prompt follows <fim_prefix> ... <fim_suffix> ... <fim_middle> <rationale> <body>. This forces the model to learn the "think-then-act" structure.
Data Curation:
The corpus consists of 968 decontaminated Python repositories from GitHub (2.6B tokens). Repositories overlapping with SWE-Bench sources are removed, and commits are restricted to pre-date SWE-Bench base commits to prevent test-time leakage.

Key Contributions

Structural Analogy: Framing function call sites as the internet-scale analogue of the agent action–observation–continuation loop, motivating function-granularity FIM as a self-supervised prior.
Pipeline Design: A mid-training pipeline combining PDG-based selection, a complexity–inferability double criterion, and embedded CoT rationales.
Robustness Validation: Demonstrating consistent gains across different model sizes (Qwen2.5-Coder 7B/14B, Qwen3-8B), different post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego), and different base model families.
Cross-Domain Transfer: Showing that a Python-only mid-training corpus improves performance on non-coding tool-use benchmarks ( $\tau$ -bench, BFCL) and preserves general coding capabilities (LiveCodeBench) that typically erode during agentic post-training.
Open Release: Releasing the 968-repository decontaminated corpus, the selection pipeline, and mid-trained checkpoints.

Experimental Results

In-Domain Agent Performance

The method was evaluated on SWE-Bench-Verified and SWE-Bench-Lite.

Qwen2.5-Coder-7B: +2.8% gain on Verified, +3.7% on Lite (vs. post-training only).
Qwen2.5-Coder-14B: +3.0% gain on Verified, +4.0% on Lite.
Qwen3-8B: +3.2% gain on Verified, +5.4% on Lite.
Pipeline Independence: Gains were observed across R2E-Gym, SWE-Smith, and SWE-Lego pipelines, indicating the prior is not specific to a single data distribution.

Capability Preservation and Cross-Domain Transfer

Agentic post-training alone was found to significantly degrade non-target capabilities (e.g., LiveCodeBench dropped by 13.1 points for the 14B model).

Restoration: Adding mid-training before post-training restored LiveCodeBench performance by +11.1 points, nearly closing the gap to the instruction-tuned ceiling.
Tool Use: Despite the corpus containing only Python code and no tool-use trajectories, mid-training improved performance on $\tau$ -bench (+3.9) and BFCL (+2.4). This serves as evidence that the function-call inductive bias transfers to general tool-use reasoning.

Behavioral Analysis

Analysis of agent trajectories revealed:

Recovery from Errors: The mid-trained agent recovered from negative observations (e.g., stack traces) at a rate of 28.8% compared to 24.8% for the baseline.
Iterate-and-Verify: The model adopted a policy of more edits and longer trajectories (23.6 steps vs. 15.1), reducing "no-patch" failures by an order of magnitude.
Granularity: Gains were most pronounced on tasks requiring reasoning about cross-function dependencies within a single file (+9.1 points on multi-function tasks), confirming the structural isomorphism hypothesis.

Significance and Claims

The paper claims that source code contains an "internet-scale supply of agent-relevant signal" in the form of function call sites. By explicitly training models to reason bidirectionally about these dependencies via function-aware FIM, the method installs a structural prior that:

Improves Agent Performance: Directly boosts success rates on software engineering benchmarks.
Mitigates Catastrophic Forgetting: Prevents the erosion of general coding and tool-use capabilities often caused by specialized agentic post-training.
Generalizes: The benefits transfer across model families and to non-coding tool-use tasks, suggesting the learned inductive bias is fundamental to the agent reasoning loop rather than specific to Python code generation.

The authors note limitations, including the reliance on a Python-only corpus, dependency on a frontier teacher (Gemini-3-Flash) for CoT generation (though self-CoT recovers most gains), and the assumption of modular code structures.