💬 NLP

Scalable Supervision for Software Agents via Patch Reasoning

This paper introduces R4P, a reasoning-based method that enables scalable supervision for software agents by providing scaffold-agnostic rewards through group-wise patch verification, thereby overcoming the limitations of test-based supervision and demonstrating significant performance improvements in training execution-free agents like Mini-SE.

Original authors: Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, sprawling landscape of modern software, code is the foundation upon which almost everything digital is built. When a program malfunctions or needs a new feature, human developers must diagnose the problem and write a precise fix, known as a patch. In recent years, artificial intelligence has begun to assist in this process, with large language models acting as digital assistants that can read code, understand errors, and propose solutions. However, teaching these AI assistants to become truly reliable has hit a wall. The traditional method for training them relies on running a battery of automated tests, much like a mechanic running a car through a diagnostic machine to see if a repair worked. While effective for small, controlled examples, this approach breaks down in the real world. Most software projects on the internet do not have these comprehensive test suites, and building the complex environments needed to run them for every single problem is slow, expensive, and prone to crashing. Without a way to verify fixes quickly and cheaply, the potential for AI to scale up and solve the millions of open software issues remains out of reach.

Researchers at The Chinese University of Hong Kong, Shenzhen, and ByteDance have proposed a new way forward that bypasses the need for these heavy, automated tests entirely. They introduced a method called R4P, which treats the verification of a software fix not as a mechanical check, but as a reasoning task. Instead of running code to see if it works, the system asks an AI to act like a senior engineer reviewing a colleague's work. It looks at a software problem and a group of different proposed solutions side by side. By comparing these solutions against one another, the AI can spot subtle mistakes or logical flaws that might be invisible if it were looking at a single patch in isolation. This approach allows the system to learn from a much wider variety of real-world problems, including those that have never been tested before, because it relies on the AI's ability to understand code logic rather than the existence of a pre-written test script.

The core of this innovation is a training technique where the AI evaluates multiple patches at once. Imagine a panel of judges reviewing several different entries for the same contest. If the judges look at each entry alone, they might struggle to decide which is best or might miss a small error. But when they look at all the entries together, they can see that one solution changes a variable in a way that contradicts the others, or that another solution handles a specific edge case that the rest ignore. The researchers found that this group-based comparison provides a much richer signal for learning than simply asking if a single patch is right or wrong. It helps the AI learn to distinguish between a solution that is merely plausible and one that is genuinely correct, even without the safety net of a test suite.

To prove this method works in practice, the team built a lightweight software agent called Mini-SE. This agent was designed to solve software issues without ever running a test or building a complex environment. It relies entirely on the R4P system to tell it if its proposed fixes are good. In experiments, Mini-SE was able to improve its performance significantly, solving about 26.2 percent of real-world software issues on its first try. This represented a 10.0 percentage point improvement over the original Qwen3-32B model. Furthermore, when the researchers used the R4P system to select the best solution from a larger pool of candidates generated by the agent, the success rate climbed to 32.8 percent. These results demonstrate that a system can learn to write better code by reasoning about the quality of its own work, rather than waiting for an external test to confirm it.

The study also highlighted that this method is incredibly fast. While running a traditional test suite for a single software issue can take minutes or even hours due to the time needed to set up the environment, the R4P system verifies a patch in less than a second on average. This speed, combined with the ability to work on issues that lack any tests at all, suggests a path toward scaling software engineering assistance to the vast majority of code on the internet that currently sits untouched by AI. The researchers noted, however, that this system is not a perfect replacement for human testing. It is a tool for supervision and learning, designed to unlock the potential of data that was previously too difficult to use. By shifting the burden of verification from heavy, automated execution to intelligent reasoning, the work offers a practical way to train AI agents to become more capable software engineers, even in the messy, untested reality of the open-source world.

Technical Summary: Scalable Supervision for Software Agents via Patch Reasoning

Problem Statement

The advancement of Large Language Model (LLM) agents in Software Engineering (SWE) is currently bottlenecked by the scalability of supervision. Existing approaches rely heavily on test-based verification (running unit tests in sandboxed environments like Docker) to provide rewards for Reinforcement Learning (RL) and Test-Time Scaling (TTS). The authors identify two critical limitations in this paradigm:

Low Test Coverage: In the wild, high-coverage tests are rare. The authors' analysis of open-source projects with over 500 stars reveals that only 28.11% of resolved issues have corresponding tests.
High Computational and Engineering Overhead: Building and maintaining test sandboxes for each issue instance is labor-intensive and unstable under parallel workloads, preventing the scaling of supervision to the vast amount of untested data available in the open-source community.

Consequently, the field lacks a scalable, model-based verifier that can judge patch correctness without executing tests or relying on specific agent trajectories.

Methodology: R4P and Mini-SE

To address these challenges, the paper introduces R4P (Reasoning-based Patch Verification) and a corresponding execution-free agent scaffold, Mini-SE.

R4P: Reasoning-based Patch Verification

R4P is a reasoning-based method designed to provide scaffold-agnostic rewards. It operates on the premise that patch verification is fundamentally a reasoning task, similar to how human maintainers review pull requests by analyzing code changes rather than writing new tests.

Group-wise Training Objective: Instead of verifying patches in isolation (point-wise), R4P processes a group of candidate patches ( $P = [p_1, ..., p_N]$ ) for a single issue simultaneously. It is trained to assess each patch by comparing it against others in the group. This allows the model to use the modification content of one patch as context to identify subtle errors or incompatibilities in another.
Dense Reward Signal: By treating verification as a group-level task, R4P transforms the sparse binary outcome space (Pass/Fail) into a denser signal based on group accuracy. The reward is calculated as the proportion of correctly identified patches within the group. This mitigates the risk of "reward hacking" where a model might guess correctly by chance, as coincidentally correct labels across a whole group are statistically less probable.
Scaffold-Agnostic: R4P does not require golden tests, developer patches, agent trajectories, or runtime sandboxes. It relies solely on the issue description and the patch content.

Mini-SE: Execution-Free Agent Scaffold

To demonstrate the practical utility of R4P, the authors designed Mini-SE, a lightweight agent scaffold that completely decouples patch generation from verification.

Architecture: Mini-SE follows a targeted "entity $\to$ $\to$ code $\to$ $\to$ patch" paradigm. It utilizes two primary tools:
- Code Search: Maps entity names to source code and file paths using a static code graph.
- Code Edit: Performs string replacement with syntax validation.
Execution-Free: Unlike existing agents (e.g., SWE-Agent, OpenHands) that rely on in-loop self-testing via sandbox execution, Mini-SE generates patches without any trial-and-error execution. All verification and reward signals are delegated to R4P.

Key Contributions

R4P Framework: A novel reasoning-based verification method that achieves high accuracy without test execution. It utilizes a group-wise training objective to leverage mutual context between patches, enabling dense supervision signals.
Mini-SE Agent: A fully execution-free agent scaffold trained via pure RL using R4P rewards. It proves that agents can be effectively trained and scaled without the overhead of sandbox environments.
Empirical Validation of Scalable Supervision: The paper demonstrates that despite R4P's supervision being imperfect (not 100% accurate compared to ground-truth tests), it provides stable training signals that prevent reward collapse and yield consistent performance gains.

Experimental Results

The authors evaluated R4P and Mini-SE on the SWE-bench-verified dataset and R2E-Gym.

Patch Verification Accuracy: Built upon a Qwen-2.5-Coder-Instruct-32B model, R4P achieved 72.2% accuracy in verifying patches. This performance is competitive with proprietary frontier models (e.g., Claude-4.5-Sonnet, GPT-5) and significantly outperforms existing trajectory verifiers and reward models, particularly in generalization across different agent scaffolds.
Agent Performance (Mini-SE):
- RL Training: Mini-SE, based on Qwen-3-32B and trained purely with R4P rewards, achieved a 26.2% Pass@1 on SWE-bench-verified, representing a 10.0% improvement over the base Qwen3-32B model. The training curves showed stable convergence, indicating that R4P's noisy signals did not lead to reward collapse.
- Test-Time Scaling (TTS): When R4P was used to select the best patch from a set of 16 candidates generated by Mini-SE, the resolution rate improved to 32.8%.
Efficiency: R4P verifies patches in approximately 1 second on average, significantly faster than the minute-level costs associated with building and running Docker-based test sandboxes.

Significance and Claims

The paper claims that R4P offers a practical and scalable alternative for supervising SWE agents in scenarios where well-maintained test data is unavailable.

Unlocking Data Scaling: The core hypothesis is that the benefits of scaling supervision to vast amounts of untested, real-world issues outweigh the losses from slightly noisy model-based signals. The stable scaling curves observed in Mini-SE support this, suggesting that reasoning-based supervision can unlock the potential of data scaling.
Decoupling Generation and Verification: By removing the dependency on execution environments, the approach simplifies the agent infrastructure, reduces engineering overhead, and accelerates the training and inference process.
Generalization: R4P's scaffold-agnostic nature allows it to generalize across different agents and interaction styles, unlike trajectory-based verifiers which are often tightly coupled to specific agent behaviors.

The authors acknowledge limitations, noting that R4P is not a replacement for high-quality tests and that its performance can vary based on patch diversity and repository characteristics. However, they position R4P as a critical step toward enabling continuous learning for software agents on the vast, untested majority of open-source issues.