LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
LEGO-RL is a framework that enables scalable reinforcement learning for coding agents by bridging native execution harnesses with policy-gradient optimization through in-process LLM proxying, robust sandbox orchestration, and integrated monitoring, thereby significantly improving model performance on SWE-bench Verified across diverse environments while maintaining high training-inference alignment.
Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, researchers are teaching computers to act as autonomous software engineers. These digital agents do not simply answer questions; they navigate complex codebases, use tools to fix bugs, and run tests to see if their changes work. To learn these skills, the agents use a method called reinforcement learning. Imagine a student learning to solve a puzzle: they try a move, and if it brings them closer to the solution, they get a small reward. Over time, they learn which moves lead to success. For coding agents, this process involves generating long sequences of actions, watching a computer program run, and receiving a simple "yes" or "no" signal indicating whether the code they wrote actually fixed the problem. The challenge lies in the fact that these agents operate in messy, real-world computing environments where things can go wrong in unpredictable ways. If the environment crashes, if the reward system is tricked, or if the computer recording the agent's thoughts loses track of what was actually said, the learning process breaks down. The agent might learn to exploit the system or simply stop learning because the feedback it receives is unreliable.
A team of researchers has developed a new system called LEGO-RL to solve these specific problems. Their goal was to connect existing, complex coding agents to powerful learning algorithms without forcing the agents to change how they work. Think of the agent as a skilled worker who knows exactly how to use a specific set of tools and follow a specific workflow. Previous attempts to train these workers often required rebuilding their entire workspace to fit the training software, which frequently caused errors or changed the worker's natural behavior. The LEGO-RL team instead built a bridge that lets the training software observe the worker exactly as they are, capturing every move and thought in real time while protecting the learning process from the chaos of the real world.
The core of their solution is a framework that ensures the training computer sees exactly what the agent sees and does. When an agent generates a response, a special component captures the raw stream of words and the exact probability the agent assigned to each word before any other software could alter or summarize it. This is crucial because the software that manages the agent's environment often rewrites history or compresses information to save space. If the training system relied on these rewritten versions, it would calculate the agent's learning progress based on a distorted memory of events. By capturing the data at the moment of generation, the researchers ensured that the learning signal remained faithful to the agent's actual decisions. Furthermore, they built a system to replay the specific internal choices the agent made, ensuring that even complex models with multiple specialized parts learned from the correct path.
To keep the learning process reliable, the team created a highly organized environment where each test runs in its own isolated, secure container. This prevents one broken test from crashing the entire system and stops agents from finding shortcuts to bypass the scoring system. For instance, they hid the answers from the agents during the test and ensured that the software used to grade the work could not be tampered with. They also designed the system to handle failures gracefully. If an agent gets stuck or the environment crashes, the system identifies the problem, discards that specific attempt, and moves on without letting the error corrupt the learning data. This allows the training to continue smoothly even when individual tests fail, which happens frequently in complex coding tasks.
The researchers tested this system by training a large language model using three different, existing coding agent frameworks. They found that the system worked remarkably well across all three. The model's ability to solve real-world software problems improved significantly. When using one popular framework, the success rate jumped from 64 percent to over 70 percent. With another, it rose from 62 percent to nearly 68 percent, and with a third, it climbed from 57 percent to nearly 67 percent. Crucially, the researchers verified that the learning process was honest; the mathematical relationship between what the agent did and what the training system calculated remained almost perfect, with a correlation above 99 percent. This proved that the system was not just getting lucky, but was genuinely learning from the correct data.
Beyond just improving scores, the system provided a clear window into how the agents learned. The researchers could watch the training in real time, seeing exactly why a test failed or how the agent's behavior changed over weeks of training. They observed that as the agents improved, they began to check their own work more often, reading files they had just edited to ensure their changes were correct. They also noticed that the agents started to take more steps to solve problems, engaging in longer and more complex conversations with the computer to reach a solution. This level of detail allowed the researchers to diagnose issues quickly, such as when an agent stopped using tools and started writing text instead, and to adjust the training accordingly.
The success of LEGO-RL demonstrates that scaling up the training of coding agents requires more than just faster computers or bigger models. It demands a system that respects the integrity of the agent's workflow while providing a stable, observable environment for learning. By building a framework that captures data faithfully, protects against errors, and offers deep visibility into the training process, the researchers have shown that it is possible to teach complex software agents to improve reliably. Their work suggests that the future of autonomous coding lies not in forcing agents into rigid molds, but in building flexible, robust systems that can learn from the messy reality of software development.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.