Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

Imagine a world where software is like a giant, bustling city built by millions of people. Sometimes, a building has a hidden crack, a bridge has a loose bolt, or a traffic light is stuck on green when it should be red. In the real world, we usually wait for someone to shout, "Hey, the bridge is broken!" before we send a repair crew. In the digital world, this "shout" is called an issue report. It's a detailed note from a human describing exactly where the bug is, what error message popped up, and how the software should behave.

For a long time, computer scientists have been training super-smart AI assistants, known as coding agents, to act as repair crews. These agents use powerful brain-like models (called Large Language Models) to read code and fix problems. But here's the catch: most of the training and testing these agents have done so far assumes that a perfect, detailed "shout" (issue report) is always waiting for them. It's like training a mechanic only on cars that have already been towed to the shop with a note saying, "The engine is making a knocking sound."

But in the real world, things don't always go that smoothly. Sometimes bugs are so sneaky they slip into the final product without anyone noticing until it's too late, causing massive chaos. Other times, the notes left by humans are vague, confusing, or missing crucial details. This raises a big, exciting question: Can these AI repair crews find and fix the broken parts of the city before anyone even knows they are broken? Can they act like proactive detectives, scanning the code on their own, spotting the cracks, and fixing them without a single instruction manual?

This is exactly what the paper "Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports" sets out to explore. The authors, a team of researchers from universities and independent labs, introduce a new challenge called Active-SWE. Think of this as a giant, high-stakes video game level designed to test if AI agents can be true detectives rather than just obedient mechanics.

Instead of giving the AI a list of "broken things" to fix, the researchers handed them a massive library of code—1,663 different tasks across six types of bugs and eight different programming languages—and told them, "Go find the problems and fix them. Good luck, no hints allowed."

The researchers built this challenge in two ways. First, they created a "Simple Setting" where the AI had to find and fix a single bug hidden in a codebase, much like finding a specific typo in a novel. Then, they cranked up the difficulty with a "Hard Setting," where the AI had to hunt down multiple bugs at once, like a detective trying to solve a whole string of crimes in one night. They also tested the AI on "Potential Bugs"—scenarios where the AI might find a problem that wasn't even on the original list of known issues, essentially asking, "Did you find something new that is actually broken?"

To make sure the AI wasn't just guessing, the researchers used a clever "dual-track" scoring system. One track checked if the AI found the bugs they knew were there (Recorded Bugs). The other track was trickier: if the AI claimed to find a new bug, the system forced the AI to write a test to prove it. If the AI couldn't prove the bug existed with a test, the discovery didn't count. It's like a detective claiming they found a hidden safe; they have to show you the key and open it to prove it's real.

The results? It turns out that even the smartest, most advanced AI agents today are still struggling with this proactive detective work. When the researchers tested top-tier models (like Claude Opus 4.8, GPT-5.5, and others), they found that most of them could only fix about 20% of the known bugs when left without a guide. That's a big drop from when they are given a detailed issue report.

The study suggests that while these AI agents are getting better at following instructions, they are still quite bad at looking around a messy room and figuring out what's broken on their own. They often get lost in the code, can't pinpoint exactly where the problem is (a skill called "localization"), and struggle to fix multiple problems at once. Interestingly, the AI was slightly better at finding certain types of bugs, like those related to logic or data flow, but it was almost clueless when it came to bugs involving the "state" of the system (like a traffic light getting stuck).

Perhaps most importantly, the paper shows that finding the bug is only half the battle. The data suggests that if the AI can't accurately locate the broken part of the code, it almost never fixes it correctly. It's like trying to fix a leaky pipe without knowing which pipe is leaking; you might end up tightening the wrong one.

In short, this paper doesn't claim that AI has solved software repair. Instead, it sounds an alarm that the current generation of coding agents is still too reliant on humans to point out the problems. The "proactive" dream—where AI scans code and fixes issues before they cause a crash—is still a work in progress. The researchers conclude that while these agents are powerful, they need to get much better at being independent detectives before we can trust them to keep our digital cities safe without a safety net.