SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
The paper introduces SWE-Interact, a new benchmark that evaluates coding agents on realistic, multi-turn, user-driven software engineering tasks, revealing that strong performance on traditional single-turn benchmarks does not reliably translate to the interactive, iterative workflows required for effective real-world development.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a brilliant but inexperienced architect to build a house.
The Old Way (Traditional Benchmarks)
In the past, researchers tested these AI "architects" by handing them a complete, 50-page blueprint on day one. The blueprint listed every single detail: the type of wood, the exact shade of paint, the placement of every outlet, and the brand of the faucet. The AI's job was simply to read the plan and build the house perfectly.
- The Problem: This doesn't reflect how real construction works. In the real world, clients rarely have a perfect blueprint ready. They usually say, "I want a cozy house with a big kitchen," and then change their minds as they see the progress.
The New Way (SWE-INTERACT)
The paper introduces SWE-INTERACT, a new test that simulates a real-life construction project.
- The Setup: Instead of a perfect blueprint, the AI starts with a vague request: "Build me a house."
- The Client (The Simulator): A computer program acts as the "client." It doesn't just sit back; it actively walks around the construction site.
- It looks at the foundation and says, "Hey, I didn't realize I wanted a basement."
- It looks at the kitchen and says, "Actually, I want the stove here, not there."
- It checks the wiring and says, "Wait, I need more outlets."
- The Goal: The AI has to figure out what the client actually wants by listening to these evolving comments, fixing its mistakes, and building the house piece by piece, all while the client keeps adding new requirements.
The Big Discovery
The researchers tested the world's smartest AI coding models in both scenarios. Here is what they found:
- Good at Reading, Bad at Listening: The models that were great at building houses from perfect blueprints (the "Single-Turn" test) struggled significantly when the client kept changing their mind. Their success rate dropped from about 50% to 25%.
- The "Over-Confident" Mistake: The best models (like GPT-5.5 and Opus 4.8) were brave enough to start building even with vague instructions. They could recover and fix things when the client corrected them. However, they often forgot earlier instructions or made technical errors while trying to be too clever.
- The "Give-Up" Mistake: Weaker models got confused by the vague start, gave up too early, or ignored the client's new instructions entirely.
Why This Matters
Think of it like a video game. The old tests were like a level where you know exactly where the enemies are and what the boss looks like. The new test is like a live-action role-playing game where the "Game Master" (the client) changes the story as you play.
The paper concludes that being a good coder isn't just about knowing how to write code (the blueprint); it's about having the patience and skill to work with a human who doesn't know exactly what they want until they see it. The current AI models are getting better at this, but they still have a long way to go before they can truly replace a human developer in a real-world, messy, collaborative environment.
In short: The paper built a new "gym" to train AI coders not just to follow orders, but to actually collaborate with humans who are figuring things out as they go.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.