💬 NLP

Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

This paper introduces Dialogue SWE-Bench, a novel benchmark and persona-grounded user simulator for evaluating coding agents' ability to resolve software engineering problems through dialogue, revealing that dialogue capability is a distinct dimension of performance that does not necessarily correlate with coding model strength.

Original authors: Brendan King, Jeffrey Flanigan

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brendan King, Jeffrey Flanigan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fix a broken toaster.

The Old Way (Current Benchmarks):
Right now, when we test AI coding assistants, we act like a robot giving a robot a perfect, 10-page manual. The manual says: "The toaster is broken because the heating element is loose. Please tighten it." The AI reads the manual, fixes the toaster, and we give it a gold star.

The problem? Real humans don't talk like that. We don't have perfect manuals. We say, "Hey, my toaster is acting weird," and the AI has to ask, "Is it making a noise? Is it smoking? What happens when you push the lever?"

The New Way (Dialogue-SWEBench):
This paper introduces a new testing ground called Dialogue-SWEBench. Instead of giving the AI a perfect manual, they set up a simulation where the AI has to talk to a "ghost user" to solve a real software problem.

Here is how they built it:

1. The "Ghost User" Simulator

The researchers created a special AI program to play the role of the human user.

The Persona: This ghost user has a personality (like "anxious and cautious" or "lazy and impatient").
The Knowledge: The ghost knows the whole story of the bug (like a secret script), but it is strictly forbidden from telling the AI everything at once. It must reveal details only when asked.
The Self-Correction: If the ghost user accidentally says something impossible (like "I just ran the code on my phone"), the simulator catches itself, says "Oops, I can't do that," and rewrites the message to be realistic. This ensures the test is fair.

2. The Test Run

In this new test, the AI coding agent can't just look at a file and fix it. It has to:

Listen to the user's vague complaint ("My code is crashing!").
Ask clarifying questions ("What error message do you see?").
Get the answer.
Fix the code.
Verify the fix with the user.

If the AI tries to guess the answer without asking, or if it asks too many silly questions, it fails.

3. The Big Surprise

The researchers tested the smartest coding AIs available (like GPT-5 and others) in this new "chat" environment. They found something surprising:

Being a great coder doesn't automatically make you a great conversationalist.

The "Big Brain" Trap: The largest, most powerful models (the "Big Brains") were actually worse at the conversation part. They tended to ask too many unnecessary questions or get confused by the chat flow.
The "Small but Smart" Winner: A slightly smaller model (GPT-5-mini) did a better job at the conversation, solving more problems for less money.
The Secret Sauce: The researchers built a new type of agent that uses a "Schema" (think of it like a checklist or a mental whiteboard). As the AI chats, it fills in boxes on its checklist: What is the bug? What is the expected result? What is missing?
- This "Schema-Guided" agent was the best at solving problems because it stayed organized during the conversation, rather than just guessing.

4. The Verdict

The paper concludes that we have been testing coding agents like they are robots working alone in a lab. But in the real world, they are teammates working with humans.

Coding Skill ≠ Chat Skill: A model can be brilliant at writing code but terrible at figuring out what the human actually needs.
The Solution: To make better coding assistants, we need to train them to be better listeners and question-askers, not just better coders.

In short: The paper built a "chat simulator" to test if coding AIs can actually talk to humans to fix bugs. They found that the biggest, most expensive models aren't always the best conversationalists, and that giving the AI a mental checklist helps it solve problems much better.

Technical Summary: Dialogue-SWEBench

Problem Statement

Current benchmarks for coding agents, such as SWE-Bench, evaluate systems in a fully-autonomous setting where agents receive a complete and correct problem specification at the outset. This approach fails to capture the interactive nature of real-world software engineering, where users frequently engage in dialogue to correct, reject, or clarify agent outputs. Studies indicate that users correct agent outputs 44% of the time, yet agents rarely seek clarification (1-2%). Furthermore, existing benchmarks often presume that a perfect specification is available, whereas real-world issues are frequently under-specified or vague. Consequently, there is a significant gap between current evaluation metrics and the actual capabilities required for interactive, dialogue-driven coding agents.

Methodology

Dialogue-SWEBench Benchmark

The authors introduce Dialogue-SWEBench, a benchmark designed to evaluate coding agents' ability to resolve real-world software engineering (SWE) problems strictly through dialogue with a user. The benchmark is constructed by reformulating tasks from SWE-Bench Verified:

Input Transformation: Instead of providing the full issue text as a static specification, the task begins with a fixed, initial user query ( $u_1$ ) that reflects the user's intent but omits critical details necessary for resolution.
Interactive Loop: Agents operate in an environment where they can perform standard coding actions (editing files, running tests) and a new message_user action. This action allows the agent to query the user, receiving a simulated response as an observation.
Termination: The loop ends when the agent issues a finish action, submitting a git patch. Success is determined by whether the patch passes all associated execution tests.

User Simulator

To ensure faithful and reproducible evaluation, the authors designed a persona-grounded user simulator using a quantized LLaMa 3.3 70B model.

Knowledge Grounding: The simulator is conditioned on the complete original GitHub issue text ( $I_{text}$ ) but is restricted from accessing the code repository or running code itself. It must act as a user who knows the problem but cannot implement the solution.
Self-Revision Mechanism: To prevent hallucinations and maintain environmental constraints (e.g., the user cannot run code), the simulator employs a self-revision step. A candidate user reply is generated, validated by the same LLM for violations (such as claiming to run code or breaking character), and revised if necessary before being returned to the agent.
Personas: To increase dialogue diversity, each problem is assigned a specific user persona (e.g., "cautious," "perfectionist," "pragmatic") that influences the tone and behavior of the simulated responses.

Schema-Guided Agent

The authors propose a novel schema-guided coding agent to improve dialogue performance. This agent uses prompts to maintain a structured internal representation (a "schema") of the dialogue state.

Workflow: The agent first determines the issue type (e.g., bug, feature) and drafts a schema with keys for critical details (e.g., "expected behavior," "reproduction steps").
State Management: Values are marked as UNKNOWN until discussed. The agent is instructed to fill these gaps through dialogue before proceeding to code exploration and patch generation. This structure is designed to guide the agent in asking targeted, information-seeking questions rather than making assumptions.

Evaluation Metrics

Resolution Rate: The primary metric is the percentage of dialogues resulting in a patch that passes all execution tests.
Dialogue Quality: The authors introduce automatic evaluations for Naturalness (ease of understanding) and Coherence (logical progression toward the solution) using an LLM-as-a-Judge (Gemma 4 31B-IT), validated against human annotations.

Key Contributions

Dialogue-SWEBench: A benchmark dataset transforming 500 real-world SWE problems into interactive, dialogue-driven tasks where agents must resolve issues without a complete initial specification.
User Simulator: A novel, open-source, persona-grounded simulator with a self-revision mechanism to produce realistic, constraint-compliant user responses for online evaluation.
Schema-Guided Agent: A new agent architecture that utilizes dialogue schemas to explicitly track missing information, resulting in improved performance over strong baselines.
Automatic Dialogue Evaluation: An LLM-as-a-Judge framework for assessing the naturalness and coherence of coding agent dialogues.

Results

The authors evaluated state-of-the-art closed (GPT-5, GPT-5-mini) and open-weight (Qwen 3 Coder, Devstral 2 Small) models using three agent configurations: OpenHands (off-the-shelf), OH Interactive (baseline for ambiguity resolution), and their proposed Schema-Guided agent.

Performance: The schema-guided agent achieved the highest average resolution rate (46.9%) across all models, outperforming OpenHands (32.9%) and OH Interactive (44.1%). It achieved this at the lowest average cost.
Dialogue vs. Coding Ability: The results indicate that stronger coding models do not necessarily translate to better dialogue agents. For instance, GPT-5-mini often matched or outperformed the larger GPT-5 in resolution rates while incurring a fraction of the cost and producing more natural dialogues.
Information Seeking: There is a strong correlation between the number of information-seeking dialogue moves and resolution rates. The schema-guided agent consistently utilized more information-seeking moves than baselines, which was critical for success.
Ablation Studies: Removing the user simulator (forcing single-turn resolution) drastically reduced performance (e.g., GPT-5-mini dropped from 68.0% to 40.0%), confirming that multi-turn dialogue is essential for solving these tasks.
Quality Analysis: While GPT-5 achieved high resolution on difficult tasks, it suffered from lower naturalness scores due to verbosity and repetition. The schema-guided approach generally improved coherence.

Significance

The paper argues that dialogue capability is a distinct and currently understudied dimension of coding agent performance. The findings suggest that simply scaling up coding models or providing better specifications is insufficient for real-world deployment. Instead, the ability to engage in multi-turn, information-seeking dialogue is a critical factor for resolving real-world software engineering problems. The authors posit that future research must address the intersection of coding and dialogue capabilities to build agents that can effectively collaborate with human users, rather than operating as fully autonomous, specification-dependent systems.