🤖 AI

Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks

This paper introduces CLARITI, an 8B-parameter model trained via reward-driven reinforcement learning that optimizes clarification efficiency in software engineering by prioritizing task-relevant and user-answerable information, achieving GPT-5-level resolution rates with 41% fewer questions.

Original authors: Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (the AI agent) trying to cook a complex dish based on a recipe card handed to you by a customer (the user).

The Problem:
Often, the customer's recipe card is incomplete. They might say, "Make me a spicy pasta," but they forget to mention:

How spicy is "spicy"? (Is it a hint of heat or a fire hazard?)
Do they have a gluten allergy?
What kind of pasta do they have in their pantry?

If the chef guesses wrong, they might burn the kitchen (the software fails) or waste a lot of ingredients (computation time).

The Old Way:
Previously, AI chefs would either:

Guess wildly: "I'll just add a little chili!" (Often fails).
Ask everything at once: "What is your name? What is your blood type? What is the exact temperature of your oven? Do you like cilantro? What is the capital of Peru?" (This overwhelms the customer, who gets annoyed and stops talking).

The New Solution: "CLARITI" (The Smart Waiter)
This paper introduces a new AI system called CLARITI. Think of CLARITI not as a chef, but as a highly trained, super-smart waiter who knows exactly what to ask and how to ask it to get the best result without annoying the customer.

Here is how CLARITI works, broken down into three simple steps:

1. The "What Matters" Filter (Task Relevance)

The researchers realized that not all missing information is created equal.

The Analogy: If the customer forgot to say "I'm allergic to peanuts," that is Critical Information. If they forgot to say "I prefer a blue plate," that is Low Priority Information.
The Science: The team analyzed thousands of software bugs and used a mathematical tool (Shapley values) to figure out which missing details actually stop the AI from solving the problem. They found that concrete error messages (like "The pot exploded!") are way more important than vague descriptions (like "It didn't taste right").
CLARITI's Move: Instead of asking about the plate color, CLARITI immediately asks, "Did the pot explode? If so, show me the smoke!" It prioritizes the "peanut allergy" questions over the "blue plate" questions.

2. The "Can You Answer?" Filter (User Answerability)

Asking the right question is useless if the customer doesn't know the answer.

The Analogy: A waiter shouldn't ask a customer, "What is the internal chemical structure of the flour you bought?" The customer has no idea! They can answer, "I bought the flour from the store on 5th Street," or "Here is the bag."
The Science: The team studied what makes a question easy for a human to answer. They found that good questions:
- Ask for visible evidence (e.g., "Show me the error message").
- Are specific (e.g., "Which version of Python?" instead of "What version?").
- Have a small scope (e.g., "Show me the code for this one function" instead of "Explain your whole app").
CLARITI's Move: It avoids asking impossible questions. It asks for things the user can actually see or copy-paste, like a screenshot or a specific error code.

3. The "Less is More" Strategy (Efficiency)

The most surprising finding was that asking fewer questions is better.

The Analogy: Imagine a waiter asking 10 questions to get a simple order. The customer gets tired and just says, "Whatever, you decide." But if the waiter asks just 2 perfect questions, the customer is happy and gives a great answer.
The Result: CLARITI manages to solve the same number of software problems as the giant, expensive AI models (like GPT-5), but it asks 41% fewer questions. It cuts out the fluff.

The Secret Sauce: The Reward System

How did they teach CLARITI to be this smart? They didn't just tell it "be helpful." They built a multi-stage reward system (like a video game with levels):

Level 1 (No Redundancy): If you ask a question the user already answered, you get zero points.
Level 2 (No Copy-Paste): If you ask the same generic question for every problem, you get zero points.
Level 3 (Can They Answer?): If the user can't possibly know the answer, you get zero points.
Level 4 (Does It Matter?): If you asked about the "blue plate" instead of the "peanut allergy," you get fewer points.

By training the AI to maximize points in all four levels, it learned to be the perfect waiter: asking the fewest, most critical, and most answerable questions possible.

The Bottom Line

This paper teaches us that for AI to work well with humans, it shouldn't just be "smart"; it needs to be empathetic and efficient. It needs to know what information actually solves the problem and respect the human's time and knowledge limits.

CLARITI proves that by asking the right questions (rather than more questions), AI can solve complex software problems just as well as the biggest models, but with much less friction for the human user.

1. Problem Statement

Real-world user requests, particularly in software engineering (SWE), are frequently underspecified. Large Language Model (LLM) agents often fail when instructions lack critical details (e.g., environment configuration, error logs, or specific constraints), leading to wasted computation or task failure.

While prior work has explored ambiguity resolution, it often focuses on linguistic ambiguity (e.g., resolving a referent) rather than the broader, multi-faceted underspecification found in SWE tasks. Existing agents lack a principled understanding of:

Which missing information is most valuable for task success.
What questions users can realistically answer (answerability), given their knowledge boundaries.
How to optimize clarification to maximize task success while minimizing user burden (number of questions).

2. Methodology

The authors propose a framework to quantify information impact and user answerability, using these insights to train a specialized clarification module named CLARITI (CLARIficiation with TIered rewards).

A. Information Categorization & Impact Analysis (RQ1)

Taxonomy: The authors analyzed 112 highly underspecified issues from SWE-Bench Verified to define six categories of missing information:
1. Error Information (stack traces, error messages).
2. Expected Behavior (desired output/state).
3. Implementation Details (approach/constraints).
4. External References (API docs, datasets).
5. Reproduction Steps (commands, inputs).
6. Version/Environment (config, OS, dependencies).
Shapley Value Analysis: Using 700 underspecified instances, they trained predictive models to map the presence/absence of these categories to task success. They computed Shapley values to determine the marginal contribution of each category.
- Finding: Error Information had the highest predictive value (SHAP $\approx$ 0.18), followed by Implementation Details. Surprisingly, "Expected Behavior" (the most frequently missing category) had a lower impact on success than concrete diagnostic signals.

B. Answerability & Distributional Analysis (RQ2)

Simulation: They generated clarification questions using GPT-5 and GPT-5 Nano and evaluated them against fully specified issue versions to determine if the information was available to a typical user.
Strategic Themes: Through distributional analysis, they identified four strategies that distinguish answerable questions from unanswerable ones:
1. Ground in Evidence: Request concrete artifacts (logs, stack traces) rather than abstract logic.
2. Demand Specificity: Ask for exact values (e.g., "Which Python version?") rather than abstract descriptions.
3. Minimize Scope: Request minimal reproducible examples rather than entire architectures.
4. Ensure Actionability: Focus on observable behaviors or actions the user can perform immediately.
Question Volume: They found that increasing the number of questions does not linearly improve success; beyond a certain point, success plateaus while the proportion of answerable questions drops.

C. CLARITI Training (RQ3)

Model: An 8B-parameter model (based on Qwen3) trained via Group Relative Policy Optimization (GRPO).
Multi-Stage Reward Pipeline: To avoid the computational intractability of using downstream task success as a direct reward, they designed a four-stage intrinsic reward system:
1. Non-Redundancy: Penalizes questions whose answers are already in the issue description.
2. Diversity: Penalizes generic, template-based questions that lack issue-specific grounding.
3. Answerability: Scores questions based on whether the answer exists in the fully specified issue (simulating user knowledge).
4. Task Relevance: Weights questions based on the Shapley-derived hierarchy (RQ1), prioritizing high-impact categories like Error Information.
Training Data: Constructed from SWE-Gym Raw, generating underspecified variants and supervised fine-tuning (SFT) pairs, followed by RL.

3. Key Contributions

Empirical Impact Hierarchy: Demonstrated that not all missing information is equal; concrete diagnostic signals (Error Info) are more critical for SWE success than abstract goal specifications.
Answerability Characterization: Identified specific linguistic and structural strategies (Evidence Grounding, Specificity, Scope Minimization, Actionability) that make questions answerable by typical users.
CLARITI Model: Developed an 8B-parameter clarification module that optimizes for both relevance and answerability using a principled, multi-stage reward pipeline.
Efficiency Gains: Showed that effective clarification is about quality and prioritization, not volume.

4. Results

The model was evaluated on 250 underspecified SWE-Bench issues using an OpenHands agent framework.

Task Success: CLARITI achieved a 36.8% success rate, matching the performance of GPT-5 (35.6%) and significantly outperforming the "No Clarification" baseline (22.4%).
Efficiency: CLARITI generated an average of 3.0 questions per task, which is 41% fewer than GPT-5 (5.1 questions) and GPT-5 Nano (5.2 questions).
Recovery Rate: CLARITI recovered 88% of the performance ceiling of fully specified issues.
Question Distribution: CLARITI allocated 26.4% of its questions to "Error Information" (the highest-impact category), compared to only 10.2% for GPT-5 and 6.0% for GPT-5 Nano. Conversely, it reduced questions on lower-impact categories like "Reproduction Steps."
Ablation: Removing the "Task Relevance" stage dropped performance to 32.0%, and removing "Answerability" dropped it to 24.4% (near the baseline), confirming both components are essential.

5. Significance

Paradigm Shift: The paper moves clarification from a heuristic or conversational task to an empirically grounded optimization problem. It proves that training agents to ask specific types of questions based on data-driven impact analysis yields better results than simply asking more questions.
User Burden Reduction: By asking fewer, higher-quality questions, the system reduces cognitive load on human users while maintaining high task success rates.
Generalizability: While focused on Software Engineering, the framework (quantifying information impact + characterizing answerability + reward-driven training) is applicable to any domain where agents rely on user-provided specifications to complete complex tasks.
Practical Implementation: The use of a multi-stage reward pipeline allows for scalable training without requiring full agent rollouts for every training step, making it feasible to train specialized clarification modules on open-weight models.