💬 NLP

LHAW: Controllable Underspecification for Long-Horizon Tasks

The paper introduces LHAW, a modular synthetic pipeline that systematically generates controllable underspecified long-horizon task variants by removing information across four dimensions and validating their impact through empirical agent trials, thereby providing the first framework for cost-sensitive evaluation of agent clarification behavior in ambiguous settings.

Original authors: George Pu, Michael S. Lee, Udari Madhushani Sehwag, David J. Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, Yuan Xue, Samuel Marc Denton

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: George Pu, Michael S. Lee, Udari Madhushani Sehwag, David J. Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, Yuan Xue, Samuel Marc Denton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very smart, super-fast robot assistant to run your entire business for you. You tell it, "Go make a marketing campaign for our new product."

In a perfect world, the robot knows exactly what you mean. But in the real world, your instructions are often missing tiny, crucial details. Maybe you didn't say which product, or what budget to use, or who the audience is.

If the robot is too confident, it might guess wrong and waste weeks of work. If it's too cautious, it might stop and ask you a million questions, annoying you and slowing everything down.

The Problem:
Currently, we don't have a good way to test if our AI robots know when to guess and when to ask for help. Most tests only give them perfect instructions, so they never learn to handle the messy, incomplete reality of real life.

The Solution: LHAW (The "Missing Piece" Simulator)
The authors of this paper created a tool called LHAW (Long-Horizon Augmented Workflows). Think of LHAW as a "Game Master" for AI robots.

Here is how it works, using a simple analogy:

1. The Recipe Test (Creating the Puzzle)

Imagine you have a perfect recipe for a cake (a well-specified task). LHAW takes that recipe and systematically removes ingredients or instructions to create a "broken" version.

Goal Removal: "Make a cake." (But what kind? Chocolate? Carrot? For a birthday?)
Constraint Removal: "Bake it." (But at what temperature? For how long?)
Input Removal: "Use the flour." (But which bag of flour? The one in the pantry or the one in the garage?)
Context Removal: "Decorate it." (But with what? Sprinkles? Icing? Is it for a baby shower or a funeral?)

LHAW creates thousands of these "broken recipes" and labels them:

Outcome-Critical: If the robot guesses wrong, the cake is ruined (the task fails).
Divergent: The robot might make a great cake or a terrible one depending on its guess.
Benign: The robot can figure it out on its own (e.g., "Bake at 350 degrees" is a safe default).

2. The Stress Test (Watching the Robot)

Now, LHAW gives these broken recipes to different AI models (like GPT-5, Claude, Gemini) and watches what they do.

The "Silent Failers": Some robots just guess. They make a cake, but it's the wrong flavor. They fail silently.
The "Over-Askers": Some robots stop and ask, "What flavor? What size? What pan? What oven?" They ask so many questions they annoy the user.
The "Strategic Askers": The best robots know exactly which questions to ask to save the day without wasting time.

3. The Scorecard (Measuring Efficiency)

The paper introduces a new way to grade these robots called "Gain per Question."

Imagine you are the user. Every time you answer a question, it costs you time and mental energy.
Bad Robot: Asks 10 questions to get a tiny bit of clarity. (High cost, low gain).
Good Robot: Asks 1 specific question that solves the whole problem. (Low cost, high gain).

What Did They Find?

The researchers tested the smartest AI models available today and found some interesting patterns:

GPT-5.2 is like a nervous student: It asks a lot of questions. It rarely fails, but it annoys the user by asking too much.
Gemini models are like the "silent type": They guess a lot and often get it wrong because they are afraid to ask.
The "Cost" of Asking: The paper also tested how the robot reacts if you tell it, "I'm very busy, don't bother me unless it's an emergency." The robots actually changed their behavior! They asked fewer questions when they thought you were busy, showing they can understand "cost."

Why Does This Matter?

We are moving toward a future where AI agents will do complex, long-term jobs (like managing your finances, coding software, or running a supply chain).

If these agents can't tell the difference between "I can figure this out" and "I need help," they will either:

Fail silently and cause disasters.
Nag you constantly and become useless.

LHAW is the first tool that lets us train and test these robots to be strategic. It teaches them to be the perfect employee: confident enough to work alone, but humble enough to ask for help when it truly matters.

In short: LHAW is a training simulator that teaches AI robots how to handle missing information without driving their human bosses crazy.

1. Problem Statement

Autonomous agents operating in long-horizon workflows (tasks requiring dozens to hundreds of sequential steps) face a critical challenge: distinguishing between solvable tasks and those requiring human clarification.

The Tension: In conversational AI, asking for clarification is cheap. In autonomous workflows, human interaction incurs high costs: interruption (cognitive switching), latency (workflow blocking), and trust erosion (excessive questioning).
The Gap: Current benchmarks (e.g., SWE-Bench, TheAgentCompany) evaluate agents on well-specified tasks, while existing clarification benchmarks (e.g., ClariQ) focus on short-context, low-cost scenarios. There is no systematic framework to evaluate whether agents can detect underspecification (missing critical information) and strategically decide when the value of clarification outweighs the cost of interruption.
Consequence: Without this capability, agents either fail silently by assuming missing parameters (leading to cascading errors) or ask excessive, low-value questions.

2. Methodology: The LHAW Pipeline

The authors introduce LHAW (Long-Horizon Augmented Workflows), a synthetic, dataset-agnostic pipeline designed to transform well-specified tasks into controllable underspecified variants. The pipeline operates in three phases:

Phase 1: Segment Extraction & Scoring

An LLM analyzes a well-specified prompt to identify atomic information segments that can be removed.

Taxonomy: Segments are classified into four dimensions:
1. Goal: The deliverable or success criteria.
2. Constraint: Rules, thresholds, or methods.
3. Input: Data sources, file locations, or tools.
4. Context: Domain knowledge, jargon, or implicit requirements.
Scoring: Each segment is scored for Criticality (does removal cause failure?) and Guessability (can the agent infer it?). A priority score is calculated: $Priority = Criticality \times (1 - Guessability)$ .

Phase 2: Candidate Generation

The pipeline generates underspecified prompts using three removal strategies:

Delete: Remove the segment entirely.
Vaguify: Replace with vague language (e.g., "the file").
Genericize: Replace with generic phrases (e.g., "appropriate format").
Severity Control: Difficulty is tuned by removing 1 or 2 segments.

Phase 3: Empirical Agent Trials

Instead of relying on linguistic intuition, LHAW validates variants by running actual agent trials. Based on the distribution of terminal states across $n$ trials, variants are classified:

Outcome-Critical: The agent always fails without the missing info ( $c=0$ ).
Divergent: The agent succeeds in some trials but fails in others, or produces different outcomes ( $c>0, |S|>1$ ).
Benign: The agent reliably infers the missing info and succeeds ( $c>0, |S|=1$ ).
New Task: The agent fails but converges on a single non-successful state (indicating the prompt was transformed into a valid but different task; these are filtered out).

3. Key Contributions

Synthetic Pipeline for Controllable Underspecification: A modular framework that systematically generates ambiguity across four dimensions with configurable severity, moving beyond static datasets.
Benchmark Release (285 Variants): The authors curated 285 underspecified variants from three major benchmarks:
- TheAgentCompany (TAC): 85 variants (focus on partial completion).
- SWE-Bench Pro: 100 variants (code repair).
- MCP-Atlas: 100 variants (tool use).
- These are empirically labeled as outcome-critical, divergent, or benign.
Clarification Behavior Analysis: Introduction of the Gain/Q metric (performance gain per question asked) to measure clarification efficiency, revealing distinct behavioral patterns across frontier models.

4. Experimental Results

The authors evaluated five frontier models (GPT-5.2, Claude Opus/Sonnet, Gemini Pro/Flash) across the benchmarks.

Performance Degradation: Underspecification consistently degrades performance. For example, on MCP-Atlas, baseline Pass@3 dropped from ~100% to ~40-50% without clarification.
Recovery via Clarification: The ask_user tool enables significant recovery, but models rarely return to full baseline performance.
- GPT-5.2: Shows the highest absolute recovery (+22% on SWE-Bench) but has the lowest efficiency (Gain/Q). It tends to over-clarify, asking many questions with diminishing returns.
- Gemini Models: Tend to under-clarify, often failing to ask necessary questions, leading to lower recovery rates.
- Claude Sonnet: Demonstrated high efficiency (high Gain/Q), asking fewer questions but extracting more value per interaction.
Failure Modes:
- Over-Clarification: Common in GPT-5.2 (asking about non-critical or inferable info).
- Under-Clarification: Common in Gemini models (proceeding with false confidence).
- Question Quality: A major bottleneck; agents often ask compound questions or target the wrong information dimension.
Cost Sensitivity: When user personas were introduced (e.g., "Busy Executive" vs. "Supervisor"), agents adjusted their behavior. Agents asked fewer questions to the "Busy Executive" but achieved lower overall task completion, demonstrating an ability to weigh the cost of interruption against task success.

5. Significance and Impact

New Evaluation Paradigm: LHAW shifts the focus from "can the agent do the task?" to "does the agent know when to ask for help?" This is crucial for deploying reliable autonomous systems in real-world enterprise settings.
Cost-Sensitive Decision Making: By quantifying the trade-off between interruption costs and information value, LHAW provides a framework for developing agents that make strategic, economic decisions regarding human interaction.
Safety and Trust: The work highlights that silent failure (assuming parameters) is a major risk. Reliable agents must be able to detect ambiguity and seek clarification to prevent cascading errors in safety-critical or high-stakes workflows.
Future Directions: The authors suggest future work should integrate these findings into decision-theoretic frameworks (Value of Information) and expand ambiguity types to include environmental and multi-turn interaction dynamics.

In summary, LHAW provides the first systematic, empirically validated framework for testing and improving the strategic clarification capabilities of autonomous agents, bridging the gap between theoretical reasoning and practical, cost-effective deployment.