💬 NLP

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

This paper introduces a forced-injection framework to demonstrate that the optimal timing for AI agent clarification is task-dependent—specifically, goal clarification must occur within the first 10% of execution while input clarification remains valuable up to 50%—revealing that current frontier models fail to ask at these empirically determined critical windows.

Original authors: Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen, Vamse Kumar Subbiah

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen, Vamse Kumar Subbiah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart robot assistant to build a complex house for you. You give it a list of instructions, but you accidentally leave out a few crucial details.

The big question this paper asks is: When should the robot stop and ask you for clarification?

Should it ask immediately before laying the first brick? Should it wait until the roof is halfway up? Or should it just guess and hope for the best?

The authors of this paper decided to test this by running over 6,000 experiments with different AI models. They didn't wait for the robots to figure it out on their own; instead, they played "God" and forced the missing information to appear at specific times during the building process to see how much it helped.

Here is what they found, explained simply:

1. The "What" Matters More Than the "When"

The most surprising discovery is that there is no single "best time" to ask a question. It depends entirely on what information is missing. Think of it like different types of construction errors:

The "Goal" (What are we building?): This is the most critical piece of info.
- The Analogy: Imagine the robot starts building a garage because you didn't specify, but you actually wanted a swimming pool.
- The Finding: If the robot asks about the goal after it has already laid 10% of the foundation, it's too late. The damage is done. The value of asking drops to almost zero immediately. You have to clarify the goal right at the start.
The "Input" (What materials do we have?): This is about data or resources.
- The Analogy: Imagine the robot starts building, but you forgot to tell it whether to use red bricks or blue bricks.
- The Finding: The robot can keep working for a while, maybe even building half the walls, before it realizes it needs to know the color. Clarification here is still very helpful up until the 50% mark of the project.
The "Constraints" (What rules must we follow?):
- The Analogy: Imagine the robot builds a beautiful house, but you forgot to tell it that no windows can be higher than 5 feet.
- The Finding: If the robot builds the whole house and then you tell it the rule, it might have to tear everything down. However, if you tell it the rule halfway through, it can sometimes adjust, though it's messy.

2. The "Point of No Return"

The paper found that for "Goal" questions, there is a very narrow window. If you wait until the robot is 10% done, asking "Wait, what are we building?" is useless. The robot has already committed to a path, and changing it now is too expensive (in terms of wasted time and effort).

For "Input" questions, the window is wider. The robot can explore and figure things out on its own for a while, so you have until the halfway point to step in.

3. The Robots Are Bad at Timing

The researchers also watched how these AI robots behave when they are allowed to ask questions naturally (without the researchers forcing the answers).

The Over-Askers: One model (GPT-5.2) asked questions constantly, but often too late. It was like a builder who keeps asking "What color is the paint?" after the walls are already dry.
The Under-Askers: Another model (Gemini) almost never asked questions, even when it was clearly confused. It just kept guessing.
The Reality Check: None of the current top-tier robots naturally know the "sweet spot" for asking. They either ask too late (when the goal is already set) or not at all.

4. The Cost of Waiting

The paper measured "wasted compute," which is basically the amount of work the robot does that turns out to be useless because it was based on a wrong guess.

Early Clarification: Very little wasted work.
Late Clarification: A lot of wasted work. The longer you wait to clarify the "Goal," the more the robot builds the wrong thing, and the more time is thrown away.

The Bottom Line

The paper concludes that timing is everything, but it depends on the topic.

If you are unsure about the Goal, speak up in the first few seconds.
If you are unsure about the Data, you have a bit more time (up to halfway).
If you are unsure about Rules, it's best to say them before the robot starts, but it's better late than never.

Currently, AI agents are terrible at knowing when to speak up. They need to be taught to ask the right questions at the right time, or they will keep wasting time building the wrong house.

Technical Summary: Ask Early, Ask Late, Ask Right

Problem Statement

Long-horizon AI agents execute complex workflows involving hundreds of sequential actions. When user instructions are underspecified, agents face a critical decision: proceed under uncertainty or pause to request clarification. While recent benchmarks confirm that underspecification degrades performance and that clarification channels enable partial recovery, existing research treats clarification as a binary capability (can/cannot ask) rather than a temporal one.

There is a lack of empirical understanding regarding when clarification is most valuable. Classical value-of-information (VOI) theory and HCI interruption research suggest that the benefit of clarification should vary based on the timing of the intervention. However, no prior work has measured how the value of clarification changes over the course of an agent's execution trajectory. The core problem addressed is the absence of quantitative demand curves for clarification timing across different information dimensions (goal, input, constraint, context) and the lack of evidence regarding whether current frontier models naturally ask within empirically optimal windows.

Methodology

The authors introduce a forced-injection framework to isolate the pure timing effect of clarification, independent of an agent's ability to detect ambiguity.

Experimental Setup

Datasets: The study utilizes 84 underspecified task variants drawn from the LHAW benchmark, spanning three distinct agent benchmarks:
- MCP-Atlas: Tool-use competency (36 variants, 6–20 actions).
- TheAgentCompany: Enterprise workflows (30 variants, 6–49 actions).
- SWE-Bench Pro: Code repair tasks (18 variants, 1–121 actions).
Models: Four frontier models were evaluated (GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, and DeepSeek V3.2 on MCP-Atlas only), resulting in over 6,000 experimental runs.
Forced-Injection Protocol:
- Tasks were modified to remove specific information segments (ground truth).
- Clarifications were injected at controlled points along the trajectory: 10%, 30%, 50%, 70%, and 90% of the oracle-calibrated action budget.
- The ask_user tool was disabled during these runs to ensure clarification arrived only via injection.
- Performance was measured using Pass@3 (probability of at least one success in three trials).
Natural Ask Protocol: A complementary study of 300 unscripted sessions on TheAgentCompany tasks enabled the ask_user tool to observe natural clarification behaviors, recording ask rates and the timing of the first inquiry.
Metrics:
- Pass@3: Primary success metric.
- Wasted Compute: The fraction of pre-injection actions absent from the oracle (fully specified) trace, serving as an upper bound on discarded work.
- Point of No Return: The latest injection timing where performance significantly exceeds the no-clarification baseline.
- Cross-Model Consistency: Kendall's $\tau$ rank correlation to determine if timing effects are task-intrinsic or model-dependent.

Key Contributions

Dimension-Dependent Timing Profiles: The study establishes that the value of clarification is not uniformly "earlier is better" but depends sharply on the information dimension:
- Goal Clarification: Highly front-loaded. Value drops precipitously after 10% of execution. In MCP-Atlas, Pass@3 falls from 0.78 (at 10%) to 0.39 (at 70%), effectively losing all benefit relative to the baseline.
- Input Clarification: Shows a gradual decline. Value is retained through approximately 50% of the trajectory, as agents can partially infer inputs through environment exploration.
- Constraint Clarification: Benefit depends on the existence of an oracle gap. In cases with a gap, value declines with delay, though it rarely drops below the no-clarification baseline (suggesting late constraint info is still better than none, but less effective than early).
- Context Clarification: Evidence is mixed due to limited sample sizes, but early injection shows strong potential benefits.
Quantification of the "Ask" Gap: The study quantifies the misalignment between current model behaviors and empirically optimal windows.
- GPT-5.2: Over-clarifies (52% ask rate) but asks too late for goal tasks (mean timing 43%, well past the 10% optimal window).
- Claude Sonnet 4.5: Asks selectively (23% rate) at a mean timing of 50%, which is suboptimal for goals but reasonable for inputs.
- Gemini 3 Flash: Never asks (0% rate), forfeiting all potential clarification benefits.
- Crucially, no current frontier model naturally asks within the empirically optimal window for any dimension.
Task-Intrinsic Timing Effects: Cross-model correlations (Kendall's $\tau$ ranging from 0.78 to 0.87 for models sharing task coverage) confirm that these timing profiles are substantially intrinsic to the tasks themselves, rather than artifacts of specific model architectures.
Cost of Delay: The study provides a quantitative measure of "wasted compute," showing that wasted work grows steadily with injection delay. For example, on TheAgentCompany, wasted compute rises from 0% at 10% injection to 21.7% at 90% injection.

Results Summary

Goal Information: Has a narrow early window ( $\le$ 10%). Delaying clarification past this point results in performance indistinguishable from never asking.
Input Information: Remains recoverable through roughly 50% of the trajectory.
Constraint Information: Late injection can be disruptive if it requires reconciling actions already taken that conflict with the constraint, though it generally remains above the no-clarification baseline.
No Sharp Threshold: The "point of no return" is generally continuous rather than a sharp cutoff, except for outcome-critical goal and constraint variants where a significant recovery point was detected at 30% of the trajectory.
Model Behavior: Current models exhibit strategies ranging from over-asking (inefficient) to never asking, failing to align with the demand curves derived from the forced-injection experiments.

Significance and Claims

The paper claims to provide the first empirical VOI curves for clarification timing in long-horizon agent workflows. Its significance lies in:

Foundational Data: Providing the quantitative "demand curves" that existing theoretical frameworks (e.g., value-of-information theory, metareasoning) require but have previously lacked.
Design Targets: Establishing concrete design targets for timing-aware clarification policies, specifically advising agents to validate goals within the first few actions and defer input queries only slightly longer (within the first half of the trajectory).
Methodological Generalization: The forced-injection methodology is presented as a generalizable approach for any agent benchmark with underspecified tasks to measure the temporal value of information.

The authors modestly note that this study addresses the demand side (how much performance benefits from information at specific times) but does not solve the supply side (how to make agents recognize ambiguity and ask at the right moment). They position their work as the necessary empirical foundation for building the next generation of timing-aware agents.