🤖 AI

When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence

This paper demonstrates that open vision-language models fail to make optimal decisions about when to ask humans for help due to their reliance on prompt formatting rather than actual sensor evidence, but their diagnostic accuracy and decision-making can be significantly improved by incorporating diverse sensor data and grounding their actions in measured reliability and task costs.

Original authors: Eshika Pathak, Leela Krishna

Published 2026-09-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eshika Pathak, Leela Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a robot working alongside a person drops a cup or fails to pick up a tool, a critical moment arrives before anyone speaks. The machine must decide its next move: should it try to fix the problem on its own, check its own internal sensors for clues, or stop and ask the human for help? This decision is not just about politeness; it is a calculation of risk. Asking for help costs time and interrupts the human's focus, but acting blindly on a wrong guess can lead to more damage. For years, researchers have assumed that if a robot can see a failure, it can understand why it happened, or that it should simply ask for help whenever it feels unsure. However, a new study challenges these assumptions, suggesting that the robot's ability to diagnose a problem depends entirely on which sensors it uses, and that current artificial intelligence models are surprisingly bad at knowing when they should stop and ask.

To investigate this, researchers built a controlled environment where they could create specific failures with known causes. They programmed a simulated robot arm to perform a simple task: pick up an object and place it somewhere. They then introduced three distinct types of problems. First, the robot might fail to see the object because it was hidden, missing, or the lighting was too dim. Second, the robot might try to lift the object but drop it because its grip was too weak, the object was too heavy, or the surface was too slippery. Third, the robot might fail to place the object because the target container was full or moved out of reach. Because the researchers created these failures themselves, they knew the exact cause of every mistake, allowing them to test whether a robot could actually figure it out.

The researchers first tested what information different sensors could provide. They found a sharp divide in what the robot could learn. When the failure was about not seeing the object, a camera was excellent at diagnosing the problem, correctly identifying the cause nearly every time. However, when the failure was about dropping the object, the camera was almost useless. No matter how advanced the image analysis was, the camera could not tell the difference between a weak grip, a heavy object, or a slippery surface, because these physical forces are invisible to a lens. In contrast, when the researchers gave the robot its own force data—measurements of how hard the gripper squeezed and how much weight it felt—the robot could diagnose the drop with near-perfect accuracy. This revealed a fundamental truth: a robot cannot diagnose a problem if the information about that problem is not present in the data it is looking at.

The study then turned to six different open-source artificial intelligence models, the kind of systems often used to give robots the ability to understand language and images. The researchers asked these models to look at the camera footage of a failed attempt and guess what went wrong. The results were startling. The models did not behave like cautious scientists who know when they are missing information. Instead, their behavior was dictated by the layout of the question they were asked. If the option to say "I don't know" was listed last, the models would almost always refuse to answer, claiming they could not determine the cause. If the researchers moved that same option to the top of the list, the models suddenly stopped refusing and started guessing, often with high confidence, even when they were wrong. Their confidence levels did not reflect reality; a model could be 100 percent sure of a wrong answer, or 50 percent sure of a right one, with no pattern to connect the two.

The researchers also tested whether giving the models the force data, written out as simple text, would help. For four of the six models, this extra information made a massive difference. Suddenly, they could diagnose the drop failures correctly, moving from guessing randomly to getting the answer right more than half the time. This proved that the models were not inherently incapable of understanding the physics of the failure; they were simply missing the right sensor data. However, even with this new ability, the models still failed to make the right decision about when to ask for help. They did not ask more often when the question was cheap and the risk of acting alone was high, nor did they stop asking when the question was expensive. Their strategy for asking was rigid and ignored the cost of interrupting a human.

The most effective strategy for a robot, according to the study, is not to rely on the model's own feeling of confidence, which is often misleading. Instead, the decision to ask should be based on two measurable facts: how accurate the robot's diagnosis actually is, and how costly it is to ask a human. If the robot's sensors can reliably tell it what is wrong, it should act. If the sensors cannot provide the answer, or if the robot's own diagnosis is likely to be wrong, it should ask. The study shows that a single question to a human can lift a robot's success rate from near zero to over 80 percent, but only if the robot asks at the right moment. Today, the choice to ask is often a guess, but the path forward is clear: robots need to be wired to know the limits of their own senses and the true cost of a question, rather than trusting the confidence of a machine that does not know what it does not know.

Technical Summary: When Should a Failing Robot Ask?

Problem Statement

When a robot fails at a task while operating alongside a human, its first communicative decision is whether to act on its own diagnosis, consult an onboard sensor, or interrupt the human. Current systems lack two critical capabilities to make this decision optimally: (1) knowing how much information their specific sensors reveal about the true cause of a failure, and (2) knowing the reliability of their own diagnosis. Existing literature either assumes failure causes are recoverable from observations without verification, or assumes they are not without quantifying the gap. This paper addresses the "unexamined step" between failure detection and repair: determining when a robot should initiate corrective dialogue based on audited evidence.

Methodology

1. Simulated Benchmark with Audited Ground Truth

The authors constructed a benchmark in the LIBERO physics simulator where failure causes are injected and known by construction. The study focuses on three failure families:

Detection Failures (Family B): The robot cannot find the object (causes: occlusion, absence, naming error, dim lighting).
Grasp Failures (Family A): The object slips during a lift (causes: low grip force, high mass, low friction).
Placement Failures (Family C): The object cannot be placed (causes: container closed, full, or moved).

The Audit Protocol: To determine what each sensor reveals, the authors trained classifiers and subjected them to three rigorous checks to prevent data leakage:

Shuffle Check: Labels are scrambled; accuracy must drop to chance.
Transfer Check: Models are tested on held-out camera viewpoints and scene appearances; accuracy must remain robust.
Separation Check: Training and test sets come from different simulation runs to avoid near-duplicates.
A score passing all checks is termed a "certificate," representing a reliable lower bound on the information available in that sensor modality.

2. Model Evaluation

Six open vision-language models (7B–16B parameters) were evaluated: Qwen2.5-VL, Cosmos-Reason2, Cosmos3-Nano, InternVL3, Pixtral, and Llama-3.2-Vision.

Input Conditions: Models were tested with camera frames only, force data (telemetry) as text, and combinations thereof.
Prompt Variants: The study included robustness sweeps, such as moving the "cannot be determined" option from the last to the first position in the answer list, and providing few-shot examples.
Confidence Measurement: Confidence was measured via stated ratings, token probabilities, and sample frequency across multiple runs.

3. Decision Framework

The problem is framed as a three-action decision: Act, Consult Sensor, or Ask Human.

Costs: A correct repair costs $C_r=1$ ; a wrong repair costs $C_w=10$ ; asking a human costs $C_a=3$ .
Optimal Policy: The threshold probability $p^*$ for asking is derived analytically based on the agent's measured accuracy ( $p$ ) and the stated costs. A rational agent asks if $p < p^*$ .
Scoring: Models are scored against the "reference policy" (the best fixed policy given the measured accuracy) using "regret" (realized cost minus reference cost).

Key Results

1. Sensor Audit Findings

Detection Failures: Highly diagnosable from images. A classifier achieved 0.910 accuracy, passing all transfer and shuffle checks.
Grasp Failures: Diagnosable almost perfectly from force data (0.986 accuracy) but nearly impossible from images alone. Image-based classifiers plateaued at ~0.55 (chance level for this specific setup), failing to distinguish causes like mass vs. friction which are invisible to the camera until contact.
Placement Failures: A classifier scored 0.998 on images but collapsed to 0.605 on held-out camera angles, revealing it learned the container's position in the frame rather than the failure cause. This highlights the necessity of the audit protocol.

2. Model Behavior and Diagnosis

Prompt Sensitivity: Model behavior tracked the prompt layout rather than the evidence. Moving the "cannot be determined" option to the front collapsed refusal rates from ~78–100% to 0–6% in three model-family pairs. One model (Qwen) began selecting the second option in the list regardless of content.
Accuracy: Under all prompt variants, accuracy from frames alone remained at or below the majority-class baseline. Stated confidence carried no information about correctness (correlation $\rho \approx 0$ ).
Force Data Impact: When provided with force data as text, four of the six models achieved above-baseline diagnosis accuracy (0.45–0.54), whereas they failed with frames alone. This suggests the failure was often a lack of sensor data, not a lack of reasoning ability.
Cosmos Models: Both Cosmos models consistently refused to diagnose grasp failures (near 100% refusal) across all prompt variants, despite the availability of force data in other conditions.

3. Decision-Making Performance

Asking Behavior: Models did not follow the optimal policy. Their ask rates were largely insensitive to the cost of asking ( $C_a$ $C_{a}$ ).
- Qwen asked 100% of the time regardless of cost.
- Pixtral and Cosmos3-Nano also asked almost always.
- Llama under-asked, incurring high regret when questions were cheap.
Value of Asking: When models did ask, the human answer (80% reliable) lifted their final task accuracy to 0.70–0.81, roughly matching the human's reliability. However, because they asked indiscriminately, they incurred unnecessary costs.
Confidence as a Signal: Confidence did not correlate with error rates, meaning it could not be used to trigger asking selectively.

Contributions

Audited Sensor Measurements: A benchmark quantifying which failure types are diagnosable from which sensors, including a specific case (grasp failures) where force data is sufficient (0.99) and images are insufficient (<0.55).
Evidence of Prompt Bias: Demonstration that open vision-language models' diagnosis behavior follows prompt layout (option order) rather than evidence, with accuracy remaining at or below majority-class baselines when relying solely on frames.
Optimal Decision Policy: A framework for deciding between acting, sensing, and asking based on measured accuracy and explicit costs, providing a reference policy to score off-the-shelf models.
Measurement of Asking Value: Quantification showing that while a single human question can restore high reliability (0.70–0.81), current models fail to modulate their ask rates based on the cost of interruption.

Significance and Claims

The paper claims that the decision to ask a human should be "wired to measured accuracy and stated costs, not to the model's confidence." It argues that current models lack the self-awareness to know when they are uncertain. The study demonstrates that much of the perceived inability to diagnose failures is actually a missing sensor modality (force data) rather than a reasoning deficit. Furthermore, it establishes that without explicit cost-aware training or external policy enforcement, models will either ask too frequently (wasting human attention) or too rarely (incurring high repair costs), failing to optimize the human-robot interaction loop. The authors conclude that for a robot to initiate dialogue effectively, the decision logic must be decoupled from the model's internal confidence and grounded in empirically measured sensor capabilities and task costs.