REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?

This paper introduces REI-Bench, the first benchmark for evaluating robot task planning under vague referring expressions, revealing that such vagueness significantly degrades performance and demonstrating that a task-oriented context cognition approach effectively mitigates this issue to improve accessibility for non-expert users.

The Gist

Here is an explanation of the paper REI-Bench, translated into simple, everyday language with some creative analogies.

The Big Problem: The Robot's "Mind Reading" Struggle

Imagine you have a super-smart robot butler. You tell it, "Please move the heavy stuff outside."

If you are standing in a kitchen where the only heavy thing is a giant pot, a human understands instantly: "Oh, they mean the pot." But for a robot, "heavy stuff" is a nightmare. Is it the pot? The bag of flour? The cast-iron pan?

The paper argues that while robots are getting great at following clear instructions (like "Move the pot"), they are terrible at following vague instructions (like "Move it" or "Move the heavy stuff"). This is a huge problem because real humans—especially the elderly, children, or people in a hurry—don't speak like robots. They use shortcuts, pronouns, and descriptions that rely on context.

The Solution: A New "Gym" for Robots (REI-Bench)

To fix this, the researchers built a new training ground called REI-Bench. Think of this as a "gym" for robot brains, but instead of lifting weights, the robots have to solve puzzles involving vague language.

They created a dataset of 2,700 scenarios based on real-life conversations. They tested the robots in three different "difficulty modes":

1. The "Clear" Mode: The human says, "Move the pot." (Easy peasy).
2. The "Mixed" Mode: The human says, "Move the pot," but then later says, "Now move it." (The robot has to remember what "it" refers to).
3. The "Vague & Distracting" Mode: The human says, "Move the heavy thing," while the conversation is full of noise, like mentioning a person named "Apple" (who isn't a fruit) or talking about a "heavy" book that isn't the target.

The Result: When the instructions got vague, the robots' success rate crashed. Some failed 37% more often than when the instructions were clear. They started grabbing the wrong items, like picking up a plate instead of the pot because they couldn't figure out what "the heated one" meant.

Why Do Robots Fail? (The "Distraction" Analogy)

The researchers discovered that the robots aren't "dumb"; they just get distracted.

Imagine a student taking a math test.

• Clear Instruction: "Solve for X." The student focuses on the math.
• Vague Instruction: "Solve for the thing that makes the answer happy."

The robot's brain (the Large Language Model) tries to do two things at once:

1. Understand the language (Figure out what "it" means).
2. Plan the actions (Pick up, move, put down).

When the language is vague, the robot gets so stuck trying to figure out the meaning that it forgets how to plan the actions. It's like a driver trying to read a map while driving; they get confused and crash. The robot spends all its "brain power" guessing the word and runs out of power to actually move the object.

The Fix: "The Translator" (TOCC)

The paper proposes a clever, simple fix called TOCC (Task-Oriented Context Cognition).

Instead of asking the robot to "Guess the meaning AND plan the move" at the same time, TOCC splits the job into two steps, like a Translator and a Manager.

1. Step 1: The Translator (Cognition): The robot first acts as a translator. It looks at the vague instruction ("Move the heavy stuff") and the conversation history, then rewrites it into a crystal-clear command: "Move the pot."
2. Step 2: The Manager (Planning): Now, the robot takes this clear command and simply plans the moves. No guessing, no confusion.

The Analogy:
Think of it like a chef and a sous-chef.

• Without TOCC: The chef tries to read a scribbled note from a customer ("Make the spicy red thing") while simultaneously chopping vegetables. They chop the wrong thing.
• With TOCC: The sous-chef (Translator) reads the note, asks the customer for clarification, and writes a clear ticket: "Make the Spicy Red Chili." The chef (Planner) then just follows the clear ticket perfectly.

The Takeaway

This paper teaches us that to make robots useful for real people (like grandma or a toddler), we can't just give them smarter brains. We have to teach them to translate human vagueness into clear instructions before they try to act.

By adding this "Translator" step, the researchers made the robots significantly better at understanding us, proving that sometimes, the best way to help a robot is to help it understand what we really mean.

Technical Summary

Here is a detailed technical summary of the paper "REI-BENCH: CAN EMBODIED AGENTS UNDERSTAND VAGUE HUMAN INSTRUCTIONS IN TASK PLANNING?"

1. Problem Statement

While Large Language Model (LLM)-based task planners have shown promise in enabling robots to execute complex tasks, they operate under an idealized assumption: human instructions are clear, complete, and unambiguous. In real-world Human-Robot Interaction (HRI), particularly with non-expert users (e.g., the elderly, children, or those with cognitive impairments), instructions frequently contain vagueness.

The paper focuses specifically on coreferential vagueness arising from Referring Expressions (REs). Unlike explicit REs (e.g., "the pot"), implicit REs (e.g., "it," "the heavy stuff," "the heated one") rely heavily on dialogue context and environmental reasoning to resolve their meaning. Current LLM-based planners often fail to resolve these implicit references, leading to incorrect object identification and task failure. The paper investigates how this vagueness degrades planning performance and proposes methods to mitigate it.

2. Methodology

A. REI-Bench Benchmark and Dataset

To systematically evaluate this problem, the authors introduce REI-Bench, the first benchmark specifically designed to model coreferential vagueness in robot task planning.

• Base Dataset: Built upon the ALFRED dataset (embodied household tasks) within the AI2-THOR simulator.
• Data Curation Pipeline:
  1. Seed Selection: Selected 6 task types (e.g., Pick & Place, Heat & Place) and filtered for tasks executable by "LLaMA3.1-8B + SayCan" with clear instructions.
  2. Context Memory Generation: Extended seed instructions into multi-turn dialogues using GPT-4o-mini to simulate realistic human-robot interactions.
  3. Context Variants: Created three context types to test reasoning robustness:
     • Standard: Complete, relevant context.
     • Noised: Introduced "Ambiguous Names" (e.g., a family member named "Rose" vs. a "Rose" flower) to create naming ambiguity.
     • Short: Removed partial task-relevant information to simulate memory loss or incomplete communication.
  4. Referring Expression (RE) Manipulation: Replaced explicit nouns with implicit forms to create three difficulty levels:
     • Explicit REs: All objects named directly.
     • Mixed REs: Instructions contain implicit REs, but context retains explicit names.
     • Implicit REs: Both instructions and most context use implicit REs, forcing reliance on scene inference.
• Scale: The final dataset contains 2,700 instructions across 9 difficulty levels (3 RE levels × 3 context types).

B. Proposed Solution: Task-Oriented Context Cognition (TOCC)

The authors observed that standard prompting methods (Aware Prompt, Chain-of-Thought, In-Context Learning) yield limited gains because they either fail to trigger deep reasoning or overwhelm small language models with token constraints.

They propose TOCC, a two-stage decoupling approach:

1. Cognition/Resolution Stage: The LLM is prompted to analyze the multi-turn dialogue, identify implicit REs, and rewrite the human instruction into a clear, explicit version (e.g., transforming "put the heated one in the sink" to "put the heated potato in the sink").
2. Planning Stage: The planner receives only the rewritten, explicit instruction to generate the action sequence.

• Key Innovation: This separates the linguistic understanding (resolving vagueness) from the decision-making (planning), preventing the planner from devoting excessive attention to ambiguity resolution during the generation of action steps.

3. Key Contributions

1. Systematic Modeling of Vagueness: The first work to systematically model instruction vagueness in robot task planning, categorizing it by RE types and context memory quality.
2. REI-Bench Benchmark: A comprehensive dataset and evaluation framework that reveals a significant performance gap between clear and vague instructions.
3. TOCC Framework: A simple yet effective method that significantly outperforms existing prompting strategies (AP, CoT, ICL) by decoupling context cognition from task planning.
4. Empirical Insights: Demonstrated that the primary cause of failure is object omission (planner fails to identify the target object) rather than execution errors, and that this issue is exacerbated by implicit REs.

4. Experimental Results

Baseline Performance

• Significant Drop: When moving from Explicit to Implicit REs, task success rates dropped drastically. For example, the LLaMA3.1-8B + SayCan planner saw a success rate drop from 57.7% (Explicit) to 46.9% (Standard Context, no implicit REs), and further down to ~25-30% in high-vagueness scenarios (Mixed/Implicit REs).
• Maximum Decline: Success rates dropped by up to 36.9% in the most difficult scenarios.
• Context Impact: While "Noised" and "Short" contexts affected performance, the presence of Implicit REs was the dominant factor causing failure.

Error Analysis

• Object Omission: The primary failure mode. In implicit RE scenarios, planners frequently identified the wrong object (e.g., choosing a "plate" instead of a "potato" for "the heated one").
• Execution Errors: Interestingly, execution errors (correct object, wrong action) decreased as implicit REs increased, suggesting that when the object is misidentified, the planner often fails to generate a valid plan for the wrong object entirely, rather than executing a wrong plan.

TOCC Performance

• Superiority: TOCC achieved State-of-the-Art (SOTA) performance among the tested methods.
• Improvement: On the LLaMA3.1-8B + SayCan baseline, TOCC improved the success rate by an average of 6.5% across all difficulty levels.
• Efficiency: Unlike Chain-of-Thought (CoT), which increased token usage and latency significantly (often causing timeouts), TOCC maintained efficiency with only a 3.95% increase in token usage over the baseline.
• Error Reduction: TOCC reduced the Object Omission Rate significantly (e.g., from 30.5% to 24.2% in Explicit REs, and from 24.0% to 30.6% in Implicit REs, though the absolute error rate for Implicit REs remains higher than Explicit, the relative improvement is substantial compared to other methods).

5. Significance and Future Work

• Accessibility: This work addresses a critical barrier to deploying robots for vulnerable populations (elderly, children) who naturally use vague language.
• Paradigm Shift: It challenges the assumption that embedding an LLM in a robot is sufficient for language understanding; explicit intermediate steps for "cognition" are necessary.
• Future Directions: The authors plan to extend this to long-horizon tasks, investigate other forms of vagueness (deictic, syntactic), and validate findings on physical robots with multimodal (visual/spatial) inputs, as the current study relied on text-based simulators.

In conclusion, REI-Bench establishes that coreferential vagueness is a major bottleneck for embodied agents, and TOCC offers a practical, efficient solution to bridge the gap between human natural language and robotic execution.