💻 computer science

ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI

The paper introduces ERQA-Plus, a diagnostic benchmark comprising 1,766 question-answer instances across five reasoning categories to evaluate and expose the specific limitations of current embodied AI models in spatial, procedural, and social reasoning despite their overall high accuracy.

Original authors: Hong Yang, Basura Fernando

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hong Yang, Basura Fernando

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a robot to help you in your kitchen. You don't just want a robot that can see a coffee mug; you want one that understands that the mug is on the counter, that it needs to be picked up before pouring, and that if it's full of hot coffee, it shouldn't be placed on a stack of papers.

For a long time, AI researchers have been testing robots with "Visual Question Answering" (VQA) tests. But the authors of this paper argue that these old tests are like a trick question: "If I see a red ball, what color is it?" A robot could just memorize the word "red" without ever actually looking at the ball. It's a shortcut, not true understanding.

To fix this, the researchers created ERQA-Plus. Think of this as a new, much harder "driver's license exam" for robots, designed specifically to test if they can actually reason about the physical world, not just guess based on patterns.

Here is a breakdown of what they did, using simple analogies:

1. The "Reasoning Syllabus" (The Taxonomy)

Instead of just asking random questions, the researchers organized the test into five specific "subjects," much like a school curriculum for a robot:

Perception (The Eyes): Can the robot see what objects are there and where they are relative to each other? (e.g., "Is the cup behind the laptop?")
Action (The Hands): Can it understand what is happening and what should happen next? (e.g., "The person is reaching for the handle; what will they do next?")
Social (The Teammate): Can it read human intentions? (e.g., "That person is looking at the door; are they leaving?")
Navigation (The Map): Can it understand the layout of the room and plan a path? (e.g., "How do I get to the fridge without hitting the chair?")
Common Sense (The Brain): Can it use real-world knowledge? (e.g., "If that glass is full of water, it might spill if I tilt it.")

2. The "Robot Factory" (How they built the test)

Creating 1,766 high-quality questions by hand would take forever. So, the authors built a three-agent assembly line to create the test automatically:

The Generator: This AI writes the questions based on robot camera images.
The Judge: This AI acts like a strict teacher. It looks at the question and says, "This is too easy," or "The answer isn't actually in the picture," and gives it a score.
The Reviser: This AI is the editor. If the Judge gives a low score, the Reviser fixes the question to make it clearer and harder.

They ran this loop over and over (like a coach reviewing game tape) until the questions were perfect. This ensures the test isn't full of "trick" questions that rely on language shortcuts.

3. The "Report Card" (The Results)

They put several popular AI models (the "students") through this new exam. Here is what they found:

The Smartest Student: The biggest model (Qwen3-VL-32B) got the highest overall score (83.4%), but it still struggled with the hardest subjects.
The Weak Spots: Even the smartest models failed at Time (predicting what happens next) and Path Planning (figuring out how to move around obstacles). It's like a student who can describe a car perfectly but doesn't know how to drive it.
The Specialist: One model trained specifically for robots (RoboBrain2.5-8B) was surprisingly good at the hard stuff. It beat the giant models at predicting time and planning paths. This suggests that training a robot specifically for its job is better than just making the brain bigger.
The "Talking" Problem: When the robots had to answer in full sentences (open-ended) instead of picking A, B, or C, they did much worse. They are good at recognizing patterns but bad at explaining why.

4. Why This Matters

The paper concludes that we can't just look at a single "accuracy score" to see if a robot is smart. A robot might get 90% of the questions right by guessing, but fail completely when you ask it to actually do something complex.

ERQA-Plus is a diagnostic tool. It's like a mechanic's scanner that doesn't just tell you "the car is broken," but tells you exactly which part is failing: "The engine is fine, but the transmission can't handle the gears."

By using this test, researchers can stop building robots that are just "pattern matchers" and start building robots that truly understand the world, their actions, and the people they work with.

Technical Summary: ERQA-Plus

Problem Statement

Generalist embodied agents require the ability to reason about spatial relations, actions, procedures, human intentions, and commonsense consequences based on situated visual observations. However, existing Visual Question Answering (VQA) and Embodied Question Answering (EQA) benchmarks suffer from critical limitations. Many VQA datasets allow models to achieve high performance through shortcut learning, linguistic priors, and weak visual grounding, failing to distinguish genuine embodied reasoning from pattern matching. Conversely, EQA benchmarks often prioritize navigation or task completion but lack control over the specific reasoning processes being evaluated. Consequently, current benchmarks cannot adequately disentangle different reasoning skills or provide fine-grained diagnostics of model capabilities, making it difficult to identify failure modes or guide the development of robust embodied intelligence.

Methodology

1. Reasoning Taxonomy

The authors propose a structured taxonomy for generalist embodied agents, organizing reasoning into five interdependent categories that reflect the cognitive architecture required for navigation, manipulation, and human collaboration:

Perceptual Reasoning: Includes object reasoning (identifying attributes), spatial reasoning (geometric structure, relative positioning, occlusion), and temporal reasoning (event sequences, motion dynamics).
Action-Centric Reasoning: Encompasses action recognition, intention reasoning (inferring goals from partial observations), and procedural reasoning (modeling multi-step tasks and valid action sequences).
Social and Interaction Reasoning: Focuses on interpreting human mental states, including attention, role understanding, gaze, and division of labor.
Navigation and Environmental Reasoning: Involves layout understanding and path planning (feasible trajectories avoiding obstacles).
Contextual and Commonsense Reasoning: Leverages prior knowledge for functional context, plausible event prediction, and world knowledge grounding to resolve ambiguity.

2. Dataset Creation Pipeline

ERQA-Plus is constructed using a multi-stage, iterative pipeline involving three agents to ensure high-quality, visually grounded question-answer (QA) pairs:

Generator: Transforms robot-centric visual observations (from main or side cameras) into structured multiple-choice questions. It adheres to design principles that prioritize grounded visual reasoning over linguistic shortcuts, ensuring questions target specific reasoning subtypes with balanced distractors.
Judge Agent: A meta-evaluator that automatically assesses QA instances along five dimensions: question quality (visual grounding), option quality/ground-truth validity, spatial reasoning correctness, and overall usefulness. It assigns a quality score (0–10) and outputs a structured JSON with justifications.
Reviser Agent: Refines candidate QA pairs based on the Judge's feedback, improving clarity, grounding, and reasoning depth while eliminating weak distractors.

The pipeline iterates between the Judge and Reviser agents for a maximum of $N$ rounds to generate the final dataset.

3. Dataset Statistics

The final dataset comprises 1,766 QA instances grounded in 711 robot-centric images (630 from the original ERQA dataset and 81 curated from COIN and ScanNet). The distribution is dominated by spatial and object-centric reasoning, with deliberate inclusion of higher-order skills like intention inference and path planning to evaluate advanced capabilities.

4. Experimental Setup

The authors benchmarked a range of models (primarily in the ~8B parameter regime) on ERQA-Plus using two evaluation settings:

Multiple-Choice Question (MCQ) Answering: Measured by standard accuracy.
Open-Ended Question Answering: Measured by semantic similarity using Sentence-BERT (SBERT) embeddings.

Evaluated models included general-purpose Vision-Language Models (VLMs) like LLaVA-NeXT-8B, Prismatic-7B, MiniCPM-V-4.5-8B, and Qwen3-VL variants, as well as embodied models like RoboRefer-8B, RoboBrain2.5-8B, and LEO.

Key Results

Performance Hierarchy

Top Performer: Qwen3-VL-32B-Thinking achieved the highest overall performance with 83.4% MCQ accuracy and 61.4 SBERT score, benefiting from its scale and chain-of-thought reasoning.
8B Models: Among 8B models, Qwen3-VL-8B (81.9% MCQ) and MiniCPM-V-4.5-8B (79.4% MCQ) established strong baselines. RoboBrain2.5-8B (80.8% MCQ) matched this performance, demonstrating that robotics-oriented training does not sacrifice breadth.
Embodied vs. General: RoboRefer-8B lagged significantly (69.6% MCQ), suggesting that domain-specific training does not uniformly confer generalization benefits.

Critical Findings

Persistent Weaknesses: Even the strongest models (Qwen3-VL-32B) showed significant weaknesses in Temporal Reasoning (dropping to 46.7% MCQ) and Path Planning, indicating that scale alone does not solve complex multi-image temporal understanding.
Efficiency of Specialized Training: RoboBrain2.5-8B significantly outperformed larger models on hard subtypes. It achieved 73.3% on Temporal Reasoning (beating Qwen3-VL-32B by 26.6 points) and 86.7% on Path Planning. This suggests that domain-aligned training can compensate for substantial capacity differences.
Cross-Modal Asymmetries: Social and Interaction Reasoning ranked highly in MCQ but became the hardest category in open-ended generation. Conversely, Perceptual Reasoning showed the inverse pattern. Action-Centric Reasoning remained a shared bottleneck across all architectures in open-ended generation.
Ablation on Pipeline: The generator (GPT-4o) demonstrated faster convergence and higher final quality scores compared to Qwen3-VL-32B. The Judge agent (Gemini 3 Pro) showed better alignment with human judgment (Pearson correlation 0.42) than Qwen3-VL-32B, though neither achieved strong agreement, highlighting the need for hybrid evaluation strategies.

Significance and Claims

The paper positions ERQA-Plus not as a leaderboard for aggregate scores, but as a fine-grained diagnostic framework. Its primary significance lies in:

Disentangling Reasoning Skills: By categorizing questions into a structured taxonomy, it reveals which forms of embodied reasoning models can and cannot perform reliably, moving beyond coarse accuracy metrics.
Identifying Failure Modes: The benchmark exposes specific bottlenecks, such as the inability of current models to handle temporal reasoning or procedural planning effectively, even when they excel at static perception.
Validating Specialized Architectures: The results provide evidence that specialized, domain-aligned training (as seen in RoboBrain2.5) can be more efficient and effective for specific embodied tasks than simply scaling up general-purpose VLMs.
Promoting Grounded Evaluation: The dataset and pipeline emphasize strong visual grounding and rigorous validation to prevent shortcut learning, encouraging the development of more robust, interpretable, and cognitively grounded embodied intelligence.

The authors conclude that while current models are improving, they remain inconsistent in spatial, procedural, and predictive reasoning. ERQA-Plus provides a foundation for targeted debugging and the development of agents capable of coherent, goal-directed behavior in dynamic environments.