Technical Summary: EPIC-Bench

Problem Statement

While Large Vision-Language Models (VLMs) are increasingly adopted as the perceptual backbones for embodied agents, existing benchmarks often rely on Question-Answering (QA) or Multiple-Choice Question (MCQ) formats. The authors argue that these protocols allow models to exploit linguistic priors and common-sense reasoning rather than demonstrating genuine visual grounding. Furthermore, traditional visual grounding benchmarks often focus on generic object detection with simple descriptions, failing to capture the fine-grained, task-oriented perception required for real-world embodied interactions (navigation and manipulation). There is a critical lack of a comprehensive evaluation framework that systematically assesses the visual perceptual capabilities of VLMs across the full embodied interaction pipeline.

Methodology

EPIC-Bench Design

To address these gaps, the authors introduce EPIC-Bench (Embodied PerceptIon BenChmark), a fine-grained grounding benchmark comprising 6,661 meticulously annotated tuples of (Image, Text, Mask). The benchmark is structured around three core stages of the embodied interaction pipeline:

1. Target Localization (TL): The foundation for navigation and manipulation, evaluating the ability to localize targets based on multi-dimensional attributes. This category is subdivided into:
   • Basic Attributes (BA): Recognition of object category, color, geometry, material, relative attributes (size/height), and projection.
   • Spatial-Related Attributes (SRA): Understanding orientation (including human-relative) and spatial relations.
   • Embodied Compositional Attributes (ECA): Recognizing part-whole relationships, human-related part-whole regions, object states, and colloquial descriptions.
2. Navigation (NAV): Evaluating perception for moving toward a target. Sub-tasks include:
   • Ground Detection (GD): Identifying traversable ground regions.
   • Feasible Path (FP): Reasoning about valid paths from a camera position to a target or between two targets, constrained to traversable areas.
   • Visual Matching (VM): Localizing the same target object across different viewpoints in the same environment.
3. Manipulation (MAN): Evaluating perception for direct physical interaction. Sub-tasks include:
   • Affordance Region (AR): Identifying specific operational regions or tool-use areas.
   • Contact Relationship (CR): Localizing objects based on contact states with single or multiple reference objects.
   • Placement Region (PR): Localizing target placement areas and determining the physical feasibility of the placement.

Data Construction and Annotation

The dataset was curated from 25 publicly available datasets spanning generic scenes, indoor environments, egocentric views, and robot-view imagery. The annotation pipeline involved:

• Filtering: Using an ensemble of open-source VLMs followed by manual review to ensure diversity and complexity (e.g., including distractors to prevent shortcut learning).
• Annotation: Utilizing SAM3 (Segment Anything Model 3) to generate initial mask proposals, which were then manually refined by 20 human annotators to achieve high-fidelity ground truth.
• Quality Control: A rigorous process involving at least two rounds of inspection to resolve ambiguities, correct inconsistencies between text and masks, and remove samples where targets could be trivially identified.

Evaluation Strategy

The benchmark evaluates 89 leading VLMs, categorized into Proprietary Models, Open-Source Models, and Embodied Foundation Models.

• Protocol: Unlike QA/MCQ, EPIC-Bench requires models to output masks (approximated as bounding boxes for scoring) and, where applicable, counts of target instances.
• Metrics: The scoring system comprises four components:
  1. Localization Score: Intersection over Union (IoU) between predicted bounding boxes and ground-truth masks.
  2. Counting Score: Accuracy in predicting the exact number of targets (binary accuracy for strict counts, soft penalty for deviations).
  3. Path Score: A composite metric for navigation paths evaluating start/end accuracy, traversability, distance, and continuity.
  4. Feasibility Score: A binary judgment on the physical viability of manipulation actions (e.g., placement stability).

Key Results

The extensive evaluation of 89 models reveals several critical findings:

1. General Struggles with Complex Grounding: Current VLMs universally struggle with complex visual-text alignment required for physical interactions. While advanced "thinking" models show promise, they do not yet solve the core bottlenecks.
2. Specific Bottlenecks:
   • Multi-target Counting: Models perform near-human levels when exactly one target is present but degrade significantly in zero-target or multi-target scenarios.
   • Part-Whole Relationships: Models fail to ground specific sub-regions of objects based on text, leading to poor performance in Affordance Region tasks.
   • Spatial Orientation: Performance drops substantially on tasks requiring robust spatial reasoning and reference-frame comprehension (e.g., "left of the person").
3. Model Performance:
   • Proprietary Models: Gemini-3-Pro and Gemini-3.1-Pro achieved the highest overall scores (~54.8), with Gemini-3-Pro excelling in Feasible Path tasks (70.82).
   • Open-Source Models: Qwen3-VL-235B-A22B (with thinking mode) achieved the best open-source result (50.93).
   • Embodied Foundation Models: Models specifically designed for embodied tasks (e.g., RoboBrain, RynnBrain) generally underperformed compared to general-purpose VLMs, largely attributed to their smaller parameter scales.
4. Impact of Reasoning and Prompts:
   • Models equipped with advanced reasoning modes ("thinking") consistently outperformed their standard counterparts.
   • Visual Prompts: Providing mask overlays as visual prompts significantly improved performance for reasoning-capable models, whereas less capable models sometimes struggled with the dense information, benefiting more from simpler bounding box overlays.
   • Token Constraints: "Thinking" models suffered from response truncation under tight token budgets (4096 tokens), requiring larger budgets (8192 tokens) to fully leverage their reasoning capabilities.

Significance and Claims

The paper claims that EPIC-Bench provides a robust foundation and actionable insights for advancing the next generation of vision-driven embodied models. By moving beyond linguistic shortcuts and focusing on mask-level grounding, the benchmark exposes fundamental limitations in current vision-language alignment, specifically regarding multi-object reasoning, part-whole understanding, and physical affordance detection. The authors assert that their work offers a clear path for future research by identifying specific perceptual capabilities that require improvement to support reliable embodied deployment. The benchmark is publicly available to facilitate large-scale evaluation and reproducibility.