BOP-ASK: Object-Interaction Reasoning for Vision-Language Models

1. Problem Statement

While Vision-Language Models (VLMs) have achieved significant success in spatial reasoning benchmarks (e.g., identifying "left of" or "behind"), they exhibit critical weaknesses in object-interaction reasoning required for real-world robotic applications. Current models struggle with:

• Fine-grained 3D localization: Precise pose estimation beyond 2D bounding boxes.
• Physical compatibility: Determining grasp affordances and collision-free paths.
• Multi-step planning: Understanding which objects must be moved to access a target in cluttered environments.
• Data Limitations: Existing datasets often rely on approximate annotations (e.g., monocular depth), lack 6D pose ground truth, or focus on simple multiple-choice questions rather than executable manipulation data.

2. Methodology

The authors introduce BOP-Ask, a large-scale dataset and framework designed to bridge the gap between perception and physical manipulation.

A. Data Generation Pipeline

The dataset is built upon the Benchmark for Object Pose Estimation (BOP) family of datasets (HOPE, HANDAL, YCB-V, LineMOD), leveraging their high-quality 6D object poses and 3D models. The pipeline involves:

1. World Frame Construction: Reconstructing a consistent world coordinate system by estimating the camera-to-world transformation. A planar support surface (e.g., a table) is detected using a VLM and fitted via RANSAC to define the "up" direction and origin.
2. Geometric Priors Generation:
   • Trajectories: Synthesized using a Rapidly-exploring Random Tree (RRT) planner in 3D Cartesian space to generate collision-free pick-and-place paths between objects. Paths are refined using the Ramer–Douglas–Peucker algorithm.
   • Grasps: Computed using the transformer-based parallel gripper model M2T2, which uses a dual-sampling strategy (global scene + object-centric points). Top-5 diverse grasps are retained; objects with no valid grasps due to clutter are labeled for rearrangement tasks.
   • 3D Bounding Boxes: Derived from object poses and model sizes.
3. Question-Answer (QA) Pair Generation:
   • Object descriptions (shape, color, utility) are rendered and refined via VLMs.
   • Templates are used to generate linguistically diverse questions covering six reasoning skills.
   • Disambiguation: For multiple instances of the same object, relative positional attributes (e.g., "leftmost," "topmost") are computed and appended to queries.

B. Six Core Reasoning Skills

The dataset targets six specific capabilities essential for embodied agents:

1. Object Pose Estimation: Predicting 3D bounding boxes (cuboids) rather than 2D points.
2. Object Grasp Estimation: Inferring stable 3D grasp poses (5-point formulation: center, finger bases, and tips).
3. Inter-object Motion Prediction: Generating collision-free waypoints to move a source object to a target.
4. Object Rearrangement: Identifying obstructing objects that must be moved to access a target.
5. Spatial Reasoning: Binary classification of spatial relationships (e.g., "Is A left of B?").
6. Relative Depth Perception: Binary classification of depth attributes (e.g., "Is A closer than B?").

3. Key Contributions

• BOP-Ask Dataset: A novel dataset comprising 150,000 images and 33.8 million QA pairs. It is the first to provide precise 6D pose ground truth combined with interaction reasoning (grasps, trajectories, rearrangement).
• Two Evaluation Benchmarks:
  • BOP-Ask-core: Held-out scenes from BOP with manual verification (688 QA pairs).
  • BOP-Ask-lab: Out-of-distribution (OOD) images from different labs to test generalization (240 QA pairs).
• Novel Metrics: Adoption of 3D Intersection-over-Union (IoU) for poses, Normalized Coordinate Error (NCE) for grasps, and Success Rate/Distance Error for trajectories.
• Human Baseline: A comprehensive human evaluation benchmark established to measure the upper bound of performance on these tasks.

4. Experimental Results

The authors evaluated proprietary (GPT-5, Gemini) and open-source (Molmo, Qwen-VL, NVILA) VLMs, as well as models fine-tuned on BOP-Ask.

• Performance Gains: Models fine-tuned on BOP-Ask significantly outperformed baselines.
  • Pose Estimation: NVILA (15B) improved from 27.2% to 73.5% IoU.
  • Trajectory Planning: Success rate increased from near 0% to 64.2%.
  • Grasping: NCE (error) decreased significantly, with NVILA (15B) achieving 1.40.
  • Rearrangement: Accuracy rose from ~8% to 57.7%.
• Generalization (OOD): Fine-tuned models showed strong transfer to out-of-domain benchmarks (RoboSpatial-Home, CV-Bench, SpatialBench), demonstrating that the geometric priors learned are not overfit to the training scenes.
• Real-World Robotics: In a Franka robot deployment (15 pick-and-place tasks), the base NVILA model failed 100% of tasks, while the BOP-Ask fine-tuned model succeeded in 10/15 tasks.
• Human vs. Model: While humans achieved high accuracy (e.g., 84.9% on spatial reasoning), current models still lag significantly, particularly in complex rearrangement and trajectory tasks, highlighting the difficulty of the benchmark.

5. Significance

• Bridging Perception and Action: BOP-Ask moves VLMs beyond simple scene description to executable manipulation. It forces models to reason about physics, collision, and 3D geometry, which are prerequisites for autonomous robotics.
• Precision over Approximation: By utilizing precise 6D ground truth from BOP, the dataset enables training models for millimeter-level accuracy, a requirement for real-world grasping that approximate datasets cannot support.
• Scalability: The automated pipeline demonstrates that large-scale, high-quality geometric reasoning data can be generated without massive manual annotation, paving the way for future embodied AI datasets.
• Foundation for Embodied AI: The results confirm that training on object-interaction reasoning data not only improves performance on the specific dataset but also enhances zero-shot transfer to unseen environments and tasks, validating the approach for developing general-purpose robotic agents.