Technical Summary: Probing Collision Grounding in Vision-Language Models for Safe Human–Robot Collaboration

Problem Statement

Safe human–robot collaboration (HRI) requires more than semantic visual description; a safety monitor must determine whether a robot's body is safely separated, already colliding with the scene or a person, or approaching collision. The authors identify a "collision-grounding gap": current Vision-Language Models (VLMs) often fail to bind visual evidence to the robot's physical footprint, camera placement, depth, occlusion, and temporal approach. While classical safety standards (e.g., ISO 10218) provide reactive control-level safeguards, they do not address the visual reasoning required to infer collision states from multi-view clips. Existing benchmarks often rely on human annotations, single-view RGB, or retrospective labels, failing to isolate robot-induced collision under deterministic physical ground truth.

Methodology

The paper introduces TouchSafeBench, a physics-grounded benchmark built on Habitat 3.0 to evaluate collision grounding in VLMs.

Dataset Construction

• Simulation Environment: Utilizes photorealistic HSSD-HAB scenes, SMPL-X humanoids, and rigid-body contact signals.
• Episodes: Generated 2,940 co-presence episodes across two tasks: Social Navigation (robot follows a humanoid) and Social Rearrangement (robot and humanoid manipulate objects).
• Observations: Each episode includes synchronized multi-view RGB-D streams (robot-arm ego-view, human-head ego-view, third-person robot view, third-person human view), a top-down trajectory map, and calibrated camera metadata.
• Ground Truth: Labels are derived directly from simulator state (contact flags for robot–scene and robot–human) rather than subjective annotation, ensuring physical accuracy.

Tasks

The benchmark evaluates two complementary capabilities:

1. Safety Event Classification (Task 1): Classifies the current state of a short clip as Safe (A), Robot–Scene Collision (B), or Robot–Human Collision (C).
2. Collision Early Warning (Task 2): Predicts imminent risk before contact occurs. Labels include No Risk (A), Scene Collision Imminent (B), or Human Collision Imminent (C). Crucially, "near-miss" scenarios (close but collision-free) are labeled as Safe to test for false alarms.

Experimental Protocol

The study evaluates three frontier VLMs: GPT-5.5, Gemini 3.1 Pro, and Gemini Robotics-ER 1.6.

• Stage I (Modality Sweep): Fixes the deployment-critical ego-view setting and tests nine visual representations (RGB, Depth, Colorized Depth, Predicted Depth, and their combinations).
• Stage II (Viewpoint Ablation): Fixes the input to RGB and varies the viewpoint context (Ego-only, All views including third-person, and Top-down BEV).
• Metrics: Accuracy, Macro-F1, and False Alarm Rates (FAR) specifically measured on near-miss sets to assess calibration.

Key Results

The experiments reveal that current VLMs are visually plausible but physically under-grounded, with the best average Macro-F1 remaining below 50%.

1. The Depth Paradox: Explicit depth information does not automatically translate into collision evidence. Depth-only or predicted-depth inputs often underperform compared to RGB or RGB-D. Models fail to transform depth maps into metric robot-body contact evidence unless explicitly tied to robot morphology and camera-to-body transforms.
2. Scene Contact is the Blind Spot: Robot–scene collisions are consistently harder to detect than human-contact risks.
   • Task 1 Scene Contact Accuracy: ~21.5% (vs. 39% for human contact).
   • Task 2 Scene Imminent Warning Accuracy: ~34.1% (vs. 57% for human).
   • Models often rationalize gripper-camera proximity as non-contact or demand visible penetration/deformation that RGB-D cannot provide.
3. Viewpoint Effects:
   • Third-person views improve current-state recognition (Task 1) for general models (GPT-5.5, Gemini 3.1 Pro) by resolving local occlusions, but do not necessarily improve early warning.
   • Top-down trajectory maps significantly reduce false alarms (improving calibration) for Task 2 but often degrade current-state recognition (Task 1) because layout cues alone do not reveal present-tense contact.
4. Specialization vs. Calibration: Robotics-specialized models (Gemini Robotics-ER 1.6) exhibit lower false alarm rates (better calibration) but do not achieve higher Macro-F1 scores. They remain blind to many true scene-contact cases, suggesting that specialization alone does not solve the grounding problem.
5. Failure Modes: Analysis identifies specific reasoning errors, such as the "Gripper-camera attribution fallacy" (dismissing wall contact because the camera is on the gripper) and the "Proximity–collision fallacy" (equating visual closeness to physical contact).

Key Contributions

1. TouchSafeBench: A repeatable, simulator-derived benchmark for robot-induced collision risk that separates semantic danger recognition from physically grounded collision reasoning.
2. Task Definition: A dual-task framework separating current-state monitoring from anticipatory warning, explicitly handling near-miss hard negatives.
3. Empirical Analysis: A controlled study demonstrating that geometry, viewpoint, and robotics-oriented pretraining only help when representations explicitly bind viewpoint, robot morphology, metric geometry, and future collision.

Significance and Claims

The paper argues that visual fluency does not imply physical accountability. Current VLMs can narrate indoor scenes but lack a dependable estimate of whether the robot is physically safe. The authors claim that reliable robot safety monitors require representations that explicitly bind:

• Viewpoint and camera placement.
• Robot morphology and footprint.
• Metric geometry and contact thresholds.
• Temporal motion and future collision risk.

The work highlights that the "collision-grounding gap" is a specific, actionable failure mode where models fail to connect visual evidence to the robot's physical body and rigid-body contact state. The authors conclude that simply adding depth or prompting for caution is insufficient; the next generation of safety monitors must integrate semantic understanding with a geometric substrate explicit about robot morphology.