Technical Summary: UNCOM – Zero-shot Context-Aware Command Understanding for Tabletop Scenarios

Problem Statement

Current household service robots often lack the capability for seamless, general-purpose interaction with non-expert users. While traditional robots perform single tasks (e.g., vacuuming), seamless Human-Robot Interaction (HRI) requires understanding rich, multimodal human communication that combines natural speech and gestural cues. Existing solutions face two primary bottlenecks:

1. Monolithic End-to-End Models: Vision-Language-Action (VLA) architectures offer direct input-to-control mapping but suffer from high computational overhead, "black-box" decision-making, and a lack of explainability.
2. Modular Symbolic Frameworks: While transparent, these often rely on closed-vocabulary object detectors and hand-coded semantic graphs, limiting their ability to generalize to the diverse and unstructured objects found in real-world domestic environments.

Furthermore, traditional systems struggle with ambiguity in natural language (e.g., deictic references like "this" or "that") and require extensive task-specific training data or predefined object models.

Methodology

The paper presents UNCOM (UNderstanding COMmands), a novel hybrid framework designed for zero-shot operation in tabletop scenarios. UNCOM integrates speech, gestures, and scene context to extract structured, actionable instructions without relying on task-specific training data or predefined object models.

System Architecture

The system employs a modular pipeline in Python, leveraging foundational deep learning models for complex sub-tasks and a single specialist model for gesture detection:

• Audio Transcription: Uses Whisper-large-v3-turbo for automatic speech recognition (ASR).
• Command Understanding: Uses Phi-4 (mini-instruct) to parse transcribed text into structured semantic elements: Object, Action, and Target. It distinguishes between concrete entities and deictic references (e.g., "this," "that").
• Gesture Detection: Uses MediaPipe Hand Landmarker to detect pointing gestures. The system defines a pointing vector from the base to the tip of the index finger and uses the z-coordinate to determine the active hand.
• Object Detection & Segmentation:
  • GroundingDINO (tiny): An open-vocabulary detector used to identify objects by name or to resolve ambiguity when multiple objects match a description.
  • Segment Anything Model (SAM): Used to segment the precise pixel regions of the target object and target location.
  • DINOv2: Used for depth estimation and general object detection to assist in scene understanding.
• Spatial Reasoning:
  • For relative targets (e.g., "next to the apple"), the system segments the table, applies a Voronoi diagram to partition the space into cells, and generates an occupancy map. It selects the unoccupied cell corresponding to the spatial descriptor.
  • For deictic targets (pointing only), it attempts to detect a container; if none is found, it selects the closest empty Voronoi cell.

Execution

The output is a structured representation (Object-Action-Target) with pixel-level segmentations and 3D coordinates. This data is passed to a task execution subsystem (demonstrated using MoveIt! on TIAGo and TIAGo++ robots) to perform pick-and-place operations. The system calculates the 6DoF pose of objects, falling back to centroid estimation for unknown objects, though it notes limitations with complex shapes.

Key Contributions

1. Zero-Shot Hybrid Architecture: A framework that combines foundational models (Phi-4, GroundingDINO, DINOv2) with a specialist gesture model (MediaPipe) to interpret multimodal commands without fine-tuning on household datasets.
2. Interpretability and Modularity: Unlike monolithic VLA models, UNCOM explicitly parses commands into structured representations, enhancing transparency and allowing integration with symbolic robotic frameworks.
3. Robust Disambiguation: The system effectively resolves ambiguity by fusing verbal instructions with pointing gestures and spatial reasoning (Voronoi partitioning), supporting diverse pointing styles and object references.
4. Benchmark Dataset: The authors released a dataset containing 159 videos across 22 tabletop scenarios, covering variations in object references, distractors, clutter, and target types (absolute vs. relative).

Results

The system was evaluated on a real-world dataset of human-robot interaction scenarios:

• Success Rate: UNCOM achieved an 82.39% success rate (131 out of 159 instances) in correctly identifying the object, action, and target intended by the human instructor.
• Robot Demonstration: The system was successfully deployed on TIAGo and TIAGo++ robots for pick-and-place tasks across six specific scenarios, including handling distractors, multiple similar objects, and deictic references.
• Error Analysis: The primary bottlenecks were identified in the underlying deep learning components rather than the procedural logic:
  • MediaPipe: Highly sensitive to motion blur, leading to missed hand detections.
  • Whisper: Occasional transcription errors, particularly with non-native English speakers (e.g., confusing "bowl" with "ball").
  • Vision Models: GroundingDINO, SAM, and DINOv2 demonstrated robust performance, with errors often correctable via contextual cues.

Significance

The paper claims that UNCOM demonstrates the efficacy of a modular, zero-shot approach to HRI, bridging the gap between the flexibility of foundational models and the explainability of symbolic frameworks. By decoupling command understanding from execution, the system allows for adaptability across different robot embodiments and environments without retraining. The authors emphasize that while the current performance is limited by the robustness of off-the-shelf deep learning models, the architecture provides a scalable foundation. Future work focuses on improving model robustness against noise, expanding language support, and integrating disambiguation routines to allow the robot to converse with users for clarification.