Hierarchical Policies from Verbal and Egocentric Human Signals for Natural Human-Robot Interaction

Problem Statement

Current robot policies for human-robot interaction (HRI) rely almost exclusively on language instructions to convey human intent. While language is flexible, it often fails to capture the full spectrum of human communication. In natural interactions, humans frequently rely on nonverbal signals—such as eye gaze, pointing gestures, and ongoing activities—to ground their verbal requests. Relying solely on language forces users to provide fully specified, verbose descriptions (e.g., "Give me the metal profile with the hexagonal tip located to the left of the green wrench"), which is cognitively burdensome and imprecise. Conversely, existing nonverbal approaches often struggle with transient or weak signals (e.g., brief glances) when processed through modular pipelines. The core challenge is to develop a robot framework that can jointly interpret continuous streams of verbal and nonverbal signals in real time to infer intent and execute actions naturally.

Methodology: The EDITH Framework

The authors present EDITH, a robot framework designed to integrate live streams of a human's first-person view (egocentric context), eye gaze, and speech into the robot's control policy. The system comprises two main components: a hardware setup and a hierarchical policy architecture.

1. Hardware System
EDITH utilizes Project Aria glasses worn by the human user. These glasses stream:

• Egocentric Context ($C^{ego}_t$): First-person RGB images.
• Gaze Coordinates: Real-time eye-tracking data projected onto the video frame.
• Speech: Audio transcribed into language instructions ($\ell_t$) via an API.
  These signals are synchronized with the robot's proprioceptive state and camera observations ($o_t$) via timestamps, creating an aligned multimodal stream.

2. Hierarchical Policy Architecture
To handle the richness and noise of human signals, EDITH employs a hierarchical policy consisting of a high-level planner ($\pi_h$) and a low-level executor ($\pi_l$), connected by an asynchronous task queue ($Q$).

• High-Level Policy ($\pi_h$): Implemented using a Vision-Language Model (VLM), specifically Gemini-3.1-Flash-Lite.

  • Input: It processes a sliding window of the human's egocentric context and language instructions. The input is rendered as a video clip where gaze markers are overlaid on frames and speech is added as time-aligned captions.
  • Function: $\pi_h$ infers human intent and decomposes it into a sequence of subtasks.
  • Output: Each subtask is a pair consisting of a fine-grained instruction (e.g., "Pick up the driver with the red handle") and a keyframe ($C^{key}$). The keyframe is a specific frame retrieved from the input stream where the nonverbal signal (e.g., the gaze fixation or pointing gesture) is most clearly expressed. This grounds the intent visually, resolving ambiguities that language alone cannot.
  • Mechanism: Subtasks are appended to a task queue $Q$. If no intent is detected, the queue remains unchanged.

• Low-Level Policy ($\pi_l$): Implemented by fine-tuning a pre-trained Vision-Language-Action (VLA) model ($\pi0.5$).

  • Input: It receives the robot's current observation ($o_t$), the subtask instruction from the head of the queue, and the associated keyframe.
  • Function: $\pi_l$ executes the subtask by predicting low-level robot actions ($a_t$) and a completion probability ($p_t$).
  • Mechanism: Once $p_t$ exceeds a threshold, the subtask is popped from the queue, and the next one is executed.

3. Training Strategy

• Data Collection: Demonstrations are collected where a human actor (wearing Aria glasses) interacts with a robot teleoperator. The teleoperator fulfills requests based on the human's multimodal signals. Trajectories are segmented into subtasks and annotated with fine-grained instructions, keyframes, and completion probabilities.
• Modality Dropout: To prevent the low-level policy from relying on only one modality (e.g., ignoring the keyframe if the text is sufficient), the authors employ a modality dropout strategy during training. With equal probability, they either: (i) keep both modalities, (ii) replace the instruction with an underspecified version (forcing reliance on the keyframe), or (iii) replace the keyframe with a blank image (forcing reliance on the instruction).

Key Results

The authors evaluated EDITH on three challenging HRI tasks: Muffin-Serving, Tumbler-Sorting, and Tool-Passing. In all tasks, language instructions were intentionally underspecified, requiring the robot to interpret nonverbal cues.

• Performance: EDITH achieved an average Success Rate (SR) of 59.7% and a Task Progress (TP) of 84.7%.
• Baselines Comparison:
  • Language-only baselines (end-to-end $\pi^{lang}_l$ and hierarchical $\pi^{lang}_h + \pi^{lang}_l$) performed poorly, with average SRs below 6.5%.
  • An end-to-end policy using egocentric context without the hierarchical keyframe mechanism ($\pi^{ego+lang}_l$) showed inconsistent performance, failing significantly in tasks where human attention was intermittent (e.g., Tool-Passing).
• Ablation Studies:
  • Removing the keyframe caused a ~50% drop in success rate, highlighting the necessity of visual grounding to prevent VLM hallucinations and ambiguous phrasing.
  • Removing the egocentric context (using only robot-view keyframes) led to a ~34% drop due to parallax issues, where pointing gestures are ambiguous from the robot's perspective but clear from the human's.
• Robustness: EDITH maintained performance even when humans were distracted (e.g., checking a phone), whereas end-to-end baselines degraded significantly.
• User Study: A study with 16 participants showed that EDITH significantly reduced the instruction workload (mental demand, effort, frustration) compared to language-only baselines ($p < 0.001$). Users found it much easier to convey intent via brief utterances and gestures than via detailed verbal descriptions.

Significance and Claims

The paper claims that EDITH demonstrates the promise of leveraging nonverbal signals alongside verbal instructions to enable natural human-robot interaction. Key contributions include:

1. Integration of Egocentric Signals: It successfully captures and utilizes continuous first-person view and gaze to ground robot actions, moving beyond the limitations of text-only interfaces.
2. Hierarchical Design: The separation of intent inference (high-level) and action execution (low-level), coupled with the use of keyframes, effectively handles noisy, transient nonverbal signals that modular pipelines or end-to-end models struggle with.
3. Reduced User Burden: By allowing users to rely on natural nonverbal cues, the system significantly lowers the cognitive load required to communicate with robots.

The authors acknowledge limitations, including latency due to window-based processing and reduced generalization to users with significantly different physical attributes (e.g., height) than those in the training data, suggesting future work in diverse data collection and real-time streaming VLMs.