Technical Summary: Behavior Prompting Policy (BPP)

Problem Statement

Teaching robots new skills traditionally requires exhaustive retraining or fine-tuning for each specific task, which is inefficient and limits rapid deployment. While large language models have demonstrated success in in-context learning (adapting to new tasks via few-shot examples at test time), robotics has lagged in achieving similar capabilities for manipulation. Existing approaches often rely on language descriptions or goal images, which can provide incomplete information regarding the specific spatial and temporal execution required for a task. Furthermore, many few-shot learning methods in robotics rely on meta-learning with gradient updates or explicit spatial transformations, which may not scale well to high-diversity task distributions.

The authors propose Behavior Prompting, a paradigm where a robot performs a new task at inference time given a single human demonstration (the "behavior prompt"). This prompt consists of observations, proprioception, and actions in the robot's sensorimotor space, simultaneously defining the task and providing a concrete example of how to execute it.

Methodology

1. Behavior Prompting Policy (BPP)

The core algorithmic contribution is the Behavior Prompting Policy (BPP), an in-context visuomotor architecture designed to translate a behavior prompt and the current observation into robot actions. The architecture comprises three main components:

• Prompt Chunking & Embedding: The behavior prompt (a sequence of observations, proprioception, and actions) is temporally downsampled (typically to 1Hz) for efficiency. The sequence is divided into "chunks," where each chunk contains one timestep of observation ($o$) and proprioception ($q$), along with the sequence of actions ($a$) leading to the next chunk. An Attention Pooling mechanism merges $\{o, q, a\}$ into a single chunk embedding ($p_i$).
• Prompt Encoder: This is a transformer decoder that extracts relevant information from the prompt sequence ($P = [p_0, \dots, p_n]$) conditioned on the current observation. The current observation is tokenized and processed via cross-attention with the prompt chunk embeddings. This allows the model to identify temporal correspondences and spatial differences between the prompt demonstration and the current environment.
• Action Decoder: The extracted prompt information, concatenated with the current observation and a diffusion timestep, is fed into a Diffusion Model (specifically a CNN-based architecture with FiLM conditioning). The decoder iteratively denoises to generate closed-loop actions.

Training Strategy: BPP is trained using standard behavior cloning on multi-task imitation learning datasets. For each training step, a single demonstration is sampled as the prompt, and a batch of receding horizon observations and future actions is sampled from other demonstrations of the same task. Crucially, no explicit spatial or temporal correspondence is enforced between the prompt and the training batch; the model learns end-to-end to leverage the prompt.

2. iPhUMI Interface

To address the data requirements for behavior prompting, the authors introduce iPhUMI, a handheld manipulation interface. Adapted from the UMI gripper, it replaces the GoPro with an iPhone 15 Pro. Key features include:

• Instant Localization: Utilizes on-device ARKit for real-time SLAM, eliminating the need for tedious environment mapping.
• Wireless Prompting: Allows a human operator to wirelessly transmit behavior prompts to a workstation, enabling immediate conditioning of the policy at test time for new tasks.

Evaluation and Benchmarks

The authors introduce two new benchmarks to evaluate test-time adaptation, focusing on high task diversity rather than single-task complexity:

1. DrawAnything: A drawing environment (simulated and real-world) requiring continuous, fine-grained action adaptation.
   • Sim: Trained on 2,000 procedurally generated tasks; tested on 50 unseen human drawings.
   • Real: Uses an ARX robot arm with an iPhone wrist camera. Trained on 1,000 tasks; tested on 10 unseen tasks.
2. LIBERO-Gen: An extension of the LIBERO manipulation benchmark with significantly more tasks.
   • Combination: Extends 10 tasks to 174, testing the ability to pick one of two identical bowls and place it in one of nine locations.
   • Chain: Extends 10 tasks to 321, testing sequential two-step interactions (e.g., open drawer, then place object).

Key Results

• Superior Adaptation: On unseen drawing tasks (DrawAnything-Sim), BPP achieved an 80.7% error reduction compared to a Goal-Image baseline and a 33.3% reduction compared to ICRT (a prior behavior prompt model).
• Robustness to Complexity: In LIBERO-Gen benchmarks, BPP outperformed baselines (Goal-Image, Language, and ICRT) and rivaled $\pi_0.5$ (a foundation VLA model with 100K LoRA fine-tuning steps), despite BPP having no foundation pretraining.
• Attention Mechanism: Visualization of the prompt encoder attention (Fig. 5) reveals that BPP effectively tracks task progression. For continuous tasks (drawing), attention follows the prompt closely; for discrete tasks (LIBERO), attention tracks "milestones" (e.g., next object to interact with).
• Data Requirements: Systematic ablations showed that task diversity is the primary driver of prompting capability. Collecting fewer demonstrations per task across a larger number of distinct tasks yields better test-time adaptation than many demonstrations for a few tasks.

Significance and Claims

The paper claims that behavior prompting provides a flexible and scalable way to teach robots new skills without expensive fine-tuning. By leveraging a single human demonstration as a prompt, users can specify tasks or define new robot capabilities "on the fly."

The authors emphasize that:

1. Sensorimotor prompts are more expressive than language or goal images for tasks requiring specific low-level behaviors or spatial-temporal strategies.
2. Task diversity in training data is critical for enabling zero-shot adaptation to unseen tasks.
3. iPhUMI serves as a practical interface for both collecting diverse training data and specifying behavior prompts at test time, bridging the gap between human intent and robot execution.

The work positions behavior prompting as a step toward enabling robots to adapt pretraining knowledge to new environments (e.g., a specific home) through a single demonstration, simplifying the adaptation problem by providing rich information about the target environment and desired manipulation strategy.