Technical Summary: Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation

Problem Definition: Spatially Prompted Visual Trajectory Prediction (SP-VTP)

Current robotic manipulation policies often rely on language instructions or discrete task identifiers to specify goals. However, in cluttered environments containing visually similar objects, linguistic descriptions (e.g., "pick the fork") or task IDs can be ambiguous or fail to identify the specific instance or placement region. The authors argue that in such scenarios, the goal specification is inherently spatial before it is linguistic: "pick this object and place it there."

To address this, the paper formalizes a new setting called Spatially Prompted Visual Trajectory Prediction (SP-VTP). In this framework:

• Input: The model receives a first-frame egocentric image with static spatial prompts (points or bounding boxes) indicating the target object and the target placement region.
• Task: The model must predict future end-effector (EE) trajectories based on streaming egocentric observations.
• Output: Unlike visual grounding or tracking which outputs masks or boxes, the output is a sequence of relative 6D EE motions and gripper states.
• Challenges: The problem combines four distinct difficulties:
  1. Static Task Specification: Object and target are defined only in the first frame.
  2. Dynamic Execution: The camera moves, the EE occludes the scene, and objects relocate after grasping.
  3. Instance Disambiguation: Clutter and same-category distractors require persistent instance-level tracking.
  4. Phase-Dependent Motion: The same object-target pair may require different motions at different execution phases.

Methodology

Dataset: EgoSPT

To study SP-VTP, the authors introduce EgoSPT, a dataset of egocentric spatially prompted manipulation trajectories.

• Collection: Data was collected using a modified Universal Manipulation Interface (UMI) equipped with a GoPro (170° fisheye lens) for egocentric video and an iPhone for 6-DoF EE trajectory tracking via visual-inertial SLAM.
• Content: The dataset contains 2,841 pick-and-place episodes involving five visually similar forks and nine targets (plates, bowls, cups).
• Structure: It is organized into three scenes with increasing difficulty:
  • Scene 1: Structured layouts (clean conditions).
  • Scene 2: Cluttered layouts (distractors and spatial ambiguity).
  • Scene 3: Randomly cluttered subscenes with sparse demonstrations per configuration.
• Annotations: Each episode includes first-frame object and target grounding annotations (bounding boxes) and recovered 3D EE trajectories.

Model: SPOT (Spatially Prompted Object–Target Policy)

The authors propose SPOT, a policy designed to fuse spatial prompts with visual observations to generate future trajectory chunks. The architecture consists of three main modules:

1. Task Encoder:

   • Processes the first frame ($I_0$) and the spatial prompts (object $p_{obj}$ and target $p_{tgt}$).
   • Dual Prompt Representation: Prompts are encoded in two complementary ways:
     • Visual: Bounding boxes are rendered onto the first frame.
     • Coordinate: Box coordinates are encoded as spatial tokens using Fourier positional features and an MLP.
   • Fusion: A frozen DINOv2 ViT-B/14 backbone extracts image tokens. Prompt tokens (role-embedded) query these visual tokens via cross-attention to extract task-relevant evidence, fusing global scene context with specific object/target information.

2. Observation Encoder:

   • Encodes the current egocentric frame ($I_t$) using the same frozen visual backbone to ensure a shared visual space with the task encoder.
   • Incorporates recent trajectory history ($h_t$) via a lightweight MLP and temporal position embeddings.
   • Concatenates current visual tokens and history tokens to form the execution context.

3. Trajectory Generator:

   • Takes the concatenated task and observation tokens as conditioning.
   • Predicts a future trajectory chunk ($A_t \in \mathbb{R}^{H \times 10}$) representing relative translation, 6D rotation, and gripper width.
   • Objective: The default implementation uses Flow Matching to predict the velocity field of the trajectory, though a diffusion head is also supported. The model is trained to denoise interpolated trajectory tokens.

Evaluation Protocol

The paper emphasizes scene-aware splits to prevent data leakage and overestimation of generalization. All episodes from the same scene unit are kept within a single partition (train or validation). Evaluation metrics include Final Displacement Error (FDE), Position L2, Rotation L2, and Gripper Width L1.

Key Results

Prompting Baselines

Experiments comparing different prompt modalities (No Prompt, Point, BBox, Visual, BBox+Visual) demonstrate that spatial prompts significantly improve performance over non-prompted baselines.

• The combination of Visual Prompts (rendered boxes) and Coordinate Prompts (box tokens) yields the best overall results, reducing FDE from 0.1739 (No Prompt) to 0.1147.
• Visual prompts alone are particularly effective for rotation accuracy, while coordinate prompts provide explicit geometry beneficial for translation in cluttered scenes.

Ablation Studies

• Vision Encoder: DINOv2-Base outperforms other foundation models (SigLIP, SAM, EVA2, PE) in endpoint and translation accuracy, particularly in cluttered scenes (Scene 2 and 3).
• Frozen vs. Unfrozen: Keeping the visual backbone frozen yields better generalization than fine-tuning it, suggesting that adapting the backbone to the limited task distribution leads to overfitting.
• Trajectory Head: The Flow Matching head consistently outperforms the Diffusion head across all metrics.
• History Horizon: A moderate history horizon ($K=4$) provides the best balance, offering sufficient motion context without introducing stale observations.

Qualitative Performance

Visualizations show that SPOT successfully infers the correct local motion phase from static prompts and current observations. While long-horizon stitching accumulates local drift (compounding errors), the model preserves the coarse geometric structure of the manipulation trajectory, including approach and return motions.

Significance and Claims

The paper claims to establish SP-VTP as a new, simple, and scalable task condition for egocentric manipulation. Its primary contributions are:

1. Formalization: Defining the first setting that frames egocentric manipulation as vision-centric, spatially prompted trajectory prediction, distinct from language-conditioned planning or standard tracking.
2. Dataset: Introducing EgoSPT, which provides the necessary grounding annotations and 3D motion recovery to study this problem.
3. Policy: Proposing SPOT, which demonstrates that fusing rendered and coordinate spatial prompts with visual history effectively predicts future trajectories in novel scene configurations.
4. Evaluation: Establishing a scene-aware evaluation protocol that tests true cross-scene generalization rather than layout memorization.

The authors position spatial prompting as a direct, low-ambiguity interface for specifying manipulation goals in cluttered environments, offering a practical alternative to language-based instructions where visual context is paramount.