Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation

1. Problem Statement

The paper addresses a critical limitation in current visuomotor policy learning for robotic manipulation: robustness in out-of-distribution (OOD) scenarios.

• The Challenge: Standard imitation learning models (e.g., Diffusion Policy, ACT) excel at executing tasks when the robot's state remains within the distribution of expert demonstrations. However, they lack explicit recovery mechanisms. If the robot encounters an execution error, a covariate shift, or a dynamic obstacle (OOD situations), the policy often fails to recover or replan, leading to task failure.
• Existing Solutions & Limitations:
  • Error Correction Datasets: Training on error data is costly, does not scale, and can introduce suboptimal behaviors.
  • Reward/Cost Functions: Manually designing cost functions for dynamic environments is impractical and lacks generalization.
  • High-Level Planners: While reasoning planners (e.g., LLMs) can generate intermediate goals, they often rely on low-frequency, rule-based execution modules that struggle in dynamic, continuous environments.
• Core Question: How can a policy model trained only on expert demonstrations be enhanced to robustly handle OOD situations and dynamically re-route trajectories using sparse external guidance (e.g., from a human or a high-level planner) without additional fine-tuning?

2. Methodology: ReV (Referring-Aware Visuomotor Policy)

The authors propose ReV, a closed-loop framework that integrates sparse referring points (3D spatial targets) into a diffusion-based policy to steer the robot's trajectory in real-time.

A. Core Architecture: Coupled Diffusion Heads

To handle long-horizon tasks while maintaining precision, ReV employs a two-stage diffusion architecture:

1. Global Diffusion Head (GDH):
   • Generates a sequence of sparse action anchors ($A'$) that encode the long-range motion intent and global consistency of the task.
   • Operates on a coarse temporal scale.
   • Conditioned on current observations and historical anchors.
2. Local Diffusion Head (LDH):
   • Performs temporal-dependent interpolation between adjacent anchors generated by the GDH.
   • Produces a fine-grained, smooth trajectory ready for execution.
   • Crucially, the interpolation strategy is conditioned on the specific temporal position, allowing the model to learn non-uniform densification (e.g., finer control near critical points).

B. Referring-Aware Mechanism

ReV introduces a mechanism to incorporate a 3D referring point ($P$) provided by an external source (human/planner) into the generation process:

1. Temporal-Position Prediction:
   • A Transformer-based module predicts the optimal temporal slot ($k$) within the trajectory where the robot should pass through the referring point $P$.
   • It treats the problem as an $N_1$-way classification over the trajectory timeline, considering historical anchors and extrapolated future steps.
2. Trajectory-Steering Strategy (Masked Denoising):
   • Once the temporal position $k$ is identified, the referring point is converted into a partially constrained referring action ($P_a$).
   • Constraint: The translation component of $P_a$ is fixed to match $P$, while rotation and gripper states remain free for the policy to optimize.
   • Injection: This constrained action is injected into the diffusion denoising process via a binary mask.
     • In the GDH, the mask forces the generated anchor at step $k$ to align with $P_a$.
     • In the LDH, the mask ensures the interpolated trajectory strictly passes through the constrained anchors.
   • This allows the model to "steer" the trajectory toward the new target while maintaining the smoothness and task logic learned from demonstrations.

C. Closed-Loop Inference

The model operates recurrently at every execution step:

• It continuously updates the anchor history with executed actions.
• It re-evaluates the scene and the referring point.
• It replans the remaining trajectory online, enabling real-time adaptation to dynamic changes (e.g., moving obstacles).

D. Training Strategy

• Data Efficiency: ReV requires no additional error data or fine-tuning. It is trained solely on expert demonstrations.
• Augmentation: During training, the system applies targeted perturbations to expert actions to create synthetic "referring actions." This teaches the model to handle a wide range of potential referring points without needing explicit examples of every possible error.
• Loss Functions: A combined loss of Categorical Cross-Entropy (for temporal position prediction) and Mean Squared Error (for trajectory generation) supervises the coupled heads.

3. Key Contributions

1. Referring-Aware Policy: A novel framework that enables imitation learning models to handle OOD situations by integrating sparse 3D referring cues, effectively bridging high-level reasoning and low-level control.
2. Coupled Diffusion Architecture: A dual-head design (Global + Local) that successfully balances long-horizon consistency with fine-grained, adaptive trajectory generation.
3. Trajectory-Steering via Masked Denoising: A method to inject spatial constraints into diffusion models without retraining, allowing for precise, real-time trajectory replanning.
4. Zero-Shot Adaptation: The model achieves robustness and generalization without requiring additional training data or complex post-processing fine-tuning.

4. Experimental Results

The authors evaluated ReV on both simulated benchmarks (RoboFactory, Adroit, DexArt, MetaWorld) and real-world dual-arm tasks.

• Referring-Awareness (RePR): ReV achieved 100% Region Penetration Rate across all tasks, meaning the robot consistently passed through the designated referring points. In contrast, baseline methods (ACT, DP3, CDP, MPD) largely ignored the referring points, resulting in penetration rates often below 50% or 0%.
• Success Rate (SuR): ReV significantly outperformed baselines in success rates, particularly in tasks requiring obstacle avoidance or error recovery (e.g., "Lift Barrier-via" and "Pick Meat-via").
• Generalization to OOD: When tested with referring points deviating significantly from the expert distribution (OOD-yet-feasible), ReV maintained high success rates (87-93%), demonstrating strong generalization.
• Real-World Deployment: In real-world experiments with dual arms, ReV achieved success rates of 20/30 to 21/30 across various complex tasks (e.g., stacking cards, handing over objects), far surpassing baselines which often failed to complete the task or pass the via-point.
• Ablation Studies:
  • The Coupled Diffusion Heads were proven essential for global consistency.
  • The Learnable LDH outperformed fixed interpolation or optimization-based methods, proving that adaptive densification is crucial for robotic manipulation.
  • The Temporal-Position Prediction module was shown to be robust, provided it is not updated excessively during a single trajectory generation to avoid anchor drift.

5. Significance and Impact

• Bridging the Gap: ReV effectively bridges the gap between high-level semantic reasoning (which can identify where to go) and low-level visuomotor control (which executes how to go), without requiring the heavy data overhead of training on error states.
• Safety and Robustness: By enabling robots to instantly replan trajectories based on sparse human or planner input, ReV enhances the safety and reliability of robotic systems in dynamic, unstructured environments.
• Practicality: The method's ability to work with standard expert demonstrations and require no fine-tuning makes it highly deployable in real-world scenarios where collecting massive error datasets is infeasible.
• Future Direction: The work suggests a path toward more flexible embodied agents that can collaborate with humans and AI planners to handle unforeseen physical constraints dynamically.