When would Vision-Proprioception Policies Fail in Robotic Manipulation?

1. Problem Statement

While proprioceptive information (internal robot states like joint angles and gripper opening) is critical for low-level control, recent studies have shown inconsistent results when combining it with vision in learning-based manipulation policies. Some works report performance gains, while others (e.g., Octo) observe that adding proprioception degrades performance compared to vision-only policies.

The authors identify the root cause of this failure through Modality Temporality:

• Motion-Consistent Phases: During continuous movement (e.g., moving forward), proprioception is highly effective, and vision provides redundant or noisy data.
• Motion-Transition Phases: During critical transitions (e.g., locating a target, grasping, or assembling), the robot must rely on visual cues for accurate localization. However, visual changes during these phases are often subtle (pixel-level), while proprioceptive signals are concise and low-dimensional.
• The Failure Mechanism: During training, the policy naturally gravitates toward the "easier" proprioceptive signals to minimize the loss function quickly. This leads to optimization dominance by proprioception, which suppresses the learning of the visual modality. Consequently, during motion-transition phases in testing (where object positions vary), the policy fails because it has not learned to utilize vision effectively.

2. Methodology: Gradient Adjustment with Phase-guidance (GAP)

To address the suppression of the visual modality, the authors propose GAP, an algorithm that adaptively modulates the optimization of proprioceptive features based on the estimated phase of the robot's motion.

A. Motion Representation & Phase Estimation

1. Motion Definition: The robot's motion is defined by changes in position ($p$), orientation ($\theta$), and gripper opening ($g$).
2. Motion-Consistent Segmentation: Using Change Point Detection (CPD), the trajectory is segmented into "motion-consistent phases" where the direction of motion is stable.
3. Transition Probability Estimation: Since transitions are continuous and CPD provides discrete labels, the authors employ an LSTM to process temporal differences in proprioceptive signals ($\Delta s$). This network predicts a phase indicator ($\rho$), representing the probability that a specific timestep belongs to a "motion-transition phase."

B. Gradient Adjustment

During policy learning (Behavior Cloning), the authors apply a fine-grained gradient adjustment to the proprioceptive feature chunk ($\omega_s$).

• Mechanism: In motion-transition phases (where $\rho$ is high), the magnitude of the backward gradient for proprioception is reduced.
• Formula:
  $$\omega_s^{j+1} = \omega_s^j - \lambda \cdot (1 - \rho) \cdot \eta \nabla_{\omega_s^j} L_{BC}$$
  Where $\lambda$ controls the adjustment degree, $\eta$ is the learning rate, and $\rho$ is the transition probability.
• Effect: By dampening the proprioceptive gradient during transitions, the optimization is forced to rely more on the visual modality ($\omega_v$) to minimize the loss, thereby preventing the visual features from being suppressed.

3. Key Contributions

1. Diagnosis of Failure: The paper provides a rigorous analysis demonstrating that vision-proprioception policies fail specifically during motion-transition phases due to the suppression of visual learning by proprioceptive optimization dominance.
2. GAP Algorithm: A novel, plug-and-play gradient adjustment method that dynamically balances modality contributions without requiring architectural changes to the policy backbone.
3. Versatility: The method is compatible with various policy architectures (MLP, Diffusion, Transformer), fusion strategies (Concatenation, Summation, FiLM), and model types (Conventional BC and Vision-Language-Action models like Octo).
4. Comprehensive Validation: Extensive experiments in both simulated (Meta-World, RoboSuite) and real-world (single-arm and dual-arm) environments.

4. Experimental Results

The authors evaluated GAP across diverse tasks including pick-and-place, assembly, threading, and dual-arm handovers.

• Performance Improvement:
  • In Meta-World, GAP-equipped policies achieved success rates of 94.2% (pick-place) and 94.2% (assembly), significantly outperforming standard concatenation (78.4% and 74.6%) and even surpassing vision-only policies.
  • In RoboSuite, GAP improved success rates on difficult tasks like "threading" from 33.2% (concatenation) to 53.0%.
  • In Real-World experiments (xArm 6), GAP achieved 20/20 success on "press button" and 18/20 on "handover," outperforming all baselines.
• Out-of-Distribution (OOD) Generalization:
  • Vision-proprioception policies without GAP suffered severe performance drops in OOD settings (e.g., changing object positions).
  • GAP restored generalization, achieving 88% on "assembly" (vs. 62% for concatenation) and 15/20 on real-world "handover" tasks under distribution shifts.
• Compatibility with VLA Models:
  • When applied to the Octo model (a Vision-Language-Action model), GAP reversed the typical trend where proprioception hurts performance. Octo-VP with GAP achieved an average 17% improvement over Octo-V (vision-only) and outperformed the vision-only baseline.
• Intervention Experiments:
  • Switching to a GAP-equipped policy during motion-transition phases resulted in a +11% to +13% success rate increase, confirming that GAP successfully enhances the utilization of the visual modality exactly when it is needed most.

5. Significance

This work offers a critical insight into the development of multimodal robotic policies: multimodal fusion is not just about concatenating features; it requires managing the temporal dynamics of modality reliance.

• Theoretical Insight: It challenges the assumption that adding proprioception always helps, showing that unregulated joint training can lead to "modality competition" where one modality suppresses the other.
• Practical Impact: GAP provides a simple, effective solution to unlock the potential of proprioception without sacrificing visual generalization. It enables robots to perform complex, contact-rich, and precision tasks (like threading or assembly) in dynamic, real-world environments where object positions are not fixed.
• Future Direction: The paper suggests that future research should focus on "modality temporality" and dynamic collaboration strategies rather than static fusion architectures.