Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation

1. Problem Statement

Robotic insertion tasks (e.g., peg-in-hole) require precise, contact-rich interactions that are difficult to resolve using vision alone. While deep learning-based vision policies excel at global scene understanding, they fail to capture subtle physical interactions such as micro-slippage, compliance, and misalignment during contact. They are also sensitive to occlusion and lighting.

Although tactile sensing provides crucial local contact data, naïve fusion of vision and tactile inputs (e.g., simple feature concatenation) has historically yielded negligible or even negative performance gains. This is due to the difficulty of synchronizing heterogeneous representations and the risk of diluting modality-specific cues. The core challenge is to develop a fusion strategy that respects the distinct roles of vision (global alignment) and touch (local correction) while stabilizing tactile embeddings against physical inconsistencies.

2. Methodology

The authors propose a Symmetry-Aware Visuo-Tactile Fusion Framework centered around a Cross-Modal Transformer (CMT) and a Physics-Informed Regularization mechanism.

A. Cross-Modal Transformer (CMT) Architecture

Instead of naïve concatenation, the CMT employs hierarchical attention mechanisms:

1. Intra-Modal Self-Attention (Tactile): Tactile signals from the left and right gripper fingers are first processed through self-attention. This enforces coherence within the tactile modality before fusion, normalizing for potential asymmetries.
2. Cross-Modal Attention: The architecture treats Vision as the Query (providing global context for alignment) and Tactile as Key/Value (providing fine-grained corrective feedback). This asymmetry ensures that visual guidance directs the policy, while tactile data refines the action based on physical contact states.

B. Physics-Informed Symmetry Regularization

Inspired by human motor control principles (bilateral force balance), the authors introduce an auxiliary loss function to stabilize tactile embeddings:

• Concept: In a stable grasp, forces on the left and right fingers should be symmetric (or balanced relative to a calibrated reference).
• Implementation: The right tactile map is vertically flipped and encoded ($\tilde{h}^R_t$). A Mean Squared Error (MSE) loss is computed between the left embedding ($h^L_t$) and the flipped right embedding.
• Objective: $L_{sym} = E[\|h^L_t - \tilde{h}^R_t\|^2_2]$.
• Effect: This acts as an inductive bias that penalizes asymmetric grasp forces, reducing lateral misalignments and preventing "jamming" during insertion. It forces the policy to learn balanced contact states before attempting insertion.

C. Training Framework

The system is trained as a Partially Observable Markov Decision Process (POMDP) using Proximal Policy Optimization (PPO). The total loss combines the standard PPO objective with the symmetry regularization term:
$$\mathcal{L} = \mathcal{L}_{PPO} + \lambda_{sym} \mathcal{L}_{sym}$$

3. Key Contributions

1. Methodological Novelty (CMT): Introduction of a Cross-Modal Transformer that uses hierarchical self- and cross-attention to integrate global visual context with local tactile feedback, addressing synchronization issues inherent in naïve fusion.
2. Physics-Informed Regularization: Development of a bilateral force-symmetry constraint derived from human motor control. This encodes a physical prior directly into the learning process, stabilizing grasps and reducing insertion errors without requiring explicit state estimation.
3. Empirical Validation: Demonstration that structured fusion with symmetry priors significantly outperforms both naïve fusion and gated fusion baselines, achieving performance nearly identical to "privileged" sensing configurations (which have access to ground-truth contact forces).

4. Experimental Results

Experiments were conducted on the TacSL benchmark (robotic insertion) and an extended screw task.

• Insertion Success Rates:

  • Vision Only: 93.23%
  • Tactile Only: 91.41% (Demonstrating tactile sensing is independently robust).
  • Naïve Fusion: 92.97% (Confirming that simple concatenation fails to improve performance).
  • Gated Fusion: 94.53%
  • CMT (No Symmetry): 96.22%
  • CMT + Symmetry Regularization: 96.59%
  • Privileged "Wrist + Contact Force": 96.09%
  • Result: The proposed method (96.59%) surpasses the privileged baseline (96.09%) and significantly outperforms all other fusion strategies.

• Qualitative Improvements:

  • Visualizations of force fields show that naïve fusion leads to unbalanced left-right forces, causing oscillations and re-grasping.
  • The symmetry-aware CMT maintains balanced force distributions, resulting in straighter, smoother insertion trajectories with near-zero lateral torque.
  • Efficiency: The CMT reduces the average steps to success by ~2.8% compared to naïve fusion, despite a slight increase in inference latency (6.52ms vs 5.42ms), which remains well within real-time control limits (>150 FPS).

• Screw Task Extension:

  • In a screwing task, a tactile-only policy achieved 100% success, outperforming both vision-only and privileged contact-force baselines. This highlights that for highly contact-intensive tasks, high-resolution tactile feedback can serve as a complete modality.

5. Significance

• Redefining Multimodal Fusion: The paper proves that successful visuo-tactile fusion requires more than just combining data; it requires structured attention mechanisms and physics-based constraints.
• Bridging the Gap to Privileged Sensing: By using symmetry priors, the system achieves performance comparable to (and slightly better than) systems with access to privileged ground-truth force data, making it highly viable for real-world deployment where such privileged data is unavailable.
• Robustness: The framework demonstrates that tactile sensing is indispensable for precise alignment and can sustain task execution even when vision is degraded or occluded.
• Generalizability: While tested on symmetric objects, the residual force formulation allows the method to generalize to asymmetric objects through calibration, laying the groundwork for general-purpose robotic assembly policies.