Technical Summary: Seeing Touch from Motion

1. Problem Statement

Contact-rich manipulation tasks (e.g., precision insertion, object wiping) require robots to perceive subtle physical interactions that global vision alone cannot capture. While optical tactile sensors offer high spatial resolution and multi-dimensional force sensing by monitoring the deformation of an elastic gel surface, extracting fine-grained contact states (e.g., distinguishing between "making contact" and "releasing contact") remains a significant challenge.

Existing methods typically rely on two representations that suffer from perception ambiguity:

1. Raw Tactile Images: These capture appearance changes but often exhibit deceptive similarities between distinct contact states (e.g., a stable contact and a releasing contact may look visually similar).
2. Cumulative Motion Fields: These represent the aggregate gel deformation relative to an initial non-contact frame. While they reflect overall deformation magnitude and direction, they fail to distinguish subtle variations, such as the difference between initiating contact and terminating it, as both can result in similar cumulative deformation patterns.

Furthermore, existing visuo-tactile fusion strategies struggle to balance cross-modal interaction with modality-specific preservation. Feature-level fusion (e.g., concatenation) often allows one modality to dominate, while decision-level fusion (training separate policies) fails to exploit cross-modal cues effectively.

2. Methodology

The paper proposes ViTacMotor, a unified yet modality-aware visuo-tactile policy that addresses these issues through two core innovations: a novel tactile representation and a specialized fusion architecture.

2.1 Tactile Motion Correlation (TMC)

The authors discover that the correlation between transient motion (instantaneous deformation between consecutive frames) and cumulative motion (aggregate deformation from the initial non-contact state) explicitly distinguishes fine-grained contact states.

• Transient Motion ($M_{tran}$): Optical flow between the previous frame ($T_{t-1}$) and current frame ($T_t$).

• Cumulative Motion ($M_{cumu}$): Optical flow between the initial non-contact frame ($T_0$) and the current frame ($T_t$).

• Correlation Mechanism: The method computes the pixel-wise dot product of these two motion fields:
  $$Corr_t(x) = M_{tran}^t(x) \cdot M_{cumu}^t(x)$$

  Physical Interpretation:

  • Making Contact: Both motions align in the same direction $\rightarrow$ Positive dot product.
  • Releasing Contact: Motions point in opposite directions (gel recovering) $\rightarrow$ Negative dot product.
  • Stable/No Contact: Transient motion is near zero $\rightarrow$ Dot product is near zero.
  • Sliding Contact: The dot product exhibits a spatial gradient (negative to positive) indicating sliding direction.

  This representation is sensor-agnostic (validated on both dense and sparse marker sensors) and the magnitude of the dot product is positively correlated with contact force, providing richer cues than raw appearance or cumulative motion alone.

2.2 Unified Modality-Aware Visuo-Tactile Policy (ViTacMotor)

To integrate this new tactile representation with global vision, the authors employ a Mixture-of-Transformers (MoT) architecture.

• Modality-Aware Design: Unlike standard fusion that treats all inputs uniformly, ViTacMotor treats tactile motion correlation as a distinct modality from tactile appearance and visual inputs.
• Architecture:
  1. Encoders: Separate encoders process visual inputs (wrist/third-view), raw tactile frames, and the TMC representation. Proprioceptive states are also encoded.
  2. Modality-Specific Projections: Each modality $m$ has unique learnable parameters ($W_Q^{(m)}, W_K^{(m)}, W_V^{(m)}$) to preserve its unique characteristics.
  3. Shared Global Self-Attention: All modality embeddings are processed through a shared self-attention mechanism to facilitate cross-modal complementarity.
  4. Modality-Specific Refinement: Post-attention, outputs pass through modality-specific Feed-Forward Networks (FFN) and Layer Normalization before being aggregated.
• Output: The fused embeddings are fed into a Transformer decoder to autoregressively predict action chunks, trained via a $\beta$-VAE objective to model stochasticity in human demonstrations.

3. Key Contributions

1. Tactile Motion Correlation (TMC): The paper reveals that the correlation between transient and cumulative motion explicitly resolves the ambiguity of fine-grained contact states. TMC is proposed as a robust, physically interpretable, and sensor-agnostic tactile representation.
2. ViTacMotor Framework: A unified policy leveraging the MoT architecture to achieve effective visuo-tactile fusion. It uniquely captures cross-modal interactions while preserving the specific properties of visual and tactile inputs.
3. Empirical Validation: Extensive experiments on four challenging real-world tasks demonstrate superior performance and robustness compared to state-of-the-art baselines.

4. Experimental Results

The method was evaluated on four contact-rich manipulation tasks using two types of optical tactile sensors (dense and sparse markers) and an Agilex robotic system.

Tasks:

• Tube Collection: Requires precise alignment and insertion into a rack (sensitive to occlusion).
• Lightbulb Insertion: Requires rotation and precise insertion until the light turns on.
• Whiteboard Erasing: Requires stable, controlled force to remove ink without damaging the surface.
• Pencil Sharpening: Requires alignment and sufficient downward force.

Performance Highlights:

• Success Rates: ViTacMotor outperformed baselines (ACT, Diffusion Policy, ACT+T, Policy Consensus, TactileACT) across all tasks.
  • Tube Collection: 73.3% success (vs. 60.0% for TactileACT).
  • Lightbulb Insertion: 40.0% success (vs. 40.0% for TactileACT, but with higher robustness in intermediate steps).
  • Whiteboard Erasing: 86.7% success (vs. 73.3% for TactileACT).
  • Pencil Sharpening: 60.0% success (vs. 46.7% for TactileACT).
• Ablation Studies:
  • Removing TMC caused significant performance drops, confirming its necessity for force control and state perception.
  • Replacing MoT fusion with simple concatenation or cross-attention resulted in lower performance, validating the effectiveness of the modality-aware architecture.
• Robustness: The method maintained high success rates under environmental variations (non-uniform/uniform lighting) and object variations (different erasers, replacing tubes with cylinders).

5. Significance and Claims

The paper claims that Tactile Motion Correlation provides a physically interpretable mechanism to explicitly distinguish contact states that are visually ambiguous in raw images or cumulative fields. By modeling the dynamic relationship between transient and cumulative motion, the system gains the ability to infer contact direction (making vs. releasing) and force magnitude.

Furthermore, the ViTacMotor framework demonstrates that effective visuo-tactile manipulation requires more than simple feature concatenation; it necessitates an architecture that respects modality-specific characteristics while enabling deep cross-modal interaction. The authors position this work as a step toward generalizable, contact-rich manipulation policies that can handle the subtle physical interactions required for real-world robotic tasks.

Note: The authors acknowledge limitations regarding generalization under extreme visual disturbances or significant object position variations, noting that failure cases under such conditions exist.