DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

1. Problem Statement

Bimanual dexterous manipulation requires complex coordination between two hands to perform tasks that often involve contact-rich interactions (e.g., assembly, inserting objects). While recent Vision-Language-Action (VLA) models and diffusion policies have shown promise, they face significant limitations:

• Hardware Constraints: Rigid grippers lack the dexterity and tactile feedback of human hands.
• Modality Fusion Issues: Existing approaches often fuse vision, proprioception, and tactile signals in a coupled manner (treating them equally). This fails to account for the distinct roles of each modality (e.g., vision changes rapidly with active cameras, while tactile signals are sparse and critical only during contact).
• Data Scarcity: There is a lack of large-scale, real-world datasets for bimanual dexterous manipulation that include tactile sensing.
• Efficiency: Integrating tactile data often requires retraining entire models from scratch, which is computationally expensive and inefficient.

2. Methodology: The DECO Framework

The authors propose DECO, a Decoupled Multimodal Diffusion Transformer (DiT) paradigm designed to handle heterogeneous sensory inputs efficiently. The framework operates on a two-stage training paradigm:

A. Core Architecture: Multimodal Diffusion Transformer (MMDiT)

Instead of fusing all modalities into a single stream, DECO decouples their conditioning mechanisms:

• Vision & Action (Joint Self-Attention): Visual tokens (from binocular images) and action tokens interact via joint self-attention. This allows visual information to directly guide action generation, acknowledging vision's dominant role in global planning.
• Proprioception & Conditions (Adaptive Layer Normalization - AdaLN): Robot joint states (proprioception) and task conditions are injected via AdaLN. This modulates the network's internal states (scale, shift, gate) without disrupting the attention structure.
• Tactile Signals (Cross-Attention): Tactile data is processed through a dedicated cross-attention module. This allows the model to attend to tactile cues specifically when relevant (e.g., during contact) without altering the core self-attention mechanism of the vision-action backbone.

B. Plugin Tactile Adapter

To incorporate tactile sensing into a pretrained vision-based policy without full retraining, the authors introduce a lightweight Plugin Tactile Adapter:

• Mechanism: It uses a tactile encoder to process raw tactile data (from 17 pads per hand) into embeddings. These embeddings are injected into the frozen pretrained policy via LoRA (Low-Rank Adaptation) applied to the cross-attention layers.
• Efficiency: Only the adapter parameters (approx. 7.97M parameters, <10% of the total model) are trainable. The backbone remains frozen, enabling rapid adaptation to tactile-aware behaviors.

3. Key Contributions

1. DECO Model: A novel DiT architecture that decouples multimodal conditioning. By treating vision, proprioception, and touch through specialized pathways, it achieves structured and controllable integration, outperforming coupled fusion baselines.
2. Plugin Tactile Adapter: A parameter-efficient method to upgrade existing vision-based policies with tactile capabilities. It demonstrates that tactile information can be effectively added to pretrained models with minimal overhead (<10% trainable parameters).
3. DECO-50 Dataset: A large-scale, real-world bimanual dexterous manipulation dataset collected via teleoperation on a dual-arm robot (Unitree H1-2 with Inspire RH56DFTP hands).
   • Scale: ~50 hours of data, ~5 million frames, 8,000 successful trajectories.
   • Content: 4 scenarios (Pick & Place, Material Sorting, Waste Disposal, Assembly) and 28 sub-tasks.
   • Uniqueness: It is one of the few datasets combining bimanual dexterity, active vision, and high-resolution tactile sensing.

4. Experimental Results

The authors evaluated DECO on the DECO-50 dataset and conducted over 2,000 real-world robot rollouts.

• Overall Performance: DECO achieved an average success rate of 72.25% across all tasks, a 21% improvement over the best baseline (Diffusion Policy).
• Impact of Tactile Adapter: Adding the tactile adapter (DECO.p) increased the average success rate by an additional 10.25% compared to the vision-only DECO.
• Task-Specific Analysis:
  • Weakly Tactile Tasks (Pick & Place, Sorting): Vision and proprioception were sufficient for high performance. Tactile provided marginal gains here.
  • Strongly Tactile Tasks (Waste Disposal, Assembly): These tasks involve contact-rich phases (e.g., closing a lid, inserting a plug).
    • Waste Disposal: Tactile helped distinguish between "lid closed" vs. "lid open," preventing premature hand withdrawal. Success rate on the final stage improved significantly.
    • Assembly: Tactile was crucial for detecting contact during insertion and monitoring force to avoid slipping or damaging parts. DECO.p achieved 55/80 success on assembly tasks vs. 37/80 for the vision-only version.
• Efficiency: The Plugin Tactile Adapter achieved performance comparable to training a model from scratch with tactile data (DECO.ds) but required updating only 7.97M parameters (vs. 83M+ for full training).

5. Significance and Impact

• Paradigm Shift: The paper challenges the "coupled fusion" approach in robotics, demonstrating that decoupled, modality-specific injection yields superior performance, particularly when modalities have different temporal and spatial characteristics.
• Practical Deployment: The "Plugin Adapter" approach solves a major bottleneck in robotics: the high cost of retraining large foundation models for new sensors. It allows researchers to upgrade existing vision-based policies with tactile capabilities cheaply and quickly.
• Community Resource: The release of DECO-50 fills a critical gap in the availability of real-world, tactile-rich bimanual data, enabling further research into contact-rich manipulation and tactile representation learning.
• Future Direction: The work suggests that for complex, contact-heavy tasks, tactile sensing is not just a bonus but a necessity, and that efficient integration mechanisms are key to scaling these capabilities in real-world robots.