BiGraspFormer: End-to-End Bimanual Grasp Transformer

1. Problem Statement

Bimanual grasping is critical for robots to manipulate large, heavy, or unwieldy objects that exceed the capabilities of single-arm systems. However, existing approaches face significant limitations:

• High Dimensionality: Bimanual grasping expands the action space from 6-DoF (single-arm) to 12-DoF, doubling computational complexity.
• Coordination Challenges: Simply combining two independent single-arm grasps often leads to collisions, unbalanced force/torque distribution, and instability.
• Modular Limitations: Current methods typically separate grasp generation and bimanual evaluation into distinct stages. This modular pipeline (e.g., generating single grasps first, then pairing them via heuristics or separate scoring modules) results in limited diversity, poor coordination, and high inference latency.

2. Methodology: BiGraspFormer

The authors propose BiGraspFormer, a unified, end-to-end Transformer framework that directly generates coordinated bimanual grasps from object point clouds. The core innovation is the Single-Guided Bimanual (SGB) strategy, which decomposes the complex 12-DoF search space into a sequence of tractable subproblems.

A. Network Architecture

The framework consists of four main modules:

1. Object Encoder:
   • Uses PointNet++ to extract local geometric features (crucial for contact points).
   • Employs a Transformer Encoder with self-attention to capture global object structure and spatial relationships between distant regions.
   • Outputs a set of global object features ($F'_g$) serving as the backbone.
2. Single Grasp Proposer (SGP):
   • Generates diverse 6-DoF single-arm grasp candidates ($\hat{G}$) that satisfy basic force-closure constraints.
   • Uses a DETR-style set prediction with learnable object queries and cross-attention to the object features.
3. Bimanual Pair Matcher (BPM) [Training Only]:
   • Acts as a supervisor during training. It takes the diverse single grasps from the SGP and constructs reliable ground-truth bimanual pairs.
   • It evaluates pairs using metrics from the DA2 dataset (force closure, dexterity, torque balance) and performs collision checking using the object CAD model.
   • Note: BPM is not used during inference; it is replaced by the direct prediction capability of the BGG.
4. Bimanual Grasp Generator (BGG):
   • The core inference module that refines single-grasp features into final 12-DoF bimanual poses.
   • SGB Attention Mechanism: This is the key novelty. Bimanual queries ($Q_b$) attend to the single-grasp features ($F_{sgp}$) from the SGP as keys and values.
   • This allows the model to condition the generation of the dual-arm pose on the learned features of individual grasps, effectively guiding the search for a coordinated pair without explicitly pairing them post-hoc.
   • Outputs a set of bimanual candidates with predicted poses and quality scores.

B. Training Strategy

• Loss Functions: Uses a combination of regression loss ($L_{dist}$) for grasp pose alignment and $L_1$ loss ($L_{quality}$) for quality score prediction.
• Matching: Employs the Hungarian algorithm for bipartite matching between predictions and ground truth.
• Data: Trained on the DA2 dataset, utilizing a subset of 3,417 objects.

3. Key Contributions

1. First Unified End-to-End Framework: BiGraspFormer is the first model to directly generate coordinated bimanual grasps from point clouds without separate generation and evaluation modules.
2. Single-Guided Bimanual (SGB) Strategy: A novel attention mechanism that leverages single-arm grasp features to guide bimanual generation. This reduces the complexity of the 12-DoF search space while ensuring coordination.
3. State-of-the-Art Performance: The method achieves superior success rates and diversity compared to existing modular approaches.
4. Real-Time Efficiency: The framework maintains an inference speed of < 0.05 seconds, making it suitable for real-world deployment.

4. Experimental Results

Simulation Experiments

Evaluated on the DA2 dataset using the Isaac Gym simulator under Normal Force and Disturbance Force (dropping a weighted cube) conditions.

• Success Rate: BiGraspFormer significantly outperformed baselines (e.g., GPG+DPN-GPD, CGDF+Pairing).
  • Disturbance Condition: Achieved 59.72% Top-1% success rate vs. ~36% for the next best method.
  • Normal Condition: Achieved 89.67% Top-1% success rate vs. ~76% for the next best.
• Diversity: The method generated grasps covering a wider area of the object surface (Top-50% diversity score of 37.99% in disturbance vs. ~18% for baselines).
• Speed: Inference time was 0.04s, compared to 4.39s–18.18s for modular baselines that require separate pairing stages.

Real-World Experiments

• Setup: Dual UR5e arms with Robotiq 2F-140 grippers and Azure Kinect cameras.
• Objects: Tested on 8 diverse objects (chairs, bins, frames, etc.) in various poses.
• Performance: Achieved high success rates (e.g., 10/10 on a toy box and green bin; 8/10 on a yellow chair).
• Failure Analysis: Failures were primarily due to torque imbalance causing slippage or heavy objects (yellow stair) exceeding stability limits, rather than collision or planning errors.

Ablation Studies

• SGB Effectiveness: Removing the SGB attention layer (direct generation) or the BPM training stage resulted in significant drops in success rate and diversity, confirming the necessity of the single-guided approach.
• Query Count: Increasing the number of bimanual queries improved performance, but the model achieved SOTA results even with fewer queries (256) compared to baselines generating 512 pairs, highlighting the efficiency of the SGB decomposition.

5. Significance and Limitations

Significance:

• BiGraspFormer bridges the gap between single-arm and bimanual grasping by treating coordination as an intrinsic part of the generation process rather than a post-processing step.
• It demonstrates that leveraging single-arm features as a "guide" is a more efficient and robust strategy than treating the 12-DoF space as a monolithic problem.
• The real-time inference speed enables practical application in dynamic environments.

Limitations & Future Work:

• Training Dependency: The current training pipeline relies on CAD models for collision checking and ground truth generation (via BPM), which limits applicability in scenarios lacking high-quality meshes.
• Sim-to-Real Gap: While successful, real-world failures still occur due to sensor noise and kinematic constraints not fully captured in simulation.
• Future Direction: The authors plan to incorporate learned collision proxies to remove CAD dependency and integrate action-policy-aware grasping for full manipulation tasks.