Technical Summary: Optimizing Grasping in Legged Robots via Deep Learning

Problem Statement

Quadruped robots equipped with manipulator arms offer significant potential for loco-manipulation in unstructured and complex terrains where wheeled robots fail. However, achieving precise and adaptable grasping in these dynamic scenarios remains a critical challenge. Traditional approaches often rely on rigid kinematic models, pre-programmed grasp configurations, or extensive real-world calibration, which limits flexibility and scalability. Furthermore, the coordination between locomotion and manipulation in unpredictable environments typically requires complex, context-specific solutions. The core problem addressed is the need for a robust, data-driven framework that enables quadruped robots to autonomously identify optimal grasping points on objects without relying on massive physical data collection or manual calibration.

Methodology

The proposed solution is a deep learning framework utilizing a sim-to-real methodology to minimize reliance on physical data. The approach integrates advanced perception, simulation, and robotic control into a cohesive pipeline.

1. Synthetic Dataset Generation (Genesis Simulation)

To overcome the scarcity of real-world labeled data, the authors developed a pipeline within the Genesis simulation environment:

• Scenario: A 3D model of a water bottle was placed at a fixed point.
• Data Collection: A virtual RGB-D camera sampled 1,000 distinct viewpoints (100 points on the X-axis, 10 on the Z-axis) around the object.
• Grasp Labeling: For every pixel in the object's segmentation mask, a robotic end-effector attempted a grasp. The pose was defined by the pixel's global coordinates and surface normal.
• Ground Truth: A grasp was labeled successful (1) if the gripper collided with the object without external collisions (e.g., ground contact) and failed (0) otherwise. Pixels outside the object were marked as indeterminate (-1). This generated pixel-wise annotated grasp-quality maps.

2. Deep Learning Architecture

A custom Convolutional Neural Network (CNN) with a U-Net-like encoder-decoder architecture was trained to predict optimal grasp points.

• Inputs: Multi-modal data including RGB images, depth maps, segmentation masks, and surface normal maps (480x640 resolution).
• Encoder: Based on MobileNetV2 for efficient feature extraction.
• Decoder: Utilizes transposed convolutions for upsampling and skip connections to merge fine-grained features for precise localization. GroupNorm is used for training stability.
• Output: A single-channel heatmap where the pixel with the highest value indicates the optimal grasp point. The model contains approximately 5.44 million trainable parameters.

3. Real-World Deployment Pipeline

The system was validated on a Boston Dynamics Spot quadruped robot equipped with a 6-DOF manipulator and a jaw gripper. The execution pipeline involves:

1. Navigation: The robot autonomously navigates to a target object (a thermos bottle) and positions itself 1.0m away.
2. Perception: The end-effector's onboard RGB and depth cameras capture synchronized data.
3. Processing:
   • Object Detection: A pre-trained YOLOv11 model segments the object and generates a binary mask.
   • Normal Estimation: The D2NT algorithm converts the depth map into a surface normal map using approximate camera intrinsics.
   • Inference: The pre-processed multi-modal inputs (RGB, depth, normal, mask) are fed into the trained CNN to generate a grasp-quality heatmap.
4. Execution: The system selects the optimal pixel, calculates 3D coordinates and orientation (converting normals to quaternions), and commands the manipulator via the Spot SDK to grasp the object with force-limited control to prevent slippage or damage.

Key Contributions

The paper outlines four primary contributions:

1. Simulation-Based Training Pipeline: A Genesis-based framework enabling massive parallel data collection for grasp quality mapping without requiring real-world data.
2. Advanced Perception Integration: The incorporation of D2NT for depth-based grasping to improve precision and reduce computational costs associated with machine learning execution.
3. Flexible Framework Development: A system designed to integrate with high-level control APIs and commercial robots (like Spot) that lack low-level access, facilitating broader applicability.
4. Physical Validation: Successful demonstration of the complete loco-manipulation task on a physical quadruped robot, proving the efficacy of the sim-to-real approach in real-world scenarios.

Experimental Results

The framework was successfully tested on a Boston Dynamics Spot robot tasked with grasping a generic thermos bottle.

• Autonomy: The robot autonomously navigated to the target, perceived the object, predicted the optimal grasp pose, and executed a precise grasp.
• Performance: The system demonstrated the ability to handle the full loco-manipulation cycle, from navigation to force-controlled grasping.
• Robustness: The use of force-limited control (max torque 3.0 Nm) allowed the gripper to adapt to minor misalignments and surface irregularities without excessive force application.

Significance and Limitations

The authors claim this work represents an important advancement in quadruped robotics by integrating deep learning, advanced simulation, and robotic control to create scalable and effective solutions for object handling. The approach offers a pathway toward autonomous, versatile systems capable of operating in unpredictable environments with optimized processing speeds.

However, the paper maintains a modest tone regarding limitations:

• Generalization: The model's generalization is currently limited by variations in object geometry and texture, restricting effectiveness to objects with specific characteristics (e.g., cylindrical, smooth surfaces).
• Sensor Noise: Real-world deployment faced challenges with noise in depth maps due to the low quality of the end-effector's sensor, which occasionally compromised pre-processing consistency.
• Future Directions: The authors propose expanding the training dataset to include diverse shapes and textures, implementing depth denoising filters, and testing in more complex environments with multiple objects to further validate scalability.

In conclusion, the paper demonstrates that leveraging simulated training with advanced sensing provides a viable alternative to extensive physical data collection for enabling loco-manipulation in quadruped robots.