Optimization of sim-to-real transfer in the humanoid robot NICO

Technical Summary: Optimization of Sim-to-Real Transfer in the Humanoid Robot NICO

Problem Statement
Robotic grasping requires the precise coordination of visual perception, object localization, inverse kinematics (IK), and hand control. A significant barrier to autonomous manipulation is the "sim-to-real" gap: movements planned in simulation often fail on physical hardware due to mechanical inaccuracies, actuator imperfections, sensor noise, and simplified model assumptions. While previous work by the authors successfully improved 2D reaching accuracy for the humanoid robot NICO using a low-cost haptic calibration method, extending this success to complex tabletop object grasping remains challenging. Grasping introduces additional variables, including object detection, depth estimation, and the need for reliable execution where small residual positioning errors can cause task failure.

Methodology
The authors propose a complete, low-cost grasping pipeline that relies solely on the robot's built-in low-resolution fisheye stereo cameras, eliminating the need for RGB-D cameras, motion capture, or external tracking systems. The pipeline integrates the following components:

1. Calibration-Based Correction Models: Building on prior work, the system employs three correction models to map estimated real-world object positions to simulated target coordinates:

   • M1 (Baseline): A piecewise linear mapping based on manual alignment and height interpolation.
   • M2 (Partially Nonlinear): Uses a Multilayer Perceptron (MLP) for horizontal coordinates while retaining interpolation for height.
   • M3 (Fully Nonlinear): A neural network that predicts the corrected 3D target position directly.
     These models were not retrained for this specific task but applied as correction modules to evaluate their efficacy in a grasping context.

2. Vision Module:

   • Object and Hand Detection: Utilizes the YOLO framework (specifically YOLO12s, selected for its balance of accuracy and speed) trained on a custom dataset of 442 images. The detector identifies six object classes, the robot's left hand, and right hand.
   • Stereo Vision Localization: After detection, stereo vision estimates the 3D position of the object relative to the robot. The system addresses localization drift (caused by camera overheating) by applying a correction that shifts predicted positions toward a fixed pivot point.

3. Visual Feedback Alignment (M4): As an alternative to calibration-based correction, the authors implemented a closed-loop visual feedback model. This approach:

   • Moves the hand to an approximate position above the object.
   • Re-detects the hand and object using stereo vision.
   • Iteratively adjusts the IK target using the formula $p^k_{IK} = p^{k-1}_{IK} + a_k(p_{obj} - p_{hand})$, where the step size $a_k$ decreases to prevent oscillation.
   • Continues until the horizontal distance between the hand and object is below 1 cm, at which point the grasp is executed.

4. Task-Specific Corrections: The pipeline includes manual measurements and Radial Basis Function (RBF) interpolation to correct vertical (z-axis) targets and palm yaw angles, ensuring a top-down grasping pose.

Key Results
The system was evaluated on a plush tomato placed across 22 grid positions in the robot's reachable workspace, with three attempts per position for each model (66 total attempts per model).

• Calibration Models:

  • M1 (Linear): Achieved a low overall success rate of 16.7% (11/66).
  • M3 (Fully Nonlinear): Demonstrated superior performance within the calibrated region (the "NN Correction Area"), achieving a 96.7% success rate (29/30). However, its performance dropped significantly outside this area (57.6% overall), indicating poor extrapolation capabilities. The mean prediction deviation increased from 2.14 cm inside the training area to 5.45 cm outside it.

• Visual Feedback Model (M4):

  • Achieved the highest overall success rate of 72.7% (48/66) across the entire workspace.
  • Unlike M3, its performance was not concentrated solely in the calibrated area; successful grasps occurred throughout the tabletop.
  • When stereo vision successfully localized the hand (which occurred in 17 of 22 positions), the success rate for those specific attempts rose to 94.1% (48/51).
  • The primary limitation was identified as stereo vision reliability for hand localization, particularly when the background behind the hand was distant, leading to overestimation of hand position.

Significance and Claims
The paper claims that the proposed pipeline offers a robust, low-cost solution for sim-to-real transfer in humanoid grasping without relying on expensive sensors. The authors highlight two main findings:

1. Transferability of Calibration: Haptic calibration methods optimized for 2D reaching can effectively improve object grasping, particularly within the calibrated workspace, with the fully nonlinear model (M3) achieving near-perfect performance in that specific region.
2. Robustness of Visual Feedback: The visual feedback model (M4) provides a more generalizable solution for the full tabletop workspace. It compensates for residual localization and execution errors, achieving the highest overall success rate without requiring the model to extrapolate beyond its training data.

The work demonstrates that a combination of object detection, stereo vision, and either calibration-based correction or visual feedback can overcome the sim-to-real gap in a constrained, low-cost environment, offering a practical alternative to complex learning-based policies that require large datasets or depth sensors.