Imitation of Arm Gestures by the Semi-Humanoid Robot NICO

Technical Summary: Imitation of Arm Gestures by the Semi-Humanoid Robot NICO

Problem Statement
Seamless Human-Robot Interaction (HRI) requires robots to possess perceptual and motor abilities, specifically the capacity to imitate human gestures. While learning-based approaches offer flexibility, they often demand large datasets, extensive training, and significant computational resources, limiting real-time applicability. Conversely, analytical approaches provide efficiency and interpretability but often rely on specialized sensing hardware (e.g., depth cameras) or are restricted to simplified gesture sets. The challenge addressed in this work is achieving meaningful imitation of human arm gestures using only monocular RGB input, without reliance on expensive sensors or computationally intensive learning models, while adapting to the specific kinematic structure of a semi-humanoid robot.

Methodology
The proposed system is a pipeline designed for the NICO (Neuro-Inspired COmpanion) robot, a child-size semi-humanoid with six degrees of freedom (DoF) in each arm. The framework operates through three primary stages:

1. 3D Coordinate Extraction:
   The system utilizes the MediaPipe framework to extract 2D and pseudo-3D landmarks from a single RGB camera input. It employs the pose_landmarker_heavy model for body pose (33 landmarks) and a dedicated hand model (21 landmarks). To enhance accuracy, the system prioritizes hand-model landmarks for the wrist and palm (replacing corresponding pose landmarks) and translates the hand coordinate system to align with the pose model's wrist origin. This provides 3D coordinates ($x, y, z$) for relevant arm joints and hand keypoints.

2. Joint Angle Reconstruction (Analytical Geometry):
   Using the extracted 3D coordinates, the system computes joint angles via derived geometric relations, assuming the subject's torso is approximately parallel to the camera plane. A torso yaw compensation is applied by rotating coordinates based on the shoulder line orientation. The reconstruction covers:

   • Wrist Flexion: Calculated as the angle between the forearm vector and the normal vector of the palm plane.
   • Forearm Rotation: Determined by the signed angle between a reference rotation base (derived from the upper arm vector) and the projected palm orientation vector. Special fallback logic is applied for straight-arm configurations.
   • Elbow Bend: Computed as the unsigned angle between the upper arm and forearm vectors.
   • Shoulder Pitch and Roll: Derived using atan2 functions on decomposed vector components to resolve quadrants and distinguish between front and back arm configurations. Shoulder yaw is fixed in a neutral position.

3. Angle Mapping to NICO:
   The reconstructed analytical angles are not directly compatible with the robot's motor coordinate systems. The system performs a linear mapping for each joint, clamping the reconstructed angle to a pre-determined analytical interval $[\theta_{min}, \theta_{max}]$ and mapping it to the corresponding motor positions $m(\theta_{min})$ and $m(\theta_{max})$. This handles both matching and reversed motor orientations.

Key Contributions

• RGB-Only Analytical Pipeline: The work extends analytical geometry-based methods to a broader set of arm gestures using only monocular RGB input, eliminating the need for depth sensors or motion capture systems.
• Kinematic Adaptation: The geometric formulations are specifically adapted to the NICO robot's kinematics, which differs from other platforms (e.g., NAO) by lacking an elbow roll joint but requiring specific handling of forearm rotation and wrist flexion.
• Integration of Hand Landmarks: The system explicitly integrates MediaPipe hand landmarks to improve the estimation of wrist flexion and forearm rotation, addressing limitations found when using pose landmarks alone.
• No Training Requirement: Unlike learning-based methods, the proposed approach requires no training stage or annotated datasets.

Experimental Results
The system was evaluated with six participants of varying heights performing 11 symmetric upper-limb poses at three different viewing angles ($0^\circ, 30^\circ, 45^\circ$).

• Accuracy: The method achieved moderate reconstruction accuracy for shoulder pitch, elbow bend, and wrist bend, with Mean Absolute Errors (MAE) of approximately $10^\circ$, $20^\circ$, and $23^\circ$, respectively. Shoulder roll showed lower accuracy ($34^\circ$ MAE).
• Forearm Rotation Limitation: Forearm rotation remained the primary source of error. Using pose landmarks alone resulted in a $67.1^\circ$ MAE. Incorporating dedicated hand landmarks reduced this to $50.1^\circ$ (a $\approx 25\%$ improvement), though the error remained significantly higher than for other joints.
• Wrist Dependency: Wrist bend accuracy was highly dependent on forearm orientation; errors increased noticeably for extreme wrist flexion/extension where forearm rotation estimation was poor.
• Robustness Factors: Reconstruction accuracy showed no clear correlation with participant height. Viewpoint variations ($0^\circ$–$45^\circ$) had a relatively small effect compared to the variability introduced by specific gestures and self-occlusion.
• Performance: The pipeline demonstrated high computational efficiency, averaging 0.0017 seconds per frame (approx. 590 FPS), with a worst-case of 0.0093 seconds (approx. 108 FPS).

Significance and Claims
The authors position this work as a "proof of concept" for analytical, monocular RGB-based gesture imitation. The paper claims that the approach successfully produces meaningful imitative motions without the computational overhead of deep learning or the cost of specialized hardware. However, the authors maintain a modest stance regarding the system's current robustness. They acknowledge that while the method works for several arm joints under controlled conditions, forearm rotation estimation remains a significant challenge. The paper concludes that the system is not yet a robust solution for general gesture imitation and requires further validation on larger, more diverse datasets and improvements in forearm rotation estimation before broader deployment.