Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids

1. Problem Statement

Understanding early human cognitive development (the sensorimotor stage) is critical for developmental science and robotics. However, current research faces significant limitations:

• External Perspective: Traditional analysis relies on third-person video observations, which fail to capture the infant's internal, first-person sensory experience (proprioception, touch, vestibular sense).
• Invasiveness: First-person data collection via head-mounted cameras is too invasive for dense, long-term developmental sampling.
• Data Scarcity: Neural recordings from infants are extremely difficult to obtain.
• Lack of Embodiment: Existing motion retargeting methods focus on reproducing kinematics (joint angles) for humanoid robots but ignore the rich, multimodal sensorimotor streams associated with human movement.

The authors propose a solution to bridge the gap between third-person video data and first-person sensory simulation by retargeting infant motions onto physical and virtual humanoid platforms.

2. Methodology

The proposed framework creates a pipeline to reconstruct an infant's 3D body configuration from a single video and map it onto four distinct humanoid embodiments.

A. 3D Pose Estimation

• Input: A single video of an infant (multi-view or single-view).
• Process:
  • Multi-view: 2D keypoints are extracted using ViTPose for each camera view, then triangulated to compute 3D coordinates.
  • Single-view: If multi-view is unavailable, the system fits a prior 3D model (e.g., SMIL for infants) to 2D projections using methods like Smplify-x.
• Output: A sequence of full 3D body poses (skeletal structure) for every frame.

B. Motion Retargeting to Embodiments

The reconstructed motion is retargeted to four specific platforms, each offering different sensory modalities and constraints:

1. iCub (Physical & Virtual pyCub): Modeled after a 4-year-old. Uses geometric motion retargeting (analytical method) to map 3D keypoints to joint angles, optimized for joint limits. It features 4,000 pressure-sensitive touch receptors, binocular vision, and vestibular sensors.
2. EMFANT (Embodied Model of Fetus And iNfanT): A high-fidelity musculoskeletal simulation in MuJoCo.
   • Morphology: Derived from infant CT/MRI data with realistic skin, bones, and muscle routing (17 segments, 37 DOF).
   • Retargeting: Uses OpenSim for inverse kinematics with biomechanical constraints, followed by PD control in MuJoCo for forward dynamics.
   • Sensors: 5,000 touch sensors, muscle-tendon length signals (162 muscles), and binocular vision.
3. MIMo: A simplified, fast-running MuJoCo model with geometric primitives (spheres/capsules).
   • Adaptability: Its morphology is scaled to match the specific infant's body dimensions (calibrated via average poses).
   • Retargeting: Uses MuJoCo's inverse kinematics solver with "puppeteering" (auxiliary mocap bodies) to minimize distance to target keypoints.
   • Sensors: Proprioception (joint angles/velocities/torques), vestibular (accelerometer/gyroscope), touch (virtual skin), and vision.

C. Sensorimotor Simulation

Once the motion is retargeted, the system simulates the first-person experience by recording:

• Proprioception: Joint angles, velocities, torques, and muscle-tendon lengths.
• Touch: Activation of virtual skin sensors upon self-contact or environmental contact.
• Vision: Binocular rendering of the scene from the infant's perspective.
• Vestibular: Acceleration and angular velocity data.

3. Key Contributions

1. First-Person Sensorimotor Framework: A novel pipeline that transforms passive third-person video into active, multimodal first-person sensory streams (vision, touch, proprioception, vestibular).
2. Multi-Platform Retargeting: Demonstrates the ability to map infant motion onto diverse embodiments (physical robot, high-fidelity musculoskeletal model, and simplified scalable model), allowing for comparative analysis.
3. Sub-Centimeter Accuracy: Achieves high-fidelity retargeting, particularly with MIMo, where the morphology is adapted to the infant, resulting in Mean Absolute Errors (MAE) as low as 4.6 mm.
4. Cross-Embodiment Invariance: Proves that despite different physical structures, the simulated sensory streams share a common "sensorimotor manifold," suggesting that the underlying developmental logic is robust across different embodiments.
5. Automated Behavioral Annotation: Demonstrates the utility of the framework for automatically detecting and quantifying developmental behaviors (e.g., self-touches) that are traditionally labor-intensive to code manually.

4. Results

• Accuracy:
  • MIMo achieved the highest accuracy (MAE < 0.5 cm, orientation error < 2°) due to its scalable morphology and IK solver.
  • EMFANT provided accurate biomechanical constraints and muscle signals.
  • iCub/PyCub showed higher errors (MAE ~10 cm) due to fixed, larger anthropometry (4-year-old size), but remained useful for testing real-world constraints.
• Sensorimotor Alignment:
  • Cross-embodiment analysis using Generalized Procrustes Analysis (GPA) and Principal Component Analysis (PCA) revealed a shared latent space.
  • The invariance index plateaued at ~0.19% with 20 latent dimensions, confirming that a compact low-dimensional manifold captures the consistency of sensorimotor experiences across different robots.
• Application Validation:
  • Hand Regard: Successfully simulated the visual experience of an infant looking at their own hand, including binocular disparity.
  • Self-Touch: The system automatically detected self-touches (hand-to-body collisions). The distribution of touches detected by the simulated iCub closely matched manual expert coding (e.g., predicting 13% left-hand-to-left-leg touches vs. 9% manual), validating its use for semi-automated annotation.

5. Significance and Impact

• Developmental Science: Provides a non-invasive tool to study the "sensorimotor stage" from the infant's perspective, offering insights into how body awareness, spatial understanding, and motor control emerge.
• Clinical Applications: Offers a potential pathway for the early detection of neurodevelopmental disorders by analyzing deviations in sensorimotor patterns.
• Robotics & AI:
  • Embodied Cognition: Bridges the "embodiment gap" between humans and humanoids by providing realistic, multimodal training data.
  • Foundation Models: Suggests a pathway for training robotics foundation models enriched with touch and proprioception, moving beyond purely visual or kinematic data.
• Efficiency: Drastically reduces the time required for behavioral coding in developmental studies by automating the detection of complex interactions like self-touch.

In conclusion, this work transforms static video data into a dynamic, multimodal simulation of infant development, enabling researchers and roboticists to "experience" the world as an infant does, thereby unlocking new avenues for understanding human intelligence and improving humanoid robot learning.