Infant Spontaneous Movement Noise Improves Exploration in Deep RL

Technical Summary: Infant Spontaneous Movement Noise Improves Exploration in Deep RL

Problem Statement
Exploration in deep reinforcement learning (RL) is frequently implemented using temporally uncorrelated white noise (Gaussian). While effective, recent literature suggests that temporally correlated "colored" noise can enhance exploration efficiency by generating smoother trajectories that cover the state space more effectively. However, most existing approaches utilize static noise parameters (e.g., constant pink noise). This paper investigates whether action noise inspired by the developmental trajectory of infant spontaneous movements can further improve exploration. Specifically, it addresses the question of whether mimicking the changing statistical properties of biological motor variability during early development can yield more efficient learning strategies for artificial agents.

Methodology
The research follows a three-stage methodology: characterizing biological data, generating synthetic noise, and integrating it into RL algorithms.

1. Characterization of Infant Movement:
   The authors analyzed a longitudinal dataset of video recordings from four infants (ages 8 to 30 weeks) recorded in a supine position. Using OpenPose, 2D keypoints for end-effectors (wrists and ankles) were extracted to compute velocity time series. The temporal structure of these movements was quantified using Power Spectral Density (PSD) analysis via Welch's method. The noise color, defined by the spectral exponent $\beta$ in the power-law relation $S(f) \propto f^{-\beta}$, was estimated for each session.

2. Developmental Noise Schedule:
   Analysis revealed a strong linear correlation ($R=0.70$) between the spectral exponent $\beta$ and infant age, with $\beta$ increasing from approximately 0.69 at 8 weeks to 0.88 at 30 weeks. Based on this, the authors proposed a "baby noise" schedule where the noise color $\beta(t)$ increases linearly during the training of an RL agent, ranging from 0.7 to 0.9 as a fraction of total training time $t/T$.

3. Noise Generation and Integration:
   To generate colored noise efficiently, the authors employed spectral shaping: generating white noise blocks, transforming them to the frequency domain via FFT, scaling the spectrum to match the target PSD ($f^{-\beta}$), and transforming back to the time domain. This noise was integrated into two continuous control algorithms:

   • TD3 (Deterministic): Replaced the standard additive Gaussian noise with the scheduled colored noise.
   • SAC (Stochastic): Replaced the white noise component in the reparametrization trick with the scheduled colored noise.

Key Contributions

• Quantification of Motor Development: The paper provides the first quantitative characterization of the transition from stochastic to structured behaviors in early infancy using PSD slopes, identifying a developmental increase in temporal auto-correlation ($\beta$) between 8 and 30 weeks.
• Developmentally Scheduled Exploration: It introduces a novel exploration mechanism where the noise color is annealed over training time, mirroring the observed infant developmental pattern.
• Empirical Validation: The study demonstrates that this infant-inspired strategy produces structured exploratory behavior and improves learning efficiency compared to conventional static noise strategies across multiple environments.

Results
Experiments were conducted on 12 continuous-control environments (classic control, locomotion, and mazes) using TD3 and SAC algorithms. The performance was evaluated using the Area Under the Learning Curve (AUC) of normalized returns.

• Performance: The "baby noise" strategy achieved the highest normalized AUC score on average across all algorithms and environments. It was closely followed by static colored noises (blush $\beta=0.5$, rose $\beta=0.75$, and pink $\beta=1.0$), all of which outperformed white noise ($\beta=0$). High-correlation noises (red noise $\beta=2$ and Ornstein-Uhlenbeck) yielded the worst performance.
• Consistency: While no single noise type dominated every specific environment, the baby noise strategy was the most consistent, with only three scores falling below the zero-mean across 24 conditions.
• Statistical Significance: In paired comparisons against white noise, baby noise was the only strategy with a win rate significantly above chance (56%, $p < 0.05$). Static colored noises (blush, rose, pink) showed higher average AUCs but did not achieve statistically significant win rates over white noise in the paired analysis.

Significance and Claims
The authors claim that their work adds to the growing body of evidence suggesting that human motor and cognitive development can provide useful guidance for designing learning mechanisms in artificial agents. The study posits that the developmental shift in noise color reflects a transition from exploratory spontaneous movements to goal-directed behaviors. By translating this biological insight into a curriculum for exploration dynamics, the paper demonstrates that deep RL agents can benefit from continuously adapting to increasingly structured and temporally correlated exploration. The authors maintain a modest stance, noting that while the results are promising, the next step is to confirm if these benefits extend to environments with more complex infant embodiments (e.g., growing bodies) and to further explore the connection between developmental science and artificial intelligence.