Hebbian Attractor Networks for Robot Locomotion

1. Problem Statement

Traditional robot learning follows a train-deploy paradigm: a neural network policy is trained (often via Reinforcement Learning or Evolutionary Strategies) to optimize a static set of weights for a specific task. Once deployed, these weights remain fixed. This approach lacks robustness in dynamic real-world environments where sensors drift, terrain changes, or actuators degrade.

In contrast, biological nervous systems continuously adapt via synaptic plasticity throughout an organism's lifetime. While Hebbian learning (learning based on the correlation of pre- and postsynaptic activity) offers a bio-inspired mechanism for continuous adaptation, previous applications in robotics have struggled with:

• Runaway weight growth: Unbounded increases in weight values leading to instability.
• Co-dynamic instability: Weight changes tightly coupled with motor control, often resulting in oscillatory behaviors that are difficult to stabilize.
• Lack of convergence: Difficulty in achieving stable, fixed-point weight configurations that allow for reliable locomotion.

The paper addresses the gap in understanding how the timing of Hebbian updates and the temporal structure of activation signals influence the emergence of stable attractor dynamics in embodied systems.

2. Methodology: Hebbian Attractor Networks (HANs)

The authors introduce Hebbian Attractor Networks (HANs), a class of plastic neural networks designed to induce specific attractor dynamics (fixed-points or limit cycles) through structural constraints and dual-timescale plasticity.

Core Architecture

• Network Structure: Feedforward networks with local Hebbian updates.
• Hebbian Rule: Uses a generalized ABCD rule where weight updates ($\Delta w$) are linear combinations of pre- and postsynaptic activations ($x_{pre}, x_{post}$) and their products.
• Max Normalization (MN): To prevent unbounded growth, weights are rescaled at every layer by their maximum absolute value after each update:
  $$w_{ij} \leftarrow \frac{w_{ij}}{\max |w_{ij}|}$$

Key Innovations: Dual-Timescale and Averaging

Unlike prior works that update weights at every time step, HANs introduce two critical modifications:

1. Dual-Timescale Plasticity: The Hebbian update frequency ($f_{hebb}$) is decoupled from the control loop frequency ($f_{NN}$). Updates occur only every $\tau_{hebb} = \lfloor f_{NN}/f_{hebb} \rfloor$ steps.
2. Temporal Averaging: Instead of using instantaneous activations, the learning rule operates on a Moving Average (MA) of pre- and postsynaptic activations over a window $M$:
   $$\bar{x}(t) = \frac{1}{M} \sum_{m=0}^{M-1} x(t-m)$$

Experimental Setup

• Optimization: The Hebbian coefficients ($\theta$) and learning rates ($\eta$) are optimized using Evolutionary Strategies (ES).
• Benchmarks:
  • Gymnasium: Standard continuous control tasks (Swimmer, Hopper, HalfCheetah, Walker2d).
  • Unitree Go1: High-dimensional quadrupedal locomotion in MuJoCo.
  • Morphological Adaptation: Testing on damaged robots (Ant-v5 with shortened legs).

3. Key Contributions

1. Systematic Analysis of HANs: The paper provides a rigorous ablation study comparing static baselines, standard Hebbian Neural Networks (HNNs), and HANs with varying update frequencies and averaging windows.
2. Characterization of Attractor Regimes: The authors demonstrate that HANs can induce two distinct types of weight dynamics based on hyperparameters:
   • Limit Cycles (Co-dynamic): Fast updates ($f_{NN} \approx f_{hebb}$) with no averaging lead to oscillating weights that are tightly coupled to the gait cycle.
   • Fixed-Point Attractors: Slower updates ($f_{NN} \gg f_{hebb}$) combined with large averaging windows ($M \ge 4$) induce convergence to stable weight configurations.
3. Generalization to High Dimensions: The findings are validated on a simulated Unitree Go1 quadruped, showing that the attractor dynamics observed in simple benchmarks scale to complex, high-dimensional locomotion.

4. Key Results

Performance on Gymnasium Benchmarks

• Max Normalization is Crucial: HNNs without MN fail (unbounded growth). Adding MN significantly improves performance.
• Dual-Timescale & Averaging: The best-performing configuration (Condition E: $f_{NN}=4\times f_{hebb}$, $M=10$) outperforms static MLPs and GRUs in most tasks (HalfCheetah, Walker2d) and matches PPO on simpler tasks (Swimmer).
• Comparison to PPO: While PPO with larger architectures achieves the highest absolute scores, HANs achieve comparable performance with significantly fewer parameters by leveraging within-lifetime plasticity.

Attractor Dynamics Analysis

• Fixed-Point Convergence: In Condition E (slow updates, high averaging), weight changes ($\Delta W$) drop to near zero after an initial adaptation phase. PCA projections show weights converging to a single point in weight space.
• Limit Cycle Oscillation: In Condition B (fast updates, no averaging), weights oscillate periodically, tracking the gait cycle. Interrupting these updates causes the robot to stumble, indicating the weights act as intrinsic pattern generators.
• Robustness to Perturbation: When a heavy pendulum collides with the robot during deployment, the Fixed-Point HAN (Condition E) adapts to regain stability and converges back to the same weight configuration, demonstrating a robust attractor. The Limit-Cycle HAN (Condition B) also recovers but relies on continuous oscillation.

Morphological Adaptation (Ant-v5)

• When tested on unseen leg mutilations, HANs maintained more consistent performance across seen and unseen damage compared to PPO, which overfit to the training distribution. However, in this specific high-damage scenario, even the "Fixed-Point" configuration converged to a Limit Cycle, suggesting evolution prefers oscillatory dynamics when adaptation across qualitatively different locomotion modes is required.

Unitree Go1 Quadruped

• The study successfully scaled to a 12-actuator robot.
• Condition E produced stable fixed-point attractors in 36% of successful runs, while Condition B produced limit cycles.
• Both configurations successfully achieved velocity tracking, validating the approach for high-dimensional systems.

5. Significance and Implications

• Principled Characterization of Plasticity: The paper moves beyond treating Hebbian learning as a "black box" meta-learning tool. It mathematically and empirically links the timing of plasticity (update frequency) and signal smoothing (averaging window) to specific dynamical regimes (fixed-points vs. limit cycles).
• Robustness in Embodied AI: The discovery that slower, averaged updates lead to fixed-point attractors offers a new pathway for creating robots that can self-stabilize and adapt to environmental changes without retraining.
• Bio-Plausibility: The dual-timescale mechanism mirrors biological observations where synaptic plasticity occurs on different timescales than neural firing, providing a more biologically realistic model for artificial control systems.
• Future Directions: The authors suggest that future work could explore multi-attractor HANs (switching between attractors via neuromodulation) to handle sudden environmental shifts and bridge the sim-to-real gap.

In summary, this work establishes that by carefully tuning the temporal structure of Hebbian learning, one can engineer neural networks that naturally evolve toward stable, adaptive attractor states, offering a robust alternative to static, gradient-based control policies for dynamic robotic tasks.