Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain

Technical Summary: Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain

Problem Statement
While Reinforcement Learning (RL) has enabled robust quadruped locomotion on complex terrain, current controllers face significant limitations regarding on-robot adaptation and energy efficiency. Most state-of-the-art policies are trained offline using backpropagation (BP) in massive parallel simulations and deployed as fixed policies. This approach struggles to adapt to real-world variations such as terrain changes, payload shifts, and actuator wear. Furthermore, the computational cost of frequent high-dimensional inference and gradient propagation on CPU/GPU platforms creates a substantial energy burden, particularly for small-scale robots operating under constrained power sources. Existing on-robot RL methods often require external supervision or power, limiting their utility in remote environments. There is a critical need for learning algorithms that are compatible with low-power, neuromorphic hardware substrates and can perform local, energy-aware adaptation without global backpropagation.

Methodology
The authors propose a framework that integrates Equilibrium Propagation (EP) with Proximal Policy Optimization (PPO) to train a bio-inspired quadruped controller. The system replaces conventional backpropagation-trained networks with EP-enabled local learning, where learning is driven by local neural states and network relaxation dynamics rather than global gradient transport.

The control architecture is a two-branch CPG–RES (Central Pattern Generator – Residual) system:

1. CPG Branch: Generates structured, rhythmic foot trajectories (phase, amplitude, frequency) to handle basic gait generation.
2. RES Branch: Provides residual postural adjustments to correct joint targets for uneven terrain traversal.

To adapt PPO for EP, the authors address several technical challenges:

• EP-Compatible Objective: They derive an output-nudging signal that maps the PPO policy objective onto EP's equilibrium dynamics. Instead of minimizing a loss, the network is driven to maximize the PPO surrogate objective by applying the negative gradient of the objective to the output layer neurons.
• Stabilization Mechanisms: To prevent policy collapse and unstable updates during the relaxation process, the authors introduce a two-sided ratio clipping mechanism (modifying the standard PPO clip range) and an adjusted gradient-scaling rule using $1/\sigma_i$ instead of $1/\sigma_i^2$.
• Preprocessing: An inverse discrete cosine transform (iDCT) is applied to observation vectors to accelerate convergence to equilibrium and reduce inference latency.
• Training Paradigm: The system employs a two-stage training process. Stage 1 trains the CPG policy on flat ground. Stage 2 freezes the CPG policy and trains the RES policy on randomized uneven terrain. Both stages utilize the EP-PPO framework.

Key Contributions

1. EP-Based PPO Framework: The paper presents the first demonstration of an EP-based PPO framework for continuous-control locomotion without backpropagation. It formulates EP updates for both policy learning and value regression, establishing a pathway for neuromorphic implementation of high-dimensional robotic control.
2. EP-Compatible Policy Gradient: The authors derive a specific output-layer nudging signal and a two-sided ratio clipping objective that stabilizes policy updates during EP relaxation, enabling the training of stochastic continuous-control policies without direct backpropagation through the network.
3. Bio-Inspired Controller Integration: The work integrates EP networks into a CPG–RES controller, connecting biological locomotion organization (oscillator dynamics) with local neuromorphic learning for uneven-terrain traversal.

Results
The proposed controller was evaluated on a 12-DoF Unitree A1 quadruped simulated in MuJoCo over randomized uneven terrain and compared against a standard backpropagation-trained PPO baseline.

• Performance Parity: The EP-based controller achieved performance comparable to the BP baseline in terms of success rate, forward velocity tracking, actuator power consumption, and body stability (roll/pitch angular velocity).
• Training Stability: The EP method achieved stable policy convergence in both training stages without policy collapse, and the value network steadily minimized loss.
• Efficiency: A significant finding is the improvement in memory efficiency. The EP algorithm consumed approximately 4.3× less GPU memory (3110 MB vs. 13486 MB) compared to Backpropagation Through Time (BPTT) for the same task, attributed to the elimination of global gradient storage requirements.

Significance and Claims
The paper claims that local equilibrium-based learning can support high-dimensional, contact-rich embodied locomotion, moving beyond simple neuromorphic control benchmarks. The primary significance lies in providing an algorithmic foundation for low-power, on-robot adaptation and fine-tuning. By aligning learning rules with in-memory and neuromorphic hardware dynamics, the framework offers a potential solution to the energy constraints of long-duration autonomy.

The authors remain modest regarding the current state of the work, noting that:

• The method inherits PPO's sample efficiency limitations, which are challenging for real-world robots.
• The current implementation still relies on digital computation for relaxation and infrastructure; it is an algorithmic step toward energy-aware learning rather than a complete hardware realization.
• Future work is needed to address safety, sample efficiency, and the development of system-level neuromorphic designs that integrate EP dynamics with the surrounding RL infrastructure.