Technical Summary: Training Slow Silicon Neurons to Control Extremely Fast Robots with Spiking Reinforcement Learning

Problem Statement

Autonomous robotics faces a critical energy-efficiency bottleneck. While large-scale neural networks have achieved unprecedented success, their training and inference costs (megawatt-hours and rapid battery drain) are incompatible with mobile platforms that operate on milliwatt budgets. Conversely, the human brain achieves complex motor learning with roughly 20 W of power and few training examples. While neuromorphic hardware offers a path to energy efficiency through asynchronous, event-driven processing, existing demonstrations of Spiking Neural Network (SNN) reinforcement learning (RL) have been largely confined to simplified benchmarks (e.g., 2D games with discrete actions and low-dimensional observations).

The central challenge addressed in this work is scaling neuromorphic RL to real-world robotic manipulation characterized by:

1. High dimensionality: A 6D continuous state space (puck position/velocity and end-effector coordinates).
2. Physical constraints: Precise end-effector control with mechanical latencies.
3. Temporal dynamics: Puck trajectories at 1.0–1.5 m/s requiring predictive planning with 50 Hz control and 1 ms physics integration.

The authors investigate whether a compact network of "slow" silicon neurons running on a mixed-signal neuromorphic processor can control an "extremely fast" robot in a high-speed air hockey task.

Methodology

Experimental Setup

The system operates in a Hardware-in-the-Loop (HIL) configuration using the MuJoCo simulation of an air hockey environment.

• Robotics: An anthropomorphic manipulator controls a mallet-shaped end-effector on a 1.038 × 1.948 m table.
• Control Loop: Operates at 50 Hz. The agent selects one of two discrete actions corresponding to pre-defined motion primitives (Home target or Forward target). These primitives are executed as open-loop ballistic trajectories via an embedded inverse kinematics solver.
• Physics: Integrated at 1000 Hz to model frictionless puck dynamics and elastic collisions.

Network Architecture

The system utilizes the DYNAP-SE analog-digital mixed-signal neuromorphic chip. The architecture is a reservoir computing setup with a plastic readout:

• Input Layer: 60 FPGA-based spike generators. Six state variables (puck $x, y, v_x, v_y$ and end-effector $x, y$) are encoded as population codes using Gaussian activity profiles across 10 neurons each.
• Hidden Layer (Reservoir): 1020 silicon AdEx-LIF neurons with fixed random feedforward synapses. This layer captures the temporal structure of the task without requiring weight updates.
• Readout Layer: 2 linear units representing the probability of selecting Action 0 or Action 1. The weights connecting the hidden layer to the readout are plastic.
• Learning Rule: The system employs e-prop (eligibility propagation), a local learning rule that approximates backpropagation through time. Weight updates depend only on local pre- and post-synaptic activity combined with a global reward signal, enabling fully online learning. The update rule is:
  $$\Delta w_{ik} = -\alpha \sum_{t} r_t \sum_{t' \le t} \gamma^{t-t'} (\pi_{t'}^k - \mathbb{1}_{a_{t'}=k}) \bar{s}_{t'}^i$$
  Where $\bar{s}$ is filtered activity, $r_t$ is the reward, and $\pi$ is the action probability.

Reward Function

A scalar reward $r_t$ is computed at each step to encourage forward puck motion and precise interception timing:

• +20: Terminal reward if the puck passes the target ($x_p > 1.5$ m) with forward momentum ($v_x > 0.1$ m/s).
• $0.2 v_x$: Shaping reward proportional to forward speed if the end-effector is within 0.08 m of the puck.
• -0.1: Per-step penalty to minimize episode length.

Training Protocol

Training runs for 2000 episodes per experimental run, repeated with 10 independent random seeds to account for the chip's stochasticity. The system learns "awake" only (no offline "dreaming" phase in this specific implementation), relying on real-time interaction.

Key Contributions

The paper claims three primary contributions in scaling neuromorphic RL from gaming to physical robotics:

1. Beyond Toy RL: Transitioning from simple grid-based games to real-world air hockey with adaptive-precision continuous state encoding.
2. Discrete to Continuous Control: Demonstrating neuromorphic RL for continuous motion primitives executing ballistic trajectories at 50 Hz, requiring predictive rather than frame-level decisions.
3. Generalization under Uncertainty: Achieving robust learning (96–98% success) despite randomizing puck positions and velocities (1.0–1.5 m/s), proving the system can adapt to varied initial conditions.

Results

• Learning Efficiency: The system achieved robust gameplay within 1500–2000 episodes.
• Performance:
  • Stationary Puck: 100% success within 200 trials.
  • Constant Speed: 100% success after 1000 episodes.
  • Variable Speed (1.0–1.5 m/s): Success stabilized above 96% after 1500 episodes.
  • Randomized Position & Speed: Success stabilized above 98% after 1300 episodes.
• Scalability Tests:
  • Narrow speed ranges ([0.7, 0.9] m/s) reached >97% success in ~150 episodes.
  • Wider ranges ([0.7, 1.5] m/s) required ~700 episodes and showed a modest drop in asymptotic success (97% to 93%), indicating graceful degradation rather than catastrophic failure.
• Behavioral Consolidation: Pre-training behavior was erratic and stochastic. Post-training, the agent exhibited a deterministic, temporally precise strategy characterized by early commitment to an interception trajectory followed by precise corrective maneuvers.

Significance and Claims

The authors position this work as the first affirmative answer to whether neuromorphic RL systems can scale to higher-dimensional states, continuous-time control, and tight temporal constraints typical of real-world robotics.

• Bridging the Gap: The work demonstrates that brain-inspired approaches can tackle fast-paced interaction tasks (1.5 m/s puck speeds) while supporting "always-on" learning in intelligent machines.
• Efficiency: The system achieved stable performance using only 1020 neurons, a 10x reduction compared to previous bio-inspired RL simulations for air hockey that required ~10,000 neurons.
• Practicality: By co-designing hardware and learning algorithms (using e-prop on DYNAP-SE), the system enables real-time learning with a computer and neuromorphic chip in-the-loop, validating the feasibility of training spiking neural networks for autonomous robotic systems without relying on non-local, computationally expensive learning rules.

The paper concludes that complex sensorimotor learning is achievable with a compact mixed-signal neuromorphic processor, highlighting the potential for energy-efficient, adaptive control in autonomous robotics.