Technical Summary: Neuromorphic Control of a Simulated SMA-Driven Multi-Legged Robot Using a Spiking Neural Network

Problem Statement

Conventional control strategies for soft robots, particularly those driven by Shape Memory Alloys (SMAs), predominantly rely on open-loop, pre-designed periodic signals. This approach faces significant challenges due to the high dimensionality of soft robot systems and the strong nonlinearities inherent in SMA actuators, such as thermal hysteresis and saturation characteristics. Manually designing and tuning coordinated motions for systems with many degrees of freedom is difficult, and open-loop controllers lack the adaptability to handle environmental changes (e.g., obstacles, slopes) or material degradation.

While Deep Reinforcement Learning (DRL) offers a pathway for adaptive control, standard DRL approaches often utilize Artificial Neural Networks (ANNs) that can be computationally expensive and energy-inefficient during inference. Furthermore, SMAs share mechanical response characteristics with biological muscles (e.g., threshold behavior, saturation, and temporal summation of pulse inputs), suggesting that a control architecture mimicking biological neural processing might be particularly well-suited. The specific challenge addressed is developing an energy-efficient, autonomous control method that can learn to exploit the nonlinear dynamics of SMA actuators without relying on manual signal design.

Methodology

Robot Model and Actuation

The authors modeled a starfish-inspired multi-legged soft robot in the MuJoCo physics simulator. The robot features a central torso and five legs arranged radially. Each leg consists of six hinge joints (30 total degrees of freedom) and is actuated by an SMA wire (flexor) and a passive elastic element (extensor).

• SMA Model: A simplified constitutive model based on Brinson's work was adopted, prioritizing computational efficiency. The model approximates the phase-transformation progress ($\xi$) as a function of temperature alone, neglecting explicit stress dependence. The force generated is proportional to $\xi$.
• Thermal Dynamics: The temperature evolution of the SMA wire is governed by an energy-conservation equation accounting for Joule heating and heat dissipation. The model includes thermal hysteresis by using different transition temperatures for heating and cooling processes.
• Actuation Logic: The control input is a normalized current value ($u \in [0,1]$). The system simulates the physical limits of temperature rise and cooling rates to prevent numerical instability.

Spiking Actor Network (SAN)

The core contribution is a Spiking Neural Network (SNN)-based Deep Reinforcement Learning method using the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm.

• Architecture: The actor network is replaced by a Spiking Actor Network (SAN) consisting of:
  1. Encoder: Converts continuous state observations into spike trains using population coding and a thresholding mechanism.
  2. FC SNN: A fully connected network with two hidden layers of 256 Leaky Integrate-and-Fire (LIF) neurons.
  3. Decoder: Converts the output spike trains back into continuous action values (control signals for the SMAs) via rate coding.
• Internal State Handling: Unlike standard TD3, the replay buffer is extended to store membrane potential information ($m_t$ and $m_{t+1}$) alongside state, action, and reward. This allows the SNN's internal dynamics to be accurately reproduced during the sampling phase of training.
• Training Algorithm:
  • Surrogate Gradient (SG): The non-differentiable step function of the LIF neuron is handled using the fast sigmoid function as a surrogate gradient during backpropagation.
  • Regularization: A regularization term ($\alpha \|m_n\|_2^2$) is added to the actor's loss function to minimize squared membrane potentials. This encourages the generation of actions with fewer spike firings, thereby improving energy efficiency during inference.

Experimental Setup

• Task: Autonomous locomotion toward a target goal.
• State Space: A 9-dimensional vector comprising SMA lengths, the relative position of the first ray's tip, and the vector from the torso center to the goal.
• Reward Function: A composite reward promoting progress toward the goal, penalizing distance to the goal, and penalizing the L2 norm of the action vector to encourage energy efficiency.
• Environment: MuJoCo physics simulator with a 10ms time step (1000ms observation/action interval).

Key Results

Motion Evaluation and SMA Dynamics

Initial experiments established the relationship between spike firing rates and the bending angle of a single leg. The results confirmed strong nonlinearity, threshold effects, and saturation, mirroring biological muscle behavior. A critical finding was the existence of an optimal drive period: periods longer than the relaxation time ($T_{ext} \approx 20-25$s) cause idle intervals, while periods shorter than $T_{ext}$ lead to incomplete extension and wasted energy.

Autonomous Locomotion Learning

The SAN successfully learned to control the robot in two scenarios: straight-line locomotion and goal-reaching (oblique) locomotion.

• Learning Progression:
  • Early Stage (30k steps): No clear periodicity; minimal movement.
  • Middle Stage (50k steps): Emergence of periodic firing patterns and limited forward movement.
  • Late Stage (300k steps): Continuous, stable locomotion with a coordinated gait.
• Gait Characteristics:
  • Periodicity: The learned firing period stabilized at approximately 24–26 seconds, closely matching the physical relaxation time of the SMA actuators measured in the basic experiments.
  • Coordination: The agent learned to drive the five legs with specific phase offsets rather than synchronizing them. For instance, in the straight-line task, Ray 2 and Ray 5 led, while Rays 3 and 4 were nearly anti-phase to Ray 1.
  • Adaptability: When the goal position changed, the agent adjusted the contraction timing and phase relationships of the rays dynamically, rather than relying on a fixed input pattern.

Energy Efficiency

The inclusion of the membrane potential regularization term successfully suppressed excessive spike firing during inference, aligning with the goal of energy-efficient neuromorphic control.

Significance and Claims

The paper claims that the proposed method demonstrates the potential of autonomous neuromorphic control for energy-efficient soft-robot locomotion. Specifically:

1. Autonomous Acquisition of Physical Constraints: The agent implicitly learned the thermodynamic response delays and output saturation of the SMA actuators, acquiring a drive period that matches the physical relaxation time without manual tuning.
2. Emergent Coordination: Through interaction with the environment, the SNN autonomously developed a coordinated gait with ordered phase-propagation dynamics. This mimics the empirically designed gaits of prior work but is achieved without open-loop pre-design.
3. Energy Efficiency: By leveraging the event-driven nature of SNNs and a regularization term to minimize membrane potentials, the method offers a pathway to lower-power inference compared to traditional ANN-based DRL.
4. Adaptability: The controller successfully adapted to different goal positions by dynamically adjusting firing patterns, showcasing robustness to task variations.

The authors conclude that learning-based neuromorphic control can overcome the limitations of conventional open-loop control for high-dimensional, nonlinear artificial-muscle systems. They note that while the results are promising in simulation, future work requires validating the transfer of these policies to physical robots (sim-to-real), acknowledging that real-world parameters (cooling rates, friction, material elasticity) may differ from the simulation.