⚡ electrical engineering

Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

This paper proposes a Deep Asymmetric Ternary Spiking Q-Network (DATSQN) that utilizes a novel ternary spiking neuron model to mitigate gradient estimation bias, thereby overcoming the performance degradation of existing ternary models and outperforming binary baselines in deep Q-learning tasks across seven Atari games.

Original authors: Aref Ghoreishee, Abhishek Mishra, John Walsh, Anup Das, Nagarajan Kandasamy

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aref Ghoreishee, Abhishek Mishra, John Walsh, Anup Das, Nagarajan Kandasamy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play video games. To make decisions, the robot needs a brain, and in the world of artificial intelligence, we often build these brains using "neural networks." These networks are inspired by the human brain, where tiny units called neurons pass signals to each other. In traditional computer brains, these signals are like smooth, continuous waves of electricity. But in a special kind of brain called a "Spiking Neural Network" (SNN), the neurons communicate using tiny, sharp bursts of electricity called "spikes," much like how real biological neurons fire. This method is incredibly energy-efficient, making it perfect for mobile robots or devices that can't carry heavy batteries.

However, there's a catch. Most of these efficient spiking brains only have two states: they either fire a spike (1) or they stay silent (0). It's like a light switch that can only be on or off. This limits how much information the robot can hold in its head at once. Recently, scientists tried to upgrade these switches to "ternary" neurons, which have three states: fire positively (+1), stay silent (0), or fire negatively (-1). You might think adding a "negative" switch would make the robot smarter, but surprisingly, when they tried this in a video game setting, the robots actually got worse at playing. This paper dives into that mystery, asking why having more options made the robot dumber, and how they fixed it to make the robot a gaming champion again.

The Mystery of the Dumber Robot

The researchers, a team from Drexel University, set out to test these new "ternary" neurons in a deep learning system designed to play Atari video games (the classic arcade games like Breakout and Space Invaders). They had a hunch that the problem wasn't the extra state itself, but how the neurons were learning.

Think of training a robot like teaching a dog tricks. You give the dog a command, it tries, and you give it a treat (a reward) or a "no" (a penalty). In the world of AI, this "treat" is calculated by looking at the difference between what the robot expected and what actually happened. This difference is called the "gradient," and it tells the robot how to adjust its brain to do better next time.

The team discovered that the standard ternary neurons were like a teacher who only cared about the direction of the mistake, not the size of it. If the robot made a mistake, the ternary neuron could only say, "You went the wrong way!" (+1 or -1), but it completely ignored the question of "How much did you mess up?" (the activity). In the early stages of learning a video game, the robot is often guessing wildly, so the "direction" of the error is confusing and random. Because the standard ternary neurons were so focused on direction, they got lost in the noise and stopped learning entirely. It was as if the robot was trying to navigate a foggy maze while only listening to a compass that spun randomly; it just stood still.

The Asymmetric Fix

To solve this, the authors proposed a new design called the Deep Asymmetric Ternary Spiking Q-Network (DATSQN). They realized that nature had the answer all along. In the human brain, not all neurons are created equal. About 80% of our neurons are "excitatory" (they push the brain to act), and only about 20% are "inhibitory" (they tell the brain to stop). The old ternary models treated these two types as perfect mirror images of each other, which is biologically unrealistic.

The team's new model broke this symmetry. They created a neuron where the "positive" (excitatory) and "negative" (inhibitory) thresholds were different. One way to visualize this is to imagine a door with two different locks. The old model had two identical locks; if you pushed too hard on either side, the door opened the same way. The new model has a heavy, hard-to-open lock on the "stop" side and an easy-to-open lock on the "go" side. This asymmetry meant that even when the robot was confused about the direction of its mistake, the neuron could still send a clear signal about how much it should be active. It kept the "volume knob" working even when the "direction arrow" was spinning.

The Results: From Stalled to Superstar

The team tested this new brain in seven different Atari games, pitting it against the old binary (on/off) robots and the standard ternary robots. They made the task harder by forcing the robots to make decisions faster (using a shorter simulation time window of 20 steps instead of 40), which usually makes learning much more difficult.

The results were striking. The standard ternary robots (DTSQN) performed terribly, scoring much lower than the basic binary robots. They were indeed stuck in the learning fog. However, the new asymmetric robots (DATSQN) didn't just catch up; they soared. In six out of the seven games, the new model beat the binary baseline, achieving a 30% improvement in the average game score. For example, in Beam Rider, the new robot scored 4,000 points compared to the binary robot's 2,069. In Breakout, it scored 252 points versus 207.

The researchers also checked the "heartbeat" of the training process. They found that the new model's learning signals remained strong and stable, avoiding the "vanishing gradient" problem where the robot forgets how to learn. They also observed that the neurons in their new model naturally evolved to be mostly excitatory, just like in a real human brain, with positive spikes outnumbering negative ones by a ratio of at least 60%.

What This Means

This paper suggests that simply adding more states to a neuron isn't enough; you have to design how those states learn. By mimicking the natural imbalance of excitatory and inhibitory neurons in the human brain, the team created a spiking network that is not only more energy-efficient but also smarter at learning complex tasks. While the study was conducted in simulations of video games, the findings suggest that this "asymmetric" approach could be the key to building better, more efficient AI brains for robots and other devices in the future. The authors note that while their model works great in these games, it's still a suggestion for how to build better systems, and it will be interesting to see if this approach helps in other areas like vision or language processing.

Technical Summary: Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

Problem Statement
Deep Reinforcement Learning (DRL), particularly Deep Q-Networks (DQN), faces significant challenges regarding energy efficiency and deployment on mobile platforms. Spiking Neural Networks (SNNs) offer an energy-efficient alternative by mimicking biological neurons that communicate via discrete spikes. However, SNNs currently underperform compared to classical Deep Neural Networks (DNNs) in complex, high-dimensional tasks for two primary reasons: the non-differentiable nature of spike generation complicates training, and binary spiking neurons (outputting {0, 1}) possess limited representation capacity.

While recent studies introduced ternary spiking neurons (outputting {-1, 0, 1}) to enhance representation capacity and improve performance in computer vision and natural language processing, this paper identifies a critical contradiction: in Deep Q-learning tasks, existing symmetric ternary models perform substantially worse than their binary counterparts. The authors hypothesize that this degradation is caused by gradient estimation bias during training, specifically the loss of informative learning signals when spike signs are undetermined.

Methodology
The authors develop a mathematical framework to analyze the training dynamics of ternary SNNs, modeling spikes as random variables to derive expected gradients.

Analysis of Symmetric Ternary Neurons:
The paper demonstrates that for a symmetric ternary Leaky Integrate-and-Fire (LIF) neuron (where positive and negative thresholds are equal), the expected gradient during backpropagation splits into a "sign channel" and an "activity channel." In the context of early Q-learning, where target values are uninformative regarding polarity, the sign channel vanishes. Because the symmetric design forces the "activity channel" (which determines whether the neuron should fire or remain silent) to zero, the neuron receives a near-zero, high-variance gradient. This leads to training stagnation, explaining the observed performance degradation.
Proposed Solution: Asymmetric Ternary Spiking Neuron:
To resolve this, the authors propose a new neuron model with asymmetric thresholds ( $v_{thp} \neq v_{thn}$ ). In this model:
- The positive threshold ( $v_{thp}$ ) is fixed.
- The negative threshold ( $v_{thn}$ ) is a trainable parameter.
- This asymmetry ensures that the sensitivities to positive and negative membrane potentials ( $c_p$ and $c_n$ ) are unequal.
- Consequently, the "activity channel" in the gradient equation remains non-zero even when the sign channel vanishes, providing a continuous learning signal.
Stability Analysis:
Using the concept of block dynamic isometry, the authors mathematically prove that networks built with these asymmetric ternary neurons avoid both vanishing and exploding gradient problems. They show that the dynamic isometry of the proposed model is at least as robust as that of networks using ReLU activations.
Network Architecture (DATSQN):
The proposed neuron is integrated into a Deep Asymmetric Ternary Spiking Q-Network (DATSQN). To ensure compatibility with neuromorphic hardware, the input encoding uses rate-based Bernoulli sampling to convert raw pixels into spike trains, eliminating the non-spiking, floating-point first layer found in some prior works. The simulation time window is reduced to 20 steps (compared to 40 in baselines) to increase task difficulty and reduce latency.

Key Contributions

Identification of Gradient Bias: The paper provides the first evaluation of ternary neurons in Deep Q-learning, revealing that increased representation capacity does not automatically translate to better RL performance due to gradient estimation bias in symmetric models.
Novel Neuron Model: The introduction of the asymmetric ternary spiking neuron, which maintains high representation capacity while ensuring a non-zero expected gradient through trainable, asymmetric thresholds.
Theoretical Guarantees: Mathematical proofs demonstrating that the proposed model prevents gradient vanishing/exploding and achieves dynamic isometry comparable to ReLU networks.
Biological Plausibility: The design aligns with biological evidence showing that brains exhibit a dominance of excitatory neurons (approx. 70-80%) over inhibitory ones, a property naturally captured by the asymmetric thresholds.

Experimental Results
The authors evaluated DATSQN against binary (DSQN) and symmetric ternary (DTSQN) baselines across seven Atari games from the Gym environment.

Performance: DATSQN achieved a 30% improvement in mean game score over the binary DSQN baseline across the seven games.
Comparison: While the symmetric ternary model (DTSQN) showed a clear performance drop in every game compared to the binary baseline, the asymmetric model (DATSQN) outperformed the binary baseline in six of the seven games.
Training Dynamics: Experiments confirmed that DTSQN suffered from low average gradient norms (indicating the "stall" predicted by theory), whereas DATSQN maintained stable gradient norms and achieved higher cumulative rewards in the later stages of training.
Ablation Studies: Experiments showed that simply making the threshold trainable in a symmetric model was insufficient; the specific asymmetry (fixing one threshold while training the other) was crucial for performance. Furthermore, making both thresholds trainable did not yield significant gains over the single-trainable-threshold design, justifying the simpler canonical design.

Significance
The paper claims that the performance degradation of ternary neurons in Q-learning is not an inherent flaw of ternary encoding but a consequence of symmetric thresholding which blinds the network to activity-based learning signals in early RL stages. By introducing asymmetry, the authors successfully recover the benefits of higher representation capacity while ensuring training stability. The results suggest that asymmetric ternary neurons are a viable and superior alternative to binary neurons for deep spiking reinforcement learning, offering a path toward more efficient and capable neuromorphic agents. The authors note that this approach warrants further evaluation in other SNN settings, such as vision-based tasks and language modeling.

Technical Summary: Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

More like this