⚡ electrical engineering

Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic

This paper proposes a Lyapunov-guided training framework that enforces monotone state projection to suppress fixed-point overflow-induced instability, thereby enabling reliable low-precision neural network training and inference on resource-constrained hardware.

Original authors: Anis Hamadouche, Amir Hussain

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anis Hamadouche, Amir Hussain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize pictures of handwritten numbers (like "1", "2", or "3"). To make this robot fast and cheap enough to run on a small battery-powered device (like a smartwatch or a sensor), you want to shrink its brain. You do this by reducing the precision of its math, switching from complex, high-precision numbers (like a scientist's calculator) to simple, low-precision numbers (like a child's abacus). This is called quantization.

However, the researchers in this paper discovered a hidden trap in this process.

The Trap: The "Wrap-Around" Accident

Most people assume that if a number gets too big for the robot's small memory, it just gets "clipped" or cut off at the maximum limit (like a speedometer stuck at 120 mph).

But in many real-world computer chips, the math works differently. It uses something called two's-complement wrapping. Imagine a clock face. If you keep turning the hands forward past 12, they don't stop; they snap back to 1 and start over.

In this robot's brain, if a number gets too big and positive, it suddenly snaps to become a huge negative number.
If it gets too big and negative, it snaps to become a huge positive number.

The paper calls this "overflow wrapping." It's like a rollercoaster that, instead of slowing down when it hits the end of the track, suddenly flips upside down and shoots in the opposite direction. This confuses the robot completely, causing it to forget what it was learning and perform at the level of random guessing.

The Solution: The "Lyapunov Safety Net"

The authors, Anis Hamadouche and Amir Hussain, propose a new way to train these robots so they don't fall off the track. They use a mathematical concept called Lyapunov stability.

Think of the robot's brain as a ball rolling down a hill (the layers of the neural network).

The Problem: Without a safety net, the ball might roll faster and faster, gaining so much energy that it flies off the cliff (the overflow wrap).
The Fix: They introduce a "Monotone Lyapunov Projection." Imagine a gentle, invisible hand that checks the ball's speed at every step. If the ball is moving too fast or has too much energy, the hand gently pushes it back to a safe speed, ensuring it never gains more energy than it started with.

This "hand" doesn't just clip the numbers; it actively monitors the "energy" of the robot's thoughts (hidden states) and forces them to stay within a safe, bounded circle. If the robot tries to jump out of bounds, the math forces it back in immediately.

The Experiment: A Race on MNIST

The team tested this on a famous dataset called MNIST (thousands of handwritten digits). They used a modern, compact type of AI called a Transformer (the same technology behind chatbots, but scaled down).

They compared four scenarios:

Standard Training: The robot learns normally but crashes when the numbers get too big.
Post-Training Quantization: They shrank the numbers after training. It worked okay at high precision, but failed miserably at low precision.
Quantization-Aware Training (QAT) without Safety: They tried to teach the robot to handle small numbers from the start. Disaster. Even with 16 bits of precision (which should be plenty), the robot's brain kept wrapping around, and it learned nothing, scoring near 10% (random chance).
The New Method (QAT + Lyapunov Safety): They taught the robot with the "invisible hand" safety net.

The Results

The results were dramatic:

Without the safety net: The robot failed completely. At 8 bits of precision, it got only 10.88% accuracy, and over 13% of its internal thoughts were corrupted by the "wrap-around" accident.
With the safety net: The robot learned perfectly. At 12 bits of precision, it achieved 86.55% accuracy (very close to the full-precision version). Most importantly, the "overflow" accidents dropped to almost zero (0.009%).

The Big Takeaway

The paper proves that simply giving a robot more bits (more precision) isn't enough if its internal logic is unstable. The "wrap-around" math of real hardware chips can destroy a robot's learning process.

By using this Lyapunov-guided training, they created a "hardware-safe" method. It's like teaching a driver not just how to steer, but also how to respect the speed limits of the road they are actually driving on, ensuring they never spin out of control. This allows us to build smaller, faster, and cheaper AI devices that won't crash when they hit the limits of their own memory.

Technical Summary: Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic

1. Problem Statement

The deployment of deep neural networks on resource-constrained hardware (e.g., FPGAs, embedded processors) relies heavily on low-precision arithmetic to reduce memory footprint, bandwidth, and energy consumption. While standard quantisation research focuses on rounding errors, clipping, and calibration, it often overlooks a critical failure mode inherent to practical fixed-point hardware: two's-complement overflow wrapping.

Unlike saturating arithmetic, where overflowed values are clipped to the maximum or minimum representable integer, wrapping arithmetic causes values to "wrap around" the finite integer range. Consequently, a large positive value may become a large negative value, and vice versa. In deep architectures like ResNets and Transformers, which can be viewed as depth-indexed dynamical systems, this wrapping introduces discontinuous sign and magnitude errors in hidden activations. These errors corrupt the state trajectory, leading to unstable numerical error propagation and catastrophic accuracy degradation, even when the nominal bit-width appears sufficient. Current quantisation-aware training (QAT) methods, which typically assume idealised clipping or ignore wrapping dynamics, fail to prevent this instability, often collapsing to near-chance accuracy under hardware-style wrapping.

2. Methodology

The paper proposes a Lyapunov-stabilised quantisation framework designed to explicitly control the internal state trajectory of neural networks operating under wrapping arithmetic. The core methodology treats the network as a discrete-time dynamical system and enforces stability through a layer-wise energy constraint.

System Model: The network is modeled as a sequence of hidden states $h_\ell$ updated by nonlinear transformations. The authors define a normalised layer-wise Lyapunov energy function:
$V(h_\ell) = \frac{1}{TD} \|h_\ell\|_F^2$
where $T$ is the number of tokens and $D$ is the embedding dimension. This energy metric represents the average magnitude of the hidden state.
Monotone Lyapunov Projection: To prevent the hidden state from entering overflow-prone regions, the authors introduce a hard projection mechanism. For an unconstrained updated state $z_{\ell+1}$ , the projected state $h_{\ell+1}$ is constrained to satisfy:
$V(h_{\ell+1}) \leq \min\{V(h_\ell), V_{\max}\}$
Here, $V_{\max}$ is a safe energy threshold derived from the 99th percentile of the initial training-set token energy, scaled by a safety margin ( $\gamma = 1.20$ ). The projection scales the state vector if its energy exceeds the target, ensuring the energy is both bounded and non-increasing across depth.
Hardware-Aware Forward Pass: A critical aspect of the method is the ordering of operations to account for wrapping. The proposed forward pass executes as follows:
1. Compute the unconstrained full-precision update.
2. Apply fixed-point quantisation with wrapping arithmetic (simulating hardware write-back).
3. Apply the Lyapunov projection to the wrapped state.
4. Perform a "repair projection" after a final quantisation step to ensure the fixed-point write-back does not violate the Lyapunov constraint.
Training Objective: In Quantisation-Aware Training (QAT), the standard cross-entropy loss is minimised subject to the layer-wise projected dynamics defined above. Unlike soft regularisation, this approach enforces a hard constraint on the state trajectory.

3. Key Contributions

The paper makes four primary contributions:

Problem Formulation: It frames fixed-point neural network quantisation with overflow wrapping as a hidden-state stability problem, distinguishing it from standard quantisation error analysis.
Mechanism Design: It introduces a monotone Lyapunov projection mechanism that enforces bounded and non-increasing layer-wise energy, directly constraining the hidden state rather than just the quantisation noise.
Hardware Awareness: The framework explicitly models two's-complement wrapping and fixed-point write-back, acting as a stabilisation mechanism specifically for hardware-style arithmetic.
Empirical Evidence: It demonstrates that activation overflow, rather than bit-width reduction alone, is a dominant failure mode in low-precision networks, and that stability control can restore learning performance.

4. Experimental Results

The method was evaluated on the MNIST dataset using a compact patch-based transformer (4 layers, 16 tokens, 64-dimensional embeddings) under signed fixed-point arithmetic with bit-widths ranging from 4 to 16 bits.

Unconstrained Failure: Standard QAT with wrapping arithmetic collapsed to near-chance accuracy (approx. 10–11%) across 6–16 bits, despite increasing precision. Activation overflow rates in these unconstrained models exceeded 11% (peaking at ~13.9% at 8 bits).
Stabilisation Success: The proposed Lyapunov-projected QAT restored stable learning. At 12 bits, the projected model achieved 86.55% ± 0.65% accuracy, compared to the unconstrained wrapped QAT's ~9.87%.
Overflow Suppression: The projection successfully suppressed activation overflow to 0.009% at 12 bits (compared to 13.6% in the unconstrained case).
Energy Control: The maximum normalised hidden-state energy was reduced from 6.26 (in full-precision unconstrained training) to 0.36 in the projected model, confirming the enforcement of a safe energy set.
PTQ Performance: Post-training quantisation (PTQ) also benefited significantly; at 6 bits, accuracy improved from 64.65% to 80.83% with the projection.

5. Significance and Claims

The paper argues that robust low-precision learning requires more than just quantisation-aware optimisation; it necessitates explicit control of the internal state trajectory. The authors claim that:

Stability is Critical: Unstable hidden-state growth in deep networks leads directly to overflow-induced state corruption under wrapping arithmetic, which standard quantisation methods fail to address.
Lyapunov Control as Safety: Lyapunov-based state control acts as an effective hardware-aware safety mechanism. By keeping the hidden-state energy within a "safe set," the network avoids the discontinuous sign changes caused by wrapping.
Trade-off: The method offers a meaningful stability-accuracy trade-off. While full-precision unconstrained training yields the highest accuracy (89.15%), it allows dangerous energy growth. The Lyapunov-constrained model sacrifices a modest amount of accuracy (approx. 2.2 percentage points in full precision) to achieve a 17.4x reduction in maximum hidden-state energy and enables reliable training at low bit-widths (e.g., 12 bits) where unconstrained methods fail completely.

The authors conclude that viewing low-precision inference and training as a dynamical stability problem, rather than solely a numerical approximation problem, is essential for deploying reliable neural networks on fixed-point hardware. Future work is suggested for larger datasets, deeper architectures, and hardware-in-the-loop implementations.