🤖 machine learning

Regularity and Stability Properties of Selective SSMs with Discontinuous Gating

This paper establishes rigorous stability and regularity guarantees for selective State-Space Models (SSMs) like Mamba using control-theoretic tools such as passivity and Input-to-State Stability, and translates these findings into a differentiable training-time regularizer that significantly reduces stability violations with negligible impact on prediction accuracy.

Original authors: Nikola Zubić, Davide Scaramuzza

Published 2026-07-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikola Zubić, Davide Scaramuzza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Smart Memory" Problem

Imagine a robot trying to remember a long story while listening to a narrator. Modern AI models like Mamba are like these robots. They have a "memory" (a hidden state) that updates as they hear each new word.

The tricky part is that these models are selective. They don't just store everything equally. They have a "gatekeeper" that decides, word by word, what to remember and what to forget. If the gatekeeper is too chaotic, the robot might get confused, forget the beginning of the story, or go crazy when the narrator gets loud.

This paper asks: How do we make sure this gatekeeper is stable? How do we ensure the robot's memory doesn't explode or vanish, even when the rules for what to remember change instantly?

The Core Idea: Energy and Stability

The authors use tools from control theory (the math used to keep airplanes stable and robots balanced) to analyze these AI models.

They treat the AI's memory like a battery or a water tank:

Input: The new words coming in.
Output: The robot's understanding.
Storage: The memory inside the robot.

The paper argues that for the system to be stable, the "battery" shouldn't create energy out of nowhere. It should only store what it gets from the input, and it should naturally lose a little bit of energy over time (like a battery slowly draining) so that old, irrelevant memories fade away. This concept is called Passivity and Dissipativity.

The Two Signals: The "Switch" vs. The "Driver"

A key insight in the paper is separating two things that usually get mixed up:

The Driver (Input): The actual data (the words) pushing the system forward.
The Switch (Selection): The internal decision-making that changes how the system reacts to the data.

Analogy: Imagine a car.

The Driver is you pressing the gas pedal (the input).
The Switch is the transmission shifting gears (the selection).
The paper says: "Let's analyze the car's stability by looking at how the transmission shifts independently of how hard you press the gas." This makes the math much cleaner and reveals hidden rules about how the transmission must behave to keep the car from crashing.

The Main Findings

1. The "Forgetting" Rule

If the system is designed correctly (specifically, if it has a "strict dissipativity" property), it will naturally forget old information exponentially fast when there is no new input.

Analogy: Think of a leaky bucket. If you stop pouring water in, the bucket doesn't stay full forever; it drains. This is good! It means the AI won't get stuck remembering the first word of a sentence forever while trying to process the last word. The paper proves mathematically that this "leak" happens automatically if the system is built right.

2. The "Freeze" Test

The authors looked at what happens if they "freeze" the selection switch (stop changing the rules) and just let the input flow. They found that even with this frozen setting, the system has a natural, built-in "energy structure" (a mathematical shape called a quadratic storage function).

Analogy: Even if you take the gears out of a car and lock them in place, the engine and wheels still have a specific physical relationship. The paper shows that Mamba has a similar "natural shape" to its memory, which explains why certain ways of initializing the model (like HiPPO) work so well—they respect this natural shape.

3. The "Irreversible Forgetting" Rule

This is a fascinating structural finding. The paper suggests that once the system decides a piece of information is "useless" (it falls into a "kernel" or a zero-energy zone), no amount of switching the gates can bring it back.

Analogy: Imagine a shredder. Once a document goes into the shredder and is cut up, you can't just flip a switch and make the paper whole again. The system has a "point of no return" for certain types of information. This prevents the AI from getting confused by trying to resurrect old, irrelevant memories.

From Theory to Practice: The "Regularizer"

The authors didn't just stop at math. They built a practical tool called a Regularizer.

The Problem: In real life, we can't check the infinite math of the continuous world. We have to check the discrete steps the computer actually takes.
The Solution: They created a "speed bump" for the training process. During training, the model calculates a specific number (an eigenvalue) that tells it if it's violating the stability rules. If it is, the training process adds a tiny penalty to push it back toward stability.
The Result: They tested this on seven different real-world datasets (like weather, traffic, and electricity usage).
- Success: The tool reduced the number of "stability violations" by 92%.
- Cost: The accuracy of the predictions barely changed (less than 0.02% difference).
- Robustness: When they added noise or "spikes" to the data (like a sudden glitch), the model with the tool stayed calmer internally, though it didn't magically become perfect at predicting the future under chaos.

What the Paper Does Not Claim

The authors are very honest about the limits of their work:

It's not a magic fix for all errors: The tool makes the internal memory of the Mamba part of the AI more stable, but it doesn't fix the whole AI network (which has other parts like normalization and routing).
It doesn't guarantee perfect predictions under chaos: While the internal memory became more stable when the data was noisy, the final prediction accuracy didn't get significantly better. The tool makes the engine run smoother, but it doesn't necessarily make the car drive faster in a storm.
It's a "soft" rule: The tool encourages stability, but it doesn't force the model to be perfectly stable in a mathematical certificate sense. It just makes it much less likely to break.

Summary

This paper takes a complex, modern AI architecture (Mamba) and analyzes it using the physics of energy and stability. They proved that these models have a natural "leak" that helps them forget, and a "shredder" that prevents them from resurrecting useless memories. They then built a simple, low-cost tool that helps train these models to respect these rules, making them more stable and reliable without hurting their ability to predict the future.

Technical Summary: Regularity and Stability Properties of Selective SSMs with Discontinuous Gating

Problem Statement

Selective State-Space Models (SSMs), such as Mamba, have become central to long-sequence modeling. However, their stability properties remain poorly understood. Unlike classical Linear Time-Invariant (LTI) or smooth nonlinear systems, selective SSMs feature state-space coefficients ( $A, B, C$ ) that are modulated online by a token-dependent gating signal. This creates a system that is neither LTI nor classically nonlinear, but rather a hybrid system with discontinuous, data-dependent parameters. Existing analyses often focus on expressivity or treat the recurrence as a fixed dynamical system per input, lacking a unified energy-based account of how these models forget initial conditions, store energy, or maintain stability under perturbations.

Methodology

The authors approach the problem through the lens of classical control theory, specifically passivity, dissipativity, and Input-to-State Stability (ISS). A key conceptual innovation is the explicit separation of two signals:

Selection Signal ( $x(\cdot)$ ): Schedules the dynamics by determining the time-varying matrices $A, B, C$ .
Driving Input ( $u(\cdot)$ ): The port input that drives the dynamics and appears in the supply rate.

While token-selective architectures often set $x \equiv u$ , the authors maintain this separation to clarify the quantifiers in passivity and ISS analyses.

Theoretical Framework

The paper analyzes continuous-time selective SSMs defined by:
$\dot{h}(t) = A(\Delta(t), x(t))h(t) + B(\Delta(t), x(t))u(t)$
$y(t) = C(\Delta(t), x(t))h(t)$
where $\Delta(t)$ is an auxiliary gating signal (typically piecewise-constant). The analysis assumes mild $L^p_{loc}$ regularity, accommodating the discontinuities introduced by gating.

Key theoretical steps include:

Strict Dissipativity: Establishing conditions under which the system exhibits exponential forgetting of initial conditions.
Frozen-Selection Analysis: Freezing the selection signal ( $x(t) \equiv 0$ ) to reveal a passive Linear Time-Varying (LTV) subsystem. Leveraging recent results by Morandin & Hinsen (2024), the authors prove this subsystem admits a minimal quadratic storage function with Locally Absolutely Upper Semicontinuous (AUC $_{loc}$ ) regularity, even under discontinuous gating.
Universal Quadratic Storage: Investigating the structural constraints imposed if a single quadratic storage function must certify passivity across all admissible selection schedules. This leads to a parametric Linear Matrix Inequality (LMI) and the concept of "irreversible forgetting" (once a state direction enters the kernel of the storage matrix, it cannot be made observable again).
Global ISS: Deriving sufficient conditions for global Input-to-State Stability with respect to the driving input $u(\cdot)$ , uniformly over selection schedules.

Discrete Bridge and Regularization

To make the theory actionable for modern architectures, the authors translate the continuous-time parametric LMI into a sampled block LMI for the Mamba selective-scan recurrence.

They derive a one-step discrete dissipativity inequality.
They define a differentiable regularizer, $R_{LMI}$ , which penalizes the largest positive eigenvalue of the sampled block matrix $L_k(P, \beta)$ at each token step.
This regularizer is integrated into the training loop of a real Mamba-based architecture (SST) as a soft constraint.

Key Contributions

Two-Signal Model: A clean formulation of selective SSMs separating the selection signal from the port input, enabling rigorous passivity and ISS analysis.
Exponential Forgetting: A formal proof that strict dissipativity ( $\beta > 0$ ) combined with quadratic bounds implies exponential decay of homogeneous trajectories, uniformly over admissible selection schedules.
Canonical Energy Structure: Demonstration that the frozen-selection subsystem possesses a minimal quadratic storage function with AUC $_{loc}$ regularity. This provides an energy-based justification for principled initializations like HiPPO.
Structural Constraints & Irreversible Forgetting: Under the hypothesis of universal quadratic storage, the authors derive a parametric LMI and prove that the rank of the storage matrix is non-increasing. This formalizes "irreversible forgetting": state directions discarded by the energy structure cannot be recovered by the gating mechanism.
Global ISS Conditions: Sufficient conditions for global stability with respect to driving inputs, uniformly over selection schedules.
Discrete LMI Regularizer: A practical, differentiable training-time regularizer derived from the continuous theory, applicable to the Mamba selective-scan core.
Empirical Validation: Extensive evaluation on seven time-series datasets and four prediction horizons, demonstrating the regularizer's ability to reduce LMI violations without degrading clean accuracy.

Results

The empirical evaluation was conducted on the SST architecture (a routed mixture of Mamba and short-range attention) across 28 dataset-horizon pairs (ETTh1/2, ETTm1/2, weather, electricity, traffic).

LMI Violation Reduction: The discrete LMI regularizer reduced the mean sampled Mamba-core LMI violation by approximately 92% (from 0.0358 to 0.0029) across all 28 pairs.
Clean Accuracy: The reduction in LMI violations came at a negligible cost to clean Mean Squared Error (MSE), with a difference of less than 0.018% compared to the unregularized baseline.
Conservativeness Ablation: Replacing the fixed identity storage matrix ( $P=I$ ) with a learnable diagonal matrix ( $P=\text{diag}(q)$ ) yielded a strict Pareto improvement (lower violations, same accuracy) in all 28 pairs, confirming the framework's flexibility.
Robustness to Perturbations: Under injected perturbations (Gaussian noise, spikes, masks, etc.), the regularizer improved internal Mamba-core diagnostics (reduced LMI violations by ~60% and state norms by ~3.4%). However, it did not produce a measurable improvement in robust forecasting MSE. The authors explicitly note this as an "honest negative," attributing it to the fact that the regularizer constrains only the Mamba core, while the surrounding network (normalization, residuals, routing) absorbs the perturbation effects on the final output.
Router Stability: An ablation attempting to learn an energy-aware routing schedule (penalizing Mamba usage when local LMI violations are high) resulted in a neutral outcome. While Mamba weights were slightly nudged downward, the correlation between Mamba weight and LMI violation was essentially zero, suggesting that training on clean data does not induce energy-aware routing.

Significance and Claims

The paper positions itself as a bridge between classical control theory and modern deep learning architectures. Its significance lies in:

Theoretical Clarity: Providing a unified energy-based account for selective SSMs, moving beyond Lyapunov exponent diagnostics to structural properties like passivity and ISS.
Justification of Initializations: Offering a theoretical explanation for why initializations like HiPPO are effective (they provide a well-conditioned quadratic energy landscape).
Practical Tooling: Introducing a low-cost, differentiable regularizer that can be dropped into existing training pipelines to enforce structural stability on the SSM core.
Honest Scoping: The authors are explicit about the limitations of their claims. They do not claim the regularizer certifies the passivity of the entire deep network, nor do they claim it guarantees robust MSE improvements on its own. They frame the results as a "substantial reduction in violation" rather than "certified passivity," and acknowledge that the gap between core stability and end-to-end robustness remains an open research direction.

In summary, the paper successfully translates continuous-time control-theoretic guarantees into a verifiable, differentiable training criterion for selective SSMs, demonstrating that such structural constraints can be enforced with negligible accuracy cost while significantly improving internal stability diagnostics.