⚡ electrical engineering

On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems

This paper theoretically analyzes and empirically demonstrates the ability of Selective State-Space Models (SSMs) to perform online filtering for unknown systems, providing generalization bounds and comparing their performance against Transformer architectures.

Original authors: Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to predict the movement of a pendulum, but you don’t have the math formulas for it. You only have a video of it swinging. You have to watch the video and, in real-time, guess where the pendulum will be in the next second.

This paper explores a new way for computers to do exactly that. It compares two "brains" (AI architectures) to see which one is better at playing this game of "guess the next move" for mysterious, unknown systems.

The Two Competitors

1. The Transformer (The "Goldfish with a Library" approach):
Transformers (like the tech behind ChatGPT) are incredibly smart, but they have a specific way of thinking. To understand what is happening now, they look back at everything that happened before. Imagine a person trying to predict a dance move by looking at a giant stack of photos of every single move made so far.

The Problem: As the dance goes on longer, the stack of photos gets too heavy to carry. If the dance lasts too long, the Transformer "runs out of memory" or gets overwhelmed by the sheer amount of history it has to re-read every single second.

2. The Selective SSM (The "Experienced Athlete" approach):
The Selective State-Space Model (SSM), specifically a version called Mamba, works differently. Instead of carrying a massive stack of photos, it maintains a "mental summary" of the movement. It’s like an athlete who doesn't need to re-watch the whole game to know the rhythm; they just feel the momentum in their muscles.

The Advantage: Because it only carries a "summary" (a hidden state) rather than the whole history, it never gets overwhelmed. It can watch a dance that lasts for hours or even days without breaking a sweat.

What the Researchers Found

The researchers put these two "brains" through four different tests to see how they handle "unknown" systems:

Test 1: The Smooth Ride (Linear-Gaussian): Everything is predictable and clean.
- Result: Both did well, but the SSM did it using much less "brainpower" (fewer parameters) than the Transformer.
Test 2: The Sudden Twist (Dynamics Switch): Halfway through, the rules of the game suddenly change (like the pendulum suddenly getting heavier).
- Result: The Transformer got confused. The SSM, however, "felt" the change in rhythm and adapted much faster.
Test 3: The Static/Noise (Colored Noise): The "video" is very grainy and shaky.
- Result: The SSM was better at seeing through the "fuzz" to find the true pattern.
Test 4: The Long Haul (Length Generalization): The models were trained on short videos but then asked to predict much longer ones.
- Result: The Transformer eventually lost its way as the sequence got longer. The SSM kept its cool and stayed accurate.

The "Secret Sauce" (The Math Part)

The most important part of this paper isn't just that the SSM worked, but that the authors proved why it works using math.

They proved that because the SSM is "stable" (it doesn't let old, tiny errors snowball into giant mistakes), its ability to learn actually gets exponentially better as it sees more data. They provided a mathematical "guarantee" that the SSM is fundamentally better suited for tracking moving, changing systems than the Transformer is.

The Bottom Line

If you want an AI to write a poem or chat with you, a Transformer is great. But if you want an AI to pilot a drone, monitor a power grid, or predict the movement of a complex machine in real-time, the SSM is the superior tool because it is faster, lighter, and doesn't get "lost in the past."

Technical Summary: On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems

1. Problem Statement

The paper addresses the challenge of online filtering and trajectory prediction for unknown dynamical systems. In many control applications, the underlying system dynamics are unknown or difficult to model explicitly. While classical methods like the Kalman Filter (KF) or Particle Filters are optimal when a model is known, they fail when the model is unavailable.

Recent advancements have seen Transformer-based architectures used for "in-context learning," where the model infers system structure from observed input sequences. However, Transformers face limitations in control settings, such as fixed context windows (which can lead to safety/accuracy risks when using sliding windows) and quadratic computational complexity. The authors investigate whether Selective State-Space Models (SSMs), specifically the Mamba architecture, can serve as a more robust and efficient alternative for these filtering tasks.

2. Methodology

The authors frame the problem as an Empirical Risk Minimization (ERM) task.

Architecture: They utilize the Selective SSM, which uses a linear time-varying (LTV) state-space model. Unlike Transformers, SSMs operate via a recurrent map, meaning they have no fixed context window and can theoretically handle arbitrarily long sequences.
Training (Algorithm 1): The SSM is trained on a diverse set of $M$ trajectory samples drawn from a distribution of systems $\mathcal{D}_{sys}$ . The goal is to minimize the loss between the predicted next-step output $\hat{y}_t$ and the true output $y_t$ .
System Assumptions: To facilitate theoretical analysis, the authors assume the unknown systems are:
1. Starting from a zero initial condition.
2. Output-Lipschitz continuous.
3. Incrementally exponentially input-to-state stable (ISS), ensuring that perturbations in noise or initial states do not cause unbounded divergence.

3. Key Contributions

The paper provides both theoretical and empirical evidence for the efficacy of SSMs:

Theoretical Generalization Bounds: This is the paper's primary contribution. The authors derive the first generalization guarantees for SSMs in filtering applications.
- SSM Robustness (Theorem 1): They prove that SSMs are robust to input perturbations, showing that the error decay is governed by the stability of the state matrix $A$ and the contraction properties of the model.
- Excess Risk Bound (Theorem 2): They establish a bound on the excess risk (the difference between the trained SSM and the optimal SSM). Crucially, they show that the excess risk for SSMs decays at an exponential rate $\mathcal{O}(\rho^T \sqrt{T})$ , which is significantly stronger/faster than the sub-exponential decay found in Transformers.
Empirical Validation: They compare the Selective SSM against a Transformer (GPT-2) and a classical Kalman Filter across four distinct scenarios.

4. Results

The empirical evaluation uses Root-Mean-Square (RMS) prediction error across several settings:

Linear-Gaussian Setting: The Kalman Filter is optimal (as expected), but the SSM and Transformer both converge to stable performance. Notably, the SSM achieves this with significantly fewer parameters (662k vs. 3.3M for GPT-2).
Dynamics Switch (Robustness): When the system dynamics abruptly change at $T/2$ , the Kalman Filter (using a mismatched model) fails. Both learned models adapt, but the SSM recovers more quickly and achieves lower error than the Transformer.
Colored Noise (Non-white Noise): When noise is temporally correlated (violating standard KF assumptions), the SSM and Transformer outperform the mismatched KF by capturing the noise structure.
Length Generalization (Extrapolation): When tested on sequences longer than the training horizon, the Transformer's error increases significantly, whereas the SSM remains stable, demonstrating superior ability to extrapolate beyond the training window.

5. Significance

This work is significant because it bridges the gap between deep learning sequence models and classical control theory. By providing a rigorous mathematical foundation for why SSMs generalize well in dynamical contexts, the authors justify their use in safety-critical or long-horizon applications where Transformers might struggle. The findings suggest that SSMs are not just efficient alternatives to Transformers, but are theoretically better suited for the inherent stability and continuity requirements of dynamical system filtering.