⚡ electrical engineering

A Deep State-Space Model Compression Method using Upper Bound on Output Error

This paper proposes a provable compression method for deep state-space models that derives an output error upper bound based on layerwise $h^2$ -norms, enabling a gradient-based optimization approach that reduces trainable parameters by approximately 60% without retraining while maintaining performance on the IMDb task.

Original authors: Hiroki Sakamoto, Kazuhiro Sato

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hiroki Sakamoto, Kazuhiro Sato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, complex machine (a "Deep State-Space Model") that reads long stories and understands them perfectly. This machine is made of a chain of smaller, specialized workers (layers). Each worker takes a message, processes it, and passes it to the next person in line.

The problem is that this machine is huge and expensive to run. You want to shrink it down to make it faster and cheaper, but you're terrified that if you cut out too many parts, the final message will be garbled nonsense.

This paper introduces a clever new way to shrink the machine without having to retrain it from scratch, while guaranteeing that the final output remains accurate. Here is how they did it, explained simply:

1. The "Domino Effect" of Mistakes

The authors realized that when you shrink one of these workers, they make a tiny mistake. In a normal machine, you might think, "Well, if I shrink every worker a little bit, the total error is just the sum of all those tiny mistakes."

But this machine is special. It's like a game of telephone where the message gets amplified as it passes down the line.

The Discovery: They proved mathematically that a mistake made by a worker at the beginning of the chain (the "shallow" layers) causes a much bigger disaster at the end than a mistake made by a worker at the very end of the chain.
The Analogy: Imagine a leaky bucket brigade passing water. If the first person spills a cup, the last person might end up with an empty bucket. If the last person spills a cup, the bucket is already mostly full, so it doesn't matter as much.

2. The "Upper Bound" Map

Instead of trying to guess how the whole machine will behave (which is incredibly hard), the authors created a mathematical map (an "upper bound").

Think of this map as a "worst-case scenario" calculator. It tells you: "If you shrink the workers in this specific way, the final error cannot be worse than this number."
This map showed them exactly how to prioritize. To keep the final error low, you must be very careful with the early workers and can afford to be more aggressive with the late workers.

3. The "Smart Compression" Strategy

Using this map, they developed a new compression method.

Old Way: Shrink every worker by the same amount (e.g., cut everyone's size in half). This often leads to a broken machine because the early mistakes piled up.
New Way: Shrink the early workers only a little bit (keep them large and powerful) and shrink the late workers significantly.
The Result: They managed to cut the total number of "moving parts" (parameters) in the machine by 60% (from ~207,000 down to ~83,000).

4. The "One-Shot" Miracle

Usually, when you shrink a complex AI, you have to teach it all over again (retraining), which takes days of computing power.

The Paper's Claim: Because their method is based on this strict mathematical guarantee, they didn't need to retrain. They just took the trained machine, applied their "smart shrinking" rules, and it worked immediately.
The Test: They tested this on a task involving reading movie reviews (IMDb). The original machine got about 86.6% accuracy. Their tiny, 60%-smaller version got 86.7% accuracy. It was actually slightly better, and it did it without a single extra hour of training.

Summary

The paper is like a blueprint for shrinking a complex factory. Instead of randomly cutting machines out of the assembly line, they figured out that the first few machines are the most critical. By keeping the early machines big and shrinking the later ones, they built a smaller, cheaper factory that produces the exact same high-quality product as the giant one, without needing to retrain the workers.

Technical Summary: A Deep State-Space Model Compression Method using Upper Bound on Output Error

Problem Statement

Deep State-Space Models (Deep SSMs), such as S5 and Mamba, have demonstrated state-of-the-art performance in handling long-range dependencies without attention mechanisms. However, deploying these models often requires reducing their parameter count while maintaining performance. Existing Model Order Reduction (MOR) techniques for Deep SSMs typically apply compression independently to each linear state-space block (layer-wise) based on local criteria (e.g., Balanced Truncation or $H_2$ norms).

The authors identify a critical limitation in these existing approaches: minimizing the approximation error of individual linear blocks does not guarantee the minimization of the overall output error of the entire Deep SSM. Because Deep SSMs involve cascaded layers with nonlinear interactions (specifically quadratic outputs and Layer Normalization), local errors can accumulate or propagate unpredictably. Consequently, there is a lack of a theoretical framework that directly links layer-wise reduction strategies to the global output performance of the Deep SSM.

Methodology

1. Theoretical Framework: LQO Systems and Error Bounds

The paper models Deep SSMs as a cascade of Linear Quadratic-Output (LQO) systems. An LQO system is defined by a linear state-space update followed by a quadratic output map, which generalizes architectures like S5 where a quadratic activation follows a linear block.

The authors derive a provable upper bound on the $\ell_\infty$ output error between a full-order Deep SSM and a reduced-order Deep SSM.

Key Insight: The global output error is bounded by a weighted sum of the $h_2$ -norm approximation errors of the individual LQO systems at each layer.
Layer Sensitivity: The derived bound reveals that the contribution of a layer's approximation error to the total output error is not uniform. Specifically, errors in shallow layers (earlier in the network) have a significantly larger impact on the final output error than errors in deeper layers. This is due to the multiplicative effect of Lipschitz constants associated with Layer Normalization and the propagation of errors through subsequent layers.

2. Optimization Formulation

Based on the derived upper bound, the authors formulate a Model Order Reduction (MOR) problem. Instead of minimizing the direct output error (which is non-smooth and lacks a closed-form characterization), the optimization targets the minimization of the derived upper bound.

Objective: Minimize the weighted sum of the $h_2$ -norm errors between the original and reduced LQO systems at each layer.
Design Principle: To minimize this bound, the method prescribes assigning larger reduced state dimensions to shallow layers and smaller dimensions to deeper layers.

3. Algorithm: Gradient-Based MOR

The paper develops a gradient-based optimization algorithm (Algorithm 1) to solve the MOR problem:

Gradient Derivation: The authors derive closed-form gradients for the $h_2$ -norm error with respect to the parameters of the reduced LQO systems (matrices $\hat{A}, \hat{B}, \hat{C}, \hat{M}$ ). These gradients are adapted for discrete-time, complex-valued systems with diagonal state matrices.
Stability Guarantee: The algorithm employs a backtracking line search with an Armijo condition. Crucially, it explicitly enforces Schur stability (all eigenvalues inside the unit disk) for the reduced state matrix $\hat{A}$ at every iteration, preventing the generation of unstable models which can occur in standard MOR methods.
Initialization: The method can be initialized with existing time-limited MOR techniques (e.g., Time-Limited Balanced Truncation or Time-Limited $H_2$ reduction) but refines them using the proposed gradient descent to satisfy the specific Deep SSM constraints.

Key Contributions

Theoretical Bound: The paper provides the first theoretical derivation of an upper bound on the output error of a Deep SSM in terms of the $h_2$ -norm errors of its constituent LQO blocks. This establishes a direct link between layer-wise reduction and global performance.
Layer-Wise Sensitivity Analysis: The analysis proves that reducing approximation errors in shallow layers is more effective for minimizing the global output error bound than reducing errors in deeper layers.
Optimization Algorithm: A novel gradient-based MOR algorithm is proposed that minimizes the derived upper bound while guaranteeing the stability of the reduced model and preserving the diagonal structure of Deep SSMs.
Retraining-Free Compression: The method is designed to construct accurate compressed models from pretrained weights without the need for retraining (one-shot compression).

Experimental Results

The authors evaluated the method on the IMDb sentiment analysis task from the Long Range Arena (LRA) benchmark and the WikiText-103 language modeling task.

IMDb Task:
- A pretrained Deep SSM with 207,490 parameters achieved 86.66% test accuracy.
- Using the proposed method with a specific layer-wise dimension allocation ( $r_{list} = [96, 48, 36, 12]$ ), the model was compressed to 83,650 parameters (a ~60% reduction) while maintaining a test accuracy of 86.69%.
- The results confirmed that allocating larger dimensions to shallow layers (e.g., $[96, 48, 36, 12]$ ) yielded significantly better accuracy than uniform allocation (e.g., $[16] \times 4$ ) for the same total parameter count.
- The proposed method outperformed existing retraining-free compression baselines (like those in [12]) at comparable state dimensions.
WikiText-103 Task:
- Similar trends were observed where minimizing the derived upper bound correlated with improved Perplexity (PPL).
- Models with non-uniform dimension allocation (larger in shallow layers) achieved PPL close to the original pretrained model, whereas uniform allocation resulted in higher PPL.

Significance and Claims

The paper claims that its primary significance lies in providing a theoretically grounded, retraining-free compression strategy for Deep SSMs. By shifting the focus from local block approximation to a global error bound, the method offers a systematic way to determine where to compress a model.

The authors emphasize that their approach enables low-cost, high-performance deployment in scenarios where retraining is impractical due to computational constraints. The method validates that the "shallow-layer-first" reduction strategy is not just heuristic but is supported by the derived output error bound, offering a practical alternative to existing MOR techniques that often fail to preserve the overall system performance when applied layer-wise without global consideration.