🤖 machine learning

Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

This paper proposes a controllability-observability framework that treats trained deep neural networks as nonlinear dynamical systems to empirically identify and eliminate redundant hidden states via balanced realisation, achieving significant parameter compression and inference speedup on MNIST and CIFAR-10 with negligible accuracy loss.

Original authors: Anis Hamadouche, Amir Hussain

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anis Hamadouche, Amir Hussain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a massive, incredibly detailed factory to sort mail. This factory has thousands of workers (neurons) and endless conveyor belts (layers). The problem is, even though the factory works perfectly, it's huge, expensive to run, and slow. You suspect that many of the workers are just standing around doing nothing, or that some conveyor belts are carrying boxes that never actually get opened.

This paper proposes a new way to shrink that factory without breaking the sorting process. Instead of just randomly firing workers or making the conveyor belts thinner, the authors use a "control theory" approach to figure out exactly which parts of the factory are actually doing the heavy lifting.

Here is the breakdown of their method using simple analogies:

1. The Factory as a Journey

The authors view the neural network not just as a static list of numbers, but as a journey.

The Input: A piece of mail enters the factory.
The Hidden States: As the mail moves through the factory, it passes through different rooms (layers). In each room, the mail gets processed and changed.
The Output: The final sorted result comes out the other side.

The question is: How many rooms do we actually need, and how big does each room need to be?

2. The Three Tests (A, B, and C)

To answer this, the authors run three specific tests on the factory while it's running with real mail. They call these tests A, B, and C.

Test A: The "Who Shows Up?" Test (Reachability)
Imagine asking, "Which workers actually get called to action when mail arrives?"
- If a worker is never touched by the mail, they are useless.
- This test measures which parts of the factory are actually "excited" by the data. It finds the busy workers.
Test B: The "Who Matters?" Test (Observability)
Imagine asking, "If we tweak a specific worker's action, does it change the final result?"
- Some workers might be very busy, but if you change what they do, the final sorted mail looks exactly the same. They are busy but irrelevant.
- This test measures which workers actually influence the final output.
Test C: The "Sweet Spot" Test (Balanced)
This is the magic combination. It asks: "Which workers are BOTH busy with the mail AND actually change the final result?"
- If a worker is busy but doesn't matter, cut them.
- If a worker matters but is never busy, cut them.
- Keep only the workers who are both active and essential.

3. The Result: A "Realized" Mini-Factory

Most compression methods are like taking a photo of the factory and saying, "We only need 20% of this space." But you still have to keep the whole building.

This paper goes a step further. They take the results of Test C and actually build a new, smaller factory based on those numbers.

If Test C says the first room only needs 100 workers instead of 1,000, they build a room with exactly 100 workers.
They then train this new, tiny factory from scratch (or teach it by watching the big one).

4. What They Found

They tested this on two famous "mail sorting" datasets (MNIST and CIFAR-10).

The MNIST Experiment:
- Before: A factory with 1,024 "hidden" workers per room.
- After: They realized they only needed 277 workers.
- Result: They cut the size of the factory by 73%. The new, tiny factory was almost as accurate as the giant one (95.45% vs 96.60%).
The CIFAR-10 Experiment:
- Before: A massive factory with 4,608 workers per room.
- After: They realized they only needed 1,339 workers.
- Result: They cut the size by 71%. The new factory was just as accurate as the big one, but it ran 3 times faster on a computer chip.

Why This Matters

Usually, when people try to shrink AI, they just chop off random weights or lower the precision of numbers (like switching from high-definition to low-definition video). This paper is different because it looks at the flow of information.

It's like realizing that in your house, you have a giant dining room, but you only ever eat at a small table. Instead of just putting a smaller table in the big room, this method says, "Let's just build a small house with a small table and get rid of the empty dining room entirely."

In short: The paper provides a scientific way to find the "useful" parts of a neural network and rebuild the network to be much smaller and faster, without losing its ability to do the job.

Technical Summary: Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability–Observability Tests

1. Problem Statement

Deep neural networks (DNNs) frequently exhibit substantial redundancy in their hidden states, yet most existing compression techniques operate directly on weights, neurons, or numerical precision without explicitly characterizing the dynamical role of internal states. Standard methods such as pruning, quantization, and low-rank factorization often fail to answer a fundamental state-space question: which hidden-state directions are actually excited by the data distribution, and which of those directions are relevant to the output? Consequently, these methods may reduce parameter counts or storage but do not necessarily reduce the effective hidden-state order or the dense inference latency associated with wide layers.

2. Methodology

The paper proposes a control-theoretic framework for empirical state-order reduction, treating a trained feedforward DNN as a depth-indexed nonlinear dynamical system.

2.1 State-Space Formulation

The network is modeled as a sequence of nonlinear transitions:
$h_{\ell+1} = f_\ell(h_\ell), \quad z = g(h_L)$
where $h_\ell$ represents the hidden state at layer $\ell$ , and $z$ denotes the output logits. The input $x$ is treated as the initial state $h_0$ . The goal is to estimate the "useful hidden-state order"—the number of hidden directions that are both reachable from the data and observable at the output.

2.2 Empirical A/B/C Tests

The authors introduce three layer-wise tests based on empirical Gramians derived from hidden-state snapshots and output Jacobians:

Test A (Reachability): Measures which hidden directions are excited by the input data. It computes the empirical reachability Gramian ( $G_{c,\ell}$ ) as the covariance of centered hidden-state snapshots. The rank is determined by the spectral energy required to capture the data-induced variance.
Test B (Observability): Measures which hidden directions influence the output logits. It computes the empirical observability Gramian ( $G_{o,\ell}$ ) using the output Jacobians ( $J_\ell = \partial z / \partial h_\ell$ ) with respect to hidden states. The rank reflects directions that significantly affect the classifier output.
Test C (Balanced Reachable–Observable): Combines both criteria to identify directions that are simultaneously reachable from the data distribution and observable through the output. It constructs a balanced matrix $B_\ell = G_{c,\ell}^{1/2} G_{o,\ell} G_{c,\ell}^{1/2}$ . The resulting C-balanced rank ( $r_\ell$ ) estimates the minimal number of hidden states required to preserve the input-output behavior.

2.3 Realised vs. Projected Compression

A critical distinction is made between two modes of reduction:

Projected Reduction: Maps hidden states to lower-dimensional coordinates using learned bases. This diagnoses redundancy but retains the original full network weights and projection bases, failing to reduce storage or dense inference latency.
Realised Compression: Uses the C-balanced ranks ( $r_\ell$ ) directly as the actual hidden-layer widths of a new, compact DNN architecture. This new network is trained from scratch or via knowledge distillation, resulting in a genuine reduction in parameters and state order.

3. Key Contributions

State-Based Formulation: The paper formulates feedforward DNNs as depth-indexed nonlinear state-space systems, defining "hidden-state order" as a distinct compression-relevant quantity separate from parameter count.
Empirical Controllability–Observability Framework: It proposes data-driven A/B/C tests to estimate layer-wise useful hidden-state dimensions using empirical Gramians and Jacobians.
Realised C-Balanced Compression: It introduces a method where estimated C-balanced ranks are converted into the physical widths of a new, compact network, bridging the gap between theoretical model reduction and deployable architecture design.
Comprehensive Evaluation: The method is benchmarked against projection-based reduction, unstructured/structured pruning, low-rank SVD, dynamic INT8 quantization, and linear baselines on MNIST and CIFAR-10.

4. Experimental Results

The framework was evaluated on fully connected SiLU DNNs trained on MNIST and CIFAR-10.

4.1 MNIST Results

Compression: A 4-layer DNN with a full state order of 1024 was reduced to a state order of 277 (72.95% state compression).
Parameters: The parameter count dropped from 400,906 to 106,323 (73.48% compression, approx. 3.77 $\times$ reduction).
Accuracy: The realised C-balanced model achieved 95.45% accuracy, compared to 96.60% for the full model (a 1.15 percentage point drop).
Latency: Inference latency remained comparable to the full model due to the reduction in dense matrix dimensions.

4.2 CIFAR-10 Results

Compression: A larger DNN with a full state order of 4608 was reduced to 1339 (70.94% state compression).
Parameters: The parameter count decreased from 9.97M to 1.69M (83.09% compression, approx. 5.91 $\times$ reduction).
Accuracy: The realised model preserved accuracy at 54.44%, virtually identical to the full model's 54.45%.
Latency: CUDA inference latency was reduced by approximately 3.0 $\times$ (from 0.001582 ms/sample to 0.000525 ms/sample).

4.3 Comparison with Baselines

Pruning: Unstructured pruning (50%) maintained accuracy but did not reduce state order or dense latency. Structured pruning to C-balanced ranks without fine-tuning failed (34.50% accuracy), but with fine-tuning, it recovered to 91.75% (MNIST) or 55.22% (CIFAR-10), suggesting the C-balanced ranks provide valid target dimensions.
SVD: Low-rank SVD (0.5 fraction) caused significant accuracy degradation (85.50% on MNIST, 39.06% on CIFAR-10), indicating that layer-local weight factorization does not preserve the nonlinear input-output geometry as effectively as the state-based C-balanced approach.
Quantization: INT8 quantization reduced storage but left the state order and architecture width unchanged.

5. Significance and Claims

The paper claims that balanced reachable–observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures.

Theoretical Bridge: The work connects control-theoretic model reduction (specifically balanced truncation) with neural network compression, adapting empirical Gramians for nonlinear, data-driven systems.
Practical Utility: Unlike projection-based methods that only diagnose redundancy, the proposed "realised" approach converts these diagnoses into deployable, smaller networks with fewer parameters and reduced latency.
State-Order Focus: The results demonstrate that significant compression can be achieved by targeting the hidden-state order (the dimension of internal representations) rather than just the weight count, often with little to no loss in accuracy.
Modesty on Limitations: The authors acknowledge that the ranks are empirical and data-dependent, lacking worst-case theoretical error bounds for nonlinear systems. They also note the computational cost of estimating observability Gramians and the necessity of retraining the compressed architecture. The current experiments focus on fully connected networks to provide a controlled setting, with extensions to CNNs and Transformers noted as future work.

In summary, the paper argues that by viewing DNNs through the lens of controllability and observability, one can identify and remove redundant hidden-state directions that are either unexcited by data or irrelevant to the output, thereby enabling the construction of efficient, high-performance neural networks.