📊 statistics

Gibbs randomness-compression proposition

This paper proposes and experimentally validates the "Gibbs randomness-compression proposition," which establishes a computable connection between model compression and directed randomness by demonstrating a high correlation between learning performance and Gibbs entropy measured over the remaining weights of compressed deep learning models.

Original authors: M. Süzen

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: M. Süzen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, chaotic library into a tiny backpack. You have to throw out books, but you want to keep the most important stories so you can still tell a great tale later. This is the heart of data compression: making things smaller without losing the magic. For decades, scientists have wondered about a strange connection between this "packing up" and randomness. Usually, we think of randomness as pure chaos—like static on an old TV or the unpredictable roll of a dice. But in the world of math and physics, there's a deep idea that the way we organize information (compression) and the way things behave randomly are actually two sides of the same coin. This paper steps into that conversation, asking a specific question: If we squeeze a smart computer brain (a neural network) to make it smaller, does the "randomness" inside it change in a predictable way? And can we use that change to tell us how well the computer will still work?

The authors of this paper, led by M. Süzen, propose a new idea called the Gibbs randomness-compression proposition. Think of a neural network as a giant, intricate web of connections, like a city with millions of roads. To make the city smaller (compression), they use a special method called Dual Tomographic Compression (DTC). It's a bit like taking a 3D scan of the city from two different angles at once, figuring out which roads are barely used, and then carefully removing them while the city is still running. They do this over and over, shrinking the city step-by-step.

Here is the big discovery: As they shrink the network, they measure two things. First, they check how well the network still does its job (like recognizing pictures of numbers). Second, they measure the "Gibbs entropy," which is a fancy math way of measuring how "random" or "disordered" the remaining connections look. The paper suggests a surprising rule: these two things move together in a highly synchronized way. As the network gets smaller and the "randomness" (entropy) drops, the performance drops in a very predictable, synchronized dance.

The authors tested this on a classic computer vision task: teaching a computer to recognize handwritten numbers from the MNIST dataset. They compared their fancy DTC method against two simpler ways of shrinking the network: just cutting out random roads (random pruning) and cutting out the smallest, weakest roads (magnitude pruning). The results showed that their method worked very well, keeping the computer smart even when the network was shrunk significantly.

Most importantly, they found a very strong link between the "randomness" they measured and the computer's performance. In fact, the correlation was so high—specifically 0.9174 for the DTC method and 0.9412 for random pruning—that it suggests a deep, mathematical connection: a lossy compression process (one that throws some information away) is essentially a form of "directed randomness." It's not just random chaos; it's a guided process where the amount of randomness tells you exactly how much the model's ability to learn has changed. The paper offers a logical proof and experimental evidence showing that randomness and compression are tightly bound, with a highly correlated relationship under specific mathematical bounds. By treating the shrinking of a neural network as a series of steps where randomness is carefully measured, the authors show that we can predict how a model will behave just by looking at its entropy. It's like realizing that if you know exactly how much the "messiness" of your backpack has changed, you can predict exactly how many books you can still read from it. This idea bridges the gap between the physics of entropy and the practical world of making AI smaller and faster.

Technical Summary: Gibbs Randomness-Compression Proposition

Problem Statement
The paper addresses the theoretical gap between the concepts of randomness, data compression, entropy, and information, particularly within the context of deep learning model compression. While connections between randomness and compression have been explored in the algorithmic information framework (Solomonoff-Kolmogorov-Chaitin), these relationships are often generic and lack strict mathematical conditions. Furthermore, as deep learning models grow in size, there is a practical need for efficient compression methods that preserve performance. The authors propose to establish a formal, quantitative bridge between the degree of randomness in a compression process and the resulting learning performance, utilizing Gibbs entropy as a metric.

Methodology
The study introduces a "train-compress" framework where compression is applied iteratively during the training process. The core methodology involves three main components:

Inverse Compressed Sensing (ICS): Unlike standard compressed sensing which reconstructs a signal from measurements, ICS constructs a hypothetical measurement process from known weights. The authors define a measurement vector $y = \Phi \cdot w$ (where $w$ are the learned weights) and reconstruct a sparse projection $w_r$ by minimizing an unconstrained $\ell_1$ -regularized objective. This generates "weight rays" representing sparse projections of the network weights.
Dual Tomographic Compression (DTC): This is a novel, complex pruning approach applied at the neuron level. For a layer with incoming and outgoing connections, the method sums the weight matrices to create input ( $w_p$ ) and output ( $w_q$ ) weight vectors. These are reconstructed via inverse compressed sensing in a dual fashion. The sum of these reconstructed vectors ( $w_p^r + w_q^r$ ) determines the ranking of neurons for pruning. Neurons are clipped based on quantile values corresponding to target sparsity levels.
Gibbs Entropy Measurement: At each compression cycle, the remaining weights serve as measurement vectors. The authors compute the Gibbs entropy ( $G_i$ ) of the normalized distribution of these vectors. This entropy serves as a proxy for the "degree of randomness" in the model state.

The authors compare DTC against two baseline compression methods:

Random Pruning: Randomly dropping neurons.
Magnitude Pruning: Removing neurons based on the magnitude of their weights (small weights removed first).

Key Contributions

Gibbs Randomness-Compression Proposition: The paper formulates a theorem stating that there is a highly correlated, comonotonic relationship between the learning performance ( $f_i$ ) and the Gibbs entropy ( $G_i$ ) of the measurement vectors during a directed, iterative compression process.
Formal Theorem: The authors provide a logical proof (Theorem 1) asserting that if the degradation in performance ( $|f_i - f_{i-1}| < \epsilon$ ) and the reduction in entropy ( $|G_i - G_{i-1}| < \delta$ ) remain small across consecutive cycles, the functions $f_i(s_i)$ and $G_i(s_i)$ are highly correlated. This suggests that a lossy compression process preserving performance is equivalent to "directed randomness" that maintains information content within specific entropy bounds.
Dual Tomographic Compression (DTC): The proposal of DTC as a more sophisticated compression mechanism than random or magnitude pruning, utilizing compressed sensing principles to guide the pruning process.

Experimental Results
The proposition was validated using a canonical vision task (MNIST classification) with a deep learning model starting at 512 hidden neurons. The experiments involved iterative pruning cycles with varying sparsity levels.

Performance: DTC demonstrated superior or comparable performance to random and magnitude pruning across various sparsity levels. Notably, DTC outperformed magnitude pruning, and while random pruning showed resilience at very high sparsity (>95%), DTC maintained robust performance.
Correlation: The study computed Pearson correlations between performance degradation and Gibbs entropy reduction. The results showed very high correlations: 0.9174 for DTC and 0.9412 for random pruning.
Entropy Evolution: Visual inspection of the data confirmed that as model complexity (sparsity) increased, the Gibbs entropy of the measurement vectors decreased, supporting the comonotonic behavior described in the theorem.

Significance and Claims
The paper claims to establish a "computable connection" between compression (probed by performance degradation) and randomness (measured via entropy). The authors argue that this finding provides a formal mathematical condition linking these concepts, moving beyond the generic statements of the SKC framework.

The significance lies in:

Theoretical Insight: It suggests that preserving learning performance during compression is intrinsically linked to the preservation of specific entropy characteristics, framing lossy compression as a form of "directed randomness."
Practical Utility: It offers a quantitative metric (Gibbs entropy) to probe the quality of compression processes in deep learning.
Methodological Advancement: It introduces DTC as a viable, complex pruning strategy that leverages compressed sensing principles.

The authors remain modest, noting that the proposition is demonstrated on a baseline model and that the "directed randomness" is guided by mathematical rules for reducing model size. The work aims to contribute to the ongoing discussion regarding the role of randomness in learning from a statistical physics perspective.

Technical Summary: Gibbs Randomness-Compression Proposition

More like this