Entropy Reveals Block Importance in Masked Self-Supervised Vision Transformers

Technical Summary: Entropy Reveals Block Importance in Masked Self-Supervised Vision Transformers

Problem Statement
Masked self-supervised vision transformers (e.g., MAE, VideoMAE) have established a dominant paradigm for pretraining, learning rich visual representations from unlabeled data. However, their substantial model size creates significant barriers for deployment in resource-constrained environments (e.g., edge devices) and efficient transfer learning. A fundamental question remains: are all transformer blocks equally important for downstream performance?

Existing solutions face critical limitations. Sensitivity-based pruning, while effective, requires iterative block removal and full finetuning, incurring prohibitive computational overhead. Data-driven pruning methods rely on downstream data or activation statistics, introducing dataset bias and limiting applicability when data is unavailable or restricted. Unstructured pruning often results in irregular sparsity patterns that are difficult to deploy efficiently on modern hardware. Consequently, there is a need for a principled, data-free, and computationally efficient method to estimate block importance and identify redundancy without access to training data or gradients.

Methodology: Gardener and Weight-Number Entropy
The authors propose Gardener, a data-free, one-shot, block-level pruning algorithm for masked self-supervised vision transformers. The core insight is that block importance is encoded within the pretrained parameters themselves, specifically in the structural distribution of weights.

The method relies on a novel metric called weight-number entropy. Unlike conventional entropy definitions that operate on weight magnitudes or activations, this formulation characterizes the distribution of parameter counts across value intervals within a transformer block.

1. Formulation: For a block $B_l$ with $N_l$ scalar weights, the weight value range is partitioned into $K$ disjoint intervals. The empirical probability mass $p_l(i)$ is calculated as the ratio of weights falling into interval $i$ to the total number of weights in the block.
2. Entropy Calculation: The weight-number entropy $H(B_l)$ is computed as $-\sum p_l(i) \log p_l(i)$. This metric captures the structural diversity of weights within a block while remaining invariant to weight scaling.
3. Pruning Strategy: Gardener computes this entropy for every block in a single pass over the pretrained parameters. Blocks are ranked in ascending order of entropy. Lower entropy indicates a more concentrated distribution of parameters, which the authors hypothesize corresponds to lower importance for transfer learning. The algorithm removes the lowest-ranked blocks based on a target pruning ratio in a single shot, without requiring forward/backward propagation, data access, or iterative finetuning.

Key Contributions

• Problem Formulation: The paper defines the challenge of data-free block importance estimation for masked self-supervised vision transformers under resource-constrained settings.
• Theoretical Insight: It introduces weight-number entropy as an information-theoretic criterion computed solely from pretrained parameters, demonstrating a strong correlation with oracle sensitivity (derived from iterative block removal and finetuning).
• Algorithm Proposal: The authors propose Gardener, a linear-time complexity, one-shot pruning algorithm with negligible memory overhead that operates entirely without data.
• Empirical Validation: Extensive experiments on VideoMAE-B across multiple pruning ratios and downstream video recognition benchmarks (UCF101) show that Gardener consistently matches or outperforms existing data-free baselines and closely approaches the performance of sensitivity-based (oracle) pruning.
• Structural Analysis: The work provides empirical evidence of systematic structural heterogeneity in masked self-supervised models, revealing that early and intermediate blocks tend to have higher entropy (capturing diverse low/mid-level patterns), while deeper blocks show lower entropy (specializing in semantic abstraction).

Results
Experiments conducted on VideoMAE-B (pretrained on Kinetics-400) and finetuned on UCF101 yield the following findings:

• Block Heterogeneity: Removing different transformer blocks leads to drastically different downstream performance, confirming that blocks are not equally important. For instance, removing Block 1 caused a massive drop in Weighted Average Recall (WAR) to 46.73%, whereas removing Block 12 resulted in a WAR of 91.29% (compared to the original 91.78%).
• Correlation with Oracle: The ranking of blocks by weight-number entropy strongly aligns with the sensitivity-based oracle ranking. Blocks with higher entropy consistently correspond to larger performance drops when removed.
• Pruning Performance: Gardener outperforms other data-free criteria (including magnitude-based, variance-based, and random selection) across all pruning ratios. Notably, even after pruning up to 91.7% of the blocks, the pruned model retains competitive transfer performance (WAR of 68.81%), closely matching the sensitivity-based pruning result at the same ratio.
• Efficiency: The method incurs negligible computational overhead, operating in a single pass over parameters, making it orders of magnitude more efficient than iterative sensitivity-based methods.

Significance and Claims
The paper claims that information-theoretic analysis of pretrained weights offers a principled and efficient pathway for model compression and resource-efficient transfer learning. By demonstrating that block importance can be accurately estimated without any access to data, the authors challenge the necessity of data-driven or iterative pruning for masked self-supervised transformers.

The work highlights that substantial block-level redundancy exists in these models and that weight-number entropy captures intrinsic representational properties learned during pretraining. This allows for the identification of redundant blocks through simple measurements, enabling rapid model adaptation. The authors position Gardener not as a perfect replacement for all pruning scenarios but as a robust, low-cost surrogate that approximates oracle behavior under strict zero-data constraints, facilitating the deployment of large-scale foundation models in environments where data and compute are limited.

Limitations
The authors acknowledge that weight-number entropy is a coarse-grained, static measure that does not account for input-dependent behavior or task-specific activation patterns, which may limit adaptability to highly specialized downstream tasks. Furthermore, the current study focuses on block-level pruning in ViT-based masked autoencoders; extending these criteria to finer-grained structures (e.g., attention heads) remains an open question. Finally, while the correlation with sensitivity is strong, it is not guaranteed to be perfect across all architectures or training regimes.