🤖 machine learning

Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features

This paper identifies a "Selection Plateau" in one-shot neural network pruning where all rank-monotone scorers converge to identical accuracy, and proposes the Sparsity-Information-Complexity Spectrum (SICS) hypothesis to demonstrate that escaping this plateau requires non-monotone features with complexity levels specifically adapted to the target sparsity.

Original authors: Guangqi Li, Yongxin Li

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guangqi Li, Yongxin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Good Enough" Wall

Imagine you are trying to trim a very large, overgrown hedge (a neural network) to make it smaller and faster. You need to decide which branches to cut and which to keep.

For decades, researchers have invented hundreds of different "scissors" (algorithms) to decide which branches to cut. Some scissors look at the thickness of the branch, others look at how much sunlight it gets, and others look at how much it shakes in the wind.

The Surprise: Even though these scissors look completely different on paper, when people use them, they all end up cutting the hedge to the exact same shape and size. They all hit a "performance wall." No matter how fancy the scissors are, they can't seem to do better than a specific level of accuracy.

The authors call this the Selection Plateau. It's like everyone is climbing a hill and hitting a flat, grassy top where no matter how hard you push, you can't go higher.

The Discovery: It Depends on How Much You Cut

The authors asked: Why do we hit this wall, and can we ever break through it?

They discovered that the answer depends entirely on how much of the hedge you are trying to cut away (the "sparsity"). They found three distinct zones, like different weather conditions for your gardening:

The "Easy" Zone (Cutting a little bit):
- The Situation: You only need to trim 30–40% of the branches.
- The Result: The simplest scissors work perfectly. Trying to use fancy, complex scissors actually makes things worse or stays the same. The "flat top" of the plateau is actually the best you can do here.
- Analogy: If you just need to trim the tips of the leaves, a basic pair of shears is perfect. Bringing out a laser-guided robotic arm is overkill and might just get in the way.
The "Critical" Zone (Cutting a medium amount):
- The Situation: You need to cut away about 70% of the branches.
- The Result: The simple scissors fail. You need a "smooth, wavy" pattern to decide what to cut. You need a tool that isn't just looking at thickness, but has a gentle curve that matches the specific point where you start cutting.
- Analogy: Now you are cutting deep into the bush. You need a tool that knows exactly where the "safe zone" ends and the "danger zone" begins. A simple straight cut won't work; you need a tool with a gentle bump that aligns with that boundary.
The "Extreme" Zone (Cutting almost everything):
- The Situation: You need to cut away 80% or more.
- The Result: The "smooth, wavy" tools now fail! They actually perform worse than the simple ones. To survive here, you need a tool with high-frequency "wiggles"—a chaotic, jittery pattern that looks like static on an old TV.
- Analogy: You are now trying to keep only the very strongest, most unique branches. A smooth curve isn't precise enough. You need a tool that can detect tiny, chaotic fluctuations that the smooth tools miss.

The "SICS" Rule

The authors named this discovery the Sparsity-Information-Complexity Spectrum (SICS).

Think of it like a video game difficulty setting:

Level 1 (Easy): Simple rules work.
Level 2 (Medium): You need a smooth, curved strategy.
Level 3 (Hard): You need a chaotic, jittery strategy.

If you use the wrong "weapon" for the level, you lose. If you use a "jittery" weapon on an easy level, you get confused. If you use a "smooth" weapon on a hard level, you miss the tiny details you need.

The "Fake" Scissors Test

One of the most interesting parts of the paper is a test they ran to see what actually matters.

They created a "fake" set of scissors that looked complex and wavy, but the waves were just a mathematical trick based on the branch thickness.

Result: These fake scissors failed miserably.
Lesson: It's not enough to just look complex. The complexity must be independent of the basic size of the branch. The "wiggle" has to come from a different source of information, not just a re-arrangement of the same old data.

The "Chaos" Ingredient

To create these complex "wiggly" tools, the authors used a mathematical concept called Chaos (specifically the Logistic Map).

They didn't claim that "chaos" is magic.
They just used chaos as a convenient way to generate shapes that were "wiggly" enough to pass the test.
They even tried making "hand-crafted" wiggles (like drawing a perfect bell curve) to see if they could get the same result.
Result: The hand-crafted bumps worked a little bit, but the "chaos" shapes worked much better. This suggests that the specific, messy, complex shape of the chaos tool contains hidden information that simple, perfect shapes miss.

Summary

The Problem: Different pruning methods all get stuck at the same accuracy level.
The Cause: They are all using the same basic "ranking" logic, which hits a limit.
The Solution: To break the limit, you must change your tool based on how much you are cutting.
- Low Cut: Keep it simple.
- Medium Cut: Add a smooth curve.
- High Cut: Add chaotic "wiggles."
The Takeaway: There is no single "best" pruning method. The best method depends entirely on how much of the network you are trying to remove.

Technical Summary: Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features

1. Problem Statement

Despite decades of research and the proliferation of diverse weight importance criteria for one-shot neural network pruning (ranging from magnitude-based methods to Taylor importance and Hessian-based approaches), empirical results consistently show that these functionally distinct methods converge to nearly identical downstream accuracies under matched sparsity budgets. Practitioners have long observed this convergence, yet a satisfying theoretical explanation has remained elusive. Furthermore, it is unclear under what conditions feature engineering beyond simple magnitude ranking can successfully improve pruning performance, or when such efforts are futile.

2. Core Phenomenon: The Selection Plateau

The authors formalize the Selection Plateau phenomenon. They demonstrate that for any rank-monotone scorer operating on weight magnitude (after standard per-layer rank normalization), the resulting channel selection is identical regardless of the scorer's specific functional form.

Mathematical Basis: Lemma 1 establishes that if two scorers $s_1$ and $s_2$ are strictly monotone functions of the per-layer magnitude rank, they induce the exact same Top-K subset of channels for any given sparsity $S$ .
Empirical Observation: Consequently, all such rank-monotone methods converge to a common accuracy plateau, denoted as $\Pi(S)$ . The height of this plateau varies sharply with sparsity but is universal across the construction of the scorer.

3. Methodology and Experimental Setup

To investigate when and how this plateau can be escaped, the authors propose the Sparsity-Information-Complexity Spectrum (SICS) hypothesis and conduct a systematic empirical study.

Architecture & Dataset: The study focuses on ViT-Small on CIFAR-10. The authors explicitly state they abstain from cross-domain claims at this stage, treating this as a single (architecture, dataset) setting.
Pruning Task: Structured channel pruning of MLP $fc1$ layers across 12 transformer blocks.
Sparsity Levels: Four levels were tested: $S \in \{0.5, 0.6, 0.7, 0.8\}$ (fraction of channels removed).
Feature Classes: The experiment tests nine distinct feature classes across four sparsity levels (36 cells total), categorized by a complexity metric $\kappa$ $κ$ :
- $\kappa=0$ (Rank-Monotone): Strictly monotone functions of rank (e.g., V1a peak_smooth, RandomSpline).
- $\kappa=1$ (Smooth Non-Monotone): Features with low-frequency non-monotone structure (e.g., smoothed Logistic Map orbit statistics like V1b var_smooth).
- $\kappa=2$ (High-Frequency Wiggle): Features containing fine-scale oscillations (e.g., raw, unsmoothed Logistic Map orbit statistics like A5).
- Controls: Includes a "fake" non-monotone scorer derived deterministically from magnitude (PureMag4) and non-chaos non-monotone controls (NL_sin, Weier4).
Optimization: Scorer fusion weights are searched using the Dung Beetle Optimizer (DBO) on a proxy validation set.

4. Key Contributions and Hypotheses

The SICS Hypothesis

The paper proposes that there exists a sparsity-dependent minimum feature complexity $\kappa(S) \in \{0, 1, 2\}$ required to escape the Selection Plateau $\Pi(S)$ . The hypothesis predicts three distinct regimes:

Low Sparsity ( $S \lesssim 0.65$ , $\kappa=0$ ): Rank-monotone scoring is empirically optimal. Non-monotone features add noise or degrade performance.
Critical Sparsity ( $0.65 \lesssim S \lesssim 0.75$ , $\kappa=1$ ): Smooth non-monotone features whose shape aligns with the channel-rank threshold provide a measurable accuracy escape.
Extreme Sparsity ( $S \gtrsim 0.75$ , $\kappa=2$ ): Only features containing high-frequency "wiggle" components succeed; smooth non-monotone features degrade below the plateau.

Key Findings and Counter-Intuitive Sub-Findings

The 36-cell experiment provides statistically significant evidence for the SICS structure:

Regime Verification:
- At $S=0.5$ and $0.6$, all features cluster at the plateau (confirming $\kappa=0$ ).
- At $S=0.7$ , smooth non-monotone features escape the plateau by +6.6% (relative to the plateau anchor), while high-frequency features perform similarly.
- At $S=0.8$ , only high-frequency (raw chaos) features escape (+2.6%); smooth features fall below the plateau.
Magnitude-Independence is Crucial: A "fake" non-monotone scorer derived purely from weight magnitude (PureMag4) underperforms even the gradient baseline. This indicates that arbitrary non-monotonicity is insufficient; the non-monotone structure must be statistically independent of the magnitude ranking.
More Indicators are Not Better: Increasing feature dimensionality (e.g., adding more orbit statistics) can lead to overfitting or noise accumulation, degrading performance (e.g., A5_Log6 underperformed A5).
Rank-Alignment is Necessary but Insufficient: Handcrafted parametric bumps (e.g., Gaussian) aligned with the channel-keep boundary at $S=0.7$ achieved only a +0.006 escape, compared to the +0.046 escape of the chaos-derived feature. This suggests that while alignment is necessary, the specific structural complexity of the chaos-derived shape provides additional, non-additive value.
Optimization Landscape: In the $\kappa=0$ zone, the optimization landscape exhibits extremely low variation (seeds converge to nearly identical weights), whereas the $\kappa=1$ zone shows significantly higher variation, suggesting a more structured landscape.

5. Mechanistic Interpretation

The paper offers a Taylor-expansion argument to explain the non-decreasing nature of $\kappa(S)$ :

Low $S$ : The pruning perturbation is small; first-order terms (magnitude/Taylor importance) dominate.
Critical $S$ : Second-order Hessian terms become significant. Capturing the coupling between channels requires non-monotone features, but the dominant eigenmodes are low-rank, so smooth ( $\kappa=1$ ) features suffice.
High $S$ : Third-order terms and fine-scale interactions become significant, requiring high-frequency oscillatory features ( $\kappa=2$ ) to resolve.

The authors also discuss a rank-alignment hypothesis, where the non-smooth features must align with the channel-keep boundary. However, they explicitly note that this alignment alone is insufficient to reproduce the full escape observed with chaos-derived features.

6. Significance and Limitations

Significance:

Unifying Explanation: SICS provides a framework explaining why diverse pruning methods cluster at similar accuracies (they fall into the same $\kappa$ regime) and predicts when feature engineering can or cannot improve upon baseline ranking.
Design Guidance: It suggests that future selection algorithms should adapt their feature complexity to the target sparsity rather than assuming a single "best" scorer exists for all sparsity levels.
Methodological Rigor: The paper includes a direct falsification test of a subsidiary hypothesis (that chaos is merely a convenient substrate for shape generation), updating its claims to reflect that the specific structure of the chaos-derived features is load-bearing.

Limitations and Scope:

No Cross-Domain Claims: The authors explicitly state they do not claim universality across architectures or datasets. The results are specific to ViT-Small/CIFAR-10.
Mechanism Verification: While the regime structure is empirically verified, the mechanistic explanations (Taylor expansion, rank-alignment) are presented as "conjectural mechanisms" or "candidate explanations" requiring further verification (e.g., direct Hessian spectrum measurement).
Future Work: The paper outlines future directions including testing on other architectures (ResNet, Swin), datasets (ImageNet), and selection problems (token pruning, KV cache compression, MoE routing), as well as developing a SICS-Adaptive Pruner algorithm.

In conclusion, the paper identifies a fundamental limitation in current pruning heuristics (the Selection Plateau) and proposes a sparsity-dependent hierarchy (SICS) that dictates the minimum complexity of features required to overcome this limitation, supported by rigorous empirical evidence and a nuanced analysis of feature independence and structural complexity.