🤖 machine learning

Task Relevance Is Not Local Replaceability: A Two-Axis View of Channel Information

This paper proposes a two-axis framework distinguishing between task relevance and local replaceability, demonstrating that a channel's ability to be substituted by its peers (local replaceability) is a more reliable predictor of pruning success than its direct contribution to the task.

Original authors: Houman Safaai, Andrew T. Landau, Celia C. Beron, Yasin Mazloumi, Bernardo L. Sabatini

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Houman Safaai, Andrew T. Landau, Celia C. Beron, Yasin Mazloumi, Bernardo L. Sabatini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are managing a large, busy kitchen (a neural network) where dozens of chefs (channels) are working together to cook a specific dish (solve a task, like recognizing a cat in a photo).

For a long time, when people wanted to know which chefs were "important" and which could be fired to save money (pruning the network), they used a single score. They asked: "How good is this chef at cooking the dish?" If a chef was great at making the sauce, they were kept. If they were okay, they were fired.

This paper argues that this single question is misleading. It hides a second, crucial question: "If we fire this chef, can the other chefs in the same kitchen station pick up the slack?"

The authors call this second property Local Replaceability.

Here is the breakdown of their discovery using simple analogies:

1. The Two Axes: "Relevance" vs. "Replaceability"

The paper proposes looking at every chef through two different lenses, or "axes":

The Target Axis (Relevance): This measures how much a chef contributes to the final flavor of the dish. It asks, "Does this chef know the secret recipe?"
The Local Axis (Replaceability): This measures how unique the chef is compared to their immediate coworkers. It asks, "If this chef leaves, do the other chefs standing right next to them already know how to do their job?"

The Big Surprise:
In a trained kitchen, these two questions often have nothing to do with each other.

Scenario A: You have a "Star Chef" who knows the secret sauce perfectly (High Relevance). But, they have three other chefs standing right next to them who are practically clones of them. If you fire the Star Chef, the others can instantly take over with no loss in quality. Verdict: Fire them! (High Relevance, but High Replaceability).
Scenario B: You have a "Junior Chef" who only adds a tiny pinch of salt (Low Relevance). But, they are the only one who knows how to chop onions in a specific way. No one else can do it. If you fire them, the whole dish fails. Verdict: Keep them! (Low Relevance, but Low Replaceability).

2. The Kitchen Evolves

When the kitchen first opens (random initialization), the chefs are all untrained. At this stage, being "good at the dish" and being "unique" are the same thing. The two axes are tightly locked together.

However, as the kitchen learns and trains:

The chefs start to specialize.
Some chefs become experts at the main task.
Others start to overlap and copy each other to make the workflow smoother.
Result: The two axes "uncouple." The chefs who are best at the task are no longer necessarily the ones who are hardest to replace. The paper shows that this separation happens quickly during training.

3. The "Safety Net" of Peers

The paper introduces a concept called Peer Support. Imagine a safety net made of your coworkers.

If you are a "Redundant Chef" (High Peer Support), your coworkers are already doing your job. You are a "local duplicate."
If you are a "Unique Chef" (Low Peer Support), your coworkers are looking at you, waiting for you to do your specific part.

The authors found that Peer Support is a better predictor of who can be fired than "How good are you at the task?"

4. The Experiment: Firing Chefs

To prove this, the researchers ran two types of tests:

The "Sudden Firing" Test (Lesions): They fired one chef at a time without retraining the kitchen. They found that firing a "Star Chef" often caused no damage because the "clones" nearby took over. But firing a "Junior Chef" with no backup caused a disaster.
The "Restructuring" Test (Pruning): They tried to cut down the number of chefs while keeping the kitchen running at the same speed (FLOPs-matched). They found that using a strategy based on Local Replaceability (firing the redundant ones) kept the kitchen running much better than using a strategy based on Task Relevance (firing the "less important" ones).

5. The Exception: The "VGG" Kitchen

The paper notes one exception. In a specific type of kitchen layout called VGG-16, the "Star Chef" score (magnitude/weight size) actually works quite well. This is because in that specific layout, the "Star Chefs" happen to be the ones who are also unique. But in modern kitchens (like ResNet or MobileNet), the old way of judging chefs by their "star power" alone is no longer accurate.

The Bottom Line

The paper concludes that we need to stop asking just one question: "Is this channel important?"

Instead, we must ask two:

What does this channel say about the task? (Relevance)
If we remove it, can its neighbors do the job? (Replaceability)

Pruning is not about finding the "best" channels; it is about finding the channels that are "safe to remove" because their neighbors are ready to step in. The most "important" channel isn't always the one you should keep; the one you should keep is the one that no one else can replace.

Technical Summary: Task Relevance Is Not Local Replaceability

Problem Statement

In vision networks, channel importance is typically summarized by a single scalar score. This practice conflates two distinct questions:

Task Relevance: How much information does a channel convey about the target task?
Local Replaceability: If a channel is removed, can the remaining channels in the same layer supply the necessary information to maintain network function?

The authors argue that these properties need not align. A highly predictive channel may be easily removable if peers carry overlapping information, while a less predictive channel may be critical if no peer covers its specific role. Existing metrics (magnitude, gradient sensitivity, reconstruction error, etc.) often compress these effects into a single value, failing to explicitly distinguish between task contribution and recoverability by same-layer peers.

Methodology

The paper introduces a two-axis view of channel information, separating task relevance from local replaceability. The analysis is performed post-hoc on frozen networks trained on CIFAR-100 (ResNet-18, VGG-16, MobileNetV2) using a fixed calibration set.

The Two Axes

Local Axis (Replaceability):
- Input Capture ( $I_X$ ): A proxy for how strongly a channel captures input variation, calculated via the Rayleigh quotient of the weight vector and input covariance ( $w_i^\top \Sigma_X w_i$ ).
- Peer Overlap ( $\bar{R}_X$ ): A measure of how much a channel's signal overlaps with same-layer peers, derived from the pairwise correlation matrix of channel activations.
- Definition: A channel with high $I_X$ and low $\bar{R}_X$ carries distinctive local structure and is harder to replace.
Target Axis (Relevance):
- Task Mutual Information ( $I(T; Y)$ ): Measures the information a channel holds about the task margin (ground-truth logit margin).
- Target Redundancy/Excess: Uses Gaussian Partial Information Decomposition (PID) proxies. The paper notes that under the scalable Gaussian-MMI (Maximum Mutual Information) definition, target redundancy collapses to singleton task relevance, making it distinct from local peer overlap.

Experimental Protocol

Structural Analysis: Measuring correlations and clusterings (Adjusted Rand Index) between the two axes across layers and seeds.
Training Trajectories: Tracking the alignment of the axes from random initialization to convergence to determine if separation is learned.
Gradient Analysis: Deriving a Gaussian linear model to explain the geometric decoupling of the two axes via residualized gradient directions.
Intervention Tests:
- Lesions: Removing individual channels without fine-tuning to measure immediate loss increase.
- Pruning: Structured pruning with FLOPs-matched recovery (fine-tuning) to evaluate accuracy retention.
- Peer Replacement: Replacing a removed channel's activation with a linear reconstruction from top peers to test recoverability.

Key Contributions

1. Structural Separation of Axes

The study demonstrates that in trained networks, the local and target axes are weakly aligned (Spearman $\rho \approx 0.01$ ) and define different channel groupings (low Adjusted Rand Index). This contrasts with random initialization, where the axes are strongly coupled ( $\rho \approx 0.7$ ). The separation is a learned phenomenon, not an artifact of initialization.

2. Mechanism of Decoupling

The authors propose a Gaussian residualization mechanism. In a linear-Gaussian model, the gradient for input capture follows the projected signal-power direction, while the gradient for task relevance follows a residualized task-covariance direction. When the task-covariance direction is weakly aligned with dominant input-power directions, the two gradients become nearly orthogonal, explaining the observed decoupling during training.

3. Pruning Predictability

Under a fixed FLOPs-matched pruning protocol, local-axis metrics are more reliable predictors of removability than target-axis metrics.

Lesion Evidence: Within bins of matched task relevance, peer-reconstruction $R^2$ (local support) predicts recovery from damage, whereas task relevance and input capture do not.
Pruning Results: Across ResNet-18, VGG-16, and MobileNetV2, the best local-axis methods outperform the best target-axis methods by significant margins (e.g., +8.0% to +10.1% AUC).
Architecture Dependence: While local replaceability is generally superior, the paper notes that in VGG-16, weight magnitude remains a strong baseline, suggesting that architecture-specific allocation details still matter.

4. Higher-Order Structure

The analysis reveals that local replaceability shifts from "singleton duplicates" in early layers to "distributed hulls" in deeper layers. Conversely, target-side information increasingly requires triplet-level synergy in deeper layers, suggesting middle layers are harder to compress from the target side due to distributed synergistic information.

Results

Weak Alignment: The correlation between input capture ( $I_X$ ) and task relevance ( $I(T; Y)$ ) drops sharply during training, settling near zero in final models.
Cluster Divergence: Channels grouped by local features (redundancy) and target features (relevance) show near-random overlap (ARI $\approx 0.02$ ), confirming they identify different subsets of channels.
Pruning Performance:
- ResNet-18: Best local method (+8.0 pp) > Best target method.
- VGG-16: Best local method (+6.6 pp) > Best target method, though magnitude (a norm-based baseline) remains competitive.
- MobileNetV2: Best local method (+10.1 pp) > Best target method.
- Stress Tests: The local-over-target hierarchy holds on CIFAR-10, Tiny-ImageNet, ImageNet-100, and a ConvNeXt-T pilot, though the degree of axis separation varies.
Gradient Orthogonality: Empirical measurements confirm that the SGD update direction becomes nearly orthogonal to both information gradients after early training, supporting the residualization hypothesis.

Significance and Claims

The paper claims an axis-level distinction rather than a universal ranking of pruning scores.

Core Insight: Pruning is fundamentally a replaceability intervention. It asks whether a channel is still needed when the rest of the layer is available. Task relevance explains what a channel represents, but local replaceability explains whether the network needs it.
Limitations: The strongest evidence is derived from CNNs on CIFAR-100. While stress tests on ImageNet and ConvNeXt support the trend, the authors do not claim universal applicability across all architectures (e.g., Transformers) without further validation.
Practical Implication: The findings suggest that pruning heuristics should explicitly account for local peer support. Norm-based scores (like magnitude) may remain competitive in specific architectures (like VGG-16) where they act as a proxy for local input capture, but they fail to capture peer overlap, which is the critical factor for removability.

The work reframes channel importance not as a single scalar but as a two-dimensional property, offering a more nuanced diagnostic for understanding network redundancy and guiding pruning strategies.