🤖 machine learning

Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy

This paper proposes an unsupervised method for selecting essential neurons in overparameterized neural networks by minimizing mapping entropy, a metric based on hidden-activation statistics that effectively identifies informative subnetworks and enhances predictive performance under strong compression without relying on labels or gradients.

Original authors: Margherita Mele, Andrea Castagna, Roberto Menichetti, Raffaello Potestio, Alessandro Ingrosso

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Margherita Mele, Andrea Castagna, Roberto Menichetti, Raffaello Potestio, Alessandro Ingrosso

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems, particularly the deep neural networks that power everything from image recognition to language translation, are built with a surplus of parts. To learn a task, these systems often contain far more internal processing units, called neurons, than strictly necessary. This overabundance is not a mistake; it is a feature that allows the network to learn complex patterns and generalize to new situations. However, this abundance creates a puzzle: if the network has so many extra parts, which ones are actually doing the work, and which are merely redundant? Understanding this distinction is crucial for making these systems more efficient, faster, and easier to understand. The challenge lies in figuring out which neurons are essential without looking at the final answer the network produces or using the labels that tell the network what it got right or wrong. Instead, researchers are asking if the internal activity of the network itself—how its neurons fire and interact—holds the secret to identifying the most important components.

A team of researchers from the University of Trento and Radboud University has tackled this question by treating the network's internal state as a landscape that can be simplified. They focused on the hidden layer of a neural network, the middle section where raw data is transformed into abstract features. Their goal was to find a way to select a smaller group of neurons from this large crowd that could still tell the difference between different inputs just as well as the full group. To do this, they developed a method based on a concept called mapping entropy. Imagine trying to describe a complex scene to someone who can only see a few pixels at a time; if you choose the wrong pixels, you lose the ability to tell a cat from a dog. The researchers used a mathematical measure to quantify exactly how much information is lost when a specific set of neurons is removed. By searching for the specific combination of neurons that minimizes this loss of information, they could identify the most informative subset without ever needing to know what the network was supposed to be classifying.

The researchers tested this approach in two different ways. First, they used a controlled setup where they knew exactly how the network was supposed to be organized. In this scenario, a "teacher" network defined the correct way to process information, and a "student" network tried to learn it. When the student network perfectly copied the teacher, the method successfully identified the smallest possible group of neurons that still captured the full structure of the task. However, when the student network was not a perfect copy and had some extra, slightly different variations, the method automatically selected a larger group of neurons to account for that extra variability. This showed that the technique is sensitive to the actual statistical structure of the data, not just a pre-set idea of what the answer should be.

In a second, more complex experiment, the researchers trained a network to distinguish between two types of patterns that differed in how their parts were correlated. As the network learned, its neurons naturally split into two distinct groups: some focused on specific, localized parts of the input, while others responded to a broader, oscillating pattern across the whole input. The researchers found that the method did not just pick a random mix of these two groups. Instead, at the beginning of training, it selected almost entirely the localized neurons. As training progressed, the method shifted its preference, eventually selecting the oscillating neurons as the most informative group for a larger subset size. This demonstrated that the technique could track how the internal organization of the network changed over time, identifying which type of representation was currently the most efficient way to describe the data.

To see if these selected groups of neurons were actually useful, the researchers pruned the networks, keeping only the neurons chosen by their method and removing the rest. They then tested how well these smaller, trimmed networks performed on the original tasks. The results were clear: the networks pruned using this method consistently performed better than networks where neurons were removed at random. This advantage was most pronounced when the network was heavily compressed, meaning only a small fraction of the original neurons remained. In these tight conditions, the method's ability to find the right neurons made the difference between a network that could still recognize patterns and one that failed. The study also applied this technique to a standard image recognition task involving handwritten digits, where the network was trained to distinguish between the numbers one and seven. Even in this realistic setting, the method outperformed random selection, particularly when the network was forced to operate with very few neurons.

The findings suggest that the statistical patterns of how neurons fire contain enough information to identify the most critical parts of a neural network, without needing to look at the final output or the correct answers. This offers a new, completely unsupervised way to understand and compress artificial intelligence. It implies that the "intelligence" of a network is not just in its final decision, but in the specific way its internal parts are arranged to distinguish between different possibilities. While the method does not guarantee the absolute best possible reduction for every single task, it provides a reliable guide for finding efficient subsets of neurons. This approach could be valuable for creating smaller, faster models that can run on devices with limited computing power, and it offers a new lens for scientists to understand how these complex systems organize themselves to solve problems.

Problem Statement

Modern neural networks are typically overparameterized, possessing far more hidden units than strictly necessary to encode the target input-output relation. This raises fundamental questions regarding the internal organization of these networks: which neurons are essential for predictive performance, and can this distinction be identified without access to task labels or gradients? Existing approaches to efficiency and interpretability often rely on supervised signals (e.g., weight magnitude, saliency, or gradient-based scores) or random pruning strategies. The authors posit that the statistical structure of the hidden representation itself may encode which units are functionally important, independent of the specific task loss.

Methodology

The authors frame neuron selection as a coarse-graining (CG) problem. The hidden layer of a feed-forward network is treated as a microscopic configuration space. A coarse-grained description is defined by retaining a subset of $n_{cg}$ neurons out of a total width $K$ , effectively decimating the hidden layer.

To evaluate the quality of a specific subset (mapping), the authors employ the Mapping Entropy (ME) optimization workflow (MEOW).

Binary Configuration: Continuous hidden activations are binarized by their sign ( $s_i = \text{sign}(h_i)$ ) to define a binary configuration $\phi$ . This conservative choice avoids arbitrary binning and ensures robustness.
Mapping Entropy ( $S_{map}$ ): For a given mapping $M$ that projects microscopic configurations $\phi$ to reduced configurations $\Phi$ , the ME is defined as the Kullback–Leibler (KL) divergence between the original empirical distribution $p(\phi)$ and a back-mapped distribution $\bar{p}_M(\phi)$ . The back-mapped distribution is constructed by redistributing the probability of a reduced state uniformly among all compatible microscopic states.
$S_{map}(M) = \sum_{\phi} p(\phi) \log \frac{p(\phi)}{\bar{p}_M(\phi)}$
A lower $S_{map}$ indicates that the reduced representation preserves more of the statistical distinguishability of the original ensemble.
Optimization: The goal is to find the subset of neurons that minimizes $S_{map}$ . For small hidden layers, this is done via exhaustive search; for larger layers, stochastic simulated annealing (using the EXCOGITO software) is employed.

Experimental Setup

The study evaluates the MEOW criterion across three distinct scenarios:

Teacher-Student (TS) Systems: A controlled setting where a "student" network (width $K$ ) learns from a "teacher" network (width $M < K$ ). The student's hidden units are either analytically constructed as replicas of teacher units with a tunable mismatch parameter $\eta$ , or trained online. This allows for explicit monitoring of alignment between student neurons and teacher features.
Non-Linear Gaussian Process (NLGP): A synthetic binary classification task where hidden neurons spontaneously organize into two distinct functional classes: localised (weights concentrated on specific inputs) and oscillatory (extended alternating-sign profiles). This tests the method's ability to detect heterogeneity without explicit replication.
MNIST Binary Classification: A practical benchmark using digits 1 and 7 with translation augmentation, trained on a single hidden layer network.

Key Results

1. Recovery of Minimal Representations in Teacher-Student Systems
In the exact replicated limit ( $\eta=0$ ), where student neurons are perfect copies of teacher neurons, ME minimization selects a subset of size $n_{cg} = M$ that is perfectly balanced across all teacher modes ( $\Delta = 1$ ). This demonstrates that ME can identify the minimal teacher-consistent representation using only hidden-activation statistics.
When a mismatch is introduced ( $\eta > 0$ ), the optimal $n_{cg}$ required to maintain a balanced selection increases. The authors interpret this shift as a quantification of the residual variability in the hidden layer; as replicas become statistically distinguishable, more neurons must be retained to resolve the additional information. In trained networks, the effective mismatch $\eta_{eff}$ decreases over time, and the ME-selected subsets evolve accordingly, consistently recovering balanced mappings at sizes corresponding to the current level of alignment.

2. Detection of Functional Classes in NLGP
In the NLGP task, hidden neurons split into localised and oscillatory classes. ME minimization does not select a random mix of these classes but rather selects coherent mappings drawn from a single representational class.

Early Training: The optimal mappings are almost exclusively composed of localised neurons.
Late Training: As training progresses, the landscape shifts. For larger subset sizes, the optimal mappings transition to being dominated by oscillatory neurons.
Wang-Landau Sampling: Analysis of the density of states (DoS) confirms that ME minima lie at the edges of the mapping space (purely localised or purely oscillatory) rather than in mixed regions. The method successfully tracks the evolution of the "preferred" representational mode during learning.

3. Predictive Performance of Reduced Networks
The functional significance of the selected subsets was validated by pruning the networks and re-optimizing only the output bias.

NLGP: ME-selected subsets consistently outperform random subsets of the same size, particularly in the strongly compressed regime (e.g., retaining 8–12 neurons out of 30). The advantage diminishes as the subset size approaches the full layer width.
MNIST: Similar results were observed on the translation-augmented MNIST task. ME-selected subsets systematically achieved higher accuracy than random subsets for small to intermediate $n_{cg}$ , demonstrating that preserving configurational distinguishability translates to preserved predictive performance.

Significance and Claims

The paper claims that Mapping Entropy minimization provides a strictly unsupervised criterion for identifying informative subsets of neurons.

Unsupervised Nature: Unlike pruning methods based on weight magnitude or gradients, ME relies solely on the statistical structure of hidden activations. It does not require access to labels or the loss function.
Link to Performance: The work establishes a concrete link between information-theoretic optimality (minimizing loss of distinguishability) and predictive performance. ME-selected sub-networks degrade less than random ones under strong compression.
Interpretability: The method acts as a probe for the internal organization of the network, capable of revealing functional heterogeneity (e.g., distinct neuron classes in NLGP) and the evolution of representations during training.
Complementarity: The authors position MEOW as a complementary approach to standard supervised pruning techniques, offering a way to identify critical units based on the intrinsic geometry of the data representation rather than task-specific signals.

The paper concludes that the preservation of configurational distinguishability is a robust proxy for preserving task-relevant information, suggesting that the statistical structure of the hidden layer alone encodes which units are functionally important.

Problem Statement

Methodology

Experimental Setup

Key Results

Significance and Claims

More like this