🔬 condensed matter

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

This paper identifies three distinct phases of learning in fully-connected neural networks under pruning—eumentia, dementia, and amentia—characterized by power-law loss scaling and a Berezinskii-Kosterlitz-Thouless-like transition, thereby establishing a statistical mechanics framework for understanding neural network behavior.

Original authors: Haining Pan, Nakul Aggarwal, J. H. Pixley

Published 2026-03-16

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Haining Pan, Nakul Aggarwal, J. H. Pixley

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, over-enthusiastic student preparing for a big exam. This student has millions of brain cells (neurons) and is so over-prepared that they could probably pass the test even if they forgot half of what they studied. This is how modern AI works: it's "over-parameterized," meaning it has way more parts than it strictly needs.

To make this student faster and cheaper to run, we use a technique called pruning. We try to remove the "useless" parts. But how much can we cut before the student stops working entirely? Does the student just get slightly slower, or do they suddenly forget everything?

A team of physicists decided to treat this AI student like a physical system (like water freezing into ice) to see if there are sudden "phase transitions" when we start cutting things out. They discovered three distinct "states of mind" for the AI, which they named after human cognitive conditions: Eumentia, Dementia, and Amentia.

Here is the breakdown of their findings using simple analogies:

1. The Three States of Mind

The researchers tested the AI by randomly turning off neurons (a process called Dropout) at two different times:

During Training: Turning off neurons while the student is studying.
During Testing: Turning off neurons while the student is taking the exam.

By adjusting how many neurons they turned off, they found three distinct phases:

🧠 Eumentia (The Healthy Brain)

What it is: The AI is healthy, smart, and learning perfectly.
The Analogy: Imagine a student who is studying with a few distractions. Even if they get distracted occasionally, they learn the material well. If you give them more textbooks (more data), they get even smarter.
The Science: In this phase, the more data you feed the AI, the better it gets. The "loss" (a measure of how wrong the AI is) goes down. This is the "Goldilocks" zone where the network is robust and learns effectively.

🧠 Dementia (The Forgetful Brain)

What it is: The AI studied hard and learned the material, but when you start the exam, you randomly turn off its neurons. It gets confused and starts making more mistakes as you give it more data.
The Analogy: Imagine a student who memorized the textbook perfectly. But, right before the exam, you start blindfolding them and covering their ears randomly. They know the answers, but they can't access them. Strangely, the more questions you ask them, the more they seem to forget, because the random "blindfolds" confuse their retrieval process.
The Science: Here, the AI is trained well, but the "evaluation dropout" (turning off neurons during the test) is too high. The AI fails to generalize. The more data you give it, the worse the error rate becomes. It's like a brain that has learned but lost the ability to recall.

🧠 Amentia (The Blank Slate)

What it is: The AI never learned anything in the first place.
The Analogy: Imagine a student who was so distracted during the study session (too many neurons turned off while learning) that they never actually absorbed the information. No matter how many textbooks you give them, they remain blank. They can't learn because the "noise" during training was too loud.
The Science: The training dropout is so high that the network cannot form any useful connections. The error rate stays flat and high, regardless of how much data you provide.

2. The "Magic" Transition (The BKT Transition)

The most exciting part of the paper is what happens between the Healthy Brain (Eumentia) and the Forgetful Brain (Dementia).

In physics, when water turns to ice, it happens at a specific temperature. It's a sharp line. The researchers found that the AI behaves similarly. As they slowly increased the "blindfolding" (dropout rate) during the test, the AI didn't just slowly get worse. It hit a critical tipping point.

The Analogy: Think of a bridge. You can add weight (data) to it, and it holds fine. But once you cross a specific weight limit, the bridge doesn't just sag; it snaps.
The Physics: They found this transition looks like a specific type of physics event called a Berezinskii-Kosterlitz-Thouless (BKT) transition. This is usually seen in 2D materials (like superconductors) where tiny whirlpools (vortices) suddenly unbind.
What it means for AI: It suggests that the "intelligence" of the network is a fragile, collective state. As long as you are above the line, the neurons work together in a coordinated dance. Once you cross the line, that coordination breaks down instantly, and the network falls apart.

3. Why This Matters

This study is a big deal because it treats AI not just as code, but as a physical system governed by laws similar to thermodynamics.

It explains "Over-parameterization": Why do we build huge AI models? Because having extra neurons makes the "Healthy Brain" phase wider and more stable. It gives the AI a buffer against errors.
It helps us prune safely: If we want to shrink AI models to save money and energy, we need to know exactly where the "tipping point" is. If we prune too much, we don't just get a slower model; we get a model that forgets everything (Dementia) or never learned (Amentia).
It connects to biology: The names (Eumentia, Dementia, Amentia) are a nod to how biological brains work. Just like our brains need a certain density of connections to function, AI networks have a "critical mass" of neurons required to maintain intelligence.

Summary

The paper tells us that AI networks have three moods: Learning, Forgetting, and Never-Knowing. There is a sharp, physics-like line between learning and forgetting. If you cut too many connections, the AI doesn't just get "a little dumb"; it undergoes a catastrophic phase change where its ability to learn or remember collapses entirely. Understanding this line helps us build smaller, cheaper, and smarter AI without accidentally breaking it.

1. Problem Statement

Modern neural networks are heavily overparameterized, containing far more parameters than necessary to fit training data. While pruning (removing redundant neurons or connections) is a standard technique for compression, the fundamental physics of this process remains poorly understood. Specifically, it is unclear whether pruning induces sharp phase transitions or smooth crossovers, and if sharp transitions exist, to which universality class they belong.

Current pruning methods (e.g., magnitude-based pruning) are often deterministic and coupled to learned weights, making it difficult to isolate the effect of pruning itself. The authors aim to address this by treating the pruning rate as a tunable control parameter to map the phase diagram of a neural network, viewing the network's ability to learn and generalize through the lens of statistical mechanics.

2. Methodology

The authors investigate Fully-Connected Neural Networks (FCNNs) trained on the MNIST dataset.

Architecture: FCNNs with varying depths ( $L=2$ to $5$) and widths ( $n_h=50$ to $200$), using ReLU activation functions.
Control Parameters: Instead of standard magnitude pruning, the authors employ Neuron Dropout as a stochastic pruning mechanism. Crucially, they independently vary two rates:
1. Training Dropout Rate ( $p_{train}$ ): Controls the network's ability to learn during the training phase.
2. Evaluation Dropout Rate ( $p_{eval}$ ): Controls the removal of neurons during the testing phase to probe the robustness of the learned state.
Metrics:
- Cross-Entropy Loss ( $L_{CE}$ ): The primary diagnostic metric, analyzed for its scaling behavior with respect to the training dataset size ( $N$ ).
- Classification Accuracy ( $A$ ): Used as a secondary metric.
Scaling Analysis: The core of the analysis involves studying the power-law scaling of the test cross-entropy loss with the dataset size $N$ :
$\langle L_{CE} \rangle \sim N^{\alpha_{CE}}$
The exponent $\alpha_{CE}$ serves as the order parameter to distinguish phases.

3. Key Contributions

Identification of Three Distinct Phases: The authors define three phases based on the sign of the scaling exponent $\alpha_{CE}$ $α_{C E}$ , drawing analogies to clinical neuroscience:
- Eumentia (Healthy): $\alpha_{CE} < 0$ . The network learns and generalizes effectively; loss decreases as dataset size increases. This corresponds to Quasi-Long-Range Order (QLRO) in statistical mechanics.
- Dementia (Cognitive Decline): $\alpha_{CE} > 0$ . The network trains successfully but fails to generalize when neurons are removed at evaluation. Counter-intuitively, increasing the dataset size increases the loss, indicating the network is "confused" by the pruning.
- Amentia (Developmental Failure): $\alpha_{CE} \approx 0$ . High training dropout prevents the network from learning altogether; the loss is insensitive to dataset size.
Phase Diagram Mapping: The authors map the phase boundaries in the $(p_{train}, p_{eval})$ plane, showing that the transitions are sharp and robust across different network architectures.
Universality Class Identification: The transition between the Eumentia and Dementia phases is identified as a Berezinskii-Kosterlitz-Thouless (BKT)-like transition.
- Unlike standard second-order transitions where the correlation length diverges algebraically ( $\xi \sim |p-p_c|^{-\nu}$ ), the authors find evidence of a stretched exponential divergence:
  $\xi \sim \exp\left( \frac{a}{|p - p_c|^\sigma} \right)$
- Finite-size scaling analysis yields a critical exponent $\sigma \approx 0.8 - 1.0$ , which is distinct from the standard BKT value of $\sigma = 0.5$ but consistent with a generalized QLRO scenario.

4. Key Results

Power-Law Scaling: The cross-entropy loss follows a strict power law with dataset size in all three phases, but the sign of the exponent dictates the phase.
Critical Behavior:
- At the transition point between Eumentia and Dementia, data for different dataset sizes $N$ cross at a specific critical evaluation dropout rate ( $p_{eval, c}$ ).
- Finite-size scaling collapse using the BKT ansatz (Eq. 6 in the paper) successfully collapses the data, confirming the nature of the transition.
- The goodness-of-fit ( $\chi^2_\nu$ ) is minimized for the BKT-like stretched exponential form, whereas a conventional algebraic fit requires an extremely large correlation length exponent ( $\nu \gtrsim 8$ ), effectively supporting the BKT interpretation (as $\nu \to \infty$ corresponds to the BKT limit).
Robustness: The three-phase structure persists across varying network widths ( $n_h$ ) and depths ( $L$ ). While the exact boundary locations shift, the existence of the phases and the BKT-like nature of the transition remain generic.
Accuracy vs. Loss: While accuracy drops sharply in the Dementia phase, the cross-entropy loss provides a more sensitive probe, revealing the "cognitive decline" (loss increasing with data) before accuracy hits random guessing levels.

5. Significance

Statistical Mechanics of Deep Learning: This work provides a concrete setting where neural network behavior can be rigorously understood using statistical mechanics concepts (phases, order parameters, critical exponents, and universality classes).
Beyond Lottery Ticket Hypothesis: While the Lottery Ticket Hypothesis suggests pruning reveals a "winning ticket," this paper characterizes the phase transitions that occur when the network is pushed beyond its capacity to maintain order, offering a more fundamental physical understanding of overparameterization and pruning limits.
Biological Analogy: The terminology (Eumentia, Dementia, Amentia) and the findings draw a compelling parallel to synaptic pruning in biological brains, suggesting that the mechanisms governing learning, forgetting, and developmental failure in artificial networks may share universal principles with biological neural systems.
Methodological Innovation: By decoupling training and evaluation dropout, the authors provide a clean experimental protocol to study network stability and phase transitions without the confounding factors of weight-dependent pruning strategies.

In conclusion, the paper establishes that pruning in neural networks is not merely a gradual degradation of performance but involves sharp, universal phase transitions, with the transition from learning to forgetting belonging to a BKT-like universality class characterized by quasi-long-range order.