🤖 machine learning

Revisiting the Volume Hypothesis

This paper resolves the apparent contradiction in the volume hypothesis by demonstrating that the generalization advantage of gradient-based learning over random sampling diminishes as training dataset size increases, suggesting that the hypothesis holds true primarily in smaller data regimes.

Original authors: Ari Pakman, Lior Kreimer, Yakir Berchenko

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ari Pakman, Lior Kreimer, Yakir Berchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Mystery: Why Do Super-Complex AI Models Work?

Imagine you are trying to find a specific key in a giant, dark warehouse filled with millions of keys. Most of these keys don't fit the lock at all. But, modern AI models are like a magical key-finder that, even though the warehouse is huge and full of useless keys, almost always finds a key that works perfectly.

This is strange because these AI models have way more "knobs" (parameters) than they need to solve the problem. In fact, they could easily just memorize the training data (like memorizing the answers to a specific test) and fail completely on new questions. Yet, they usually don't. They generalize well.

For a long time, scientists thought the magic was in how the AI learned (a process called "Stochastic Gradient Descent" or SGD). They believed the learning algorithm was a smart guide that steered the AI away from bad keys and toward good ones.

The New Theory: The "Volume Hypothesis"

A newer idea, called the Volume Hypothesis, suggests a different reason. It says: "Maybe the learning algorithm isn't that special. Maybe the 'good' keys just occupy a much bigger area of the warehouse than the 'bad' keys."

If the "good" area is huge and the "bad" area is tiny, then even if you just threw darts at the warehouse wall (randomly picking keys), you would be very likely to hit a good key simply because there's so much space to hit.

The Confusion: Two Experiments, Two Different Answers

Recently, two different groups of scientists tested this idea and got opposite results:

The "Small Data" Group: They tried to find good keys by randomly guessing in a warehouse with very few items (small datasets). They found that random guessing was terrible. The "smart" learning algorithm was still much better.
- Conclusion: The learning algorithm is the hero; the "Volume Hypothesis" is wrong.
The "Big Data" Group: They looked at warehouses with millions of items (large datasets). They found that the "good" area was indeed massive. Random guessing actually landed on good keys quite often, almost as well as the smart algorithm.
- Conclusion: The "Volume Hypothesis" is right; the learning algorithm isn't doing much heavy lifting.

The Paper's Solution: It Depends on the Size of the Warehouse

The authors of this paper realized that the two groups were looking at different sizes of warehouses. They decided to test the "middle ground"—warehouses of medium size—to see what happens as you add more items.

They used a special statistical tool (called the Wang-Landau algorithm) to map out the "volume" of the warehouse. Instead of just throwing darts, they calculated exactly how much space the "good" keys and "bad" keys took up at different dataset sizes.

Here is what they found:

In Small Warehouses (Small Datasets): The "good" keys are hidden in a tiny, hard-to-find corner. The "bad" keys are everywhere.
- Result: If you guess randomly, you will almost certainly pick a bad key. You need the smart learning algorithm (SGD) to guide you to the tiny good spot.
In Large Warehouses (Large Datasets): As you add more data, the "good" keys expand. The "bad" keys shrink. The "good" area becomes a giant, obvious island.
- Result: Now, if you guess randomly, you are very likely to land on a good key. The advantage of the smart learning algorithm shrinks and almost disappears.

The Final Takeaway

The paper solves the mystery by saying: Both sides were right, but they were looking at different stages of the process.

When you have little data, the "smart learning" is the most important thing. It acts like a flashlight in the dark, finding the few good solutions that exist.
When you have lots of data, the "architecture" (the design of the AI) does the heavy lifting. The "good" solutions naturally take up so much space that even a random guesser can find them.

The Analogy of the Needle in a Haystack:

Small Data: Finding a needle in a haystack is impossible by random chance. You need a magnet (the learning algorithm).
Large Data: Imagine the haystack grows so big that the needle turns into a giant steel beam. Now, you don't need a magnet; you just need to walk into the haystack, and you'll almost certainly bump into the beam.

The paper concludes that the "Volume Hypothesis" is true, but only when you have enough data for the "good" solutions to grow large enough to be found by chance. In the beginning, the learning algorithm is the hero; in the end, the sheer size of the data makes the architecture the hero.

Technical Summary: Revisiting the Volume Hypothesis

Problem Statement
Modern deep neural networks often possess far more parameters than necessary to fit training data, yet they generalize impressively to unseen data. While the prevailing explanation attributes this to the implicit bias of Stochastic Gradient Descent (SGD), a competing "volume hypothesis" suggests that within low training-loss regions, basins leading to strong generalization simply occupy a much larger volume of the weight space than poor-generalizing basins. Recent literature presents contradictory findings regarding this hypothesis:

Peleg & Hein (2024) found that randomly sampling weights until achieving zero training error (Guess-and-Check) yields poor generalization compared to SGD, suggesting optimization bias is dominant.
Yang et al. (2026) used molecular dynamics to estimate parameter densities, finding that for low training loss, the density peaks at generalization values similar to or better than optimized SGD, supporting the volume hypothesis.

The authors observe that these studies operated in different data regimes: Peleg & Hein used small datasets (up to 32 samples), while Yang et al. used large datasets. The central problem addressed is whether the validity of the volume hypothesis depends on the size of the training dataset, potentially reconciling these divergent results.

Methodology
To investigate the joint density of states over training accuracy ( $A$ ) and test accuracy ( $Q$ ), the authors employ the Replica Exchange Wang–Landau (REWL) algorithm. This technique, adapted from statistical physics, allows for efficient sampling of the parameter space to estimate the density of states $g(A, Q)$ , which represents the volume of network configurations yielding specific accuracy pairs.

Setup: Experiments were conducted on binary neural networks (weights $\in \{\pm 1\}$ ) across three architectures (Base, Deeper, Wider) and two datasets (MNIST, Fashion-MNIST).
Data Regimes: Training set sizes ( $D$ ) were varied systematically at 30, 300, and 600 samples to explore the transition from low-data to intermediate/high-data regimes.
Algorithm: The Wang-Landau algorithm performs a random walk in the weight space, reweighted by $1/g(A, Q)$ to ensure a flat histogram of visited $(A, Q)$ states. The modification factor $f$ is reduced iteratively until convergence.
Comparison: The peak of the density curve for zero training error ( $A=D$ ), denoted as $Q^* = \arg\max_Q g(A=D, Q)$ , represents the expected generalization of random sampling. This is compared against the generalization performance of standard (stochastic) gradient descent.

Key Contributions

Reconciliation of Contradictory Results: The paper bridges the gap between previous studies by demonstrating that the relationship between random sampling and gradient-based training is strongly data-dependent.
Data-Dependent Transition: The authors identify a transition regime where the generalization advantage of SGD over random sampling diminishes as the training data size increases.
Refined Volume Hypothesis: The study clarifies that the "volume hypothesis" is not a universal primitive independent of optimization. Instead, volume effects (architectural bias) emerge and concentrate as data constraints increase, while optimization-induced bias dominates in small-data regimes.

Results

Diminishing SGD Advantage: As the dataset size grows from 30 to 600, the gap in generalization accuracy between random sampling (randomly finding a zero-training-error solution) and SGD narrows significantly. In small-data regimes (e.g., $D=30$ ), SGD significantly outperforms random sampling. In larger regimes (e.g., $D=600$ ), the typical random interpolating solution approaches the performance of SGD.
Sharpening of Density Curves: Across all architectures and datasets, the curvature of the generalization accuracy density sharpens with increasing dataset size. This indicates that the probability volume of well-generalizing solutions becomes increasingly concentrated around a narrow range of test accuracies as more data is observed.
Architecture Effects: Wider networks consistently showed higher generalization accuracy in random sampling. The effects of depth were mixed, suggesting these architectural details are secondary to the primary data-size dependency of the volume hypothesis.

Significance and Claims
The paper claims to provide a unified explanation for the role of optimization versus architecture in overparameterized generalization. It posits that:

In small-data regimes, interpolating solutions are abundant but heterogeneous; SGD's implicit bias is essential to navigate to the rare, high-performing regions of the weight space.
In large-data regimes, the constraints imposed by the data concentrate the volume of valid interpolating solutions into a narrow band of high generalization. Consequently, the "typical" solution found by random sampling becomes competitive with SGD, and architectural volume effects become the dominant factor.

The authors conclude that architectural and optimization biases are complementary mechanisms whose relative importance shifts with dataset size. They emphasize that their findings are specific to binary-weight networks and moderate-scale architectures, serving to isolate fundamental geometric effects rather than to model modern large-scale continuous-weight training directly.