🤖 machine learning

How the Hessian-Spectrum of Neural Networks Depends on Data

This paper derives the eigenvalues of the Hessian matrix for linear networks with arbitrary architectures and datasets, revealing that solution sharpness in classification tasks is directly determined by the maximum proportion of samples in any single class, while demonstrating that these theoretical insights remain robust even when relaxing simplifying assumptions and introducing nonlinearities.

Original authors: Jasraj Singh, Enea Monzio Compagnoni, Antonio Orvieto

Published 2026-07-16

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Jasraj Singh, Enea Monzio Compagnoni, Antonio Orvieto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize cats, dogs, and birds. You don't just hand it a picture and say "learn"; you give it a massive, invisible landscape of hills and valleys. Every time the robot guesses wrong, it slides down a slope toward a better answer. This landscape is called the "loss landscape," and the robot's journey across it is "optimization." But here's the tricky part: the landscape isn't just bumpy; it's a wild, twisting mountain range with cliffs, flat plains, and sharp peaks. To understand how the robot moves, scientists look at a mathematical tool called the Hessian matrix. Think of the Hessian as a topographical map that tells you exactly how steep the ground is at any given spot. If the ground is very steep (sharp), the robot might bounce around wildly; if it's flat, the robot might get stuck or move too slowly. Understanding this "steepness" helps scientists build better robots that learn faster and make fewer mistakes.

Now, a team of researchers at the University of Basel and the ELLIS Institute in Tübingen decided to dig into the math behind this map. They wanted to know: How does the shape of the data itself change the steepness of the landscape? They built a mathematical model of a "neural network" (a type of AI) and asked, "If we feed this network different kinds of data—some with many samples, some with weird features, some with imbalanced labels—how does the Hessian change?" They didn't just guess; they derived exact formulas for the "eigenvalues" (the numbers that tell you the steepness) of this map. Their big discovery? The steepness of the solution isn't just about how complex the network is; it's directly tied to the distribution of the data. Specifically, if one class of data (like "cats") is way more common than others, the solution becomes "sharper." They found that while their math was built on some idealized assumptions (like perfectly round data clouds), the rules they discovered held up surprisingly well even when they broke those rules and added real-world messiness like non-linear activations.

The Shape of the Learning Landscape

To understand what these researchers found, let's first meet the players. They studied a "linear neural network," which is a simplified version of the AI brains we use today. Imagine a factory assembly line where a raw material (the input data) passes through several stations (layers) to become a finished product (the prediction). The "weights" are the settings on the machines at each station. The goal is to adjust these settings so the product matches the target perfectly. The researchers used a "Mean-Squared Error" (MSE) loss, which is just a fancy way of saying they measured the distance between the robot's guess and the real answer, squared it, and tried to make that number as small as possible.

To see how the robot moves, they looked at the Hessian, a giant grid of numbers that describes the curvature of the error landscape. Instead of calculating the exact, messy Hessian (which is computationally heavy), they used a clever shortcut called the Generalized Gauss-Newton (GGN) approximation. Think of this as using a satellite photo to estimate the terrain instead of hiking every inch of it. As the robot learns and the error gets smaller, this satellite photo becomes incredibly accurate.

The Rules of the Game

The researchers started by setting up a very clean, ideal world to solve the math. They assumed the data was "isotropic," meaning the features were spread out perfectly evenly in all directions, like a perfectly round cloud of points. They also assumed the network layers were "strongly balanced," meaning the settings in one layer were perfectly aligned with the next, like a synchronized dance troupe.

Under these perfect conditions, they discovered a beautiful pattern. For a simple two-layer network, the steepness of the landscape (the eigenvalues) is determined by the sum of the squared "strengths" (singular values) of the weights in each layer. It's like saying the total steepness of a mountain is the sum of the steepness of its two main slopes. They found that the sharpest point of the landscape is simply the sum of the squared sizes of the weights in the first and second layers. This contradicted an earlier idea that suggested the sharpness was just the largest of the two, proving that both layers contribute to the total steepness.

When they extended this to deeper networks (more than two layers), they found that if the layers remain "balanced" (the dance troupe stays in sync), the steepness follows a specific formula involving the number of layers and the strength of the weights. A key finding here is that most of the landscape is actually flat! Out of thousands of possible directions the robot could move, only a tiny fraction are actually steep; the rest are near-zero. This explains why AI models often seem to have a "bulk" of flat directions, a phenomenon observed in real-world experiments.

How Data Shapes the Terrain

The most exciting part of the paper is how the data itself dictates the shape of this landscape. The researchers asked: "What happens if we change the dataset?"

Dataset Size: Surprisingly, if you keep the data points consistent, the sharpness of the solution does not depend on the number of samples. Whether you have 100 pictures or 10,000, the steepness of the final solution remains the same. This challenges some previous beliefs that more data always leads to a sharper (or flatter) landscape in a specific way.
Depth: The number of layers matters. If the input data is "smaller" than the output labels, making the network deeper actually makes the solution sharper. It's like adding more steps to a staircase; if the steps are uneven, the whole structure becomes more precarious.
Feature Magnitude: If your data features are large and spread out (high variance), the solution becomes sharper. Imagine trying to balance on a tightrope that is stretched very tight; it's more sensitive to movement than a loose rope.
Label Distribution (The Big Discovery): This is the paper's "smoking gun." For classification tasks (like sorting cats, dogs, and birds), the sharpness of the solution is directly related to how imbalanced the classes are. If one class has a disproportionately large number of samples (e.g., 90% cats, 10% dogs), the solution becomes sharper.
- Wait, isn't a dataset with one dominant class easier to learn? Intuitively, yes. It's easier to guess "cat" if almost everything is a cat. However, the math shows that this "easy" solution sits on a sharper peak.
- This contradicts a previous idea that simpler datasets lead to "flatter" (more robust) solutions. The authors suggest that while the learning might be easier, the mathematical landscape is actually more precarious (sharper) when the data is imbalanced.

Testing the Theory in the Real World

The researchers knew their math relied on "perfect" assumptions (round clouds of data, balanced layers). So, they did something brave: they broke the rules one by one to see if their theory survived.

Breaking the "Round Data" Rule: They used real, messy data (like images of MNIST digits and CIFAR objects) that wasn't perfectly round. Even though the exact math didn't hold perfectly, the trend remained. The sharpness still correlated with the imbalance of the labels.
Breaking the "Balanced Layers" Rule: They initialized the network randomly (the standard way AI is built) instead of forcing it to be balanced. Again, the theory held up. As the network trained, it naturally became more balanced, and the sharpness still followed their predictions.
Adding Non-Linearity: They added "Tanh" activation functions (a common non-linear twist in AI) to make the network more like a real brain. The results were slightly different from the perfect math, but the qualitative behavior was the same. The imbalance of labels still drove the sharpness.

The Takeaway

In simple terms, this paper tells us that the "steepness" of an AI's learning journey isn't just a property of the AI itself. It is deeply rooted in the geometry of the data. If your data is lopsided—with one class dominating the others—the AI finds a solution that is mathematically "sharper," even if that solution is easier to find. The authors suggest that this sharpness is a direct consequence of the data's structure, specifically the maximum proportion of samples belonging to any single class.

While their findings are derived from linear networks and specific mathematical setups, the fact that these rules persist even when they add real-world complexity (non-linearities, unbalanced weights, messy data) suggests that this relationship between data distribution and landscape sharpness is a fundamental truth of deep learning. It's a reminder that in the world of AI, the data you feed the machine doesn't just teach it what to learn; it shapes how the learning landscape looks, determining whether the robot slides gently into a solution or balances precariously on a sharp peak.

Technical Summary: How the Hessian-Spectrum of Neural Networks Depends on Data

Problem Statement

The Hessian matrix is a critical tool for analyzing loss landscapes, optimization dynamics, and generalization in deep learning. However, prior research has largely relied on either empirical observations or theoretical treatments under overly simplified settings. This work aims to precisely characterize the full spectrum of the Hessian for neural networks, specifically investigating how the geometry of the data influences the sharpness of the learned solution. Unlike previous works that prioritize minimal assumptions at the cost of model complexity, this paper leverages standard deep learning assumptions to derive results for networks with arbitrary widths and depths and datasets with arbitrary numbers of samples, features, and labels.

Methodology

The authors analyze $L$ -layer linear neural networks defined as $f(x; W_{1:L}) = xW_1^\top \dots W_L^\top$ , trained on inputs $X$ and outputs $Y$ using Mean-Squared Error (MSE) loss.

Hessian Approximation: The study utilizes the Generalized Gauss-Newton (GGN) approximation of the Hessian. As training reduces the loss, the GGN becomes increasingly precise. The non-zero eigenvalues of the GGN are shown to be identical to those of the Neural Tangent Kernel (NTK) matrix, which is analytically more tractable.
Theoretical Assumptions:
- Assumption 1 (Isotropic Features): The feature matrix $X$ is full-rank and isotropic ( $\Sigma = \Sigma_{11}I$ ), representing an overdetermined system ( $n \ge d_0$ ).
- Assumption 2 (Strongly Balanced Layers): Consecutive layers satisfy $W_\ell W_\ell^\top = W_{\ell+1}^\top W_{\ell+1}$ . This implies alignment between the left singular vectors of one layer and the right singular vectors of the next. The authors note that while arbitrary parameter points need not be balanced, gradient flow or weight-decay training tends to induce this balance, particularly at the edge-of-stability (EoS).
Analytical Derivation: The authors derive bounds and exact characterizations for the eigenvalues of the NTK (and thus the Hessian) by utilizing the Singular Value Decomposition (SVD) of the feature matrix and the singular values of the weight matrices.

Key Contributions and Results

1. Characterization of Hessian Eigenvalues

Shallow Networks ( $L=2$ ): The non-zero eigenvalues are characterized by the pairwise sums of the squared singular values of the two weight layers. Specifically, $\lambda_{ij} \le \frac{\|X\|_2^2}{n} (\sigma_i^2(W_1) + \sigma_j^2(W_2))$ . The bound is attained under Assumption 1.
Deep Networks ( $L \ge 2$ ): Under the assumption of strongly balanced layers, the non-zero eigenvalues are determined by a specific ratio of powers of the singular values of the weights. The number of non-zero eigenvalues is bounded by $(\min \{d_0, \dots, d_L\})^2$ , explaining the empirical observation that the bulk of Hessian eigenvalues are near-zero.

2. Dependence on Data Geometry

The paper derives explicit relationships between the spectral norm (sharpness) of the Hessian and data properties:

Dataset Size: For balanced solutions, the sharpness is independent of the dataset size $n$ , provided the entries of $X$ and $Y$ scale as $\Theta(1)$ . This contrasts with findings suggesting sharpness increases with $n$ .
Depth: Sharpness increases with depth $L$ when $\|X\|_2 \le \|Y\|_2$ .
Feature Magnitude: Larger feature magnitudes (higher variance) lead to increased solution sharpness.
Label Distribution (Key Finding): For classification tasks with one-hot encoded labels, the sharpness is directly related to the empirical distribution of labels. Specifically, the sharpness is proportional to the maximum proportion of samples belonging to any single class ( $\alpha_{max}$ ). The model learns sharper solutions when the dataset is imbalanced (one class dominates) compared to a uniform distribution.

3. Empirical Validation and Robustness

The authors validate their theoretical predictions against numerical computations on MNIST, FashionMNIST, and CIFAR10.

Approximation Accuracy: The theoretical bounds closely match the top eigenvalues of the Hessian as training progresses.
Ablation Studies: The paper systematically sheds assumptions to test robustness:
- Non-isotropic Inputs: Removing the isotropic assumption causes the theory to break down immediately, indicating that input geometry strongly influences peak sharpness.
- Unbalanced Weights: Using standard Kaiming initialization (unbalanced) still preserves the qualitative trend that sharpness increases with label imbalance.
- Nonlinearities: Incorporating Tanh activations yields results that are qualitatively similar to the linear theory, allowing the authors to extend their conclusions regarding label distribution to nonlinear networks.

Significance and Claims

The paper claims to provide a precise theoretical characterization of the Hessian spectrum that bridges the gap between simplified theoretical models and practical deep learning setups. Its primary significance lies in identifying a direct link between label imbalance and solution sharpness.

The authors observe that models trained on datasets with a disproportionately large number of samples in a single class converge to sharper solutions. This finding challenges the intuition (and some prior literature, e.g., Cohen et al.) that simpler datasets (like those with extreme class imbalance) should result in lower peak sharpness. Instead, the authors argue that the geometry of the data, specifically the concentration of labels, dictates the sharpness of the minimizer.

While the theoretical derivations rely on strong assumptions (isotropic features, balanced weights), the empirical ablation studies suggest that the core conclusion regarding the relationship between label distribution and sharpness remains robust even when these assumptions are relaxed or when nonlinearities are introduced. The work provides a framework for understanding how data geometry shapes the optimization landscape, offering insights relevant to generalization measures and second-order optimization algorithms.