A Function-Centric Perspective on Flat and Sharp Minima

This paper challenges the conventional view that flat minima inherently ensure better generalization, arguing through extensive empirical studies that sharpness is a function-dependent property — sharper minima often correlate with improved performance, robustness, and calibration when models are properly regularized, though distinguishing task-driven sharpness from memorization-driven sharpness remains an open practical question.

The Gist

The Big Idea: It's Not About the "Valley," It's About the "Map"

For years, scientists studying AI believed a simple rule: Flat is good, Sharp is bad.

Imagine training an AI is like a hiker trying to find the lowest point in a mountain range (the "loss landscape").

• Flat Minima: A wide, gentle valley. If you take a small step in any direction, you stay at roughly the same low height. The old theory said this was great because the AI wouldn't get confused by tiny changes in data.
• Sharp Minima: A narrow, jagged peak or a deep, thin canyon. If you take a tiny step, you fall up the side immediately. The old theory said this was bad because the AI was "memorizing" the training data too perfectly and would fail on new data.

This paper argues that this old rule is wrong.

The authors say: "Stop looking at the shape of the valley. Look at the map you are trying to draw."

They propose a Function-Centric Perspective. This means the shape of the solution (flat or sharp) isn't a sign of "good" or "bad" AI. Instead, it's a reflection of how complex the task is.

---

The Analogies

1. The "Perfectly Smooth" vs. "Roughly Detailed" Painting

Imagine you are an artist trying to paint a picture.

• Task A: Paint a simple blue sky.
  • The Result: You can paint this with broad, sweeping, flat brushstrokes. The "valley" is wide and flat. This is easy.
• Task B: Paint a hyper-realistic portrait of a face with every pore and hair visible.
  • The Result: To get this right, you need very precise, tight, and detailed brushstrokes. You can't just make broad, flat strokes; you have to be exact. The "valley" you end up in is narrow and sharp.

The Paper's Point: If you are trying to paint a complex face (a hard task), ending up in a "sharp" valley is actually a sign of success! It means your AI found a solution precise enough to handle the complexity. If you forced a "flat" solution on a complex task, you'd get a blurry, bad painting.

2. The "Rubber Band" vs. The "Steel Wire"

Think of the AI's decision boundary (the line it draws to separate cats from dogs) as a rubber band.

• Flat Minima: A loose, floppy rubber band. It's easy to wiggle, but it might not fit the data tightly.
• Sharp Minima: A tight, steel wire. It's rigid and hard to wiggle.

The paper shows that when we use Regularization (tools like "Data Augmentation" or "Weight Decay" that help the AI learn better), the AI often ends up with that tight steel wire.

• Old View: "Oh no! The wire is too tight! It's sharp! It must be broken!"
• New View: "Actually, the wire is tight because the task requires it to be precise. The AI is doing a better job because it learned to be precise, not because it got lucky." However, a crucial caveat remains: a tight wire can still sometimes indicate that the AI has simply memorized the data. The paper doesn't rule that out. The key takeaway is that sharpness alone is not a reliable signal either way—it doesn't automatically mean the model is broken, nor does it automatically mean it's perfect.

---

What Did They Actually Do?

The authors didn't just guess; they ran three types of experiments to prove their point:

1. The "Toy" Test (Single-Objective Optimization)
They gave the AI simple math problems to solve.

• Some problems naturally had wide, flat solutions (like a sphere).
• Some problems naturally had narrow, sharp solutions (like a twisted curve called "Rosenbrock").
• Result: The AI found the "sharp" solution for the twisted problem and the "flat" solution for the sphere. The shape depended on the math problem, not on whether the AI was "smart" or "dumb."

2. The "Decision Boundary" Test
They made the AI separate two groups of dots.

• When the dots were far apart, the AI made a wide, flat decision line.
• When they forced the dots to be very close together (making the task harder), the AI made a tight, sharp decision line.
• Result: Even though the line was "sharp," the AI still got 100% of the answers right. The sharpness was just a sign that the dots were close together, not a sign of failure.

3. The "Real World" Test (Images)
They trained big AI models (like ResNet and VGG) on image datasets (CIFAR, Tiny ImageNet) using standard tricks to make them better (Regularization).

• The Surprise: The models that performed the best (most accurate, most robust, best at guessing correctly) were often the ones with the sharpest minima.
• The "flat" models were actually the ones that performed the worst!

---

Why Does This Matter?

For a long time, researchers thought: "If I want a better AI, I must force it to find a flat valley." They built special tools (like SAM) to do exactly that.

This paper says: "Wait a minute. Sometimes, forcing the AI to be 'flat' makes it worse. If the task is complex, the AI needs to be sharp to get it right."

The Takeaway:

• Sharpness is not always a bug — sometimes it's a feature. A sharp minimum often means the AI has learned a complex, precise function that fits the data perfectly.
• However, sharpness can still coincide with memorization in some cases; it is simply not a guaranteed sign of it.
• A flat minimum might just mean the AI is taking the "easy way out" and missing the details.

The "Goldilocks" Conclusion

The authors conclude that there is no single "perfect" shape for an AI solution.

• If the task is simple, a flat solution is fine.
• If the task is complex, a sharp solution is necessary.

Trying to force every AI to be "flat" is like trying to force a surgeon to use a butter knife because "knives are dangerous." Sometimes, you need the sharp edge to do the job right. The paper calls for us to stop judging AI by how "flat" its valley is, and start judging it by how well it actually solves the problem.

However, a critical practical question remains open: identifying exactly when sharpness reflects legitimate function complexity versus when it reflects memorization is still an unsolved problem. The paper successfully reframes the relationship between sharpness and generalization, showing that sharpness is not automatically a defect, but it does not yet provide a diagnostic tool to distinguish between a "good" sharp solution and a "bad" one in practice.

Therefore, sharpness should not be treated as an automatic defect to be eliminated. It can reflect complex, well-generalizing solutions, but it can also still reflect memorization in some cases. Distinguishing between these two scenarios in real-world applications remains an open challenge for the field.

Technical Summary

1. Problem Statement

The prevailing paradigm in deep learning posits that flat minima (regions in the loss landscape with low curvature) are inherently superior to sharp minima because they correlate with better generalization, robustness, and adherence to Occam's Razor (Minimum Description Length). This view suggests that sharp minima often indicate overfitting or memorization.

However, recent theoretical work (e.g., Dinh et al., 2017) demonstrated that sharpness is not intrinsic to the learned function but can be arbitrarily manipulated via reparameterization. Furthermore, empirical exceptions have emerged where sharp minima coincide with strong performance. The paper identifies a critical gap: Is sharpness a universal indicator of poor generalization, or is it a property dependent on the complexity of the learned function and the training conditions? The authors argue that the current "flatness is good" narrative is overly simplistic and fails to account for the relationship between solution geometry, function complexity, and reliability metrics (calibration, robustness, consistency).

2. Methodology

The authors employ a multi-scale empirical approach, moving from controlled toy problems to high-dimensional real-world datasets, utilizing reparameterization-invariant sharpness metrics to ensure robustness against architectural transformations.

A. Sharpness Metrics

To avoid the pitfalls of Hessian-based metrics (which are not reparameterization invariant), the study focuses on:

1. Fisher-Rao Norm: An information-geometric measure of model complexity.
2. Relative Flatness: A reparameterization-invariant metric based on the trace of Hessian blocks.
3. SAM-Sharpness: The average loss difference under small adversarial perturbations (local sharpness).

B. Experimental Stages

1. Single-Objective Optimization (Toy Setting):
   • Trained MLPs on standard benchmark functions (Sphere, Rosenbrock, Rastrigin, etc.).
   • Goal: Determine if global minima of different functions inherently possess different geometries (flat vs. sharp) regardless of optimality.
2. Synthetic Non-Linear Binary Classification:
   • Used the "make circles" dataset.
   • Experiment 1 (Memorization): Introduced random noise to labels to force memorization, observing the resulting sharpness.
   • Experiment 2 (Decision Boundary Tightness): Systematically reduced the margin between classes (increasing decision boundary tightness) while maintaining perfect generalization (100% test accuracy).
   • Goal: Decouple sharpness from memorization to see if tight decision boundaries alone induce sharpness.
3. High-Dimensional Deep Learning (Vision):
   • Datasets: CIFAR-10, CIFAR-100, and Tiny ImageNet.
   • Architectures: ResNet-18, VGG-19, and Vision Transformer (ViT).
   • Controls: Used matched-seed training (10 seeds) where initial weights and data order were identical across conditions to isolate the effect of regularizers.
   • Regularization Conditions: Baseline, Weight Decay (WD), Data Augmentation (AUG), Sharpness Aware Minimization (SAM), and combinations thereof.
   • Evaluation: Measured Generalization Gap, Test Accuracy, Expected Calibration Error (ECE), Corruption Robustness, and Functional Consistency (Prediction Disagreement).

3. Key Contributions

1. Function-Centric Interpretation of Sharpness: The paper proposes that minima geometry reflects the complexity of the learned function rather than serving as a universal proxy for generalization quality. Sharpness indicates how tightly constrained or complex the decision boundary is.
2. Sharpness as an Unreliable Indicator of Memorization: The study demonstrates that sharpness can arise from legitimate structural complexity (such as tight decision boundaries and perfect generalization), proving that sharpness is not a reliable indicator of memorization. However, the authors clarify that sharpness can still coincide with memorization in certain cases; the key finding is that the presence of sharpness alone cannot be used to definitively diagnose overfitting.
3. Regularization Induces Sharpness: Contrary to the belief that regularization flattens the landscape, the study finds that standard regularizers (Weight Decay, Augmentation, SAM) often drive models toward sharper minima that simultaneously achieve better generalization and reliability.
4. Reconciling SAM: Clarified that Sharpness Aware Minimization (SAM) promotes local robustness but does not necessarily result in globally flat minima under reparameterization-invariant metrics.
5. No "Goldilocks" Zone: Argued against a universal "optimal" level of sharpness. The ideal geometry is task-dependent, influenced by the inductive bias of the architecture and the complexity of the data.

4. Key Results

A. Toy and Synthetic Experiments

• Optimization Landscapes: Different objective functions (e.g., Sphere vs. Rosenbrock) have global minima with inherently different curvatures. A "flat" solution is not always achievable or desirable for every function.
• Decision Boundaries: In the "make circles" experiment, increasing the tightness of the decision boundary (reducing the margin between classes) led to a significant increase in sharpness (Fisher-Rao and Relative Flatness), even when the model generalized perfectly (100% accuracy). This proves sharpness can arise from structural complexity, not just overfitting.

B. High-Dimensional Experiments (CIFAR/Tiny ImageNet)

• Regularization vs. Flatness: Models trained with Weight Decay, Augmentation, or SAM consistently converged to sharper minima (higher Fisher-Rao norm and Relative Flatness) compared to the unregularized Baseline.
• Performance Correlation: Despite being sharper, these regularized models outperformed the flatter Baseline models across:
  • Generalization: Lower generalization gaps.
  • Reliability: Better calibration (lower ECE), higher corruption robustness, and higher functional consistency.
• SAM Behavior: While SAM is designed to find flat minima, the results show it often leads to sharper solutions (under invariant metrics) that generalize better. This suggests SAM guides the model to regions robust to local perturbations, even if the global landscape is sharp.
• Simpson's Paradox: Aggregating results across different architectures or seeds can mask trends. For instance, while regularization generally increased sharpness and performance, specific combinations (e.g., WD+SAM on Tiny ImageNet with VGG) showed cases where flatter solutions were preferred, highlighting the lack of a universal rule.

5. Significance and Implications

• Reappraisal of Geometric Intuition: The paper challenges the dogma that "flat is always better." It suggests that sharpness is a necessary consequence of learning complex, highly constrained functions with tight decision boundaries.
• Reliability Focus: The findings link sharper minima to improved reliability properties (calibration and robustness), suggesting that seeking flatness might inadvertently sacrifice the structural complexity needed for reliable deployment.
• Methodological Rigor: By using matched-seed controls and reparameterization-invariant metrics, the study provides a more causal and robust understanding of the geometry-performance relationship than previous works.
• Open Practical Question: While the paper reframes the relationship between sharpness and generalization, identifying WHEN sharpness reflects memorization versus legitimate function complexity remains an open practical question. The study does not provide a diagnostic tool to distinguish between these two cases in practice, highlighting a critical area for future research.
• Future Directions: The authors call for a shift in research focus from "flattening" the loss landscape to understanding how training controls (regularizers) shape the complexity of the learned function. They suggest that the "right" geometry is determined by the inductive bias of the model and the specific demands of the task, rather than a universal preference for flatness.

In conclusion, the paper argues that sharpness should not be treated as an automatic defect. While it can reflect complex, well-generalizing solutions, it can also still coincide with memorization in certain cases, and distinguishing between the two in practice remains an open problem. The geometry of a minimum should be interpreted relative to the function being learned, not as an absolute indicator of model quality.