Benign interpolation and Occam's razor

Technical Summary: Benign Interpolation and Occam's Razor

Problem Statement
Contemporary deep learning methods achieve perfect fit on training data (interpolation) while maintaining strong generalization performance on unseen data, a phenomenon termed benign interpolation. This contradicts the classical intuition of Statistical Learning Theory (SLT), which posits that perfect fit to training data in high-capacity models signals overfitting. Classical SLT relies on bounds derived from the capacity of a model class (e.g., VC dimension) to guarantee that empirical error approximates true risk. However, modern deep neural networks operate with model classes so large that these classical bounds become vacuous, failing to explain why these "overparameterized" models generalize well. The central puzzle is: What explains the predictive success of methods that perfectly interpolate training data without explicit regularization?

Methodology and Theoretical Framework
The authors employ a conceptual analysis of the literature in statistics and machine learning, contrasting the classical framework of SLT with emerging explanations for benign interpolation.

1. Classical SLT Review: The paper reviews the standard framework where generalization is guaranteed if a model class has finite capacity (uniform convergence). This underwrites Methodological Norm M1 (prefer simple model classes) and Methodological Norm M2 (trade fit for simplicity via explicit regularization, e.g., Structural Risk Minimization). Crucially, in the classical view, "simplicity" is a property of the model class (its capacity), and theorems provably connect this property to generalization.
2. Analysis of the Puzzle: The authors examine empirical observations (e.g., Zhang et al., 2017) showing that deep networks can fit random labels perfectly (high test error) while fitting natural labels perfectly (low test error), despite having the same architecture. This implies the algorithm does not rely on a simple model class or explicit regularization to generalize.
3. Critique of New Explanations: The paper analyzes the "basic strategy" proposed in recent literature to explain benign interpolation. This strategy posits that learning algorithms possess an implicit inductive bias that selects "simple" individual models (e.g., those with low norm, high smoothness, or low Kolmogorov complexity) among the many that perfectly fit the data. Proponents label this an application of Occam's razor.

Key Contributions and Arguments
The primary contribution of the paper is the identification of an explanatory gap in the new accounts of benign interpolation.

• Shift in the Object of Simplicity: The authors argue that the new accounts shift the locus of "simplicity" from the model class (as in classical SLT) to individual models.
• Lack of Theoretical Justification: While classical theory provides theorems connecting the capacity of a model class to generalization, there are no known theorems connecting properties of individual models (such as low norm or descriptional complexity) to generalization in the interpolation regime.
• The Explanatory Gap: The authors contend that the new explanations rely on an unargued, substantive assumption: that the specific inductive bias toward "simple" individual models is the "right" bias for the task. By labeling this bias "simplicity" and invoking "Occam's razor," these accounts create the appearance of a methodological principle (like M1 or M2) without the underlying mathematical justification. The name "simplicity" does the work that a theorem used to do in classical theory.
• Evaluation of Proposed Solutions: The paper surveys attempts to close this gap, including:
  • Metaphysical assumptions: The world is simple (rejected as too strong and unproven).
  • Constructivist views: Data construction processes embed simple patterns (plausible but shifts the explanation to data curation).
  • Local assumptions: Specific domain structures (e.g., power-law scaling in images) make the bias effective (a retreat from a general theory of induction to piecemeal domain-specific analysis).

Results and Claims
The paper concludes that while the "basic strategy" (implicit bias toward simple individual models) is a dominant narrative in the literature, it currently fails to provide a complete explanation for benign interpolation.

• The classical theory offers a provable connection between class-level simplicity and generalization.
• The new accounts offer a nominative connection between individual-model simplicity and generalization, lacking a corresponding theorem.
• Consequently, the appeal to Occam's razor in the context of benign interpolation is not a straightforward extension of classical theory but a distinct, unproven assumption. The debate remains open regarding whether a general "new theory of induction" can be formulated or if the explanation must remain local to specific data domains.

Significance
The paper's significance lies in clarifying a philosophical and methodological confusion in the machine learning community. It warns against conflating the name of a principle (Occam's razor) with its justification. By highlighting the lack of a theorem connecting individual model properties to generalization, the authors argue that current explanations for deep learning success are incomplete. They urge a distinction between the methodological norms of classical SLT (which are theorem-backed) and the implicit biases of modern deep learning (which currently lack such backing), suggesting that the field must either find a new theoretical foundation or accept that the success of deep learning relies on specific, domain-dependent assumptions rather than a universal simplicity principle.