📊 statistics

Statistical learning theory and Occam's razor: Regularization

This paper provides a statistical learning theory justification for regularization and Occam's razor, arguing that trading off fit for simplicity is a necessary methodological means to achieve theoretical reliability and "what-you-see-is-what-you-get" guarantees, without relying on pragmatic preferences or ontological assumptions about the simplicity of truth.

Original authors: Tom F. Sterkenburg

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tom F. Sterkenburg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: Why Less is Often More

Imagine you are a detective trying to solve a mystery, but instead of clues, you have a mountain of data. In the world of science and computers, this is called machine learning. The goal is to teach a computer to find patterns in data so it can make smart guesses about new things it hasn't seen before. Think of it like teaching a dog to recognize a "ball" by showing it a thousand different balls. If the dog learns too strictly, it might think only that specific red ball is a ball, and miss the blue one. If it learns too loosely, it might think a round cookie is a ball. This balancing act is the heart of the problem.

For decades, scientists have argued about a rule called Occam's Razor. It's an old idea that says when you have two explanations that fit the facts equally well, you should pick the simpler one. But why? Is the universe just naturally simple? Or is it just that simple things are easier to handle? This has been a tricky question for philosophers and computer scientists alike. They've tried to prove that simple models are better, but often the proof felt like it was circular—assuming the world is simple just to prove that simple models work.

The Paper's Big Idea: Trading Fit for a Safety Net

This paper, written by Tom F. Sterkenburg, dives into the math behind machine learning to find a solid, non-circular reason to use Occam's Razor. The author doesn't just say "simplicity is good"; he uses a framework called Statistical Learning Theory to show that trading a little bit of "perfect fit" for a lot of "simplicity" is actually a clever survival strategy for computers.

Here is the story of what he found:

1. The Trap of the Perfect Fit
Imagine you are trying to draw a line through a scatter of dots on a graph. If you have a very flexible ruler (a complex model), you can wiggle it so perfectly that it touches every single dot. It fits the data perfectly. But here's the catch: if you get a new set of dots tomorrow, that wiggly line will probably miss them all. It memorized the noise (the random squiggles) instead of the pattern. In the paper's language, this is called overfitting.

The paper explains that if you try to use the most complex model possible (one that can fit anything), you lose your ability to trust your results. You get a guarantee that says, "If you have infinite data, you might be right," but in the real world with limited data, that guarantee is useless.

2. The "What-You-See-Is-What-You-Get" Promise
The author introduces a concept called Uniform Convergence. Think of this as a "truth-in-advertising" label for your model. It promises that if your model looks good on the data you have (the training set), it will likely look good on new data (the test set).

However, the paper proves a strict rule: You can only get this "truth-in-advertising" promise if you limit how complex your model is allowed to be. If your model is too flexible (too complex), the promise breaks. You can't trust that what you see is what you get. So, the first lesson is: Keep your model simple enough so that you can trust your results.

3. The Real Magic: Structural Risk Minimization (SRM)
But wait, what if the truth is complicated? What if the pattern really is a wiggly line, and a straight line (a simple model) just won't cut it? If we just stick to simple models, we might miss the answer entirely. This is the "bias-complexity trade-off."

The paper's main discovery is a method called Structural Risk Minimization (SRM). This is the computer's way of playing it smart. Instead of picking one model and sticking with it, SRM looks at a whole family of models, ranging from very simple to very complex.

Here is the clever trick: SRM doesn't just look for the model that fits the data best. It looks for the model that fits the data well enough while staying as simple as possible. It adds a "penalty" for complexity.

If a complex model fits the data slightly better than a simple one, but the complexity penalty is huge, SRM says, "No thanks, stick with the simple one."
If a complex model fits the data much better, the penalty is worth it, and SRM says, "Okay, let's go complex."

4. Why This Isn't Just a Guess
The paper argues that this isn't just a lucky guess or a philosophical hunch. It's a methodological justification. The author shows that even if we don't know if the world is simple or complex, using this "trade-off" strategy is the smartest way to learn.

He uses a concept called "Luckiness." Imagine you are betting on a horse race.

If you bet on a simple horse and the race is actually simple, you win big.
If you bet on a simple horse and the race is complex, you lose, but not by much more than if you had bet on a complex horse blindly.
But if you bet on a complex horse and the race is simple, you lose big because you overcomplicated things.

By using SRM (the trade-off), you protect yourself from the worst-case scenario. You gain a lot if you are "lucky" (the truth is simple), and you don't lose much if you are "unlucky" (the truth is complex).

5. What the Paper Says It Is NOT
The author is very careful to say what this is not.

It is not a proof that the universe is simple. We don't need to believe the world is simple for this to work.
It is not just a pragmatic rule (like "simple things are easier to write down"). It's about getting better accuracy.
It is not a magic bullet for every single modern technique. The paper admits that in the very latest field of "deep learning," things get weird (sometimes super-complex models work surprisingly well), and this specific math doesn't fully explain those new phenomena yet.

The Bottom Line
So, why do we prefer simplicity in machine learning? According to this paper, it's not because the universe is simple. It's because simplicity is a safety net. By trading a little bit of "perfect fit" for a lot of "simplicity," we get a mathematical guarantee that our computer's guesses will actually work on new data. It's the difference between memorizing a script and understanding the story. The paper proves that this trade-off is the most reliable way to learn, whether the truth is simple or complicated.

Problem
The paper addresses the enduring philosophical and methodological challenge of justifying Occam's razor—the principle of preferring simplicity in inductive inference—within the context of machine learning and statistical learning theory. While the literature often cites Occam's razor as a heuristic, a rigorous justification that avoids collapsing into a purely pragmatic preference (simplicity for convenience) or an ontological assumption (the truth is simple) has remained elusive. Specifically, the paper seeks to justify the procedure of regularization (trading off empirical fit for simplicity) not merely as a practical tool, but as a methodological norm derived from theoretical guarantees.

Methodology
The author employs the framework of Statistical Learning Theory (SLT), specifically focusing on binary classification, to construct a "means-ends" argument. The analysis proceeds through the following logical steps:

Core Argument (ERM and VC Dimension): The paper first reviews the standard justification for Empirical Risk Minimization (ERM). It establishes that for a hypothesis class to possess the uniform convergence property (ensuring that empirical error is a reliable indicator of true risk), the class must have a finite Vapnik-Chervonenkis (VC) dimension. This leads to a methodological norm (N1): "Keep the hypothesis class simple." However, the author notes this norm is too restrictive for modern machine learning, where complex, data-driven models (like deep learning) are preferred, and where the "best" hypothesis in a simple class may still have high approximation error (underfitting).
Structural Risk Minimization (SRM): To overcome the limitations of N1, the paper introduces Structural Risk Minimization (SRM). This involves organizing a potentially complex "superclass" of hypotheses into a countable sequence of nested subclasses ( $H_n$ ) with increasing complexity (VC dimension). The SRM rule selects a hypothesis by minimizing a bound that includes both the empirical error and a regularization term dependent on the VC dimension of the subclass and a weight function.
Theoretical Justification Analysis: The paper critically examines the theoretical guarantees of SRM, including:
- Non-uniform learnability: SRM allows learning from classes that are not PAC-learnable as a whole but are countable unions of VC classes.
- Oracle Bounds: The reliability of SRM depends on an "oracle" bound relative to the specific subclass containing the best hypothesis.
- Model-Relativity: The author highlights that theoretical guarantees are relative to the prior choice of the inductive model (the specific sequence of subclasses and weights). This poses a potential circularity: does the justification for simplicity merely assume the world is simple?
Means-Ends Argument for Regularization: The author constructs a new argument to resolve the circularity. The argument posits that to achieve specific theoretical guarantees (reliability and "what-you-see-is-what-you-get" or WYSIWYG bounds) for a complex hypothesis class, one must implement a preference for simplicity via regularization.

Key Contributions and Results

Formalization of Norm N2: The paper articulates a second methodological norm (N2): "Trade fit for simplicity." Unlike N1, which restricts the choice of the hypothesis class itself, N2 applies to the learning process within a chosen (potentially complex) class.
Justification of Regularization: The paper argues that regularization is not just a heuristic but a necessary condition for obtaining finite-sample reliability and WYSIWYG guarantees when working with complex hypothesis classes (specifically, countable unions of VC classes).
Resolution of Circularity: The author argues that the justification for preferring simplicity is methodological, not ontological. Even if the "truth" is complex, using SRM (regularization) is a "clever" strategy because:
- Luckiness: If the data-generating process is simple (lucky case), regularization yields significantly tighter error bounds than ERM on the full superclass. If the process is complex (unlucky case), the bounds are only slightly worse than ERM.
- WYSIWYG: Regularization provides a sharper indication of whether a selected hypothesis is good or bad (via the tightness of the bound for early subclasses), allowing the learner to act accordingly (e.g., discard the model if the bound is loose).
Distinction from Pragmatism: The paper distinguishes this justification from mere pragmatic convenience (e.g., computational ease). The preference for simplicity is tied to the epistemic end of predictive accuracy (minimizing true risk) and the ability to assess that accuracy.

Significance and Claims
The paper claims to provide a genuine methodological justification for Occam's razor in the form of regularization. It asserts that this justification:

Is non-circular: It does not require the assumption that the world is simple; rather, it shows that simplicity is a necessary "means" to achieve the "end" of theoretical reliability and diagnostic capability in learning.
Is robust: It holds even when the inductive model (the choice of subclasses) is not perfectly matched to the true distribution, provided the model is a countable union of VC classes.
Clarifies the role of complexity: It formalizes simplicity not as the number of parameters or description length (which can be unstable), but as capacity (VC dimension), which is a robust, invariant measure of a class's flexibility.

The author concludes modestly, acknowledging that the argument is confined to the specific framework of statistical learning theory (predictive accuracy under i.i.d. assumptions) and does not cover all aspects of scientific simplicity. Furthermore, the paper notes that contemporary phenomena like "benign interpolation" and "double descent" in deep learning challenge the relevance of these norms, suggesting that while the argument clarifies the role of Occam's razor in classical theory, its application to modern deep learning requires further investigation. The paper does not claim to solve the problem of simplicity in all of science, but specifically to justify the procedure of regularization within the mathematical framework of SLT.

The Detective's Dilemma: Why Less is Often More

The Paper's Big Idea: Trading Fit for a Safety Net

More like this