🤖 machine learning

Singular Learning and Occam's Razor in Deep Monomial Networks

This paper utilizes tools from polynomial algebra, specifically Mason's Theorem, to demonstrate that in deep monomial networks with sufficiently high activation degrees, critical points of the optimization landscape correspond precisely to subnetworks with inactive or redundant neurons, thereby providing a mathematical explanation for the implicit bias toward simpler functions in deep learning.

Original authors: Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why AI Prefers "Simple" Solutions

Imagine you are teaching a robot to draw a picture. You give it a massive box of crayons, a giant canvas, and a very complex set of instructions. You might expect the robot to use every single crayon and draw the most complicated, detailed masterpiece possible.

However, in the real world, deep neural networks (the "robots" of AI) often do the opposite. During training, they tend to ignore many of their own internal parts and settle on a much simpler solution. This is known as Occam's Razor: the idea that the simplest explanation is usually the best.

This paper asks: Why does the AI do this? Is it just luck, or is there a mathematical rule forcing it to be simple?

The authors, a team of mathematicians, say it's a mathematical rule. They prove that for a specific type of AI network, the "trouble spots" where the learning process naturally gets stuck are exactly the spots where the network has become simpler by turning off or merging unnecessary parts.

The Cast of Characters

To understand the paper, let's meet the main players using a metaphor:

The Neural Network (The Factory): Think of the AI as a factory with many assembly lines (layers) and workers (neurons). Each worker takes an input, does a math operation, and passes it to the next worker.
The "Monomial" Activation (The Special Rule): In this specific factory, every worker follows a very strict rule: they must multiply their inputs by themselves a specific number of times (like squaring or cubing a number). The authors call this a "monomial" activation. It's a simplified version of real-world AI, but it allows mathematicians to use powerful algebra tools to see what's happening under the hood.
The "Critical Points" (The Traffic Jams): When the factory tries to learn, it moves along a path to find the best way to work. Sometimes, it hits a "traffic jam" or a "dead end" where the instructions get confusing. In math, these are called critical points. The paper argues that these traffic jams are the most important places to look because they reveal the AI's hidden biases.
The "Subnetwork" (The Lean Team): A subnetwork is what happens when some workers in the factory are either:
- Inactive: They have no tools (weights are zero), so they do nothing.
- Redundant: They are doing the exact same job as a neighbor, so one of them is unnecessary.
- Result: You could fire these workers, and the factory would still produce the exact same output.

The Discovery: Traffic Jams = Lean Teams

The core discovery of the paper is a direct link between the Traffic Jams (Critical Points) and the Lean Teams (Subnetworks).

The authors used advanced math (specifically a tool called Mason's Theorem, which is like a super-powered rule for checking how numbers and shapes fit together) to prove a surprising fact:

If the factory's workers are following the "Monomial" rule and the math is complex enough, the only places where the learning process gets "stuck" (critical points) are exactly the places where the factory has redundant or inactive workers.

The Analogy:
Imagine you are trying to navigate a maze. Usually, you can walk anywhere. But in this specific maze, the only places where you get stuck in a corner are the spots where the walls have already been knocked down, leaving you with a shorter, simpler path.

The paper proves that the AI doesn't just happen to find simple solutions; the mathematical structure of the network makes it impossible to get stuck anywhere except at the simple solutions.

Why This Matters (The "Occam's Razor" Connection)

In the world of AI, "stuck" points aren't always bad. In fact, Singular Learning Theory (SLT) suggests that these stuck points act as a magnet. The learning process is naturally pulled toward them.

Because the paper proves that these "magnets" are located exactly where the network is simpler (with fewer active neurons), it explains why AI naturally prefers simplicity. It's not a conscious choice by the AI; it's a mathematical law. The AI is physically forced by its own architecture to prune away the extra, complicated parts and settle on a leaner, more efficient version of itself.

The "How" (A Glimpse at the Math)

The authors didn't just guess this; they proved it using Polynomial Algebra.

They treated the AI's output as a giant mathematical equation (a polynomial).
They looked at the "Jacobian," which is a fancy way of measuring how sensitive the AI's output is to tiny changes in its settings.
They found that when the AI is "complex" (no redundant workers), the math works smoothly.
But the moment the AI becomes "simple" (redundant workers appear), the math hits a singularity (a rank-deficient point).
Using Mason's Theorem (a tool from number theory usually used to study prime numbers), they showed that for large enough complexity, these singularities only happen when the network is simple.

Summary

The Problem: Why do deep neural networks naturally become simple and ignore unnecessary parts?
The Method: The authors studied a simplified AI model using advanced algebra.
The Result: They proved that the mathematical "bumps" in the learning path (critical points) occur only when the network has redundant or inactive parts.
The Conclusion: The AI's architecture mathematically forces it to converge toward simple, efficient solutions. This is a mathematical justification for Occam's Razor in deep learning.

Note: The paper focuses strictly on this mathematical proof for networks with "monomial" (power-based) activations. It does not claim this applies to every single type of AI in existence (like those using ReLU), but it provides a strong theoretical foundation for understanding why simplicity is a natural outcome of deep learning.

Technical Summary: Singular Learning and Occam's Razor in Deep Monomial Networks

Problem Statement
The paper investigates the nature of critical points in the optimization landscape of deep neural networks. Specifically, it addresses the phenomenon where critical points arise not from the data or the loss function, but from the model's architecture itself. These points occur where the Jacobian of the model's parametrization is rank-deficient, leading to singularities in the parameter space. Within Singular Learning Theory (SLT), such singularities are posited to act as implicit biases, potentially explaining why deep neural networks tend to converge toward "simpler" functions (a form of Occam's Razor). The authors focus on deep fully-connected networks with monomial activation functions ( $\sigma(z) = z^r$ ), where the end-to-end function is a homogeneous multivariate polynomial. The core question is: For sufficiently large activation degrees, what characterizes the critical points of the parametrization map?

Methodology
The authors employ tools from polynomial algebra and algebraic geometry, operating within the framework of "Neuroalgebraic Geometry." Their approach involves:

Formalizing the Setting: Defining the "neuromanifold" as the image of the parametrization map $\phi$ for deep monomial networks. They analyze the relationship between the parameter space, the function space, and the fibers of $\phi$ .
Characterizing Subnetworks: They define "subnetworks" as parameter configurations containing inactive neurons (vanishing weights) or redundant neurons (proportional incoming weights). They establish that removing such neurons yields a smaller network realizing the same function, implying larger fibers in the parameter space.
Algebraic Tools: Unlike previous work on identifiability in monomial networks (e.g., [FRWY24]) which relied on a polynomial version of Fermat's Last Theorem (Newman-Slater), this work leverages Mason's Theorem (a polynomial analog of the abc-conjecture). This tool allows for a rigorous analysis of the divisibility and degrees of polynomials defined by the network layers.
Kernel Analysis: The authors analyze the kernel of the Jacobian $J_W\phi$ . They prove that for non-subnetwork parameters, the kernel is spanned only by specific symmetries (diagonal scaling of weights) that correspond to the generic fiber dimension, whereas subnetworks possess a larger kernel.

Key Contributions and Results
The primary contribution is Theorem 5.1, which establishes a precise equivalence for deep fully-connected monomial networks with sufficiently large activation degree $r$ :

A parameter configuration $W$ is a critical point of the parametrization map $\phi$ if and only if $W$ is a subnetwork (i.e., it contains at least one inactive or redundant neuron).

The proof strategy proceeds as follows:

Forward Direction: It is shown that if $W$ is a subnetwork, the fiber $\phi^{-1}(f_W)$ has a dimension larger than the generic fiber. By the fiber-dimension theorem, this implies $W$ is a critical point.
Reverse Direction: The authors prove that if $W$ $W$ is not a subnetwork, it is a regular point. This is achieved by demonstrating that the kernel of the Jacobian for non-subnetworks is exactly the dimension of the generic fiber.
- Lemma 5.3 utilizes Mason's Theorem to bound the degrees of polynomial quotients, showing that non-proportional outputs from the network layers cannot satisfy certain linear dependencies unless the activation degree $r$ is small.
- Lemma 5.4 and Proposition 5.5 extend this to show that for $r > 9(D(D-1))^{L-1}$ (where $D$ is the maximum width and $L$ the depth), any linear dependence in the Jacobian's kernel implies the existence of redundant or inactive neurons.

Significance and Claims
The paper claims to provide a mathematical justification for the implicit simplicity bias observed in deep learning. By proving that critical points coincide exactly with subnetworks (configurations where neurons can be pruned without changing the function), the authors link the geometric singularities of the model to the structural simplicity of the solution.

In the context of Singular Learning Theory, where critical points are associated with lower Local Learning Coefficients (LLC) and thus higher Bayesian posterior probability, this result suggests that the learning dynamics are implicitly biased toward "pruned" or simpler network architectures. The authors frame this as a mathematical explanation for the tendency of deep neural networks to implement a form of Occam's Razor.

Limitations and Scope
The authors are explicit about the scope of their results:

The equivalence holds strictly for deep fully-connected networks with monomial activations of sufficiently large degree.
The proof relies on complex scalars ( $\mathbb{C}$ ), though the result extends to real scalars ( $\mathbb{R}$ ) due to the compatibility of the definitions.
The paper acknowledges that for other architectures (e.g., convolutional networks, ResNets, attention mechanisms) or non-monomial activations (e.g., ReLU), the relationship between criticality and subnetworks may differ or remain an open problem. For instance, they note that convolutional networks with monomial activations may have no critical points other than those producing the zero function.