🤖 machine learning

Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networks

This paper employs algebraic geometry to analyze the function spaces of polynomial neural networks, establishing their identifiability and dimensionality while characterizing singularities as arising from sparse subnetworks to explain the geometric origins of sparsity bias in MLPs.

Original authors: Vahid Shahverdi, Giovanni Luca Marchetti, Kathlén Kohn

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vahid Shahverdi, Giovanni Luca Marchetti, Kathlén Kohn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to recognize cats. You give the robot a massive instruction manual (the "parameters") that tells it how to process images. However, there's a catch: many different versions of this manual can actually produce the exact same result. One manual might say "turn left, then turn right," while another says "turn right, then turn left," but both end up at the same destination.

This paper is like a map of all the possible instruction manuals a specific type of robot (a neural network) can use. The authors call this map a "neuromanifold." They wanted to answer two big questions about this map:

Identifiability: If I see the robot's final behavior, can I figure out exactly which manual it is using? Or are there many manuals that look the same?
Singularities: Are there "danger zones" or "cliffs" on this map where the rules of geometry break down?

Here is a breakdown of their findings using simple analogies.

1. The Robot's Brain: MLPs vs. CNNs

The paper studies two types of robot brains:

MLPs (Multi-Layer Perceptrons): Think of these as a standard, fully connected brain where every neuron talks to every neuron in the next layer. It's like a dense web of phone calls.
CNNs (Convolutional Neural Networks): These are specialized for images. They use "filters" that slide over the image, looking for patterns like edges. It's like a team of inspectors walking through a factory, each checking a specific section.

The authors tested these robots using a special type of "activation function" (the rule that decides if a neuron fires). Instead of using standard rules like "if the number is positive, turn on," they used polynomials (mathematical curves like $x^2$ , $x^3$ , etc.). They found that if you use a "generic" (randomly chosen, complex) polynomial, the math becomes much cleaner and easier to analyze.

2. The Mystery of "Who Did It?" (Identifiability)

The first question is: If the robot solves the problem, can we reverse-engineer the exact manual it used?

For MLPs (The Web): The authors found that for almost every behavior the robot produces, there are only a finite number of manuals that could have created it.
- The Analogy: Imagine you see a cake. You can't be 100% sure if the baker used a specific brand of flour or a slightly different one, but you know it wasn't any random recipe. There are only a few specific recipes that result in that exact cake. The authors proved that for these networks, the "recipe space" is exactly the right size—no hidden, infinite redundancies.
For CNNs (The Inspectors): The result is even stronger. For almost every behavior, there is only one single manual that could have created it.
- The Analogy: If you see a specific pattern on a factory floor, there is only one specific way the inspectors could have arranged themselves to create it. The CNN is much more "unique" in its construction.

3. The "Cliffs" and "Dead Ends" (Singularities)

In geometry, a "singular point" is a place where the surface isn't smooth—like the tip of a cone or the edge of a star. In the world of training robots, these are dangerous spots where the learning algorithm (gradient descent) might get stuck or behave strangely.

The authors discovered that these "cliffs" are created by sparse subnetworks.

The Analogy: Imagine a massive highway system (the full network). A "subnetwork" is a scenario where you close down several lanes, leaving only a few open.
- The Finding: When a robot effectively "turns off" a chunk of its neurons (making it a subnetwork), it lands on a "cliff" in the geometry of the map.
- Why it matters: These cliffs are special because they act like magnets for the learning process.

4. The "Sparsity Bias": Why Robots Like to Turn Off Neurons

This is the most practical part of the theory. The authors explain why robots often end up using fewer neurons than they have available (a phenomenon called "sparsity").

For MLPs (The Web): The "cliffs" created by turning off neurons are critically exposed.
- The Analogy: Imagine the learning process is a ball rolling down a hill. In an MLP, the "cliffs" (where neurons are turned off) are like deep valleys or traps. Once the ball rolls near them, it gets stuck there. The math shows that the training process naturally pulls the robot toward these sparse configurations. The robot wants to be a smaller, simpler version of itself.
For CNNs (The Inspectors): The "cliffs" exist, but they are not critically exposed.
- The Analogy: In a CNN, the "cliffs" are just sharp edges on a flat plain. If the ball rolls near them, it doesn't get stuck. It can roll right past them. The training process does not naturally force the CNN to turn off its filters in the same way it does for MLPs.

Summary

The paper uses advanced math (algebraic geometry) to prove that:

MLPs have a few duplicate manuals, but mostly unique ones, and they naturally get "stuck" in simpler, sparser configurations during training because of the shape of their mathematical landscape.
CNNs have almost perfectly unique manuals, and while they have "cliffs" (singularities), the training process doesn't get trapped in them the same way MLPs do.

Essentially, the paper explains why standard neural networks (MLPs) tend to become sparse and efficient on their own, using the geometry of their mathematical "map" as the reason.

Technical Summary: Learning on a Razor's Edge

Problem Statement

This paper investigates the geometric properties of neuromanifolds—the function spaces parametrized by deep neural networks—specifically focusing on Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs). The study addresses two fundamental problems in theoretical deep learning:

Identifiability: Determining the extent to which a function in the neuromanifold corresponds to a unique set of network parameters. This relates to the "redundancies" or symmetries of the parametrization and directly influences the model's sample complexity.
Singularity and Optimization Bias: Characterizing the singular points of the neuromanifold (where the manifold fails to be smooth) and understanding how these singularities relate to the training dynamics, specifically the phenomenon of sparsity bias (the tendency of networks to converge to sparse subnetworks).

While previous works have analyzed specific activation functions (e.g., ReLU, Tanh, Sigmoid, monomials), this paper targets a broader class: sufficiently generic polynomial activation functions of large degree. This choice is motivated by the fact that polynomials can approximate arbitrary continuous functions and, crucially, allow the application of algebraic geometry tools, as polynomial neuromanifolds reside in finite-dimensional ambient spaces.

Methodology

The authors employ tools from algebraic geometry, specifically leveraging the Zariski topology and properties of semi-algebraic varieties.

Genericity: The analysis focuses on "generic" parameters and activation functions. In the Zariski topology, a property holding for a generic object implies it holds for "almost all" objects (as non-empty open sets are dense).
Parametrization Maps: The study analyzes the fibers of the parametrization map $\phi: \mathcal{W} \to \mathcal{M}$ , where $\mathcal{W}$ is the parameter space and $\mathcal{M}$ is the neuromanifold.
Subnetworks: The concept of subnetworks is central. For MLPs, a subnetwork is defined by setting specific rows/columns of weight matrices to zero (inactive neurons). For CNNs, subnetworks are defined by padding filters with zeros.
Critical Exposedness: The paper introduces a geometric notion called critically exposed sets. A subset of parameters is critically exposed if the set of target functions (derived from datasets) for which these parameters are critical points of the loss function has a non-empty interior. This formalizes the idea that training dynamics are biased toward these sets.

Key Contributions and Results

1. Identifiability and Dimension

MLPs: The authors prove that for generic polynomial activations of sufficiently large degree, the parametrization map of an MLP has finite fibers for almost all functions. This means that almost all functions in the neuromanifold correspond to only finitely many parameter choices (specifically, discrete symmetries like neuron permutations).
- Consequence: The dimension of the MLP neuromanifold equals the total number of parameters ( $\sum d_i d_{i-1}$ ). This resolves a dimension conjecture previously established only for monomial activations.
CNNs: The result is stronger for CNNs. The parametrization is shown to be generically one-to-one (injective) for almost all functions.
- Consequence: The dimension of the CNN neuromanifold also equals the number of parameters, but without the discrete symmetries found in MLPs.

2. Singularities and Subnetworks

MLPs: The paper proves that subnetworks (where a subset of neurons is inactive) correspond to singular points of the neuromanifold. Specifically, if the architecture allows for a "bottleneck" (a layer smaller than subsequent layers), the functions parametrized by strict subnetworks are singular.
CNNs: Similarly, subnetworks in CNNs (defined by zero-padded filters) are proven to be singular points. However, the nature of these singularities differs; they are "nodal" (arising from self-intersections) rather than the more degenerate singularities found in MLPs.
Completeness: For CNNs, the authors provide a complete characterization: all singular points of the neuromanifold (excluding the zero function) arise from proper subnetworks. For MLPs, it is shown that subnetworks define singularities, but a complete characterization of all singularities remains an open problem.

3. Critical Exposedness and Sparsity Bias

This is the paper's most significant geometric insight regarding training dynamics:

MLPs: The authors prove that sets of strict subnetworks in MLPs are critically exposed. This means that for a generic dataset, the training process (gradient descent) will encounter critical points corresponding to these sparse subnetworks with positive probability.
- Implication: This provides a geometric explanation for the sparsity bias in MLPs. The training dynamics are naturally biased toward sparse solutions because these solutions correspond to singularities that act as attractors (or equilibria) for the optimization flow.
CNNs: In contrast, the authors prove that subnetworks in CNNs are not critically exposed (once the zero-function fiber is excluded).
- Implication: Pure single-channel CNNs do not exhibit the same geometric bias toward sparsity as MLPs. This aligns with empirical findings that CNNs initialized with near-zero weights often recover and learn effectively, rather than collapsing into sparse subnetworks.

Significance and Claims

The paper claims to provide a geometric explanation for the empirically observed sparsity bias in neural networks, distinguishing between architectures that exhibit this bias (MLPs) and those that do not (CNNs).

Theoretical Unification: By using generic polynomial activations, the work unifies and extends previous identifiability results (which were limited to specific activations like Tanh or monomials) to a broad class of functions.
Neuroalgebraic Geometry: The work contributes to the emerging field of "neuroalgebraic geometry," bridging algebraic geometry and deep learning theory. It demonstrates that algebraic tools can rigorously analyze symmetries and singularities in deep networks.
Distinction from Singular Learning Theory (SLT): The authors clarify that their notion of singularity (classical algebro-geometric singularities of the neuromanifold) is distinct from the singularities defined in Singular Learning Theory (degeneracy of the Fisher information metric). They argue their results extend SLT-like ideas to the function space perspective.

Limitations and Future Work

The authors acknowledge several limitations:

Polynomial Restriction: The results rely on polynomial activations. Extending these rigorous proofs to non-polynomial activations (like ReLU or Tanh) requires polynomial approximation arguments, which are sketched but not fully proven in the functional analysis sense.
Incomplete Characterization: While subnetworks are shown to be singular, a complete classification of all singularities for polynomial MLPs is not yet achieved.
Critical Point Types: The analysis identifies critical points but does not distinguish between local minima, maxima, or saddles. Understanding whether subnetworks are specifically local minima is left for future work.
Complexity Bounds: The degree of the polynomial required for the theorems to hold is extremely large and likely loose, though its existence is sufficient for theoretical purposes.