🤖 machine learning

On the Geometry and Optimization of Polynomial Convolutional Networks

This paper employs algebraic geometry to analyze convolutional neural networks with monomial activation functions, establishing that their parameterization is generically an isomorphism, characterizing the dimension, degree, and singularities of the resulting neuromanifold, and deriving an explicit formula for the number of critical points in regression optimization.

Original authors: Vahid Shahverdi, Giovanni Luca Marchetti, Kathlén Kohn

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vahid Shahverdi, Giovanni Luca Marchetti, Kathlén Kohn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize patterns. To do this, you give the robot a set of adjustable knobs (parameters) that control how it processes information. As you turn these knobs, the robot's behavior changes. If you could map every possible setting of these knobs to the robot's actual output, you would get a giant, multi-dimensional shape. In the world of machine learning, this shape is called a "neuromanifold."

This paper, written by researchers from KTH Royal Institute of Technology, explores the geometry of this shape specifically for a type of AI called a Convolutional Neural Network (CNN) that uses simple "monomial" (power-based) math instead of the usual complex activation functions.

Here is a breakdown of their findings using simple analogies:

1. The "Perfect Map" (Parameterization)

Usually, when you adjust the knobs on a machine, different knob settings can lead to the exact same result. It's like having two different keys that open the same lock. This creates "redundancy" or confusion in the system.

The authors discovered that for these specific polynomial CNNs, the map from knobs to results is incredibly efficient.

The Analogy: Imagine a factory where every unique product requires a unique combination of machine settings. In most factories, you might have multiple settings that produce the exact same widget (waste). In this specific factory, once you ignore the fact that you can just "turn up the volume" (rescaling) on a machine, every single setting produces a unique product.
The Claim: The researchers proved that, almost everywhere, there is a one-to-one, smooth relationship between the settings and the output. There are no "dead zones" or confusing overlaps, making the system mathematically "regular" and optimal.

2. The Shape of the Machine (Geometry)

The researchers wanted to know: How "big" is this shape? How complex is it?

Dimension (Width): They found that the "width" of this shape grows linearly as you add more layers to the network. Think of it like adding a new room to a house; the house gets bigger, but in a predictable, straight-line way.
Degree (Complexity/Curvature): However, the "curvature" or complexity of the shape grows super-exponentially.
The Analogy: Imagine a piece of clay. As you add layers to your network, the clay doesn't just get slightly more complex; it starts folding in on itself in wild, intricate ways, filling up the available space with incredible detail. This explains why deep networks are so powerful: they can represent a massive variety of functions (high degree) without needing a massive number of parameters (low dimension).

3. The "Cracks" in the Shape (Singularities)

In geometry, a "singularity" is a point where a shape gets weird, like the tip of a cone or a place where two surfaces cross each other.

The Finding: The researchers found that the only "cracks" or weird points in this shape happen when parts of the network effectively turn off (weights become zero).
The Analogy: Imagine a bridge. Most of the bridge is smooth and safe. The only "rough spots" are where a small side-bridge connects to the main one. If you remove that side-bridge, the main bridge is still fine. The researchers showed these rough spots are simple "knots" (nodal singularities) caused by the network simplifying itself into a smaller version of itself.

4. Finding the Best Settings (Optimization)

When we train a neural network, we are trying to find the "lowest point" in a valley (the best settings) to minimize errors. This is like trying to find the bottom of a foggy bowl.

The Problem: Sometimes, there are many "local bottoms" (pits) where the robot might get stuck, thinking it has found the best solution when it hasn't.
The Solution: The researchers used a tool from algebraic geometry called the Euclidean Distance Degree. Think of this as a way to count how many "peaks and valleys" exist on the surface of the shape before you even start looking.
The Result: They derived a formula that gives an upper limit on the number of these "traps" (critical points) for a large dataset.
The Good News: They proved that the "rough spots" (singularities) mentioned earlier are not traps. If you are optimizing, you won't get stuck at these weird points (unless the network is completely broken/zero). This means the path to the best solution is relatively clear of these specific obstacles.

Summary

In short, the paper argues that polynomial Convolutional Neural Networks are mathematically "well-behaved."

No Redundancy: Their settings map cleanly to their outputs.
High Power: They can represent incredibly complex patterns despite having a manageable number of settings.
Safe Optimization: The weird points in their geometry don't act as traps for the learning process.

The researchers used advanced math (algebraic geometry) to prove these properties, suggesting that these networks are structurally sound for learning tasks, at least when using these specific mathematical functions.

Technical Summary: On the Geometry and Optimization of Polynomial Convolutional Networks

Problem Statement
Deep neural networks define a space of functions, known as a "neuromanifold," as their parameters vary. Understanding the geometry of these manifolds is fundamental to analyzing learning dynamics, as training corresponds to minimizing a functional distance over this manifold. While algebraic geometry has been applied to fully-connected polynomial networks and linear CNNs, the geometric properties of Convolutional Neural Networks (CNNs) with monomial (polynomial) activation functions remain largely unexplored. Specifically, there is a lack of comprehensive understanding regarding the parametrization regularity, the dimension and degree of the resulting neuromanifold, the nature of its singularities, and the count of critical points arising during optimization.

Methodology
The authors employ tools from algebraic geometry, specifically focusing on projective geometry and birational geometry, to analyze the neuromanifolds of polynomial CNNs.

Parametrization Analysis: The study treats the CNN parameters (filters) up to rescaling, working within projective spaces to remove scaling symmetries.
Factorization: The parametrization map of the polynomial CNN is factorized into a linear projection and a Segre–Veronese embedding. This leverages the fact that convolutions can be rephrased as polynomial multiplications, and the composition of layers with monomial activations corresponds to tensor products of symmetric powers.
Invariants: The authors compute the dimension and degree of the neuromanifold to measure expressivity. They utilize the Euclidean Distance Degree (EDD), an algebraic invariant that counts the number of critical points of the distance function from an external point to the variety, to bound the number of critical points in the optimization landscape.
Complexification: While the results are primarily derived over complex coefficients ( $\mathbb{C}$ ) to utilize the full power of algebraic geometry, the authors demonstrate that most results extend to the real case ( $\mathbb{R}$ ).

Key Contributions and Results

Regularity and Isomorphism of Parametrization:
The paper proves that, up to rescaling the filters, the parametrization map of a polynomial CNN is regular (its differential has maximal rank) and is an isomorphism almost everywhere (birational).
- Significance: This contrasts with fully-connected networks, which often exhibit large fibers and critical points in their parametrization, leading to redundancy. For polynomial CNNs, the parameters are in a one-to-one regular correspondence with the associated functions almost everywhere.
- Singularities: The only non-singleton fibers (singularities) arise from "subnetworks" where specific weights vanish. These singularities are of the nodal type (self-intersections), which are geometrically simple.
Dimension and Degree of the Neuromanifold:
The authors derive explicit formulas for the dimension and degree of the neuromanifold:
- Dimension: Grows linearly with the number of layers ( $L$ ). Specifically, $\dim = |k| - L + 1$ , where $|k|$ is the sum of filter sizes.
- Degree: Grows (super-)exponentially with depth.
- Implication: This provides a geometric explanation for the importance of depth: deeper networks remain low-dimensional (implying low sample complexity for learning) while efficiently filling their ambient function space (implying high expressive power).
Closedness in Zariski Topology:
The neuromanifold of a polynomial CNN is proven to be a closed algebraic variety in the Zariski topology (and thus in the Euclidean topology). This ensures the manifold contains its asymptotic limits, a property that does not necessarily hold for linear CNNs.
Optimization and Critical Points:
For a generic large dataset, the square-error regression loss is rephrased as a distance minimization problem over the neuromanifold.
- Using the theory of the Euclidean Distance Degree for Segre–Veronese varieties, the authors derive an explicit formula that upper-bounds the number of (complex) critical points of the loss function.
- Because the parametrization is regular, this bound applies to the critical points in the parameter space (up to scaling) as well.
- The analysis shows that singular points of the neuromanifold (excluding the zero function) are not critical for the distance function in generic settings, suggesting they do not act as stable equilibria for gradient descent in typical scenarios.

Significance and Claims
The paper claims to be the first to provide a comprehensive description of both the parametrization and the neuromanifold of a polynomial CNN model. By establishing that the parametrization is regular and birational, the work suggests that polynomial CNNs are "parametrized in an optimal way," avoiding the redundancy found in other architectures.

The derived formulas for dimension and degree offer a geometric interpretation of why deep networks are effective: they balance low sample complexity (via low dimension) with high expressivity (via high degree). Furthermore, the explicit bound on critical points via the Euclidean Distance Degree provides a theoretical tool to quantify the complexity of the optimization landscape for these specific architectures.

The authors note that while their results are specific to polynomial activations, the methodology using algebraic geometry could potentially be extended to non-polynomial networks (e.g., ReLU) via polynomial approximation, though this remains a direction for future work rather than a current result.