🤖 machine learning

Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defects

This paper refutes the minimal unimodal conjecture for polynomial neural networks by presenting a counterexample discovered through frontier search and certified via recursive dimension bounds, which notably features subarchitectures with large defects contrary to prior observations.

Original authors: Kevin Dao, Jose Israel Rodriguez

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kevin Dao, Jose Israel Rodriguez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the most efficient "factory" possible to turn raw materials (input data) into a finished product (output predictions). In the world of machine learning, this factory is a Neural Network.

This paper is about a specific type of factory called a Polynomial Neural Network (PNN). Instead of using standard switches, these factories use mathematical "power functions" (like squaring or cubing numbers) to process information.

The researchers were investigating a popular belief about how these factories should be built. Here is the story of what they found, explained simply.

1. The "Hourglass" Belief (The Conjecture)

For a long time, engineers and scientists believed that the most efficient factory layout looked like an hourglass.

The Idea: You start with a small input, expand the factory to be very wide in the middle (to capture complex patterns), and then narrow it back down to a small output.
The Rule: The width of the layers should go up and then go down, never dipping in the middle and rising again. This is called being "unimodal" (one peak).

The researchers called this the "Minimal Unimodal Conjecture." They thought: "If you want the smallest possible factory that can still do the job perfectly, it must look like an hourglass."

2. The Surprise: A "Wobbly" Factory

The authors, Kevin Dao and Jose Israel Rodriguez, decided to test this rule. They used a clever search method (like a treasure hunt) to find the smallest possible factories that could do the job perfectly.

They found a counterexample.

They discovered a factory layout that works perfectly but does not look like an hourglass. It looks more like a wobbly mountain range with multiple peaks and valleys.

The Layout: The widths of the layers were: 2 → 3 → 4 → 5 → 4 → 6 → 4 → 1.
Why it breaks the rule: Notice the middle? It goes up to 5, drops to 4, shoots up to 6, and drops again. It has two peaks (at 5 and 6).
The Result: This proves that the "hourglass" rule is false. You can build a perfectly efficient, minimal factory that is "wobbly" in the middle.

3. How They Proved It (The Detective Work)

How do you prove a factory is efficient? You check if it can produce every possible output it is theoretically capable of.

The Ambient Space: Imagine a giant warehouse of all possible products the factory could make.
The Neurovariety: This is the actual set of products the specific factory layout can make.
The Goal: If the factory can make everything in the warehouse, it is called "filling."

The researchers used powerful computer math (specifically checking the factory's "blueprints" over different number systems) to prove:

This wobbly factory can make everything in the warehouse (it is "filling").
If you shrink any single layer of this factory, it suddenly fails to make everything (it stops being "filling").
Therefore, it is a Minimal Filling Architecture (MFA): the smallest possible version that still works, and it breaks the hourglass rule.

4. The "Defect" Discovery

While studying this wobbly factory, they noticed something strange about its smaller parts (sub-factories).

Defect: This is a measure of how much "expressive power" a factory loses. Usually, when you shrink a factory, it loses a little bit of power, but not much.
The Finding: In this wobbly factory, some of the smaller versions lost a massive amount of power. It's like taking a small piece out of a machine, and suddenly the whole thing loses 90% of its ability to work, rather than just 10%.
Significance: The paper notes that these are some of the first examples of factories with such "large defects," which is a new and surprising discovery in the field.

5. The Search for More

The authors didn't just stop at one example. They ran a massive computer search to find all the "minimal" factories for different sizes.

They found 13 minimal factories for a certain size, and only one was wobbly (the one they found first).
When they looked at slightly larger factories, they found 82 minimal ones, and 20 of them were wobbly.
This suggests that while the "hourglass" shape is common, the "wobbly" shape is definitely possible and becomes more common as factories get more complex.

Summary

The paper shatters a "common sense" rule in machine learning design. It proves that the most efficient, smallest neural networks do not always have to look like smooth hourglasses. Sometimes, the most efficient design is a bumpy, multi-peaked structure that defies intuition. This discovery helps mathematicians understand the hidden geometry of how these networks learn and express themselves.

Technical Summary: Minimal Filling Architectures of Polynomial Neural Networks

Problem Statement
The paper addresses a fundamental question in neuroalgebraic geometry regarding the structural properties of Polynomial Neural Networks (PNNs). Specifically, it investigates the "Minimal Unimodal Conjecture" (Conjecture 12 in [8]). The conjecture posits that for PNNs with power activation functions ( $\sigma(z) = z^r$ ), any Minimal Filling Architecture (MFA)—defined as the smallest network configuration capable of spanning the entire functional space (neurovariety) for fixed input/output dimensions—must possess unimodal hidden layer widths. That is, the sequence of layer widths should increase to a peak and then decrease ( $d_0 \le \dots \le d_i \ge \dots \ge d_L$ ). This hypothesis aligns with the "conventional wisdom" in machine learning that networks expand capacity to capture patterns and then narrow to refine them.

Methodology
The authors employ a combination of algebraic geometry, symbolic computation, and algorithmic search to test the conjecture:

Algebraic Framework: The study utilizes the parameter map $\Psi_{d,r}$ , which maps network weights to the space of homogeneous polynomials (the neurovariety $V_{d,r}$ ). An architecture is "filling" if the dimension of its neurovariety equals the dimension of the ambient space (the full symmetric tensor space).
Dimension Verification: To determine if an architecture is filling, the authors compute the generic rank of the Jacobian of the parameter map. Since exact symbolic rank computation is difficult for large networks, they utilize finite field computations (over $\mathbb{F}_p$ ) to determine the generic rank. If the rank over a finite field matches the ambient dimension, the architecture is filling.
Recursive Bounds: To prove minimality (that no subarchitecture is filling), the authors apply recursive dimension bounds derived from Lemma 2.4 in [8]. This allows them to bound the dimension of a subarchitecture by summing the dimensions of its constituent parts minus the interface width.
Frontier Search Algorithm: To discover MFAs, the authors developed a "frontier search" algorithm (Algorithm 1). This algorithm iteratively proposes candidate hidden layer widths, pruning the search space based on:
- Naive dimension bounds.
- Known results from previous searches (maintaining sets of minimal filling and maximal non-filling architectures).
- The definition of minimality: if a candidate is filling, all larger architectures are pruned; if not filling, all smaller architectures are pruned.

Key Contributions and Results

Counterexample to the Minimal Unimodal Conjecture: The primary contribution is the discovery and certification of a counterexample to the conjecture. The authors identified the architecture $d = (2, 3, 4, 5, 4, 6, 4, 1)$ (with $r=2$ , input 2, output 1, depth 7) as a Minimal Filling Architecture that is not unimodal.
- Proof of Filling: They verified that the neurovariety dimension is 65, matching the ambient space dimension $\dim(\text{Sym}^{2^6}(\mathbb{R}^2) \oplus \mathbb{R}^1)$ .
- Proof of Minimality: They demonstrated that every subarchitecture obtained by reducing a single hidden layer width by one fails to be filling. This was achieved by showing that the dimensions of these sub-architectures are strictly less than 65, using both finite field computations and recursive bounds.
Discovery of Large Defects: In analyzing the counterexample and its sub-architectures, the authors observed "large defectivity." Defect is defined as the difference between the expected dimension (based on parameter counting) and the actual dimension of the neurovariety. Unlike prior examples where defects were small, several sub-architectures of the counterexample exhibited significant defects, challenging the assumption that defects are negligible in typical configurations.
Systematic Search Results: Using their frontier search algorithm, the authors cataloged MFAs for $d_0=2, d_L=1, r=2$ up to depth $L=7$ .
- For $L=7$ , they found 13 MFAs, only one of which (the counterexample above) was non-unimodal.
- For $L=8$ , they found 82 MFAs, 20 of which were non-unimodal.
- They provided a table of verified MFAs for $L \le 7$ , noting that while they verified completeness for $L \le 7$ , a general method for proving completeness for larger $L$ remains an open problem.
Additional Counterexample: The authors identified a second, deeper counterexample with $d_0=d_L=2$ and $L=9$ ( $d=(2, 3, 4, 4, 10, 17, 11, 12, 4, 2)$ ), further illustrating that non-unimodal MFAs exist in various configurations.

Significance and Claims
The paper claims to provide the first counterexample to the Minimal Unimodal Conjecture for polynomial neural networks, thereby disproving the idea that minimal expressivity always requires a unimodal width sequence.

The authors modestly frame the significance of their work as follows:

Theoretical Correction: The results correct the "conventional wisdom" regarding network architecture efficiency in the context of algebraic geometry.
Defectivity Insight: The discovery of large defects in sub-architectures suggests a lack of systematic understanding regarding which architectures are defective, opening a new avenue for research into the geometry of function spaces.
Practical Implication: The authors suggest that understanding MFAs for power activations could inform architecture selection, particularly in pairing low-degree polynomial activations (which are computationally efficient but may limit expressivity) with architectures that are a priori highly expressive.
Future Directions: The paper identifies several next steps, including generalizing to broader polynomial activation families (e.g., Chebyshev bases), applying neuroalgebraic geometry to $\Pi$ -nets, and comparing the training dynamics of MFAs versus maximal non-filling architectures.

The authors do not claim to have solved the general problem of characterizing all MFAs for arbitrary depths or to have fully explained the geometric causes of large defects, but rather to have established the existence of non-unimodal minimal fillings and provided the computational tools and data to explore this phenomenon further.