Technical Summary: Algebraic Networks and Architectural Degenerations

Problem Statement

The paper investigates the geometry of polynomial neural networks (PNNs) with monomial activation functions and no bias. While neural networks are typically analyzed through their parameter spaces, the fundamental geometric object is the neurovariety: the Zariski closure of the image of the realization map from the parameter space to the space of polynomial functions.

The central problem addressed is the relationship between the singularities of these neurovarieties and the degenerations of the underlying network architecture. Recent literature suggests a guiding principle: singularities in the function space should arise from structural degenerations in the network (e.g., rank-deficient layers or inactive neurons). This paper aims to formalize this principle within a rigorous algebraic framework, specifically for fully connected networks, by defining algebraic networks, subnetworks, and architectural degeneracy loci.

Methodology

The authors employ tools from algebraic geometry, specifically the study of affine varieties, group actions, and local ring theory. The methodology proceeds through the following structural steps:

1. Algebraic Network Framework:

   • An algebraic network is defined as a triple $X = (N, W, r)$, where $N$ is a sequence of vector spaces (layers), $W$ is an irreducible algebraic parameter space (subsets of linear maps), and $r$ is a vector of activation degrees.
   • The realization map $\Phi_X$ maps parameters to polynomial functions. The affine neurovariety $F(X)$ is the Zariski closure of the image of $\Phi_X$.
   • Morphisms between networks are defined rigidly to ensure compatibility with linear layers and activation maps, inducing closed embeddings between neurovarieties.

2. Subnetworks and Degeneracy:

   • Subnetworks are defined categorically as morphisms where hidden layer maps are inclusions of subspaces spanned by subsets of neurons.
   • The Architectural Degeneracy Locus $D_{arch}(X)$ is defined as the image of parameters that are not "full." A parameter is "full" (in $W^\circ$) if all weight matrices have maximal rank and no hidden neuron has a zero outgoing column.
   • $D_{arch}(X)$ represents functions admitting at least one degenerate representation (rank-deficient or inactive neurons).

3. Symmetries and Identifiability:

   • The paper analyzes the symmetry group $G(X)$ generated by permutations of hidden neurons and rescalings compatible with activation degrees.
   • It constructs the affine quotient $W(X) = \text{Spec}(\mathbb{C}[W]^{G(X)})$.
   • Identifiability is formulated geometrically: a network is generically finitely identifiable if the induced map from the quotient parameter space to the neurovariety has zero-dimensional general fibers.

4. Regularity Analysis via Layerwise Reconstruction:

   • The core technical argument involves a layerwise formal reconstruction. The authors introduce auxiliary parameter spaces $Q_{a,B,r}(E)$ parametrizing decompositions of sums of powers with vector coefficients.
   • They define regular triples $(a, b, r)$ where the secant parameterization induces an isomorphism on completed local rings (a local version of identifiability).
   • By iterating this local isomorphism argument across layers, they establish an isomorphism between the completed local ring of the neurovariety at a point $\Phi_X(w)$ and the completed local ring of the quotient parameter space at $[w]$.

Key Contributions

1. Formalization of the Singularity-Degeneracy Principle

The paper provides a precise geometric framework linking singularities of neurovarieties to architectural degenerations. It proves that for fully connected networks with non-increasing widths, the singular locus of the neurovariety is contained within the architectural degeneracy locus.

2. Definition of Architectural Degeneracy Locus

The authors define $D_{arch}(X)$ as the closure of the union of neurovarieties of proper rank-constrained subnetworks. They prove that for fully connected networks satisfying mild width conditions, this locus coincides exactly with the set of functions admitting a degenerate representation.

3. Regularity Theorem for Full Parameters

A primary result is that for fully connected networks with non-increasing widths and scalar output, full parameters (those with maximal rank matrices and no zero columns) map to smooth points of the neurovariety, provided explicit layerwise regularity conditions are met. Consequently, singularities can only occur at degenerate parameters.

4. Classification of Regular Triples

The paper classifies when the local reconstruction argument holds (i.e., when triples are "regular"):

• For activation degree $r \ge 3$, the triple is regular if $a \ge b \ge 1$.
• For activation degree $r = 2$, the triple is regular if $a = b$.

5. Connection to Identifiability and Reducedness

The authors demonstrate that the existence of "good parameters" (where the kernel of the differential of the realization map equals the Lie algebra action) is equivalent to generic finite identifiability. Furthermore, they prove that fully connected generically finitely identifiable networks are reduced (their neurovariety cannot be realized by a proper rank-constrained subnetwork).

Main Results

Theorem 35 (Main Theorem):
Let $X$ be a fully connected network with non-increasing widths and scalar output. Suppose that for every layer $t$, either the activation degree $r_t \ge 3$ or ($r_t = 2$ and $\dim N_t = \dim N_{t+1}$).
If $\Phi_X(w)$ is a singular point of the neurovariety $F(X)$, then the parameter $w$ must contain:

1. A matrix with non-maximal rank, OR
2. A matrix with a vanishing column.

Furthermore, under these conditions, the network $X$ is reduced and generically finitely identifiable.

Corollary:
For such architectures, the singular locus of the neurovariety is strictly contained within the architectural degeneracy locus $D_{arch}(X)$. The "good" parameters (those in $W^\circ$) correspond to smooth points of the variety.

Significance and Claims

The paper claims to provide the first formal instance of the principle that singularities of neurovarieties arise from architectural degenerations.

• Theoretical Contribution: It moves beyond single-layer analysis to deep architectures, establishing a dictionary between geometric properties (singularities, smoothness) and learning-theoretic concepts (identifiability, reducedness, subnetworks).
• Scope: The framework is intentionally general, separating the formal mechanism from the specific fully connected case treated. The authors express hope that the language of algebraic networks, subnetworks, and quotient parameter spaces can be adapted to other architectures, such as convolutional, equivariant, or tensor-network models.
• Modesty: The paper does not propose new training algorithms or experimental validations. Instead, it offers a rigorous geometric characterization of when and why singularities occur in polynomial neural networks, grounding statistical and optimization properties in algebro-geometric invariants.