On the Stable Euclidean Distance Degree of Algebraic Layers

Technical Summary: On the Stable Euclidean Distance Degree of Algebraic Layers

Problem Statement
This paper investigates the projective geometry of algebraic neural layers, specifically families of maps induced by polynomial activation functions. The central object of study is the generic Euclidean Distance degree (gED) of the associated "neurovarieties." The gED is a projective invariant that measures the number of optimal approximations of a general point in the ambient space with respect to a general metric. While the Euclidean Distance (ED) degree is a metric-dependent invariant, its value stabilizes for a sufficiently general scalar product, becoming a projective invariant equal to the sum of the polar classes of the variety.

The specific problem addressed is the behavior of the gED for shallow neural networks (single hidden layers) as the dimensions of the input and output spaces vary, while the architectural width and the degree of the activation polynomial remain fixed. Previous work [9] provided closed formulas for the gED when the output dimension varies, but the stable functional properties regarding both input and output dimensions for general polynomial activations remained open.

Methodology
The authors employ a uniform intersection-theoretic approach grounded in algebraic geometry. The methodology proceeds through the following steps:

1. Geometric Realization: The set of functions in an algebraic neural layer is realized as a projective variety (the $k$-th secant variety of a specific Segre-like product). The paper establishes that for a fixed support of the activation polynomial, the associated variety is projectively equivalent regardless of the specific coefficients, provided the support is identical.
2. Nash Blow-up and Resolution: To handle the singularities of these varieties, the authors utilize the Nash blow-up. They construct a resolution of singularities $Z_S$ for the neurovariety $X_S$ as a projective bundle over a product of Grassmannians, $B_S = \text{Gr}_{N_S-k}(V_S) \times \text{Gr}_k(W)$. This space parametrizes the rank-$k$ linear maps defining the layer along with their kernels and images.
3. Chern-Mather Classes and Polar Degrees: The gED is expressed in terms of the Chern-Mather classes of the variety (Theorem 2.18). Using the Nash blow-up, the authors relate these classes to the tautological bundles over the Grassmannian product $B_S$. This reduces the computation of the gED to an intersection number (an integral) over $B_S$.
4. Equivariant Localization: To evaluate these integrals and prove polynomiality, the authors apply equivariant localization techniques. They consider the action of an algebraic torus on the Grassmannians. The integral is expressed as a finite sum over the fixed points of this action.
5. Schubert Calculus and Stabilization: The paper analyzes the structure of the integrand using Schubert calculus. By establishing uniform bounds on the degrees of the Schubert classes contributing to the intersection numbers, the authors demonstrate that for sufficiently large dimensions, the specific support of the activation polynomial becomes irrelevant, and only its degree matters.

Key Contributions and Results

• Stable Polynomiality (Theorem 4.5): The primary result establishes that for a fixed layer width $k$ and a fixed activation degree $r$, the generic Euclidean Distance degree $gED(m, n, S, k)$ is eventually given by a polynomial function in the input dimension $n$ and the output dimension $m$. This holds for any finite support $S$ of the activation polynomial with $\max(S) = r$. The proof relies on the fact that the localization formula yields symmetric sums of weights, which are known to be polynomial in the number of variables for large $N$ (Proposition A.1).
• Reduction to Monomial Case (Theorem 5.4): The second major result proves that the gED depends only on the degree of the activation polynomial, not on its specific monomial support. Specifically, for sufficiently large dimensions, the gED of a layer with a general polynomial activation $P$ is identical to the gED of a layer with a monomial activation $z^{\deg(P)}$.
  • This is achieved by showing that the difference between the intersection integrals for different supports involves Schubert classes of high codimension that vanish in the stable range (Lemmas 5.1 and 5.2).
  • This implies a drastic simplification for computational purposes: one can replace complex polynomial activations with simple monomials without altering the asymptotic algebraic complexity (the gED).
• Explicit Intersection Formulas: The paper provides a framework to express the gED as an integral over products of Grassmannians involving the Chern classes of tautological bundles (Remark 4.2). This formulation is independent of the specific coefficients of the activation function, depending only on the dimension of the space of symmetric powers involved.

Significance and Claims
The paper claims to extend the understanding of the projective geometry of neurovarieties beyond specific cases to a general stable regime. By proving stable polynomiality, the work places the algebraic complexity of neural layers within a broader paradigm where structural parameters are fixed while ambient dimensions grow, drawing a parallel to results in Gaussian models and semidefinite programming via Schubert calculus on varieties of complete quadrics [11].

The reduction to the monomial case is highlighted as a significant theoretical simplification. It suggests that for the purpose of computing the generic ED-degree—a measure of the number of critical points of the distance function—the specific algebraic structure of the activation function (beyond its degree) does not influence the count in the stable limit. This allows researchers to focus on monomial models when studying the asymptotic algebraic complexity of shallow neural networks.

The work relies on standard tools from intersection theory, including the Nash blow-up, Chern-Mather classes, and equivariant localization, applied systematically to the specific geometry of algebraic neural layers. It does not propose new experimental applications or future algorithmic implementations but rather solidifies the theoretical foundation for analyzing the expressive power and geometric complexity of these models.