Technical Summary: Topology and Geometry of the Learning Space of ReLU Networks

Problem Statement

The paper addresses two fundamental pathologies in the training dynamics of feed-forward ReLU networks defined on general Directed Acyclic Graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. While classical results often focus on layered architectures, modern networks frequently employ flexible DAG structures (arising from design, pruning, or compression). The authors argue that training with gradient flow on networks with homogeneous activations (like ReLU) restricts learning trajectories to a specific algebraic variety, termed the invariant set. Understanding the geometric and topological properties of this set is critical for analyzing training dynamics, as it dictates which parameter configurations are reachable and how the network's expressivity is constrained by its initialization.

Methodology

The authors employ a combination of algebraic geometry, graph theory, and dynamical systems analysis:

1. Mathematical Formulation:

   • The network is modeled as a DAG $G=(V, E)$. The parameter space $\Theta$ is identified with $\mathbb{R}^{|E|}$.
   • Leveraging the rescaling symmetry of homogeneous activations (where input weights of a neuron can be scaled by $\alpha$ and output weights by $\alpha^{-1}$ without changing the function), the authors derive conservation laws under gradient flow.
   • These laws are formulated using the incidence matrix $\tilde{B}$ of the graph (with rows for input/output nodes removed). The conservation law is expressed as $\tilde{B}(\theta \odot \dot{\theta}) = 0$, which integrates to $\tilde{B}\theta^2 = c$, where $c$ is a vector determined by initialization.

2. Invariant Set Analysis:

   • The invariant set $H_G(c)$ is defined as the solution set to the system of quadratic equations $\tilde{B}\theta^2 = c$. This set represents the manifold on which gradient flow trajectories are confined.
   • The authors analyze the connectedness of $H_G(c)$ using results from the topology of intersections of quadrics (specifically adapting de Medrano, 2023).
   • They analyze singularities by examining the rank of the Jacobian matrix $J_G(\theta) = 2\tilde{B}\text{diag}(\theta)$. A singularity occurs when the rank drops, which corresponds to specific edges having zero weights.

3. Regularization Strategy:

   • To actively drive the network toward singular configurations (which correspond to disconnected sub-networks), the authors propose a nuclear norm regularizer on the Jacobian $J_G(\theta)$. This serves as a smooth surrogate for minimizing the rank, thereby promoting "self-pruning."

Key Contributions

1. Characterization of Connectedness

The paper provides necessary and sufficient conditions for the connectedness of the invariant set $H_G(c)$ in general DAG architectures.

• Bottleneck Nodes: Disconnectedness is shown to arise specifically from bottleneck neurons (nodes with only one input or one output edge).
• Balance Conditions: Theorem 1 establishes that $H_G(c)$ is connected if and only if specific "balance" conditions are met for subsets of pure ancestors (for out-bottlenecks) and pure descendants (for in-bottlenecks). Specifically, the sum of balance values $c_v$ in certain stable sets must be non-negative or non-positive.
• Implication: If these conditions are violated at initialization, the optimization space splits into disconnected components. A network initialized in one component cannot reach optima in another, potentially rendering certain functions (e.g., those requiring a sign change in a bottleneck weight) unreachable regardless of the loss landscape.

2. Analysis of Singularities

The authors identify singularities in the invariant set as configurations where a subset of neurons becomes effectively disconnected from both input and output nodes.

• Topological Correspondence: A singularity corresponds to a drop in the rank of the Jacobian, which occurs if and only if a set of edges with zero weights disconnects a subgraph of hidden neurons from the rest of the network.
• Unreachability: The paper proves that under standard gradient flow from generic initializations, singularities are unreachable in finite time.
  • Reason 1: The condition for a singularity requires an exact equality $\sum_{v \in W} c_v = 0$ for some subset of nodes $W$. For random initializations, the probability of this exact equality is zero.
  • Reason 2: Even if such a configuration exists, Proposition 7 shows that the rank of the Jacobian is invariant under gradient flow; thus, a trajectory cannot transition from a regular point to a singular point.

3. Differentiable Pruning via Regularization

Since singularities are unreachable via standard gradient flow, the authors propose using regularization to induce them.

• Nuclear Norm Regularizer: By penalizing the nuclear norm of the Jacobian, the training dynamics are encouraged to converge toward singular configurations, effectively performing differentiable, structure-agnostic pruning.
• Empirical Observation: Numerical experiments on MLPs show that this regularizer successfully disconnects a significant portion of hidden neurons (e.g., 18 out of 20). Surprisingly, standard L1 regularization (typically associated with weight sparsity) empirically induces similar singular behavior and effective lossless pruning, suggesting a deeper, unexplained connection between L1 sparsity and singularity induction.

Results

• Theoretical: The paper establishes that the topology of the invariant set is dictated by the network's graph structure (specifically bottlenecks) and the initialization-dependent balance vector $c$. It proves that disconnectedness prevents global optimization reachability in certain topologies.
• Singularities: It is proven that singularities correspond to "killed" sub-networks. While they are theoretically part of the variety, they are dynamically inaccessible under standard training unless the initialization is pathological or specific regularization is applied.
• Experiments:
  • In a toy DAG with an out-bottleneck, the authors demonstrate that if the initialization violates the balance condition, the network fails to learn a target function requiring a sign flip in the bottleneck weight.
  • On the Breast Cancer dataset, nuclear norm regularization successfully drives the Jacobian rank down, disconnecting neurons. L1 regularization yields comparable results, while L2 and unregularized training maintain full connectivity.
  • Pruning experiments confirm that networks driven to singularities allow for lossless pruning (removing disconnected neurons without performance degradation), whereas L2-regularized networks suffer performance drops when neurons are removed.

Significance and Claims

The paper claims to shed light on the interplay between network topology and optimization geometry. By characterizing the invariant set as an algebraic variety constrained by quadratic balance conditions, the authors provide a principled framework for understanding why certain training paths are blocked (disconnectedness) and how singularities relate to sub-networks.

The primary significance lies in offering a principled pathway for designing pruning mechanisms. Rather than relying solely on heuristic sparsity constraints, the authors suggest exploiting the structure of the optimization space (specifically, driving the system toward singularities) to achieve effective, lossless model compression. The observation that L1 regularization implicitly promotes these singular configurations suggests that standard sparsity-inducing methods may already be leveraging these geometric properties, even if the mechanism was previously understood only in terms of weight sparsity.

The authors remain modest regarding limitations, noting that their theoretical guarantees rely on gradient flow and unregularized losses with homogeneous activations. They acknowledge that discrete-time optimizers (like Adam) and explicit regularization terms may violate the derived conservation laws, and that the practical application of the nuclear norm regularizer is currently limited by computational constraints.