🤖 machine learning

The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks

This paper demonstrates that the gradient flow for learning deep linear convolutional networks can be formulated as a Riemannian gradient flow on function space for any initialization, provided the network uses multi-dimensional convolutions or one-dimensional convolutions with strides greater than one, with the resulting Riemannian metric depending on the specific initialization.

Original authors: El Mehdi Achour, Kathlén Kohn, Holger Rauhut

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: El Mehdi Achour, Kathlén Kohn, Holger Rauhut

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Navigating a Foggy Mountain

Imagine you are trying to find the lowest point in a vast, foggy valley (this is the loss function, or the "error" of your AI). You are blindfolded and can only feel the slope under your feet. You decide to take small steps downhill. This process is called Gradient Descent.

In the world of deep learning, the "valley" is shaped by the architecture of the neural network. The paper asks a specific question: Does the path we take depend on how we describe the map, or just on the terrain itself?

The Two Types of Maps: Full vs. Convolutional

The paper compares two types of neural networks:

Fully Connected Networks (The "Full Map"): Imagine a giant spreadsheet where every single cell is connected to every other cell. It's flexible but messy.
Convolutional Networks (The "Pattern Map"): Imagine a stamp that slides over an image, looking for specific patterns (like edges or textures). This is how Convolutional Neural Networks (CNNs) work. They are structured and efficient.

The Problem: The "Balanced" vs. "Unbalanced" Dilemma

In the past, researchers found a weird rule for the "Full Maps" (fully connected networks).

The Rule: If you start your journey with a perfectly "balanced" backpack (where the weight is distributed evenly across all layers), the path you take downhill is smooth and predictable. It follows a specific geometric rule called a Riemannian Gradient Flow.
The Catch: If your backpack is unbalanced (even slightly), this smooth geometric rule breaks down. The path becomes chaotic and depends entirely on the specific details of your backpack, not just the terrain.

This meant that for standard networks, the "geometry" of learning was fragile. It only worked under very specific, perfect starting conditions.

The Discovery: The "Pattern Map" is Robust

The authors of this paper discovered something amazing about Convolutional Networks (the pattern maps).

The Analogy of the Puzzle:
Imagine you have a long, complex puzzle that you built by snapping together several smaller puzzle pieces (layers).

In a Fully Connected Network, there are millions of ways to snap those pieces together to make the same final picture. If you start with a slightly different arrangement of pieces, the way the puzzle "unfolds" as you solve it changes completely.
In a Convolutional Network, the pieces are shaped like specific stamps. Because of their rigid shape, there are very few ways to snap them together to make the same final picture.

The Main Finding:
The authors proved that for Convolutional Networks, it doesn't matter how you start.

Whether your "backpack" is balanced or unbalanced, the path the network takes to learn is always the same smooth, geometric flow.
The "terrain" (the Neural Tangent Kernel, or NTK) is determined only by the final picture you are trying to draw and a few simple numbers describing the initial weight distribution. It does not care about the specific arrangement of the internal pieces.

Why Does This Happen? (The "Stride" Secret)

The paper explains why this happens using a concept called Stride (how much the stamp jumps when it moves).

High-Dimensional Signals (2D images, 3D video): If you are working with images or higher dimensions, the "stamps" are so rigid that there is essentially only one way to build the final picture from the pieces. The geometry is locked in.
1D Signals (Simple lines): If the "stamps" jump by more than one step (Stride > 1), they are still rigid enough to lock the geometry in.
The Exception: If you are on a 1D line and the stamps don't jump at all (Stride = 1), the pieces can slide past each other. In this specific, rare case, the geometry does become messy and depends on the starting arrangement, just like the Fully Connected networks.

The "ReLU" Twist

The paper also briefly touches on ReLU networks (networks with a "switch" that turns off negative numbers). They found that even with these switches, if the network is "shallow" (not too deep) and working with images, the same robust geometric rules apply. The "switches" don't break the pattern.

The Takeaway: Why Should We Care?

Predictability: We now know that Convolutional Networks have a built-in "self-correcting" geometry. Even if you initialize them poorly, they tend to follow a smooth, predictable path to a solution. This explains why CNNs are so reliable in practice.
Structure is King: The rigid structure of CNNs (the way they slide filters over data) acts as a guardrail, preventing the learning process from getting lost in the "fog."
A New Lens: The authors used advanced math (Algebraic Geometry) to prove that the "shape" of the learning process is independent of the specific parameters in most cases. This allows us to study how these networks learn without getting bogged down in the messy details of every single weight.

In short: Fully connected networks are like a pile of loose LEGOs; if you start with them in a weird order, building the castle is hard and unpredictable. Convolutional networks are like a set of specialized LEGO bricks that only fit together one way; no matter how you hold them, they snap together into a smooth, predictable path to the solution.

1. Problem Statement

The paper investigates the geometric properties of gradient flows used to train deep linear neural networks, specifically comparing fully-connected (FC) architectures with convolutional (CNN) architectures.

Context: In deep learning, the loss landscape is non-convex, making convergence analysis difficult. Recent work on linear FC networks showed that under a specific "balanced initialization" condition, the gradient flow on the parameter space induces a Riemannian gradient flow on the function space (the space of end-to-end mappings). In this regime, the Neural Tangent Kernel (NTK) becomes independent of the specific parameterization (i.e., it depends only on the function being learned).
The Gap: It was unknown whether this geometric property (parameter-independence of the NTK) holds for convolutional networks without requiring balanced initialization, and how the presence of strides (downsampling) affects this geometry.
Core Question: Does the gradient flow of linear CNNs induce a Riemannian gradient flow on the function space regardless of initialization, and how does this depend on the network's stride and dimensionality?

2. Methodology

The authors employ tools from algebraic geometry and differential geometry to analyze the "neuromanifold" (the image of the parameter space under the network map).

Algebraic Formulation: They model the composition of linear convolutional layers as the multiplication of sparse polynomials (in 1D) or multivariate polynomials (in $D$ -dimensions). This allows them to treat the network parametrization map $\mu$ as an algebraic map.
Invariants: They utilize the balancedness invariants (denoted $\delta_l$ or $\Delta_l$ ), which are quantities that remain constant throughout the gradient flow. For CNNs, these are defined as $\delta_l = \|w_{l+1}\|_F^2 - \|w_l\|_F^2$ .
NTK Analysis: The Neural Tangent Kernel (NTK), defined as $K_\theta = J_\theta J_\theta^\top$ (where $J_\theta$ is the Jacobian), is analyzed to determine if it depends on the specific parameters $\theta$ or only on the end-to-end function $v = \mu(\theta)$ and the invariants $\delta$ .
Stratification: The authors analyze the smooth locus (regular points) versus the singular locus of the neuromanifold. They investigate the fiber structure of the map $\mu$ (i.e., how many parameter sets map to the same function) under different stride configurations.

3. Key Contributions & Results

The paper establishes a sharp dichotomy between fully-connected networks and convolutional networks regarding the parameter-independence of the NTK.

A. Linear Convolutional Networks (Main Result)

The authors prove that for linear CNNs, the NTK is parameter-independent (depending only on the end-to-end function and the invariants $\delta$ ) in the following cases:

High-Dimensional Signals ( $D \ge 2$ ): For any strides, the NTK is parameter-independent on the smooth locus of the neuromanifold.
1-Dimensional Signals ( $D=1$ ) with Strides $>1$ : If all strides $s_1, \dots, s_{H-1}$ $s_{1}, \dots, s_{H - 1}$ are strictly greater than 1, the NTK is parameter-independent.
- Mechanism: In these cases, the projective parametrization map is birational. This means that for a generic end-to-end filter, there is a unique set of layer filters (up to scaling and signs) that produce it. Since the scaling is fixed by the invariants $\delta$ , the parameters are essentially unique, forcing the NTK to be a function of the output only.
Riemannian Metric: Consequently, the gradient flow can be written as a Riemannian gradient flow on the function space with a metric defined by the inverse of the NTK.

B. Cases of Parameter Dependence

The NTK becomes parameter-dependent (i.e., different parameter sets yielding the same function have different NTKs) in:

1-Dimensional Signals with Stride = 1: If any stride is 1, the map is no longer birational. Multiple distinct filter tuples can produce the same end-to-end filter (due to the ability to swap polynomial factors). These different tuples generally yield different NTKs.
Fully-Connected Networks: Even with fixed invariants $\Delta$ , the NTK is generally parameter-dependent. The authors provide a geometric explanation: in the balanced case ( $\Delta=0$ ), the fiber consists of orthogonal transformations (isometries), preserving the NTK. However, for generic (unbalanced) $\Delta$ , the fiber structure does not preserve the NTK.

C. Convergence Analysis

For 1D CNNs with strides $>1$ and generic data/initialization, the gradient flow converges to a critical point in the smooth locus of the neuromanifold.
Crucially, the flow avoids the zero function (a saddle point) for generic initializations, ensuring convergence to a non-trivial solution.

D. Extension to ReLU Networks

The results extend to shallow ReLU networks:

ReLU CNNs: Because the first layer of a non-zero ReLU CNN is generically a full-rank linear map on a specific domain, the factorization properties of linear CNNs apply. The NTK remains parameter-independent.
ReLU MLPs: The results do not extend to fully-connected ReLU networks. The lack of unique factorization in MLPs means the NTK remains parameter-dependent.

4. Significance and Implications

Robustness of Implicit Bias: The paper demonstrates that the "implicit geometric regularization" (where optimization dynamics follow a Riemannian flow) is more robust in structured convolutional architectures than in fully-connected ones. CNNs achieve this desirable geometric property even without the restrictive "balanced initialization" required for FC networks.
Role of Architecture: It highlights that architectural constraints (convolutions, strides) fundamentally alter the geometry of the optimization landscape. The "inductive bias" of convolutions constrains the function space so tightly that it enforces unique parametrizations (up to scaling), leading to well-defined Riemannian metrics.
Finite-Width Behavior: Unlike standard NTK theory which relies on infinite-width limits where the kernel becomes constant, this work shows that for finite-width CNNs, the kernel is still well-behaved (parameter-independent) due to algebraic structure, not just width.
Theoretical Foundation: The work bridges algebraic geometry (birationality, singular loci) with optimization theory (gradient flow, NTK), providing a rigorous framework to understand why certain deep learning architectures are easier to train than others.

Summary Table of Findings

Architecture	Condition	NTK Parameter-Independent?
Fully-Connected	Balanced ( $\Delta=0$ )	Yes
Fully-Connected	Generic ( $\Delta \neq 0$ )	No
1D Convolution	All Strides $> 1$	Yes (for any $\delta$ )
1D Convolution	Some Stride $= 1$	No
High-Dim Conv ( $D \ge 2$ )	Any Strides	Yes (for any $\delta$ )

In conclusion, the paper establishes that linear convolutional networks possess a superior geometric structure compared to fully-connected networks, allowing their training dynamics to be described by a parameter-independent Riemannian gradient flow under much broader conditions.