🤖 machine learning

Geodesics in the Deep Linear Network

This paper derives a general system of ODEs and explicit solutions for geodesics between full-rank matrices in deep linear network geometry, identifying specific horizontal straight lines in the balanced manifold that act as geodesics under Riemannian submersion.

Original authors: Alan Chen

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alan Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Concept: The "Deep Linear Network"

Imagine you are trying to learn how to play a complex song on a piano. You don't just hit one key; you have to coordinate your left hand, your right hand, and the timing between them.

In Artificial Intelligence, a Deep Linear Network (DLN) is like that piano. It’s a mathematical model where information passes through several "layers" (like your hands) to produce a final result (the song). The "layers" are just matrices (grids of numbers), and the "song" is the final output.

The problem is that there are many different ways to coordinate your hands to play the exact same song. This is called overparameterization. You could play a C-major chord with a heavy left hand and light right hand, or vice versa. Both result in the same sound, but the "effort" or "path" taken to get there is different.

The Problem: Finding the "Shortest Path" (Geodesics)

In geometry, the shortest distance between two points on a flat map is a straight line. But if you are flying a plane from New York to London, the shortest path isn't a straight line on a flat map—it’s a curve that follows the shape of the Earth. This shortest path on a curved surface is called a geodesic.

In AI training, we want to know: If we start with one way of playing the song (one set of weights) and want to move to a better way, what is the most efficient, "natural" path to take?

The paper explores the "geometry" of these neural networks. It treats the different ways to represent the same output not as a flat list of numbers, but as a curved, complex landscape.

The Discovery: The "Magic Elevator" (Riemannian Submersion)

The most brilliant part of this paper is a concept called Riemannian Submersion.

Imagine a massive, multi-story skyscraper (the Upstairs Space). This skyscraper represents all the possible ways to coordinate your hands (the many layers of the network). Each floor represents a different configuration of weights.

Now, imagine there is a special elevator in this building. This elevator doesn't just move you up and down; it is designed so that if you move in a straight line inside the elevator, you are actually following the most efficient, "natural" path in the world outside (the Downstairs Space, which is just the final song being played).

The paper proves that the DLN has this "Magic Elevator" structure.

It shows that if you find a simple, straight-line path in the high-dimensional "skyscraper" of the network, and you "project" it down to the actual output, that path is guaranteed to be the mathematically perfect, most efficient path (the geodesic) for the AI to learn.

The Result: A Mathematical GPS

Before this paper, calculating these "perfect paths" for deep networks was incredibly difficult—like trying to find the shortest route through a mountain range without a map.

The author provides:

The Map (The ODEs): A set of complex equations (Differential Equations) that act like a GPS, telling the network exactly how to change its weights at every micro-second to stay on the most efficient path.
The Shortcut (Explicit Formulas): For certain common scenarios, the author provides a "cheat code"—a direct formula where you can plug in your starting point and your goal, and it instantly tells you the perfect path without having to do all the heavy math.

Why does this matter?

If we understand the "geometry" of how AI learns, we can make training faster, more stable, and more predictable. Instead of the AI "stumbling" around in the dark trying to find the right weights, we can give it a "curved highway" that leads it directly to the best solution.

Technical Summary: Geodesics in the Deep Linear Network

Title: Geodesics in the Deep Linear Network
Author: Alan Chen
Subject Area: Differential Geometry, Machine Learning Theory (Deep Linear Networks)

1. The Problem

The paper investigates the geometric structure of Deep Linear Networks (DLNs). A DLN is a simplified model of neural network training where the nonlinearity is removed, leaving only a product of $N$ weight matrices $W_N \dots W_1$ .

The central mathematical challenge is to understand the geodesics (the shortest paths) in the "observable" space $\mathcal{M}_d$ (the space of $d \times d$ matrices) under a specific Riemannian metric $g_N$ . This metric is induced by the training dynamics (gradient descent) of the network. While the geometry of simpler models (like the Bures-Wasserstein geometry for $N=2$ ) is well-understood, the general case for an arbitrary depth $N$ lacks explicit solutions and a closed-form description of its geodesic paths.

2. Methodology

The author employs a combination of Hamiltonian mechanics and Riemannian submersion theory to tackle the problem:

Hamiltonian Formulation: The author derives the geodesic equations by treating the problem as a variational problem. By defining a Hamiltonian $H(X, P)$ based on the inverse metric tensor, the author uses the canonical equations of motion ( $\dot{x} = \partial_p H, \dot{p} = -\partial_x H$ ) to derive a system of Ordinary Differential Equations (ODEs) that describe the geodesic flow.
Riemannian Submersion: The author leverages the fact that the product map $\phi: \mathcal{M} \to \mathcal{M}_d$ (from the high-dimensional "balanced" parameter space to the low-dimensional observable space) is a Riemannian submersion. This allows the author to "lift" the problem: instead of solving difficult equations directly in the observable space, they find "horizontal" straight lines in the parameter space $\mathcal{M}$ and project them down to $\mathcal{M}_d$ .
Functional Calculus: To handle the derivatives of matrix powers (which appear in the metric), the author utilizes specialized matrix functional calculus, specifically defining a matrix $f^{[1]}(\Lambda)$ to manage the derivatives of singular values.

3. Key Contributions and Results

The paper provides three major mathematical results:

Theorem 2.1 (General Geodesic Equations): The author derives the general system of non-linear ODEs for geodesics in $(\mathcal{M}_d, g_N)$ . This is a Dirichlet boundary value problem that characterizes how a matrix $X(t)$ evolves between two endpoints $A$ and $B$ .
Lemma 2.2 (Condition for Straight Lines): The author identifies the specific algebraic condition under which a linear interpolation between two points in the parameter space remains within the "balanced manifold" $\mathcal{M}$ . This is crucial for the submersion method.
Theorem 2.3 (Explicit Geodesic Formulas): This is the paper's primary achievement. For a specific (but widely applicable) class of endpoints where the singular vectors are related by a single orthogonal matrix $Q$ $Q$ (e.g., when matrices are symmetric or share singular vectors), the author provides a closed-form solution for the geodesic $X(t)$ $X (t)$ .
- For the case where $A$ and $B$ are diagonal, the formula simplifies to a power-based interpolation: $X(t) = ((1-t)A^{1/N} + tB^{1/N})^N$ .
- As $N \to \infty$ , this converges to the path $X(t) = A^{1-t}B^t$ .

4. Significance

The significance of this work is twofold:

Theoretical Machine Learning: By providing explicit formulas for geodesics, the paper offers a tool to study the "natural" paths taken by neural networks during gradient descent. Understanding these paths helps explain how overparameterization affects the speed and trajectory of convergence in deep learning.
Differential Geometry: The paper generalizes known results from the Bures-Wasserstein geometry (which is a specific case of DLN) to a much broader class of Riemannian manifolds. It demonstrates a powerful application of Riemannian submersion theory to solve complex matrix-valued ODEs that are otherwise analytically intractable.

Conclusion: The paper successfully bridges the gap between high-dimensional parameter dynamics and low-dimensional observable trajectories, providing the first general explicit geodesic solutions for the Deep Linear Network geometry.