💻 computer science

Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions

This paper uses Morse theory to analyze the geometric structure of finite-depth neural networks and neural ODEs, characterizing the existence and non-degeneracy of their critical points to establish implications for universal embedding and approximation capabilities.

Original authors: Christian Kuehn, Sara-Viola Kuntz

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christian Kuehn, Sara-Viola Kuntz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a high-tech water filtration system to clean a river. The "river" is your raw data, and the "clean water" is the correct answer your AI is trying to find.

This paper, written by researchers at the Technical University of Munich, looks at the "plumbing" of two different types of AI systems: Neural Networks (which work in discrete steps, like a series of filters) and Neural ODEs (which work like a continuous flow of water through a pipe).

The researchers wanted to know: "Does the shape of our plumbing actually allow the water to reach the right destination, or are we building pipes that are physically incapable of doing the job?"

To answer this, they used a mathematical tool called Morse Functions. Here is the breakdown of their findings using everyday analogies.

1. The Concept: The "Landscape" of Intelligence

Think of an AI’s task as a mountain range. The "valleys" are the correct answers (the lowest points of error), and the "peaks" are the wrong answers.

To learn, the AI needs to navigate this landscape. A Morse Function is essentially a "perfectly shaped" landscape. In a Morse landscape, every peak, valley, and mountain pass is distinct and well-defined. There are no "flat plateaus" where the AI gets stuck, wandering aimlessly because it can't tell if it's going up or down.

2. The Problem: The "Bottleneck" and the "Flatland"

The researchers discovered that if you design your AI poorly, you accidentally create "Flatlands"—areas where the landscape becomes perfectly flat. If an AI is in a flatland, it has no "slope" to follow, so it stops learning.

They categorized AI architectures into three types:

The Non-Augmented (The Narrow Pipe): Imagine a pipe that gets narrower and narrower as it goes. If the pipe becomes thinner than the amount of water you're trying to push through, you lose information. The researchers proved that these "narrow" systems are mathematically incapable of representing complex landscapes. They are "too simple" to ever find the deep valleys of complex data.
The Augmented (The Wide Reservoir): This is the "Goldilocks" zone. By making the internal layers wider than the input (like a wide reservoir), the AI has enough "room" to create all the peaks and valleys it needs. The researchers proved that these systems are "Generically Morse," meaning that if you pick random settings, you are almost guaranteed to get a perfect, navigable landscape.
The Bottleneck (The Funnel): This is like a pipe that starts wide, gets very thin, and then widens again (like an hourglass). This creates "glitches" in the landscape. Depending on how you build the funnel, you might end up with "degenerate" points—mathematical traps where the AI gets stuck in a flat, featureless zone.

3. The Comparison: Steps vs. Flow

The paper compares two ways of processing information:

Neural Networks (The Staircase): Information moves in jumps from one layer to the next.
Neural ODEs (The Slide): Information flows continuously, like a smooth slide.

The researchers found a beautiful symmetry: The rules of the "Staircase" and the "Slide" are almost identical. Whether you move in steps or in a smooth flow, the most important thing is the dimension (the width of your pipes). If your "flow" is too narrow compared to your input, you will hit those same mathematical dead-ends.

4. Why does this matter? (The "So What?")

If you are an engineer building an AI, this paper gives you a blueprint for success.

It tells you:

Don't go too narrow: If your internal layers are smaller than your input, you are mathematically "blind" to certain patterns.
Go wide to be safe: If you want your AI to be able to learn anything (Universal Approximation), you need to "augment" it—give it extra internal dimensions to work with.
Watch your funnels: If you use "bottleneck" designs (like in Auto-encoders), be aware that you are intentionally creating "traps" in your landscape, and you need to be careful how you manage them.

Summary Table

Architecture Type	Analogy	Can it learn complex patterns?
Non-Augmented	A narrow straw	No. It's too simple; it can't create the necessary "valleys."
Augmented	A wide, flowing river	Yes! It creates a perfect, navigable landscape.
Bottleneck	An hourglass	Maybe. It's risky; it can create "flat zones" where the AI gets lost.

This paper, "Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions" by Christian Kuehn and Sara-Viola Kuntz, provides a rigorous mathematical framework to understand the expressive power of neural architectures through the lens of differential topology.

1. The Problem

The authors address a fundamental question in deep learning theory: How do the structural properties of a neural network (layer widths, bottlenecks, and phase-space dimensions) dictate its ability to represent complex functions?

While existing "Universal Approximation Theorems" state that networks can approximate any continuous function given sufficient width or depth, they often fail to explain why certain architectures (like those with bottlenecks or restricted dimensions) fail to represent specific types of functions (e.g., functions with local extrema). The authors identify a gap in understanding the geometric obstructions—specifically the existence and regularity of critical points—that prevent certain architectures from being "universal."

2. Methodology

The authors employ Morse Theory to analyze the input-output maps of two major classes of models:

Multilayer Perceptrons (MLPs): Finite-depth feed-forward networks.
Neural Ordinary Differential Equations (Neural ODEs): The infinite-depth limit of residual networks, where the state evolves according to a vector field.

Key Mathematical Tools:

Morse Functions: A scalar function is "Morse" if all its critical points (where the gradient is zero) are non-degenerate (the Hessian matrix is non-singular). Morse functions are "generic," meaning most smooth functions belong to this class.
Classification of Architectures: They categorize networks into three types:
1. Non-augmented: The hidden layer/phase-space dimension is less than or equal to the input dimension ( $n \ge m$ ).
2. Augmented: The hidden layer/phase-space dimension is greater than the input dimension ( $n < m$ ).
3. Bottleneck/Degenerate: Architectures where dimensions decrease mid-network (MLPs) or where weight matrices lack full rank (Neural ODEs).
Normal Form Theory: They develop an algorithm to transform any MLP with non-full-rank weight matrices into an equivalent "normal form" with full-rank matrices, ensuring the analysis focuses on the essential geometry.

3. Key Contributions and Results

The paper provides a complete classification of the existence and regularity of critical points for both MLPs and Neural ODEs.

A. Existence of Critical Points

Non-augmented Architectures: The authors prove that for both MLPs and Neural ODEs, if the architecture is non-augmented and weights have full rank, no critical points can exist. The gradient is always non-zero.
Augmented Architectures: Critical points can exist. For these architectures, the input-output map is generically a Morse function.
Bottlenecks/Degeneracy: In MLPs with bottlenecks, critical points can be degenerate. In Neural ODEs, if the affine transformations lack full rank (degenerate), every critical point is necessarily degenerate.

B. Regularity (Morse Property)

The authors prove that for augmented MLPs and Neural ODEs, the input-output map is a Morse function for almost all weight configurations (except for a set of Lebesgue measure zero). This means they can generically represent functions with non-degenerate local extrema.

C. Universal Embedding and Approximation

The authors link these geometric findings to the ability of networks to represent functions:

Universal Embedding (Exact Representation): Non-augmented MLPs and Neural ODEs cannot have the universal embedding property because they cannot represent functions with critical points (like a simple parabola). Augmented Neural ODEs, however, do possess the universal embedding property.
Universal Approximation: Using a distance estimation theorem, they prove that non-augmented architectures (and certain bottleneck MLPs) cannot satisfy the universal approximation property because they cannot approximate functions with bounded level sets (which require critical points).

4. Summary Table of Results

Architecture Type	MLP Critical Points	Neural ODE Critical Points	Morse Property (Generically)
Non-Augmented	None exist	None exist	Yes (Class $C^1$ )
Augmented	Can exist	Can exist	Yes (Morse Function)
Bottleneck/Degenerate	Can be degenerate	Must be degenerate	No (Not Morse)

5. Significance

This work is significant because it moves beyond "black-box" approximation results to provide a structural explanation for architectural performance.

Architectural Design: It mathematically justifies why "widening" a network (augmentation) is not just about adding parameters, but about changing the topological capability of the network to represent functions with extrema.
Theoretical Unification: It establishes a formal bridge between the discrete dynamics of MLPs and the continuous dynamics of Neural ODEs, showing they share comparable geometric obstructions.
Foundational Insight: By identifying that non-augmented networks are topologically incapable of representing certain functions, it provides a rigorous basis for why increasing the dimension of the latent space (augmentation) is a requirement for universal expressivity.