🌀 nonlinear sciences

A Dynamical Systems Perspective on the Analysis of Neural Networks

This chapter employs a dynamical systems framework to reformulate and analyze key challenges in deep learning, including information propagation, training dynamics like the edge of stability, and mean-field limits, thereby offering new insights into the behavior, stability, and explainability of neural networks.

Original authors: Dennis Chemnitz, Maximilian Engel, Christian Kuehn, Sara-Viola Kuntz

Published 2026-06-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Chemnitz, Maximilian Engel, Christian Kuehn, Sara-Viola Kuntz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: AI as a Moving Story

Imagine a Neural Network (the brain of an AI) not as a static computer program, but as a living, breathing story that changes over time.

The authors of this paper argue that to truly understand how AI works, we shouldn't just look at the math as a set of frozen equations. Instead, we should view it through the lens of Dynamical Systems. Think of a dynamical system like a river: the water (data) flows, the banks (network structure) shape the flow, and the riverbed itself can change shape as the water erodes it (learning).

The paper breaks this "river of AI" into three main chapters:

Chapter 1: The Journey of Information (The Flow)

The Analogy: Imagine a package traveling through a series of conveyor belts (layers of a network).

The Standard Conveyor Belt (Feed-Forward Networks): In a standard network, the package moves from one belt to the next in a straight line. The paper looks at different belt designs:
- Non-augmented: The belts get narrower as the package moves forward (like a funnel).
- Augmented: The belts get wider in the middle before narrowing again (like an hourglass).
- Bottleneck: The package gets squeezed through a tiny hole in the middle.
The Infinite Conveyor (Neural ODEs): What if the conveyor belt didn't stop at distinct steps but was a smooth, continuous slide? The authors show that if you make the conveyor belt infinitely long and smooth (a "Neural ODE"), it can mimic almost any shape or function you want, provided the slide is wide enough.
The Memory Lane (Neural DDEs): Sometimes, a package needs to remember where it was a moment ago to decide where to go next. This is like a conveyor belt with a "delay loop." The paper shows that if this delay is long enough and the system is stable, the network can remember complex patterns and approximate difficult tasks that standard networks might miss.

The Takeaway: The shape of the network (wide, narrow, delayed) dictates what kinds of "stories" (functions) it can tell. Some shapes are better at remembering; others are better at generalizing.

Chapter 2: The Training Process (The Sculptor)

The Analogy: Imagine a sculptor trying to carve a statue (the perfect AI) out of a block of marble. The sculptor chips away pieces (adjusting weights) based on how close the current shape is to the goal.

The Overdetermined Problem (Too Many Rules, Not Enough Clay): Imagine you have a tiny block of clay but a million rules about what the statue must look like. It's impossible to satisfy every rule perfectly. The sculptor tries to get as close as possible. The paper uses "stability" concepts to see if the sculptor will eventually stop chipping and settle on a good shape, or if they will keep shaking the block forever.
The Overparameterized Problem (Too Much Clay, Few Rules): Now imagine a mountain of clay and only a few rules. There are millions of ways to carve a perfect statue. The sculptor could stop at any of these millions of perfect spots.
- The Edge of Stability: The paper discovered something surprising. If the sculptor chips too aggressively (a high "learning rate"), they don't just settle on a smooth spot; they hover right on the edge of stability. It turns out, hovering on this "edge" often leads to a statue that looks better to people who haven't seen the original rules yet (better generalization).
The Random Sculptor (Stochastic Gradient Descent): In the real world, the sculptor doesn't see the whole block at once; they only see a small handful of marble chips at a time. This is "Stochastic Gradient Descent" (SGD). The paper treats this randomness as a "random dynamical system." They found that even with this randomness, the sculptor will eventually settle into a stable spot if the "chaos" of the random chips isn't too wild. They use a concept called "Lyapunov exponents" (a measure of chaos) to predict if the sculptor will find a good spot or get lost in the noise.

The Takeaway: Training isn't just about finding the lowest point; it's about finding a stable low point. Sometimes, being slightly unstable or "wobbly" during training actually helps the AI learn better.

Chapter 3: The Crowd and the Collective (Mean-Field Limits)

The Analogy: Imagine a stadium full of 10,000 people (neurons) all shouting and listening to each other. Tracking every single person's voice is impossible.

The Crowd Effect: Instead of tracking 10,000 individuals, the paper suggests looking at the "average voice" of the crowd. This is called a "Mean-Field Limit."
The Graph of Connections: In a real stadium, people only talk to their neighbors, not everyone. The paper uses advanced math (digraph measures) to show that even with complex, messy connection patterns (like a social network), the behavior of the whole crowd can be predicted by looking at the "average" flow of information.
Why it matters: This proves that many complex AI models (like Transformers or Recurrent Networks) are actually just special cases of a very old, well-understood class of physics problems involving interacting particles. If we understand the physics of the crowd, we understand the AI.

The Takeaway: You don't need to track every single neuron to understand the AI. By looking at the "average" behavior of the network, we can use powerful tools from physics and math to predict how the whole system will behave.

The Final Lesson: Backpropagation is Time Travel

The paper ends with a cool insight about Backpropagation (the method used to teach AI).

The Analogy: Imagine you are walking down a path and you drop a breadcrumb. To find out where you started, you can either retrace your steps forward (Forward Accumulation) or walk backward from where you are now (Reverse Accumulation).
The Insight: The paper points out that Backpropagation is essentially mathematical time travel. It's the same as calculating how a small change in the past (the starting weights) affects the future (the output), but doing it in reverse time. This is a concept mathematicians have known for centuries (like in the study of light or fluid dynamics), but AI rediscovered it as a computational trick.

Summary

This paper is a bridge. It says: "Stop treating AI as a black box of magic code. It is actually a river of data, a sculptor chipping away at marble, and a crowd of interacting particles."

By using the tools mathematicians have used for centuries to study rivers, crowds, and chaos, we can finally start to rigorously explain why AI works, when it fails, and how to make it more reliable. The authors aren't promising new AI apps; they are promising a new way to understand the AI we already have.

Technical Summary: A Dynamical Systems Perspective on the Analysis of Neural Networks

Problem Statement

Despite the practical success of machine learning (ML) and artificial intelligence (AI), a significant disparity exists between their application and the theoretical understanding of their underlying mechanisms. The authors identify a critical need to bridge this gap by re-framing ML algorithms and neural network architectures as dynamical systems. The central challenge is to determine how effectively problems regarding explainable and reliable AI can be reformulated within a dynamical framework, how to handle the high dimensionality and heterogeneity of these systems, and the extent to which rigorous analytical proofs can be established.

The paper focuses on direction (b) of the field: viewing ML algorithms themselves as dynamical systems to understand their behavior using established mathematical tools from dynamical systems theory. This involves separating the "dynamics on the network" (information propagation with fixed weights) from the "dynamics of the network" (the evolution of weights during training).

Methodology

The authors employ a multi-faceted approach rooted in dynamical systems theory, adapting concepts from ordinary differential equations (ODEs), delay differential equations (DDEs), random dynamical systems, and interacting particle systems (IPS). The methodology is structured around three primary axes:

Information Propagation (Dynamics on the Network): The input-output map of neural networks is analyzed as a dynamical system. The authors utilize the infinite-depth limit of discrete architectures to model networks as continuous-time systems (Neural ODEs and Neural DDEs). They apply tools from differential geometry, Morse theory, and functional analysis to classify the geometric structure of these maps.
Training Dynamics (Dynamics of the Network): Optimization algorithms, specifically Gradient Descent (GD) and Stochastic Gradient Descent (SGD), are modeled as discrete-time dynamical systems (or random dynamical systems for SGD). The authors analyze the stability of fixed points (minima) and invariant manifolds using spectral theory, Lyapunov exponents, and the theory of Milnor attractors.
Mean-Field Limits and Network Topology: To address the high dimensionality of large networks, the authors utilize mean-field limit techniques. They extend existing theories of interacting particle systems to heterogeneous networks using graph limits, graphops, and digraph measures (DGMs). This allows for the derivation of continuum limits (Vlasov-Fokker-Planck equations) for large-scale neural networks.

Key Contributions and Results

1. Information Propagation and Universal Properties

The paper establishes rigorous connections between network architecture and the class of functions a network can represent or approximate.

Universal Embedding vs. Approximation: The authors distinguish between universal approximation (approximating any function arbitrarily well) and universal embedding (exactly representing a function).
Augmented Architectures: It is proven that Augmented Neural ODEs (where the phase space dimension $m$ exceeds the input dimension $d$ ) possess the universal embedding property for $C^k$ functions, provided $m \geq d + q$ .
Geometric Classification: Using Morse theory, the paper classifies scalar Multi-Layer Perceptrons (MLPs) and Neural ODEs based on the existence and regularity of critical points.
- Non-augmented architectures are restricted to specific function classes (e.g., strictly monotone functions in certain cases) and generally lack universal approximation capabilities.
- Augmented architectures can represent a broader class of functions, including those with critical points.
- Bottleneck/Degenerate architectures exhibit specific limitations; for instance, Neural ODEs cannot have bottlenecks in hidden layers due to constant dimensionality over time.
Memory Dependence: For Neural DDEs, the paper demonstrates that universal approximation depends on the "memory capacity" (the product of the Lipschitz constant $K$ and delay $\tau$ ). If the memory capacity is sufficiently large, Neural DDEs can achieve universal embedding and approximation, even without increasing the phase space dimension.

2. Training Dynamics and Stability

The authors analyze the convergence properties of optimization algorithms by treating them as dynamical systems.

Overdetermined vs. Overparameterized Regimes:
- In overdetermined settings ($D < qN$), the loss landscape typically has isolated minima. The paper applies Milnor stability (based on the Lebesgue measure of the basin of attraction) to determine if a minimum is reachable. Stability is linked to the spectral radius of the Jacobian of the update map.
- In overparameterized settings ($D > qN$), the set of global minima forms a continuous manifold (interpolation solutions). The authors redefine stability to characterize convergence to specific regions of this manifold.
Edge of Stability: The analysis reveals that in overparameterized settings, Gradient Descent converges to "flat" minima where the Hessian norm satisfies $\|Hess L(\theta^*)|_{N(\theta^*)}\| \approx 2/\eta$ . This provides a dynamical explanation for the "Edge of Stability" phenomenon, suggesting that larger learning rates select for flatter minima, which are heuristically associated with better generalization.
Stochastic Gradient Descent (SGD): For SGD, the authors model the process as a Random Dynamical System. They establish that the stability of interpolation solutions in the overparameterized regime is determined by the Lyapunov exponent of the product of random Jacobian matrices. A negative Lyapunov exponent implies local stability. This result relies on the Furstenberg-Kesten theorem and assumptions of "regular" minima (strong irreducibility of the generated semigroup).

3. Mean-Field Limits and General Network Models

The paper generalizes the dynamical systems view to various network types, including Boltzmann machines, Hopfield networks, RNNs, and Transformers.

Interacting Particle Systems (IPS): Many neural network models are identified as special cases of IPS. The authors show that these systems can be analyzed via mean-field limits.
Heterogeneous Graph Limits: A significant contribution is the extension of mean-field theory to heterogeneous networks using digraph measures (DGMs). This framework allows for the derivation of nonlocal Vlasov-Fokker-Planck equations for large networks with complex topologies (e.g., graphons), moving beyond simple all-to-all coupling.
Generative Models: The paper connects generative models (GANs, Diffusion Models) to dynamical concepts. GANs are framed as zero-sum games linked to replicator dynamics, while Diffusion Models are analyzed via Stochastic Differential Equations (SDEs) and their associated Fokker-Planck equations. The robustness of score-based generative models is linked to the alignment of Oseledets spaces with the data manifold.
Backpropagation as Variational Equations: The authors clarify that backpropagation is mathematically equivalent to solving the variational equation of a dynamical system backward in time. This reframes the algorithm as a classical dynamical concept involving local linearization along trajectories.

Significance and Claims

The paper asserts that re-considering AI/ML as a dynamical problem is both natural and advantageous. Its primary significance lies in:

Unification: It demonstrates that diverse ML architectures (MLPs, ResNets, Transformers, GANs) and training methods (GD, SGD) fall under the umbrella of dynamical systems theory, specifically as adaptive networks, IPS, or random dynamical systems.
Rigorous Justification: By isolating specific aspects (e.g., memory capacity, phase space dimension, stability of minima), the authors show that rigorous analytical proofs are possible for complex ML phenomena, such as the edge of stability and the role of memory in approximation.
Explainability: The dynamical perspective offers a pathway to explain why certain algorithms work (e.g., why augmentation enables universal embedding, why flat minima generalize well) by linking them to fundamental properties like spectral radius, Lyapunov exponents, and manifold geometry.
Future Directions: The authors suggest that this framework is applicable to fundamental issues like vanishing/exploding gradients (interpreted as multiscale dynamical effects) and the development of reliable AI. They note that while full generality is challenging, the dynamical approach provides a powerful toolkit for theoretical analysis, potentially aided by computer-assisted proofs for complex non-linear systems.

The work is presented as a contribution to the DFG priority program SPP2298 "Foundations of Deep Learning," serving as a report on how dynamical systems techniques can be systematically applied to understand the theoretical underpinnings of deep learning.