⚡ electrical engineering

Information Shapes Koopman Representation

This paper proposes an information-theoretic Lagrangian formulation and a corresponding algorithm that balances mutual information and von Neumann entropy to overcome the expressivity-simplicity tradeoff in Koopman representation learning, resulting in more stable, interpretable, and high-performing models across diverse dynamical systems.

Original authors: Xiaoyuan Cheng, Wenxuan Yuan, Yiming Yang, Yuanzhao Zhang, Sibo Cheng, Yi He, Zhuo Sun

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoyuan Cheng, Wenxuan Yuan, Yiming Yang, Yuanzhao Zhang, Sibo Cheng, Yi He, Zhuo Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, the movement of a swinging pendulum, or the flow of water in a dam. These are all examples of dynamical systems—things that change over time in complex, often messy ways.

For a long time, scientists have tried to turn these messy, non-linear movements into simple, straight-line (linear) rules. This is called the Koopman operator approach. Think of it like trying to describe a chaotic dance by saying, "If you take one step forward, you will always end up exactly here." It's a powerful idea because linear math is easy to solve, but the real world is rarely that simple.

The problem, as this paper points out, is that when we try to teach computers to find these "simple rules" inside complex data, they often get stuck. They either oversimplify things (missing important details) or they get too complicated and unstable (predicting nonsense after a few steps).

Here is how the authors fixed this, explained through simple analogies:

1. The Goldilocks Dilemma: Too Simple vs. Too Complex

The authors argue that the computer's "brain" (the latent representation) needs to find a perfect balance, much like the story of Goldilocks.

Too Simple: If the computer tries to compress the data too much to make it "clean," it forgets important details. It's like trying to describe a whole movie using only one sentence. You get the gist, but you miss the plot twists. In the paper, this is called mode collapse, where the system ignores most of the possible movements and only focuses on a few dominant ones.
Too Complex: If the computer keeps every tiny detail, it gets confused and unstable. It's like trying to memorize every single leaf on a tree to predict the wind; you get overwhelmed and can't see the big picture.

2. The Two Ingredients: "Mutual Information" and "Entropy"

To solve this, the authors introduce two "ingredients" based on information theory (the math of how much data you have):

Ingredient A: Mutual Information (The "Glue")
- What it does: This measures how well the "past" connects to the "future."
- The Analogy: Imagine a chain. Mutual Information is the strength of the links. If the links are strong, knowing where the chain was yesterday tells you exactly where it will be today. This ensures temporal coherence—the system doesn't forget its own history.
- The Risk: If you make the links too strong, the chain becomes rigid and breaks into just a few stiff links. You lose the ability to bend and move in different directions.
Ingredient B: Von Neumann Entropy (The "Shaker")
- What it does: This measures how "spread out" the information is.
- The Analogy: Imagine a jar of marbles. If all the marbles are stuck in one corner, the jar is "collapsed." Entropy is like shaking the jar so the marbles spread out evenly. This prevents the system from getting stuck on just one type of movement. It ensures expressiveness—the system can handle many different kinds of motion.
- The Risk: If you shake it too much without any structure, the marbles fly everywhere and the chain falls apart.

3. The New Recipe: The "Information Lagrangian"

The authors created a new "recipe" (a mathematical formula called a Lagrangian) that mixes these two ingredients perfectly.

It tells the computer: "Keep the links between past and future strong (Mutual Information), but make sure you don't forget to spread your marbles out (Entropy)."
They also added a third rule: Structural Consistency. This is like making sure the dance steps actually follow a straight line in the computer's hidden brain, even if the real-world dance is a wobble.

4. What Happened When They Tried It?

The team tested this new recipe on three very different types of "dances":

Physical Simulations: Like predicting the chaotic swirl of a tornado (Lorenz 63) or the flow of water over a dam.
Visual Control: Watching a video of a swinging pendulum or a robot arm and trying to predict its next move just by looking at the pixels.
Graph Dynamics: Predicting how a rope or a soft robot moves, where the parts are connected like a web.

The Results:

Longer Predictions: Other methods worked well for a few seconds but then drifted off course. The new method stayed accurate for much longer, like a GPS that doesn't lose signal after a minute.
Better Stability: When they visualized the "hidden map" the computer learned, other methods looked like a crumpled piece of paper or a single line. The new method's map looked like a smooth, perfect circle (for the pendulum) or a clear loop (for the vortex), matching the real physics.
Robustness: Even when they added "noise" (like static on a TV screen) to the data, the new method kept working, while others failed.

In Summary

The paper claims that by treating the problem as a balance between keeping information connected (Mutual Information) and keeping information diverse (Entropy), they can teach computers to find simple, linear rules for complex, chaotic systems. This leads to predictions that are not only more accurate but also more stable over long periods, without needing to know the specific physics of the system beforehand.

The code for this new method is available for anyone to try, and it has been tested to work better than current state-of-the-art methods across a wide variety of physical and visual tasks.

Technical Summary: Information Shapes Koopman Representation

1. Problem Statement

The Koopman operator framework offers a powerful method for modeling nonlinear dynamical systems by representing their evolution as linear transformations in an infinite-dimensional function space. However, practical implementation requires identifying or learning a suitable finite-dimensional subspace. Current deep learning approaches, such as Variational Autoencoders (VAEs) adapted for Koopman learning, often struggle with instability, mode collapse, and unreliable long-term dynamics.

The core challenge lies in the tension between simplicity (achieving a compact, linear latent representation) and expressiveness (retaining sufficient information to faithfully reconstruct and predict complex system trajectories). Existing methods often fail to balance these competing objectives, leading to latent spaces that either collapse onto a few dominant modes (losing predictive sufficiency) or fail to maintain temporal coherence (accumulating error over time). The paper posits that these difficulties stem from suboptimal representation learning that does not explicitly address the information-theoretic constraints of the Koopman operator.

2. Methodology

The authors propose a novel framework grounded in the Information Bottleneck (IB) principle, adapted specifically for the dynamical context of Koopman learning. The methodology proceeds through three main stages:

A. Information-Theoretic Analysis

The paper establishes a probabilistic view of Koopman representation where information loss during latent propagation directly correlates with prediction error.

Error Bounds: It is proven that the discrepancy between true system trajectories and Koopman-induced trajectories is bounded by the gap between the intrinsic dynamical coupling ( $I(x_{n-1}; x_n)$ ) and the preserved latent mutual information ( $I(z_{n-1}; z_n)$ ).
Information Disentanglement: The total information in the latent space is decomposed into three spectral components:
1. Temporal-coherent information: Associated with Koopman eigenvalues near the unit circle ( $|\lambda| \approx 1$ ), representing stable, conserved dynamics.
2. Fast-dissipating information: Associated with eigenvalues inside the unit circle ( $|\lambda| < 1$ ), representing transient dependencies that decay rapidly.
3. Residual information: Unpredictable noise or anomalies with no spectral counterpart.

B. Theoretical Insights on Representation

The authors derive two critical insights regarding how information metrics shape the latent space:

Mutual Information (MI) and Mode Collapse: Maximizing latent mutual information ( $I(z_{t-n}; z_t)$ ) enhances temporal coherence and predictive relevance. However, under finite variance constraints, this optimization leads to a "water-filling" effect where variance concentrates on the few dominant spectral modes. This causes mode collapse, reducing the effective dimension of the latent space and limiting the model's ability to represent diverse dynamics.
Von Neumann Entropy (VNE) and Diversity: The von Neumann entropy of the normalized latent covariance matrix serves as a measure of effective dimension. Penalizing low entropy (or maximizing entropy) prevents the collapse of variance onto a single direction, encouraging a spectrally diverse representation that preserves the full range of Koopman modes.

C. The Information-Theoretic Lagrangian

To operationalize these insights, the authors propose a unified Lagrangian formulation that balances simplicity and expressiveness. The objective function includes:

Temporal Coherence: Maximizing mutual information between latent states at different time steps ( $I(z_n; P_n)$ ).
Structural Consistency: Enforcing the linear transition constraint via a conditional likelihood term (equivalent to minimizing $I(z_t; x_t | z_{t-1})$ ), ensuring the latent dynamics follow the Koopman operator structure.
Predictive Sufficiency: Maximizing the von Neumann entropy of the latent covariance to maintain effective dimensionality and prevent mode collapse.
Reconstruction: Standard decoder loss to ensure the latent space can reconstruct the observed states.

This formulation yields a tractable, architecture-agnostic loss function applicable to VAEs, Autoencoders, and graph-structured networks.

3. Key Contributions

Theoretical Framework: The paper provides the first information-theoretic characterization of Koopman representation learning, explicitly linking latent mutual information to error bounds and von Neumann entropy to effective dimensionality.
Resolution of the Simplicity-Expressiveness Trade-off: It identifies the specific mechanism (water-filling allocation) by which maximizing mutual information leads to mode collapse and proposes von Neumann entropy regularization as the solution to preserve spectral diversity.
General Algorithm: A new loss function derived from the Lagrangian formulation that simultaneously enforces temporal coherence, structural consistency, and predictive sufficiency without relying on domain-specific priors (e.g., symmetry or conservation laws).
Empirical Validation: Extensive experiments across physical simulations (Lorenz 63, Kármán vortex, Dam flow, ERA5 weather), visual-input control (Planar, Pendulum, Cartpole), and graph-structured dynamics (Rope, Soft Robotics).

4. Experimental Results

The proposed method, referred to as InformationKoopman, was evaluated against state-of-the-art baselines including VAEs, Koopman Autoencoders (KAE), Koopman Kernel Regression (KKR), and specialized models like Poincaré Flow Neural Networks (PFNN).

Physical Simulations: The method achieved superior performance in both short- and long-term forecasting (measured by NRMSE) and spectral distribution errors (SDE). Notably, it outperformed KAE and KKR in long-horizon predictions, maintaining stability where baselines exhibited drift.
Visual Control: In tasks requiring control from high-dimensional images, the method produced latent manifolds that closely aligned with ground-truth phase spaces (e.g., the $S^1 \times \mathbb{R}$ structure of a pendulum). In contrast, baselines like KAE collapsed into low-dimensional structures, and E2C suffered from geometric distortion.
Robustness: The approach demonstrated robustness under noisy observations and extended prediction horizons (up to 100 steps in graph dynamics and 50 steps in weather forecasting).
Ablation Studies: Removing any of the three core regularization terms (mutual information, structural consistency, or von Neumann entropy) resulted in specific failures: loss of temporal coherence, collapse of the manifold, or suppression of effective dimensions, respectively.

5. Significance and Claims

The paper claims to offer a principled, general approach to constructing finite-dimensional Koopman representations that does not rely on hand-crafted physical priors. By reframing the learning problem through the lens of the Information Bottleneck, the authors demonstrate that:

Latent mutual information is the primary driver of predictive accuracy but must be constrained to avoid collapse.
Von Neumann entropy is the critical regulator that ensures the latent space retains the necessary complexity to model rich, multi-modal dynamics.

The authors assert that their Lagrangian formulation successfully balances the trade-off between simplicity (linear dynamics) and expressiveness (rich spectral content), leading to stable, interpretable, and accurate Koopman representations across a diverse range of dynamical systems. The work suggests that information-theoretic principles provide a robust meta-framework for guiding representation learning in complex dynamical contexts.