🤖 machine learning

Learning Reduced-Order Dynamics with Singularity via Latent-Augmented Neural Ordinary Differential Equations

This paper proposes the Latent-Augmented Neural Ordinary Differential Equations (LA-NODEs) framework to resolve self-intersecting trajectory issues in industrial reduced-order modeling by augmenting neural ODEs to represent conflicting vector fields, thereby achieving superior prediction accuracy and fidelity in complex systems like IPMSM drives and distributed energy systems.

Original authors: Xiaorui Wang, Yu Zhou, Wenjie Mei, Dongzhe Zheng, Yang Bai, Masaaki Nagahara

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaorui Wang, Yu Zhou, Wenjie Mei, Dongzhe Zheng, Yang Bai, Masaaki Nagahara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the complex world of modern engineering, from the motors driving electric vehicles to the grids powering our cities, systems are often too intricate to manage in real time. These machines are governed by laws of physics that involve dozens of variables changing simultaneously, creating a high-dimensional puzzle that is computationally expensive to solve. To make these systems manageable for control and optimization, engineers use a technique called model order reduction. This process simplifies a complex machine into a smaller, faster version by focusing only on the most visible parts, much like viewing a three-dimensional object through a two-dimensional window. However, this simplification comes with a hidden trap. When the full, complex reality is squashed into a lower-dimensional view, different internal states can sometimes look exactly the same from the outside. This creates a confusing situation where a single point on the simplified map corresponds to multiple different directions of movement, a phenomenon known as a singularity. If a computer model tries to learn the behavior of such a system without accounting for this ambiguity, it gets stuck, unable to predict which way the system will go next.

Researchers have long relied on a powerful tool called Neural Ordinary Differential Equations to learn these dynamic systems directly from data. These models act as universal approximators, capable of learning intricate patterns without needing explicit physical formulas. Yet, standard versions of these models hit a hard wall when faced with the singularities caused by model reduction. Because they assume that every point in the system has only one unique direction of movement, they fail when that point actually has several possible paths. The result is a model that learns the general shape of the system but misses the critical details, leading to significant errors in prediction. This limitation is not just a minor glitch; it is a fundamental barrier that prevents accurate modeling of many real-world industrial systems where data is incomplete or simplified.

To overcome this, a team of researchers from Lanzhou University of Technology, Hiroshima University, Nanjing University, and Princeton University has developed a new framework called Latent-Augmented Neural Ordinary Differential Equations. The core idea is surprisingly straightforward: if the view through the window is too crowded to tell the paths apart, the model needs to look through a slightly larger window. The researchers proposed adding a hidden, or "latent," layer of information to the model. This extra layer acts as a temporary workspace that lifts the system into a higher-dimensional space just long enough to separate the confusing paths. By doing this, the model can distinguish between states that look identical on the surface but are moving in different directions. Once the model learns the correct path in this expanded space, it projects the answer back down to the original, simplified view, effectively resolving the confusion without needing to know the full, complex internal state of the machine.

The team proved mathematically that this approach is necessary. They demonstrated that without this extra dimension, the error in learning such systems cannot be reduced to zero, no matter how much data is provided or how long the model trains. The error is locked in by the geometry of the problem itself. However, by introducing these latent variables, the model can theoretically achieve perfect accuracy, learning the true dynamics even when the observed data is ambiguous. Crucially, the researchers did not just add random extra dimensions; they derived a specific rule for how many are needed. They found that the number of extra dimensions required depends on the complexity of the confusion at the singular points. If a point has two possible directions, one extra dimension is enough to separate them. If it has three, two extra dimensions are needed. This provides a clear, principled way to design the model, balancing the need for accuracy with the cost of computation.

To test their theory, the researchers applied this new framework to two very different industrial systems. The first was an interior permanent magnet synchronous motor, a type of electric motor used in high-performance drives. In this system, the researchers reduced the model from three variables down to two, creating a scenario where the motor's speed was hidden, causing the simplified view to show a single point with two possible velocity directions. The second system was a distributed energy network, simulating a microgrid with solar panels and batteries. Here, the reduction created a point with three possible directions. In both cases, the researchers compared their new method against existing models, including standard neural networks and other advanced differential equation models. The results were clear: the new framework consistently outperformed all others. In the motor experiment, the new model kept its prediction errors extremely low, while the older models struggled significantly as the system approached the confusing points. In the energy system test, the new model reduced the average error by more than half compared to the next best method, and it remained stable even as the system evolved over time.

The researchers also investigated how the size of the hidden layer affected performance. They found that increasing the number of extra dimensions improved accuracy up to a specific point, after which adding more dimensions provided no benefit and only slowed down the training process. For the energy system, which required two extra dimensions to resolve its three-way confusion, increasing the dimension beyond two actually caused a slight dip in performance. This confirmed their theoretical rule: the model needs just enough extra space to separate the paths, but no more. This finding is vital for practical application, as it ensures that engineers can build efficient models without wasting computational resources on unnecessary complexity.

The study concludes that this approach offers a robust solution for high-precision modeling of complex industrial systems where data is limited or simplified. By acknowledging and mathematically addressing the geometric limitations of reduced-order models, the researchers have provided a tool that can recover system features that were previously impossible to capture accurately. While the method has shown great promise in these specific simulations and experiments, the authors note that its ability to handle extremely high-dimensional systems or those with severe information loss still needs further exploration. Nevertheless, the work establishes a clear path forward for creating more reliable, data-driven models that can keep pace with the demands of modern engineering, turning the problem of singularities from a dead end into a solvable puzzle.

Technical Summary: Learning Reduced-Order Dynamics with Singularity via Latent-Augmented Neural Ordinary Differential Equations

1. Problem Statement

In practical industrial processes, high-dimensional, nonlinear models are often computationally intractable for real-time optimization and model predictive control. Consequently, model order reduction is employed to derive low-dimensional surrogate models. However, reducing the dimensionality of a system often involves projecting high-dimensional states onto a lower-dimensional observation space. This projection can result in non-injective mappings, where distinct internal states correspond to the same observed state.

At the trajectory level, this manifests as self-intersecting trajectories in the phase space of the reduced-order system. At these intersection points (singularities), a single observation state is associated with multiple distinct velocity vectors (directions of motion). Standard Neural Ordinary Differential Equations (NODEs) and their variants (such as ControlSynth NODEs or CSODEs) assume that the vector field is a single-valued function of the state. Consequently, they cannot accurately represent systems where a single state admits multiple distinct dynamics, leading to unavoidable training errors and a failure to capture the true system evolution. While time-dependent NODEs can theoretically disambiguate these directions, they require learning mappings with extreme temporal precision, which is often computationally intractable and insufficient for industrial applications where latent variables are not solely time-dependent.

2. Methodology: Latent-Augmented Neural ODEs (LA-NODEs)

To address the limitations of autonomous NODEs in the presence of singularities, the authors propose the Latent-Augmented Neural Ordinary Differential Equations (LA-NODEs) framework.

Core Architecture

The LA-NODE framework augments the observed state space with learnable latent variables. The system is modeled as a coupled set of differential equations:

Primary Dynamics: $\dot{y}(t) = \tilde{f}_\theta(y(t), z(t))$ , where $y(t)$ is the observed state.
Latent Dynamics: $\dot{z}(t) = g_\phi(y(t), z(t))$ , where $z(t) \in \mathbb{R}^k$ represents the augmented latent variables.

By lifting the system into a higher-dimensional space ( $\mathbb{R}^{p+k}$ ), the framework ensures that trajectories which intersect in the reduced observation space remain distinct in the augmented space. The latent variables $z(t)$ act as a "workspace" that resolves the velocity ambiguity at singular points. During inference, the latent state is discarded, and only the evolution of the observed state $y(t)$ is utilized.

Theoretical Foundation

The paper establishes a formal connection between the required augmentation dimension and the geometric properties of the singularities:

Lemma 1: Proves that for autonomous NODEs, the presence of singularities (where one state maps to multiple velocities) imposes a strictly positive lower bound on the training error. The error cannot be reduced to zero regardless of the network architecture because the model must assign a single velocity to a state that requires multiple.
Lemma 2: Demonstrates that by augmenting the state space with at least one latent dimension ( $k \ge 1$ ), the framework can achieve zero training error. The augmented states allow the model to distinguish between previously indistinguishable states, restoring the bijective nature of the vector field.
Theorem 1 (Key Theoretical Contribution): Derives a condition for the minimum required augmentation dimension. Let $\mathcal{V}(y^*)$ be the set of distinct velocity vectors at a singularity $y^*$ . The velocity rank $r(y^*)$ is defined as the dimension of the subspace spanned by these velocities. The theorem states that the augmentation dimension $k$ must satisfy:
$k \ge \max_{y^*} \{r(y^*)\}$
If $k$ is less than the velocity rank, the model cannot represent the full set of velocity directions, and the training error lower bound from Lemma 1 remains active.

3. Experimental Validation

The effectiveness of LA-NODEs was validated on two representative industrial models exhibiting different singularity properties:

Case 1: Interior Permanent Magnet Synchronous Motor (IPMSM) Drive

System: A 3D electromechanical system reduced to 2D ( $i_d, i_q$ ) by eliminating the electrical angular velocity ( $\omega_e$ ).
Singularity: A "two-trajectory" singularity where different $\omega_e$ values map to the same current state but produce different velocities.
Velocity Rank: $r=1$ .
Result: Theoretically, $k=1$ is sufficient. Experiments showed that LA-NODE (with $k=1$ ) achieved significantly lower Mean Squared Error (MSE) compared to CSODE, ANODE, and Transformer baselines. CSODE failed to capture the dynamics near the singularity, exhibiting large deviations, while LA-NODE maintained consistency across the state space.

Case 2: Distributed Energy System (DES)

System: A 4D dynamic system involving photovoltaic generation, energy storage, and loads, reduced to 2D (observed power outputs).
Singularity: A "three-trajectory" singularity where three distinct internal states map to the same observation with three distinct velocity directions.
Velocity Rank: $r=2$ .
Result: Theorem 1 dictates $k \ge 2$ . Experiments confirmed that a 2D augmentation was necessary. LA-NODE (with $k=2$ ) outperformed all baselines, reducing MAE and RMSE by approximately 56% and 59% respectively compared to the second-best method (ANODE).

Sensitivity Analysis

A sensitivity analysis on the DES system varied the augmentation dimension $k$ from 1 to 5.

Performance: Increasing $k$ from 1 to 2 resulted in a massive reduction in error (contributing ~97% of the total error reduction). Further increases ( $k > 2$ ) yielded marginal gains and eventually led to a slight increase in test error due to over-parameterization and optimization complexity.
Efficiency: Training time increased linearly with $k$ . The analysis concluded that selecting the minimum feasible dimension satisfying Theorem 1 offers the best trade-off between representational capability and computational cost.

4. Key Contributions

Identification of Fundamental Limitation: The paper formally demonstrates that autonomous NODEs cannot achieve zero training error when learning reduced-order systems with singularities due to the multi-valued nature of the underlying dynamics at intersection points.
Proposed Framework (LA-NODE): Introduces a latent-augmented architecture that lifts the system to a higher-dimensional space, theoretically enabling zero training error by resolving velocity ambiguities.
Theoretical Criterion for Dimension Selection: Derives a principled lower bound for the augmentation dimension based on the velocity rank of the singularity. This provides a guideline for selecting the minimal $k$ required to resolve specific topological ambiguities, balancing model expressivity and efficiency.
Industrial Validation: Validates the method on two complex industrial systems (IPMSM and DES), demonstrating superior prediction accuracy and modeling fidelity compared to state-of-the-art baselines (CSODE, ANODE, Transformer).

5. Significance and Claims

The paper claims that LA-NODEs provides an effective approach for high-precision data-driven modeling of complex industrial systems where dimensionality reduction introduces singularities. By addressing the geometric limitations of standard NODEs, the framework recovers system features that are difficult to capture using conventional approaches.

The authors emphasize that their contribution is not merely a new architecture but a theoretical understanding of why augmentation is necessary and how much is required. They assert that without satisfying the velocity rank condition, the model is fundamentally limited in its ability to learn the true dynamics.

Limitations and Future Work:
The paper modestly acknowledges that the framework relies on the existence of a finite-dimensional latent space to resolve ambiguity. For systems with severe information loss or highly complex hidden dynamics, state augmentation alone may be insufficient. Furthermore, the current validation is on low-dimensional manifolds, and the scalability to high-dimensional state spaces requires further exploration. Future work aims to extend the framework to high-dimensional systems and optimize it for real-time control under stochastic perturbations.