HyperKKL: Enabling Non-Autonomous State Estimation through Dynamic Weight Conditioning

This paper introduces HyperKKL, a novel hypernetwork-based approach that enables non-autonomous state estimation for nonlinear systems by dynamically generating KKL observer parameters from exogenous inputs, thereby overcoming the limitations of existing learning-based methods that struggle with driven dynamics.

The Gist

Imagine you are trying to guess the exact location and speed of a car driving through a thick fog. You can't see the car directly, but you have a few sensors: maybe you can hear the engine noise or see the headlights flickering. This is the problem of State Estimation: figuring out the full, hidden reality of a system based on only a few clues.

For simple, predictable cars (like a car on a straight, empty road), we have great mathematical tools to guess where it is. But what if the car is being pushed by strong, unpredictable winds, or the driver is suddenly slamming the brakes and hitting the gas? The car becomes non-autonomous—its behavior depends not just on where it is, but on these outside forces.

This paper introduces a new tool called HyperKKL to solve this problem. Here is the breakdown in simple terms:

1. The Old Problem: The "Static Map" vs. The "Moving Target"

Imagine you have a map of a city.

• Autonomous Systems: The city is static. The streets don't move. If you know the rules of the road, you can draw a perfect map to predict where a car will go.
• Non-Autonomous Systems: Now, imagine the city is shifting. The streets stretch, shrink, and twist based on the wind blowing outside. A static map you drew yesterday is useless today because the terrain has changed.

Existing AI methods tried to solve this by either:

1. Ignoring the wind: They used a map designed for calm weather. When the wind blew, the map was wrong, and the prediction failed.
2. Retraining constantly: They tried to learn a new map for every single wind speed. This is slow and expensive.
3. Curriculum Learning (The "School" Approach): They tried to teach the AI by starting with calm days and slowly introducing windier days. The paper found this failed miserably. It's like trying to teach someone to surf by starting in a bathtub, then a pool, then a lake, and expecting them to survive a tsunami just because they practiced on smaller waves. The "brain" (the architecture) wasn't built to handle the chaos of the big waves.

2. The Solution: The "Smart Chameleon" (HyperKKL)

The authors propose HyperKKL. Think of this not as a single map, but as a smart chameleon.

Instead of having one fixed map, the system has a "Master Painter" (a secondary neural network called a Hypernetwork).

• The Context: The Master Painter looks at the current weather (the external input, like the wind or wind speed).
• The Action: Based on the weather, the Master Painter instantly rewrites the rules for the main map. It doesn't just add a note to the side; it physically changes the shape of the map to match the current conditions.
• The Result: The main observer (the map) instantly adapts. If the wind blows left, the map stretches left. If the wind stops, the map snaps back to normal.

3. Two Ways to Paint the Map

The paper tests two versions of this chameleon:

• The "Static" Chameleon (Static HyperKKL): This one is good for gentle changes. It keeps the main map mostly the same but adds a small "adjustment knob" based on the wind. It works great for systems that wiggle smoothly (like a pendulum).
• The "Dynamic" Chameleon (Dynamic HyperKKL): This one is the heavy lifter. It completely reshapes the map based on the history of the wind. It remembers that the wind was gusting 5 seconds ago and adjusts the map accordingly. This is necessary for chaotic systems (like weather patterns or turbulent fluids) where the past matters just as much as the present.

4. The Experiments: What Happened?

The team tested this on four famous "toy" systems:

1. Duffing & Van der Pol: These are like swinging pendulums. The Dynamic Chameleon was amazing here, reducing errors by up to 60% compared to the old methods.
2. Rössler: A chaotic system (like a swirling smoke ring). The Dynamic Chameleon handled it well, while the old "School" method (Curriculum Learning) failed completely.
3. Lorenz: The "Butterfly Effect" system. This is extremely sensitive. Here, even the smart chameleon struggled a bit. The old "ignore the wind" method was actually the safest. Why? Because the chameleon tried to adjust so perfectly that tiny errors in its "painting" got magnified into huge mistakes. It's a reminder that sometimes, knowing too much about the wind can make you more confused than knowing nothing.

The Big Takeaway

The paper teaches us that you cannot just "train harder" to solve complex, changing problems. You need the right architecture.

• Old Way: Try to memorize every possible wind scenario (Curriculum Learning). Result: Failure.
• New Way (HyperKKL): Build a system that can instantly reshape its own brain based on the current situation. Result: Success for most complex systems.

In a nutshell: If you want to predict the future of a system being pushed by outside forces, don't just give the AI more homework. Give it a shape-shifting brain that can adapt its own rules in real-time.

Technical Summary

1. Problem Statement

The paper addresses the challenge of state estimation for non-autonomous nonlinear systems.

• Context: In many real-world applications (robotics, biology, industry), systems are driven by external inputs ($u(t)$), making them non-autonomous. Traditional state estimation methods often struggle here.
• The KKL Framework: The Kazantzis-Kravaris/Luenberger (KKL) observer is a powerful theoretical tool that immerses nonlinear dynamics into a stable linear latent space. However, its practical implementation requires solving a Partial Differential Equation (PDE) to find a transformation map $T(x)$.
• The Gap:
  • Existing learning-based KKL approximations are primarily designed for autonomous systems (where $u=0$).
  • Extending KKL to non-autonomous systems theoretically requires a time-dependent transformation $T(x, t)$ or an input-dependent map.
  • Current learning approaches fail to generalize to driven dynamics without expensive retraining or online gradient updates.
  • Simple "curriculum learning" (training on increasingly complex inputs) has not been rigorously tested against architectural changes for this specific problem.

2. Methodology: HyperKKL

The authors propose HyperKKL, a framework that uses hypernetworks to condition the KKL observer parameters on the history of exogenous inputs. This allows the observer to adapt instantly to changing inputs without retraining.

Core Architecture

The approach builds upon a physics-informed autoencoder (Encoder $\hat{T}$ and Decoder $\hat{T}^*$) and introduces two specific architectures:

1. Static HyperKKL (Stationary Approach):

   • Retains a fixed transformation map $T(x)$.
   • Augments the observer dynamics with a learned input injection term $\bar{\phi}(\hat{z}, u)$.
   • Uses an LSTM to encode the input history $u[t-w, t]$ and feed it into a small MLP that injects the input effect into the latent dynamics.
   • Best for: Systems where inputs act as bounded perturbations that do not fundamentally alter the attractor geometry.

2. Dynamic HyperKKL (Dynamic Approach):

   • Implements a fully time-varying transformation $T(x, t)$.
   • Uses a hypernetwork ($H_\psi$) that takes the input history as context and generates weight perturbations ($\Delta\theta, \Delta\phi$) for the base encoder and decoder.
   • Residual Structure: The final weights are $\theta_{base} + \Delta\theta(u)$. The base weights are frozen from a pre-trained autonomous model, ensuring stability when $u=0$.
   • Efficiency: To manage parameter space, the hypernetwork uses a "chunked prediction" strategy with low-rank decomposition rather than predicting full weight matrices.

Training Procedure (Two-Phase)

To avoid gradient conflicts and ensure stability, the authors employ a sequential training strategy:

• Phase 1 (Autonomous Pretraining): Train the base encoder and decoder on unforced dynamics ($u=0$) using a physics-informed loss (data fit + PDE residual). The resulting weights are frozen.
• Phase 2 (Hypernetwork Training): Freeze the base maps and train only the hypernetwork parameters on forced trajectories. The loss function minimizes reconstruction error and the residual of the time-dependent PDE.
• Temporal Derivative Approximation: Since the PDE involves $\frac{\partial T}{\partial t}$, the authors approximate this using finite differences between the encoder outputs of adjacent input windows, avoiding explicit differentiation through the LSTM.

Baseline Comparison: Curriculum Learning

The paper evaluates a "Training-Only" baseline where a fixed-parameter KKL observer is trained using Adaptive Curriculum Learning. This involves training on non-autonomous data with increasing spectral complexity (constant $\to$ sinusoidal $\to$ square wave) to see if data diversity alone can bridge the gap.

3. Key Contributions

1. Hypernetwork-Conditioned KKL: Introduction of a novel architecture that maps input signals to observer parameters, enabling adaptation to varying input conditions without retraining or online updates.
2. Systematic Evaluation of Training vs. Architecture: A controlled comparison demonstrating that curriculum learning alone is insufficient for non-autonomous KKL observers. In many cases, it degrades performance compared to a simple autonomous baseline.
3. Comprehensive Empirical Study: Evaluation across four benchmark nonlinear systems (Duffing, Van der Pol, Lorenz, Rössler) covering both oscillatory and chaotic regimes under various input conditions (zero, constant, sinusoidal, square wave).

4. Experimental Results

The framework was tested on four systems with metrics including RMSE and SMAPE.

• Oscillatory Systems (Duffing, Van der Pol):

  • HyperKKL Success: Both Static and Dynamic variants significantly outperformed the autonomous baseline.
  • Duffing: Static HyperKKL reduced RMSE by up to 62% under sinusoidal inputs.
  • Van der Pol: Dynamic HyperKKL achieved the best results on time-varying inputs.
  • Conclusion: For systems with smooth attractor shifts, input conditioning works exceptionally well.

• Chaotic Systems (Rössler, Lorenz):

  • Rössler: The Dynamic HyperKKL yielded the lowest errors, confirming that aggregating input history is crucial for complex attractors.
  • Lorenz (Critical Finding):
    • Curriculum Learning Failure: Performance collapsed (RMSE doubled) compared to the autonomous baseline.
    • Static HyperKKL Failure: Conditioning on instantaneous input caused catastrophic degradation (RMSE $\approx$ 16 vs. 5.5 for autonomous).
    • Dynamic HyperKKL: Improved over Curriculum and Static methods but still slightly underperformed the pure Autonomous baseline (RMSE 6.66 vs. 5.55).
  • Analysis: The Lorenz system is extremely sensitive to perturbations. The hypernetwork's modulation introduced small errors that propagated exponentially along unstable manifolds. The "noise" introduced by conditioning outweighed the signal benefit in this specific regime.

5. Significance and Conclusion

• Representational vs. Educational Bottleneck: The failure of Curriculum Learning proves that the inability to estimate non-autonomous states is a representational limitation (the architecture cannot solve the dynamic PDE), not a lack of training data diversity.
• Architecture Matters: Hypernetworks provide the necessary flexibility to solve time-varying PDEs, but they must be applied carefully.
• Trade-offs: While HyperKKL enables state estimation for driven systems, the results on the Lorenz attractor highlight a fundamental tension: Input conditioning adds capacity but introduces a pathway for error propagation. For highly sensitive chaotic systems, ignoring the input (Autonomous observer) can sometimes be safer than attempting to model it imperfectly.
• Future Directions: The authors suggest incorporating Lyapunov-informed regularization or adaptive gating to attenuate weight perturbations in high-sensitivity regions, ensuring stability while retaining the benefits of input conditioning.

In summary, HyperKKL successfully extends KKL observers to non-autonomous systems for a wide class of problems, but it reveals that for highly chaotic systems, the design of the conditioning mechanism must be rigorously constrained to prevent instability.