⚡ electrical engineering

Local Observability and Moving Horizon Estimation-based Training of Feedforward Neural Networks

This paper proposes a moving horizon estimation-based training method for feedforward neural networks with ReLU activations by reformulating them as dynamical systems, analyzing their local observability to derive sufficient conditions and input design strategies, and thereby establishing convergence guarantees for weight estimation from a control-theoretic perspective.

Original authors: Yi Yang, Victor G. Lopez, Matthias A. Müller

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Yang, Victor G. Lopez, Matthias A. Müller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Tuning a Radio with a Map

Imagine you have a complex radio (a Neural Network) that needs to be tuned to catch a specific station perfectly. The "knobs" on this radio are its weights (the numbers inside the network that determine how it processes information).

Usually, people tune these radios by trial and error, turning a knob slightly, listening, and turning it again. This is like the standard "Backpropagation" method mentioned in the paper. It works, but nobody can mathematically prove exactly when or why it will find the perfect setting, or if it might get stuck in a dead end.

This paper proposes a new way to tune the radio. Instead of just guessing, it treats the tuning process like a navigation problem. It asks: "If I play a specific song (input) and hear the result (output), can I mathematically prove exactly where the knobs are?"

The Core Idea: Turning Weights into a Moving Target

The authors take a clever shortcut. They pretend the weights of the neural network aren't just static numbers, but are actually the position of a car on a map.

The Car: The neural network's weights.
The Road: The flow of data through the network.
The Goal: To figure out exactly where the car is parked (the ideal weights) just by listening to the engine noise (the output) while driving over different bumps (the inputs).

They use a technique called Moving Horizon Estimation (MHE). Think of MHE as a detective looking at a short video clip of the car's recent movements (a "window" of data) to deduce where it is right now.

The Problem: The "Ghost" Knobs

The paper identifies a major snag. In many neural networks, different settings of the knobs can produce the exact same sound.

Analogy: Imagine you have three volume knobs. If you turn Knob A up and Knob B down by the same amount, the total volume might stay the same. If you only listen to the volume, you can't tell which specific combination of knobs is being used. The system is "unobservable." You can't distinguish the true setting from its neighbors.

The authors found that for deep, complex networks (multi-layer), this "ghost" problem is almost always true. You simply can't mathematically prove you've found the one perfect setting just by looking at the output.

The Solution: The "Special Room" and the "Magic Song"

However, the paper shows that for a specific type of simpler network (a two-layer network with fixed output settings), there is a "Special Room" (a locally observable neighborhood) where the knobs are distinguishable.

To enter this room and stay there, you need a Magic Song (a Persistently Exciting Input).

The Magic Song: This isn't just any random noise. It is a very specific, carefully designed sequence of inputs.
The Analogy: If you just tap the radio randomly, you might not hear the difference between two similar settings. But if you play a specific, complex chord progression (the PE input), the radio will react differently for every tiny difference in the knobs. This allows the "detective" (the algorithm) to pinpoint the exact location of the knobs.

The authors created a recipe for writing this "Magic Song" so that the system becomes mathematically solvable.

How the Training Works

Once they have this "Magic Song" and know they are in the "Special Room," they run their MHE training:

Look at a small window of data: They don't use the whole dataset at once (which would be too heavy). They look at a small batch (mini-batch).
Update only what they can see: If a specific part of the data doesn't help distinguish the knobs (the "unobservable" part), they freeze that part. They only update the parts of the weights that the current data can distinguish.
Repeat: They cycle through different batches of data. Over time, the "Magic Song" ensures that every part of the radio gets a chance to be distinguished.

The Results: Faster and Proven

The paper tested this on two things:

Fake Data (Synthetic): They created a perfect "teacher" radio and tried to train a "student" radio to copy it. The new method found the perfect settings much faster than the standard method (reaching the target in 1 epoch vs. 3.3 seconds of time, though the paper notes the calculation per step is heavier, the convergence is faster).
Real Data (UCI Wine Dataset): They tried to predict wine quality. Even though the network they used was the "complex" type where the math doesn't strictly guarantee a solution, the new method still performed better (lower error) than standard methods like Adam or other advanced techniques.

The Bottom Line

The authors didn't just say "try this new trick." They built a mathematical map.

They proved that for certain networks, if you use their specific "Magic Song" (input design), you can mathematically guarantee that the training will converge to the right answer.
They showed that even if you can't guarantee it for every network, using this method still works very well in practice.

In short, they turned the messy art of training AI into a rigorous science of navigation, proving that with the right map and the right song, you can always find your way to the perfect settings.

Technical Summary: Local Observability and Moving Horizon Estimation-based Training of Feedforward Neural Networks

Problem Statement
Feedforward Neural Networks (FNNs) are widely used as universal function approximators, yet their training via standard backpropagation and gradient descent (GD) often lacks rigorous theoretical guarantees regarding convergence to optimal weights. While state estimation-based methods, such as the Extended Kalman Filter and Moving Horizon Estimation (MHE), have been explored to treat network weights as system states, a critical theoretical gap remains: the assumption that these weights are observable. In general, FNN weights are not globally observable due to non-uniqueness (e.g., symmetries in weights and node permutations). Furthermore, for general multi-layer FNNs, the observability rank condition often fails to hold, leaving the convergence of MHE-based training unproven. This paper addresses the challenge of establishing local observability for FNNs to provide a theoretical foundation for MHE-based training.

Methodology
The authors reformulate the FNN training problem by treating the network weights as the static states of a dynamical system. The core methodology involves a three-step process:

Local Observability Analysis: The paper investigates the local observability of FNNs with Rectified Linear Unit (ReLU) activation functions. By defining the $k$ -observability mapping and applying the observability rank condition, the authors analyze whether the weights can be distinguished from their neighbors using input-output data.
- Two-Layer FNNs: For a specific class of two-layer FNNs with fixed output weights, the authors derive a sufficient condition for local observability. This condition relies on the intersection of the weight matrix's row space with specific orthants in the state space, ensuring the existence of a persistently exciting (PE) input sequence that satisfies the observability rank condition.
- General Multi-Layer FNNs: The analysis demonstrates that general multi-layer FNNs typically fail to satisfy the observability rank condition, implying that global or even local observability cannot be guaranteed for arbitrary deep architectures.
Persistently Exciting (PE) Input Design: Based on the observability analysis, the paper develops a systematic method to design PE input sequences. For a locally observable state, the authors construct a "locally observable neighborhood" where all states share the same Jacobian structure of the observability mapping. Within this neighborhood, a single PE input sequence ensures that all states are distinguishable.
MHE-Based Training Algorithm: The authors propose an MHE-based training scheme where weights are updated iteratively using a fixed-length window of input-output data (mini-batches).
- Constraint Handling: To handle cases where individual mini-batches may not be persistently exciting, the optimization problem includes a constraint that fixes the projection of the state estimate onto the unobservable subspace. Only the projection onto the observable subspace is updated.
- Convergence Analysis: Using alternating projection theory, the paper proves that if the initial weights lie within a locally observable neighborhood and the aggregate training dataset is persistently exciting, the state estimation error converges to a neighborhood of zero. The size of this neighborhood is bounded by the fitting error (noise) in the data.

Key Contributions

Reformulation and Condition Derivation: The paper reformulates FNNs as dynamical systems and derives a sufficient condition for the local observability of two-layer FNNs with ReLU activations and fixed output weights. This condition is based on the rank of an indicator matrix derived from the sign vectors of the weight matrix's row space intersections with orthants.
PE Input Design and Neighborhood Construction: A systematic approach is developed to design PE input sequences for locally observable states. Crucially, the authors construct a "locally observable neighborhood," proving that if an input sequence renders a specific state observable, it also renders all states in that neighborhood observable and distinguishable.
Convergence Guarantees: The paper establishes theoretical convergence guarantees for the proposed MHE-based training method. It proves that under the assumption of local observability and periodic use of mini-batches, the weight estimation error converges to a bound determined by the data noise level.
Extension to General Architectures: While the rigorous convergence proof applies to specific two-layer architectures, the paper acknowledges that general multi-layer FNNs often lack local observability. It proposes a modified MHE approach (removing observability constraints and adding regularization) that remains competitive in practice, though without the same theoretical guarantees.

Results
The effectiveness of the proposed method is validated through numerical experiments:

Synthetic Data: In a synthetic regression task with a two-layer FNN, the MHE-based method demonstrated faster convergence of the state estimation error compared to a standard mini-batch GD method. The MHE method reached a target error threshold within one epoch, whereas GD required more time, despite the MHE method having a higher computational cost per epoch.
UCI Wine Dataset: On a real-world benchmark dataset using a general two-layer FNN (where local observability is not theoretically guaranteed), the MHE-based method achieved a lower Root Mean Square Error (RMSE) compared to standard Adam optimization and a recent Natural Gradient Descent method (NNG-MVG).

Significance and Claims
The paper claims to provide the first rigorous theoretical analysis of MHE-based FNN training grounded in control-theoretic concepts of local observability. By identifying specific conditions under which FNN weights are locally observable, the authors enable the application of MHE with formal convergence guarantees. The work highlights that while global observability is generally unattainable for FNNs, local observability is sufficient for training if the input data is appropriately designed. The authors modestly note that their convergence guarantees are contingent on the initial weights being within a locally observable neighborhood and that future work is required to extend these guarantees to general multi-layer architectures where local observability cannot be ensured.