⚡ electrical engineering

Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient

This paper proposes "Relearn LQR," a data-driven, on-policy framework that integrates recursive least squares with policy gradient methods to iteratively optimize control inputs for unknown linear systems while providing formal stability guarantees through a Lyapunov-based analysis of the resulting interconnected nonlinear dynamics.

Original authors: Lorenzo Sforni, Guido Carnevale, Ivano Notarnicola, Giuseppe Notarstefano

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Sforni, Guido Carnevale, Ivano Notarnicola, Giuseppe Notarstefano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to fly a plane, but you have no manual and no blueprint of how the plane works. You don't know how the engines respond to the controls, how the wind affects the wings, or how heavy the plane is. All you have is the plane itself, and you need to figure out the perfect way to fly it while keeping it safe.

This paper presents a new method called "Relearn LQR" to solve exactly this problem. It's a way to learn how to control a system (like a plane) in real-time, without ever stopping to "reboot" the system, while guaranteeing that the plane won't crash during the learning process.

Here is the breakdown using simple analogies:

1. The Problem: The "Blind Pilot"

In traditional control engineering, you usually need a perfect map (a mathematical model) of the system before you can design a controller.

Old Way (Off-Policy): Imagine a pilot who sits in a simulator, tries a million different maneuvers, records the data, and then tries to fly the real plane based on that data. If the real plane is slightly different from the simulator, the pilot might crash.
The Challenge: The authors want a pilot who learns while flying the real plane. This is called On-Policy. The pilot adjusts the controls, sees what happens, and immediately tweaks the next move.
The Risk: If you tweak the controls too aggressively while learning, the plane could spiral out of control. Most learning methods can't promise the plane will stay safe while they are figuring things out.

2. The Solution: The "Two-Brain" System

The authors propose a system with two brains working together simultaneously:

Brain A (The Learner): This brain is constantly guessing the "rules of the game." It looks at the plane's current speed and where the control stick is, then guesses: "If I push the stick this way, the plane will turn that much." It uses a technique called Recursive Least Squares, which is like a super-fast calculator that updates its guess every single second based on new data.
Brain B (The Pilot): This brain uses the current guess from Brain A to decide how to move the controls. It tries to minimize a "cost" (like fuel usage or turbulence). It uses Gradient Descent, which is like walking down a hill: if you feel the ground sloping down, you take a step in that direction to find the bottom (the best control strategy).

The Magic: These two brains talk to each other instantly. As the Pilot flies, the Learner gets better at guessing the plane's physics. As the Learner gets better, the Pilot gets better at flying. They happen at the same time, not one after the other.

3. The Secret Sauce: The "Wiggly Signal"

To learn the rules of the plane, the system needs to see how the plane reacts to everything. If you just hold the stick steady, you learn nothing new.

The Dither: The system adds a tiny, invisible "wiggle" or vibration to the controls (like a gentle hum). This is the dithering signal.
Why? It's like shaking a jar of mixed nuts to see how they settle. The wiggle forces the plane to react in different ways, giving the "Learner" brain enough data to figure out the true physics of the plane. The wiggle is so small that the passenger (the plane) barely notices it, but it's huge for the computer's learning.

4. The Safety Net: "Stability Certificates"

This is the most important part of the paper.

The Fear: Usually, when you combine learning and controlling, you risk a "feedback loop" where the learning gets confused, the pilot panics, and the system explodes.
The Guarantee: The authors used advanced math (specifically Lyapunov stability and Averaging Theory) to prove that their system is like a self-correcting gyroscope.
- Even though the system is constantly changing and learning, the math proves that the "errors" (the difference between where the plane is and where it should be) will shrink exponentially fast.
- They proved that the system will eventually find the perfect way to fly, and until then, it will never crash. It's like having a safety harness that tightens automatically if you start to fall.

5. The Real-World Test: The Drifting Plane

To prove it works, they simulated a highly maneuverable aircraft.

Scenario 1: They flew the plane with unknown settings. The system learned the settings and flew perfectly.
Scenario 2 (The Drift): They made the plane's physics change slowly over time (like the engines wearing out or the fuel tank emptying).
The Result: The system didn't crash. It noticed the plane was changing, updated its "Learner" brain, and adjusted the "Pilot" brain on the fly. It tracked the new optimal flight path seamlessly, without needing to stop and restart.

Summary

Think of this paper as inventing a self-driving car that learns the road rules while driving, without ever needing a driver's license or a map, and with a guarantee that it will never swerve into a tree.

It combines:

Learning (guessing the rules),
Acting (driving the car),
Safety (mathematical proof that you won't crash),

...all happening at the exact same moment. This is a huge step forward for robotics and automation, allowing machines to adapt to the real world safely and instantly.

1. Problem Statement

The paper addresses the Linear Quadratic Regulator (LQR) problem for linear systems with unknown dynamics ( $A^\star, B^\star$ ). The goal is to design a control policy $u_t = K_t x_t$ that minimizes the infinite-horizon cost function:
$J = \mathbb{E} \left[ \sum_{t=0}^{\infty} (x_t^\top Q x_t + u_t^\top R u_t) \right]$
subject to $x_{t+1} = A^\star x_t + B^\star u_t$ .

Key Challenges:

Unknown System: The matrices $A^\star$ and $B^\star$ are not known a priori.
On-Policy Constraint: The control input must be applied to the actual physical system while the policy is being learned and optimized. This contrasts with "off-policy" methods where data is collected under one policy and used to optimize another, often requiring separate data collection and optimization phases.
Stability Guarantees: Most existing data-driven methods focus on convergence or regret bounds but lack rigorous stability certificates for the entire closed-loop system (combining the learning dynamics, the optimization dynamics, and the physical plant) during the transient learning phase.

2. Methodology: Relearn LQR

The authors propose Relearn LQR (REcursive LEARNing policy gradient for LQR), a concurrent learning and optimization framework. The method operates in a single loop where data collection, system identification, and policy update happen simultaneously.

Core Components:

On-Policy Data Collection:
- The control input is defined as $u_t = K_t x_t + d_t$ , where $K_t$ is the current feedback gain and $d_t$ is an exogenous dithering signal.
- The dithering signal is generated by a stable oscillator ( $w_{t+1} = F w_t, d_t = E w_t$ ) to ensure Persistency of Excitation (PE), which is crucial for accurate parameter estimation.
Recursive Least Squares (RLS) for Identification:
- The system parameters $\theta^\star = [A^\star, B^\star]^\top$ are estimated online using a recursive least squares approach.
- The algorithm maintains estimates of the system matrices ( $\theta_t$ ) and auxiliary matrices ( $H_t, S_t$ ) updated at every time step based on the observed state-input pairs $(x_t, u_t)$ .
Direct Policy Search (Gradient Descent):
- The control gain $K_t$ is updated using a gradient descent method on the cost function $J(K, \theta)$ .
- Since the true $\theta^\star$ is unknown, the gradient is computed using the current estimate $\theta_t$ .
- The update rule is: $K_{t+1} = K_t - \gamma G(K_t, \theta_t)$ , where $G$ is the gradient of the cost with respect to $K$ .

System Architecture:

The closed-loop system is modeled as a nonlinear dynamical system with two time scales:

Fast Subsystem: The physical plant dynamics and the RLS estimation error dynamics (driven by the dithering signal).
Slow Subsystem: The evolution of the policy gain $K_t$ and the parameter estimates $\theta_t$ (driven by the step size $\gamma$ ).

3. Key Contributions

Stability-Certified On-Policy Framework:
Unlike many existing data-driven LQR methods that are off-policy or lack closed-loop stability proofs, Relearn LQR provides a formal stability certificate for the entire interconnected system (learning + control + optimization) operating on the real plant.
Theoretical Analysis via Averaging Theory:
The authors model the system as a two-time-scale nonlinear system. They utilize Averaging Theory and Lyapunov analysis to prove that:
- The origin of the error dynamics is exponentially stable.
- The estimates $\theta_t$ converge to the true parameters $\theta^\star$ .
- The gain $K_t$ converges to the optimal gain $K^\star$ .
- The system state $x_t$ remains bounded and oscillates around the origin with an amplitude determined by the dithering signal.
Concurrent Learning and Control:
The method eliminates the need for distinct phases of "data collection" and "policy optimization." The system learns and optimizes simultaneously, allowing for dynamic adaptation to time-varying parameters without restarting the algorithm.
Handling Time-Varying Systems:
The theoretical framework suggests that the method can adapt to scenarios where system matrices change over time (drifting parameters), as the closed-loop system dynamically restores optimality without requiring a restart.

4. Results

Theoretical Results:

Theorem 3.3: Proves that under specific assumptions (controllability, persistency of excitation, and sufficiently small step size $\gamma$ ), the set containing the optimal gain, true parameters, and the steady-state locus of the fast dynamics is exponentially stable.
Convergence: The algorithm guarantees exponential convergence of the cost error and parameter estimation error to zero (or a tunable neighborhood defined by the dither magnitude).

Numerical Simulations:

The authors tested Relearn LQR on a highly maneuverable aircraft model (longitudinal dynamics).

Static Parameters: The algorithm successfully converged to the optimal cost $J^\star$ and true parameters $\theta^\star$ from random initial conditions. The state trajectories oscillated around the origin due to the dithering signal but remained stable.
Drifting Parameters: The system was subjected to a slow, sigmoidal change in system matrices ( $A^\star, B^\star$ ). The algorithm demonstrated robust adaptability, tracking the changing optimal cost and parameters without losing stability or requiring re-initialization.

5. Significance and Impact

Bridging the Gap: This work bridges the gap between theoretical control stability and modern data-driven learning. It moves beyond "regret bounds" (common in RL literature) to provide Lyapunov-based stability guarantees for the closed-loop system.
Safety in Learning: By certifying the stability of the learning process itself, the method makes data-driven control safer for real-world applications (like aerospace) where instability during the learning phase could be catastrophic.
Online Adaptation: The ability to handle time-varying systems makes this approach highly relevant for real-world scenarios where system dynamics are not perfectly stationary (e.g., aging components, changing environmental conditions).
Methodological Rigor: The use of averaging theory to decouple the fast estimation dynamics from the slow policy optimization provides a rigorous mathematical toolset for analyzing complex, interconnected learning-control systems.

In summary, Relearn LQR offers a robust, theoretically grounded solution for controlling unknown linear systems, ensuring that the learning process itself does not destabilize the physical plant, while simultaneously converging to the optimal control policy.