🔢 mathematics

Global Convergence of Policy Gradient Methods for ReLU Controllers in Linear Quadratic Regulation

This paper proves that model-based policy gradient methods can globally converge to the optimal solution for a scalar LQR problem when using an overparameterized, one-hidden-layer ReLU network, by analyzing the convergence of the two effective gains induced by the network's piecewise-linear structure.

Original authors: Jhojan A. Rodriguez-Gil, César A. Uribe

Published 2026-04-27

📖 3 min read🧠 Deep dive

CC BY 4.0

Original authors: Jhojan A. Rodriguez-Gil, César A. Uribe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to balance a long pole on its finger. In the world of math and engineering, this is a version of the "LQR problem" (Linear Quadratic Regulator)—a classic puzzle about finding the perfect, smoothest way to correct a mistake so you don't drop the pole.

Traditionally, we solve this using a very simple "rulebook": If the pole tilts 5 degrees left, move your hand 2 inches left. This is a linear controller. It’s predictable, easy to calculate, and mathematically perfect.

But this paper asks a much harder, more modern question: What if, instead of giving the robot a simple rulebook, we give it a "brain" (a Neural Network)?

The Problem: The "Messy Brain" Paradox

Neural networks (like the ones in ChatGPT) are incredibly powerful, but they are "messy." They don't think in simple rules; they think in millions of tiny, interconnected signals.

When you use a neural network to control a robot, two big problems arise:

The "Identity Crisis": A neural network might find a way to balance the pole that is incredibly complicated and "wiggly," even though a simple rule would work better.
The "Training Trap": Because the brain is so complex, the robot might get stuck in a "local minimum"—like a hiker getting stuck in a small dip on a mountain, thinking they’ve reached the bottom, even though the real valley is much further down.

The Solution: The "Crowd of Experts" Analogy

The researchers studied a specific type of brain called a ReLU network. To understand how they proved it works, imagine you are trying to steer a massive ship using a crowd of 1,000 tiny sailors (the "neurons").

Each sailor has a tiny paddle. Some sailors only paddle when the ship tilts left; others only paddle when it tilts right.

The researchers proved that if you have enough sailors (this is called "overparameterization") and you start them off with a random but balanced distribution (the "initialization"), something magical happens:

The Stability Safety Net: They proved that even though the sailors are acting individually, the ship won't flip over during training. The "brain" stays stable while it's still learning.
The Great Convergence: They showed that as the sailors learn from their mistakes (using "Policy Gradient"), they don't just settle for a "good enough" wiggly movement. Instead, the crowd of sailors eventually organizes themselves so perfectly that they mimic the simple, perfect rulebook. The "messy brain" actually turns itself into the "perfect rule."

Why does this matter?

For a long time, engineers have been hesitant to use deep learning for critical things like self-driving cars or industrial robots because neural networks are "black boxes"—we don't know if they will suddenly do something crazy.

This paper provides a mathematical guarantee. It says: "If you make the neural network wide enough and start it correctly, we can prove that it will eventually find the mathematically perfect way to control the system, and it won't crash while it's learning."

In short: It proves that a complex, artificial brain can learn to be as precise and reliable as a master mathematician.

Technical Summary: Global Convergence of Policy Gradient Methods for ReLU Controllers in LQR

This paper investigates the convergence properties of model-based policy gradient (PG) methods when the controller is parameterized by an overparameterized, one-hidden-layer ReLU neural network in the context of the deterministic, scalar, discounted Linear Quadratic Regulator (LQR) problem.

1. Problem Formulation

The authors consider a scalar discrete-time system:
$x_{t+1} = ax_t + bu_t$
The goal is to minimize the infinite-horizon discounted cost:
$J_\mu = \mathbb{E}_{x_0 \sim \rho} \left[ \sum_{t=0}^{\infty} \gamma^t (qx_t^2 + r\mu(x_t)^2) \right]$
While the optimal LQR controller $\mu^\star(x) = K^\star x$ is linear, the authors parameterize the policy using a bias-free ReLU network:
$\mu_\theta(x) = \sum_{i=1}^m v_i \sigma(w_i x)$
This parameterization is non-convex and redundant. In the scalar case, this results in a piecewise-linear controller that acts with one effective gain ( $K_{1,\theta}$ ) on the positive half-line ( $x \geq 0$ ) and another ( $K_{2,\theta}$ ) on the negative half-line ( $x < 0$ ).

2. Methodology

The research bridges control theory and neural network optimization through a multi-level analysis:

Controller-Level Analysis: The authors characterize the cost landscape in terms of the two effective gains $(K_{1,\theta}, K_{2,\theta})$ . They prove that within a "stability-margin set" (where the closed-loop system remains stable), the cost function satisfies the Polyak-Łojasiewicz (PL) inequality. This implies that if the gains were optimized directly, gradient descent would converge geometrically to the optimal linear gain ( $K_{1,\theta} = K_{2,\theta} = K^\star$ ).
Weight-Level Analysis (Neural Landscape): The core challenge is that PG updates the weights $\theta = \{w_i, v_i\}$ $θ = {w_{i}, v_{i}}$ , not the gains directly. The authors use Neural Tangent Kernel (NTK)-style scaling (Gaussian initialization with width $m$ $m$ ) to show that:
1. Stability Preservation: With high probability, the initial weights produce a stabilizing controller.
2. Sign Stability: They identify a "backbone" of neurons that, due to the overparameterization and small step size, are unlikely to cross the origin ( $w=0$ ). This prevents the "crossing mass" (neurons switching from one gain to the other) from destabilizing the optimization.
Benign Width Regime: The authors define a regime where the network width $m$ is sufficiently large and the initialization variance $\beta$ is appropriately scaled, ensuring that the optimization trajectory stays within the stable, PL-satisfying region of the controller space.

3. Key Contributions

Characterization of ReLU LQR Dynamics: They derive the exact relationship between the piecewise-linear gains and the resulting quadratic cost-to-go.
Global Convergence Proof: They provide a rigorous proof that, under specific initialization and scaling, model-based policy gradient converges to the unique optimal LQR controller with high probability.
Stability Guarantee: Unlike general RL theory, this work proves that the iterates $\theta_k$ remain within the stable set $\mathcal{K}_\delta$ throughout the entire training process.
Complexity Analysis: They show that the convergence rate is geometric (linear in log-space) regarding the performance gap.

4. Results

Theoretical Result: Theorem 1 establishes that for a sufficiently wide network and small step size, the cost $J_\rho(\mu_{\theta_k})$ decreases geometrically toward the optimal cost $J_\rho(\mu^\star)$ , and the induced gains converge to $K^\star$ .
Numerical Validation: Simulations confirm that as the number of neurons $m$ increases, the controller effectively recovers the optimal linear gain. They also demonstrate that "bad" initializations (e.g., $m=2$ ) can lead to suboptimal piecewise-linear policies, highlighting the necessity of overparameterization.

5. Significance

This paper is significant because it addresses a fundamental gap in reinforcement learning theory: the intersection of neural network non-convexity and control stability. While previous works have studied linear policy gradients or general MDPs, this work specifically tackles the stability-critical nature of LQR using non-linear (ReLU) architectures. It provides a theoretical justification for why overparameterized neural networks are effective for control tasks, showing that redundancy and width can actually simplify the optimization landscape by providing a "stable path" to the optimal linear solution.