Policy Optimization of Mixed H2/H-infinity Control: Benign Nonconvexity and Global Optimality

This paper establishes that mixed H2/H-infinity control exhibits benign nonconvexity where every stationary point is globally optimal, by characterizing the feasible set's topology and employing an Extended Convex Lifting framework to bridge nonconvex policy optimization with convex reformulations for scalable, data-driven applications.

The Gist

Imagine you are the captain of a spaceship trying to get from Earth to Mars. You have two main goals:

1. Efficiency (The H2 Goal): You want to use as little fuel as possible to get there smoothly. This is like minimizing your average travel time and energy.
2. Safety (The H∞ Goal): You want to make sure that even if a massive solar storm hits (the worst-case scenario), your ship doesn't crash. You need a safety margin.

Mixed H2/H∞ Control is the mathematical challenge of finding the perfect flight path that balances these two. You want to be efficient, but you can't cut corners on safety.

The Old Way: The "Black Box" Map

For decades, engineers solved this using complex, rigid formulas (called Riccati equations or LMIs). Think of these like a pre-drawn map that only works for small, simple cities.

• The Problem: If you try to use this map for a giant, sprawling metropolis (a large-scale system like a power grid or a fleet of drones), the map becomes useless. It's too heavy, too slow, and it doesn't tell you why a path works, just that it works. It's like being told "turn left at the big red building" without understanding the geography.

The New Way: "Policy Optimization" (The GPS)

This paper introduces a modern approach called Policy Optimization. Instead of following a pre-drawn map, the captain (the algorithm) learns by trial and error, adjusting the steering wheel (the controller) to find the best path.

Usually, learning by trial and error is dangerous because the landscape is full of traps. Imagine a hilly terrain where you want to find the lowest valley (the best solution).

• The Trap: In many problems, you might get stuck in a small, shallow dip (a "local minimum") thinking you've reached the bottom, when in reality, there's a much deeper valley nearby. This is called a "spurious stationary point."

The Big Discovery: "Benign Nonconvexity"

The authors of this paper discovered something magical about the Mixed H2/H∞ problem. They found that the landscape is "Benign."

The Analogy: Imagine a mountain range where every single valley you can find is actually the same deepest valley. There are no fake, shallow dips to trick you.

• The Result: If your GPS (the algorithm) stops moving because it thinks it's at the bottom of a hill, it is guaranteed to be at the absolute best possible destination. You don't need to worry about getting stuck in a bad spot. Every "stationary point" is a "global optimum."

How They Proved It: The "Magic Elevator" (Extended Convex Lifting)

How did they prove this? They used a mathematical trick called Extended Convex Lifting (ECL).

Think of the problem as a tangled ball of yarn (non-convex). It looks impossible to untangle.

• The Trick: The authors built a magic elevator. They lifted the tangled yarn up into a higher dimension where, suddenly, the yarn untangles itself into a perfectly straight line (convex).
• The Insight: Once you solve the problem in this "lifted" world (where everything is simple and straight), you can bring the solution back down to the real world, and it remains the perfect solution. This proves that the messy, tangled problem we started with actually has a hidden, simple structure underneath.

Why This Matters

1. No More Getting Lost: Because the landscape is "benign," we can use simple, fast, gradient-based methods (like a hiker always walking downhill) to find the perfect controller. We don't need complex, slow, old-school maps.
2. Scalability: This method works for huge systems. Whether you are controlling a single drone or a massive network of thousands of robots, this approach scales up efficiently.
3. Data-Driven: Since this method is based on "learning" the landscape rather than needing a perfect mathematical model of the system, it opens the door for AI-driven control. You can learn the best controller just by observing the system, even if you don't know all the physics equations beforehand.

Summary

This paper takes a classic, difficult control problem (balancing speed and safety) and shows that, contrary to what we thought, the path to the solution is surprisingly smooth. There are no hidden traps. By using a clever mathematical "elevator," they proved that any method that finds a stable point has actually found the best possible solution. This paves the way for smarter, faster, and more robust AI controllers in the real world.

Technical Summary

Here is a detailed technical summary of the paper "Policy Optimization of Mixed H2/H∞ Control: Benign Nonconvexity and Global Optimality" by Pai, Watanabe, Tang, and Zheng.

1. Problem Statement

The paper addresses the design of Mixed $H_2/H_\infty$ controllers for continuous-time linear dynamical systems. The objective is to find a static state-feedback policy $u(t) = Kx(t)$ that minimizes an average performance cost ($H_2$ norm) while satisfying a robustness constraint against worst-case disturbances ($H_\infty$ norm).

Mathematically, the problem is formulated as:
$$\inf_{K \in \mathcal{K}_\beta} J_{\text{mix}}(K)$$
where:

• $\mathcal{K}_\beta = \{ K \mid \|T_\infty(K)\|_{H_\infty} < \beta \}$ is the set of stabilizing policies satisfying the $H_\infty$ constraint.
• $J_{\text{mix}}(K)$ is a cost function derived from the solution of a specific Riccati equation, serving as an upper bound on the $H_2$ cost.
• The formulation covers two cases: Two-channel (distinct performance outputs for $H_2$ and $H_\infty$) and Single-channel (identical outputs).

Challenges:

• Nonconvexity: The feasible set $\mathcal{K}_\beta$ is nonconvex and unbounded.
• Landscape Complexity: Classical solutions (Riccati equations or LMIs) provide optimal values but offer little insight into the optimization landscape (e.g., existence of spurious local minima).
• Scalability: Model-based LMI/Riccati methods scale poorly with system dimension, hindering application in large-scale or data-driven settings.

2. Methodology

The authors analyze the problem through the lens of modern policy optimization and nonconvex optimization theory. Their approach relies on three main pillars:

1. Geometric Characterization: They rigorously analyze the topology of the feasible set $\mathcal{K}_\beta$ and the smoothness properties of the cost function $J_{\text{mix}}$.
2. Extended Convex Lifting (ECL): They utilize the ECL framework (a recent theoretical tool) to bridge the gap between nonconvex policy optimization and convex reformulations. This involves constructing a "lifted" convex set and a diffeomorphism that maps the nonconvex policy space to a convex space.
3. Non-Strict Inequalities: Unlike classical suboptimal synthesis which relies on strict Riccati inequalities, this work employs non-strict Riccati inequalities and LMIs. This distinction is crucial for characterizing the boundary of the feasible set and proving global optimality over the entire domain, including limit cases.

3. Key Contributions

A. Landscape Properties

• Feasible Set Geometry: The set $\mathcal{K}_\beta$ is proven to be open, path-connected, and nonconvex. Its boundary is precisely characterized as the set of policies where the $H_\infty$ norm exactly equals $\beta$.
• Cost Function Smoothness: The mixed cost function $J_{\text{mix}}$ is shown to be real analytic (infinitely differentiable) within the interior of the feasible set. The authors provide explicit gradient formulas for both the general two-channel case and the single-channel case.
• Continuity: The cost function is continuous up to the boundary of the feasible set, allowing for the definition of an extended cost function on the closure.

B. Global Optimality (Benign Nonconvexity)

• Absence of Spurious Stationary Points: The central theoretical result is that every stationary point (where $\nabla J_{\text{mix}}(K) = 0$) of the mixed $H_2/H_\infty$ problem is a global minimizer.
• Existence and Uniqueness:
  • For the single-channel case, a unique stationary point always exists.
  • For the two-channel case, a stationary point may not exist if the robustness constraint $\beta$ is too tight. However, the authors prove that if $\beta$ is sufficiently relaxed (above a certain threshold), a stationary point is guaranteed to exist.

C. Extended Convex Lifting (ECL) Construction

• The authors explicitly construct an ECL for the two-channel mixed control problem.
• They define a lifted set involving the policy $K$, the cost $\gamma$, and a Lyapunov/Riccati variable $X$.
• They prove that this lifted set projects onto the non-strict epigraph of the cost function and is diffeomorphic to a convex set defined by Linear Matrix Inequalities (LMIs).
• This construction certifies that the optimization landscape is "benign," meaning local search methods cannot get stuck in suboptimal local minima.

4. Results and Numerical Experiments

The paper validates the theory through numerical experiments comparing four approaches:

1. Analytical Solution: Solving coupled Riccati equations (Single-channel only).
2. Policy Iteration (PI): Iterative updates based on the derived gradient formulas.
3. LMI-based Convex Optimization: Solving the convex reformulation via semidefinite programming (MOSEK).
4. HIFOO: A state-of-the-art nonsmooth optimization package.

Key Findings:

• Global Convergence: The Policy Iteration method successfully converges to the global optimum in cases where the constraint $\beta$ is sufficiently relaxed, confirming the "benign" landscape.
• Scalability: While LMI methods guarantee global optimality, their computational cost scales poorly with system dimension (e.g., runtime increases drastically for 60x60 and 90x90 systems). In contrast, Policy Iteration scales favorably, making it suitable for large-scale problems.
• Robustness of HIFOO: HIFOO, which relies on local nonsmooth optimization, often fails to find feasible solutions or converges to suboptimal points when constraints are tight, highlighting the advantage of the global optimality guarantees provided by the ECL framework.
• Boundary Behavior: The experiments demonstrate that for tight constraints, the optimal solution may lie on the boundary of the feasible set, a scenario where the non-strict ECL formulation remains valid while strict formulations might fail.

5. Significance

• Theoretical Insight: The paper resolves the long-standing question of the optimization landscape for mixed $H_2/H_\infty$ control, proving it shares the "benign nonconvexity" properties found in simpler problems like LQR and LQG.
• Algorithmic Design: By establishing the absence of spurious local minima, the work provides a strong theoretical foundation for using gradient-based policy optimization (e.g., Policy Gradient, Policy Iteration) in robust control. This opens the door for data-driven and model-free mixed control designs.
• Scalability: The results suggest that for large-scale systems, policy iteration methods can achieve global optimality more efficiently than traditional LMI-based convex solvers, bridging the gap between classical robust control and modern reinforcement learning techniques.
• Unified Framework: The use of the ECL framework unifies the analysis of strict and non-strict inequalities, offering a more comprehensive view of solvability and optimality conditions in robust control.