On the Global Optimality of Linear Policies for Sinkhorn Distributionally Robust Linear Quadratic Control

1. Problem Statement

The paper addresses the limitations of the classical Linear Quadratic Gaussian (LQG) regulator. While LQG provides an elegant closed-form solution for controlling linear systems with Gaussian noise, it is highly fragile when the true noise distribution deviates from the assumed Gaussian model (model misspecification).

To mitigate this, the authors propose a Distributionally Robust (DR) formulation of the finite-horizon LQG problem.

• System Dynamics: Discrete-time linear systems with additive process noise ($w_t$) and measurement noise ($v_t$).
• Uncertainty Model: The true noise distributions are unknown but are assumed to belong to an ambiguity set $\mathcal{S}$.
• Ambiguity Set Definition: The set $\mathcal{S}$ is defined using the Sinkhorn discrepancy (an entropy-regularized Wasserstein distance) centered at a nominal Gaussian distribution.
• Objective: Find a control policy $\pi$ that minimizes the expected quadratic cost under the worst-case distribution within the ambiguity set:
  $$\inf_{\pi} \sup_{P \in \mathcal{S}} \mathbb{E}_P[J(\pi, \delta)]$$
• Core Question: Does the optimal solution to this robust problem remain a linear policy (specifically, a linear feedback controller), or does the complexity of the distributional uncertainty necessitate non-linear strategies?

2. Methodology

The authors employ a "sandwich" argument to prove that linear policies are globally optimal. They construct a lower bound and an upper bound for the optimal value of the DR problem and show that these bounds coincide.

A. Purified Output Re-parametrization

To decouple the control input from the noise, the authors introduce purified outputs ($\eta_t = y_t - \hat{y}_t$), where $\hat{y}_t$ is the output of a noise-free copy of the system.

• This transformation allows the control problem to be reformulated in terms of $\eta$, which is independent of the control input $u$.
• The problem is rewritten as a zero-sum game between the controller (minimizing cost) and an adversary (choosing the worst noise distribution).

B. Lower Bound Construction (Dual Problem)

The authors restrict the adversary's choice to Gaussian distributions within the ambiguity set ($\mathcal{S}_N \subseteq \mathcal{S}$).

• Key Insight: For Gaussian distributions, the Sinkhorn discrepancy coincides with the entropy-regularized Gelbrich divergence ($G_\epsilon$), which depends only on the covariance matrices.
• This reduces the infinite-dimensional optimization over probability measures to a finite-dimensional optimization over covariance matrices subject to Gelbrich constraints.
• Since the inner problem (for fixed Gaussian noise) is a standard LQG problem, the optimal policy is known to be linear.

C. Upper Bound Construction (Primal Problem)

The authors restrict the controller to linear policies and enlarge the ambiguity set to the entropy-regularized Gelbrich set (which contains all distributions with covariances satisfying the Gelbrich bound).

• Key Theorem (Proposition 3): They prove a Gelbrich-type inequality for the Sinkhorn discrepancy: $W_\epsilon(P, Q) \geq G_\epsilon(\Sigma_P, \Sigma_Q)$ when $P$ is Gaussian.
• This inequality implies that the original Sinkhorn ambiguity set is contained within the Gelbrich ambiguity set defined by covariances.
• Consequently, the optimal cost of the restricted linear policy problem over the enlarged Gelbrich set serves as a valid upper bound for the original problem.

D. Existence of Nash Equilibrium

The authors prove that the lower bound (dual) and upper bound (primal) problems are equivalent:

1. Convexity and Compactness: They establish that the set of covariance matrices satisfying the Gelbrich constraint is convex and compact (Lemma 1).
2. Strong Duality: Using Sion's Minimax Theorem, they prove that the order of optimization (min over policies, max over distributions) can be swapped without a duality gap.
3. Result: The optimal value of the lower bound equals the upper bound, implying the existence of a Nash equilibrium where the optimal policy is linear and the worst-case distribution is Gaussian.

3. Key Contributions

1. Global Optimality of Linear Policies: The paper establishes that for Sinkhorn DR LQG, linear feedback policies are globally optimal. This is a significant theoretical result because distributionally robust problems often require complex, non-linear policies.
2. Novel Bounds and Inequalities:
   • Derivation of a lower bound for the Sinkhorn DR LQG problem using regularized optimal transport results.
   • Proof of a Gelbrich-type inequality for Sinkhorn discrepancy, showing it is lower-bounded by the entropy-regularized Gelbrich divergence. This inequality is not available in existing literature and is crucial for the upper bound.
3. Structural Properties: Proof that the entropy-regularized Gelbrich ambiguity set is convex and compact, enabling the application of minimax theorems.
4. Geometric Insight: The work highlights how the KL regularization term in Sinkhorn discrepancy fundamentally alters the geometry of the ambiguity set compared to standard Wasserstein DR (e.g., ensuring the worst-case distribution remains continuous/Gaussian rather than finitely supported).

4. Results

• Theoretical: The authors prove that the Sinkhorn DR LQG problem admits a Nash equilibrium $(u^*, P^*)$ where $u^*$ is an affine policy of the purified observations and $P^*$ is a Gaussian distribution.
• Numerical Experiments:
  • A simulation on an open-loop unstable system ($T=25$) compared the classical LQG controller against the proposed Sinkhorn DR LQG policy.
  • Nominal Performance: Under nominal Gaussian noise, the DR policy incurred a slightly higher cost (the "price of robustness").
  • Robust Performance: Under adversarial noise distributions (worst-case within the ambiguity set), the Sinkhorn DR policy significantly outperformed the classical LQG controller, demonstrating superior stability and lower average cost.

5. Significance

• Bridging Stochastic and Adversarial Control: The work successfully combines the computational tractability of stochastic LQG with the robustness of adversarial control. It demonstrates that one does not need to sacrifice the simplicity of linear controllers to achieve distributional robustness.
• Practical Applicability: By reducing the problem to a finite-dimensional conic program (specifically a Semidefinite Program), the proposed method is computationally feasible for implementation.
• Theoretical Advancement: It resolves a critical open question regarding the optimality of linear policies in DR control settings involving entropy-regularized optimal transport, extending previous results from standard Wasserstein metrics to the Sinkhorn framework.

In summary, this paper provides a rigorous theoretical foundation for using linear controllers in uncertain environments where noise distributions are unknown but close to a nominal Gaussian model, measured via Sinkhorn discrepancy.