Data-Driven Covariance Steering with Output Feedback

1. Problem Statement

The paper addresses the Covariance Steering (CS) problem for stochastic, discrete-time, linear, time-invariant (LTI) systems where the system model (matrices $A, B, C$) is unknown.

• Objective: Steer the system's output distribution from an initial Gaussian distribution $\mathcal{N}(\mu_i^y, \Sigma_i^y)$ to a terminal distribution $\mathcal{N}(\mu_f^y, \Sigma_f^y)$ over a finite horizon $N$, while minimizing a quadratic cost function.
• Constraints:
  • Output Feedback: The controller only has access to past inputs and outputs, not the full state vector.
  • No Model Knowledge: The system matrices are unknown; control must be derived solely from a single trajectory of input-output data ($D = \{u_k, y_k\}$).
  • Stochastic Disturbances: The system is subject to process noise ($w_k$) and measurement noise ($q_k$), which are i.i.d. Gaussian.
• Key Challenge: Existing data-driven CS methods assume full state availability or noise-free measurements. This work tackles the more difficult case of partial observability (output feedback) in the presence of noise, where the induced noise in the transformed state space becomes temporally correlated.

2. Methodology

The proposed framework converts the output-feedback problem into a state-feedback problem using a non-minimal state representation, then solves the mean and covariance steering problems using indirect and direct data-driven approaches, respectively.

A. Non-Minimal State Representation

To handle the lack of state measurements, the authors construct a state vector $z_k$ composed of past inputs and outputs:
$$z_k \triangleq \text{col}(u_{k-1}, \dots, u_{k-\ell}, y_{k-1}, \dots, y_{k-\ell})$$
where $\ell$ is the observability index.

• Transformation: This representation allows the system to be written in a state-space form. However, the resulting "state" noise is temporally correlated (not white).
• Dimensionality Reduction: Since the non-minimal state $z_k$ may contain linearly dependent components (violating controllability conditions for data-driven control), a transformation matrix $L$ is applied to create a reduced state $\chi_k = L z_k$. This ensures the data matrix has full row rank, satisfying Willems' Fundamental Lemma.

B. Handling Temporally Correlated Noise

A critical contribution is addressing the fact that the transformed noise $\zeta_k$ is correlated over time ($E[\zeta_k \zeta_m^T] \neq 0$ for $k \neq m$).

• The noise is modeled as an autoregressive process: $\zeta_{k+1} = \Psi \zeta_k + \eta_k$.
• The covariance propagation equations are modified to explicitly track the cross-covariance between the state $\chi_k$ and the noise $\zeta_k$ ($\Sigma_{\chi\zeta}$), which is non-zero due to the correlation.

C. Data-Driven System Identification

Two distinct approaches are used for the mean and covariance controllers:

1. Indirect Approach (Mean Steering):

   • Estimates the system matrices ($\hat{A}, \hat{B}, \hat{C}$) from data.
   • Uses Instrumental Variables (IV) or Total Least Squares (TLS).
   • Why IV? Because the regressors (past states) are correlated with the noise in the output-feedback formulation, standard Ordinary Least Squares (OLS) and even TLS can be inconsistent. The IV method uses past control inputs as instruments (which are independent of the noise) to ensure consistent estimation of the mean dynamics.

2. Direct Approach (Covariance Steering):

   • Does not explicitly identify $A$ and $B$. Instead, it parameterizes the control law directly using the collected data matrix $P$ (constructed from inputs and reduced states).
   • Uses a convex relaxation to convert the non-convex optimization problem into a Semidefinite Program (SDP).
   • Requires estimating the noise realization and its covariance from the residuals of the estimated model.

D. Optimization Formulation

The problem is formulated as an SDP to minimize the cost:
$$J = \sum_{k=0}^{N-1} \text{tr}(Q\Sigma_k) + \text{tr}(R(K_k \Sigma_k K_k^T + \Sigma_k^\nu))$$
Subject to:

• Propagation of mean and covariance (including cross-covariance terms).
• Terminal constraints on output mean and covariance.
• The relaxation introduces a slack variable $Y_k \succeq U_k \Sigma_k^{-1} U_k^T$ to handle the non-convexity of the feedback gain terms.

3. Key Contributions

1. Output-Feedback Extension: Extends data-driven covariance steering from full state feedback to output feedback, a significant step toward practical application where full state measurement is often impossible.
2. Handling Correlated Noise: Addresses the theoretical gap in handling temporally correlated noise induced by non-minimal state representations. The paper derives specific propagation equations for the cross-covariance between state and noise.
3. Robust Estimation Strategy: Proposes using Instrumental Variables (IV) for model identification in the presence of correlated noise, demonstrating via simulation that IV outperforms TLS and OLS for small-to-medium sample sizes in this specific context.
4. Direct Covariance Steering: Formulates a direct, model-free covariance steering problem that relies on data matrices rather than identified system parameters, solved efficiently via SDP.

4. Numerical Results

The authors validated the framework using numerical simulations on a MIMO system ($n=3, p=2$).

• Model Estimation:
  • Compared OLS, TLS, and IV methods.
  • Result: For small sample sizes, IV showed poor performance due to variance. However, as sample size increased, IV significantly outperformed TLS and OLS in terms of mean error and error variance, confirming its consistency in the presence of correlated noise.
• Density Steering:
  • Simulated steering a system from an initial distribution to a target Lissajous trajectory with a specific terminal covariance.
  • Result: Both mean and covariance steering performance improved with larger dataset sizes.
  • The terminal covariance ellipses successfully converged to the target shape, and the mean trajectories tracked the reference path accurately.
  • The framework successfully handled the temporal correlation of noise without requiring a known system model.

5. Significance and Future Work

• Significance: This work bridges the gap between theoretical data-driven control and practical engineering constraints (partial observability and noisy data). It provides a rigorous mathematical framework for steering probability distributions without a known model, which is crucial for safety-critical autonomous systems operating in uncertain environments.
• Limitations & Future Work:
  • The convex relaxation used for the SDP is not "lossless" when estimation errors are present; the terminal constraints are not guaranteed to be met exactly if the model estimates are imperfect.
  • Future work should focus on developing robust formulations that explicitly account for estimation uncertainty to guarantee constraint satisfaction.

In summary, this paper presents a novel, data-driven framework for controlling the statistical distribution of stochastic systems using only input-output data, effectively handling the complexities of partial observability and temporally correlated noise through advanced estimation and convex optimization techniques.