⚡ electrical engineering

3DIOC: Direct Data-Driven Inverse Optimal Control for LTI Systems

This paper proposes a direct data-driven inverse optimal control framework for linear time-invariant systems under linear quadratic control that learns objective functions directly from input-output trajectories using the Fundamental Lemma, offering both a model-free necessary condition for noise-free scenarios and a robust bi-level optimization formulation for noisy data.

Original authors: Chendi Qu, Jianping He, Xiaoming Duan

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chendi Qu, Jianping He, Xiaoming Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a master chef cook a perfect dish. You can see the ingredients they grab, the way they chop, and the final plate they serve. But you don't know the secret recipe: how much salt, how much heat, or exactly how they decided to mix the spices. Inverse Optimal Control is the art of figuring out that secret recipe just by watching the chef cook.

For a long time, scientists trying to reverse-engineer these "recipes" for machines (specifically, Linear Time-Invariant or LTI systems) had a major hurdle: they needed to know the machine's internal blueprint first. They had to build a model of how the machine works before they could guess what the machine was trying to achieve. It was like trying to guess a chef's recipe by first measuring the exact chemical composition of every pot and pan in the kitchen.

This paper, titled 3DIOC, introduces a new, "direct" way to solve this puzzle. The authors, Chendi Qu, Jianping He, and Xiaoming Duan, propose a method that skips the blueprint entirely. They don't need to know the machine's internal gears or equations. Instead, they look directly at the "input-output" trails—the data of what went in and what came out—to figure out the hidden goal.

The Magic Trick: The Fundamental Lemma

The secret sauce here is something called the Fundamental Lemma from behavioral system theory. Think of it like this: If you have a long enough video of a machine moving around, that video contains all the possible ways the machine could move. You don't need to know the physics of the machine; the video itself is the map.

The authors use this idea to create a "model-free" rule. They derived a mathematical condition (called a KKT condition) that says: "If the machine is acting optimally, the data it leaves behind must fit this specific pattern." By checking if the data fits this pattern, they can work backward to find the hidden weights (the "salt and pepper" of the machine's objective function) that made it act that way.

Two Ways to Solve the Puzzle

The paper doesn't just offer one tool; it offers two, depending on how messy the data is.

1. The "Perfect World" Solver (KKT-based 3DIOC)
If the data is clean—like a video recorded in a studio with no static or glitches—the authors use a method based on the KKT condition. This is like solving a jigsaw puzzle where every piece fits perfectly.

How it works: They set up a math problem that asks, "What weights make the data fit this perfect pattern?"
The Catch: There's a tiny trick. The math can't tell the difference between a recipe with "1 cup of sugar" and one with "2 cups of sugar" if the machine just scales everything up by two. So, the solution isn't one single number, but a whole family of solutions that are just scaled versions of each other. The paper proves that if you have enough data (specifically, if the "horizon" or length of the observation is long enough), this family of solutions is unique.
The Result: In simulations, this method was incredibly fast and accurate, needing only a tiny amount of data (one offline trajectory of length 50 and one optimal trajectory) to find the answer. It beat other methods that tried to build a model first, which were slower and less accurate.

2. The "Messy World" Solver (Bi-level Optimization)
Real life is rarely a studio. Data often has noise—glitches, static, or random errors. When the data is noisy, the "perfect world" solver gets confused and might fail.

The New Approach: The authors switch to a Bi-level Optimization strategy. Imagine a game of "Hot and Cold."
- The Inner Loop: You guess a recipe (the weights).
- The Outer Loop: You see how far off the machine's actual behavior is from the expert's behavior you are trying to copy.
- The Goal: You keep adjusting your guess to get "hotter" (closer to the expert).
Why it's better: This method is designed to handle noise. The paper proves mathematically that as you get more and more data, this method will eventually find the best possible guess, even if the data is noisy. It's like a detective who keeps refining their theory as they gather more clues, even if some clues are misleading.

What This Paper Says "No" To

The authors are very clear about what their method is not.

No System Identification: They explicitly argue against the old way of "identifying the system first." They show that trying to build a model of the machine before guessing the goal introduces errors and wastes data. Their method is "direct," meaning it goes straight from data to the goal.
No Magic with Too Little Data: They warn that if you don't watch the machine long enough (if the "horizon" $N$ is too short), the problem is impossible to solve. There is a specific mathematical threshold (related to the size of the machine's inputs and outputs) that the data must exceed, or the secret recipe remains hidden.
No State Observation Needed: Unlike many other methods that require you to see the machine's internal state (like the exact position of every gear), this method only needs to see the inputs and outputs. It works even if you can't see inside the machine.

How Sure Are We?

The authors are confident, but they are also careful.

Proven: They have mathematically proved that their method works in a noise-free world and that it has a unique solution if enough data is collected. They also proved that their "messy world" method will converge to the right answer as data increases.
Simulated: The performance numbers come from computer simulations. They tested their method on randomly generated machines with 3 states and 2 inputs. In these tests, their method was faster and more accurate than the "System Identification" and "Maximum Entropy" baselines.
Robustness: They showed through simulations that their method handles different types of noise (like random spikes or uniform errors) well, though the error does go up if the noise is very loud.

The Bottom Line

This paper presents a clever, direct way to reverse-engineer the goals of a machine just by watching it move. It skips the boring step of building a model and goes straight to the point. If the data is clean, it solves the puzzle instantly. If the data is noisy, it uses a smart, iterative guessing game to find the best answer. While these results are currently based on simulations, the math behind them is solid, offering a promising new path for robots and autonomous systems to learn from demonstrations without needing a manual for how they work.

Technical Summary: 3DIOC – Direct Data-Driven Inverse Optimal Control for LTI Systems

Problem Formulation
This paper addresses the Direct Data-Driven Inverse Optimal Control (3DIOC) problem for Linear Time-Invariant (LTI) systems under Linear Quadratic (LQ) control. The objective is to identify the underlying weighting matrices ( $Q$ and $R$ ) of the system's cost function directly from measured input-output trajectories, without prior knowledge of the system dynamics (matrices $A, B, C, D$ ). The authors distinguish their approach from traditional methods that rely on System Identification (SI) as a pre-processing step, noting that identification errors can propagate and degrade the learning of the objective function. The problem is formulated using a collected offline stochastic trajectory ( $w_d$ ) and an optimal input-output trajectory ( $w_o$ ) generated by an unknown forward LQ controller.

Methodology
The proposed methodology leverages the Fundamental Lemma from behavioral system theory, which allows an LTI system to be represented entirely by its input-output trajectories under persistent excitation conditions. The paper develops two distinct algorithmic frameworks:

KKT-based 3DIOC (Single-Level):
- Derivation: The authors derive a model-free Karush–Kuhn–Tucker (KKT) condition (Optimality Necessary Condition or ONC) for the forward LQ problem using the input-output representation. This establishes a linear relationship between the unknown weighting matrices and the observed optimal trajectories.
- Simplification: To improve efficiency, redundant variables (specifically the Lagrangian multipliers and initial state dependencies) are eliminated. This results in a simplified formulation (Problem 3) that requires solving a convex optimization problem (specifically a Quadratic Program or Semi-Definite Program) to find the scaling-invariant set of $Q$ and $R$ .
- Noise Handling: In the presence of observation noise, the strict equality constraint becomes infeasible. The authors reformulate the problem as a convex optimization (Problem 4) minimizing the residual error, subject to positive definiteness and trace constraints. A sensitivity analysis is provided, linking estimation error to the spectral gap of the data matrix.
Bi-Level Optimization (BLO-IOC):
- Formulation: The IOC problem is cast as a bi-level optimization problem (Problem 5). The lower level solves the forward LQ control problem for a given set of weights, while the upper level minimizes the discrepancy (loss function) between the expert's trajectory and the trajectory generated by the learned weights.
- Algorithm: An iterative gradient descent algorithm (Algorithm 2) is proposed to solve this bi-level problem. The authors prove that under specific conditions (summable tolerance sequences), the algorithm converges to a stationary point.
- Robustness: For noisy data, the BLO formulation is shown to possess asymptotic risk consistency, meaning the estimator converges to the best possible prediction as the amount of data increases. A numerical enumeration method (Algorithm 3) is also introduced to approximate the global optimum by discretizing the parameter space.

Key Contributions

Model-Free Framework: The paper proposes a novel direct data-driven approach that bypasses system identification, utilizing only input-output trajectories to derive model-free KKT conditions for LQ control.
Identifiability Analysis: The authors provide rigorous identifiability conditions (Theorem 1 and Proposition 3) ensuring the uniqueness of the solution up to a scaling factor. They also establish a condition for "intention protection," showing how horizon length affects the ability to uniquely identify the cost function.
Dual Formulations: The paper establishes the relationship between the KKT-based and BLO-based formulations. It demonstrates that in noise-free scenarios, the KKT solution is a subset of the BLO solution set. In noisy scenarios, the BLO approach offers convergence and risk consistency guarantees, whereas the KKT approach relies on sensitivity analysis.
Efficiency: The simplified KKT-based algorithm reduces computational complexity by removing redundant parameters, requiring only one offline trajectory and one optimal trajectory.

Results
Simulation experiments were conducted on randomly generated LTI systems with dimensions $n=3, m=2, p=3$ .

Accuracy: The proposed KKT-based 3DIOC achieved lower estimation errors compared to baselines involving System Identification (Sys-ID IOC) and Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL).
Data Efficiency: The proposed method required significantly less data (e.g., $T=50$ ) compared to baselines (e.g., $T=1000$ ) to achieve comparable accuracy.
Robustness: Under noisy conditions, the BLO-IOC algorithm demonstrated stable convergence and maintained low estimation errors across various noise distributions (Gaussian, Laplace, uniform, and sparse outliers).
Computation: The KKT-based approach solved via SDP/QP was shown to be computationally efficient (sub-second execution), while the BLO approach, though iterative, provided robustness in noisy environments.

Significance and Claims
The authors claim that 3DIOC offers a practical solution for learning objective functions in scenarios where system dynamics are unknown or difficult to model accurately. By avoiding the "identify-then-optimize" pipeline, the method mitigates compounding errors inherent in traditional two-stage approaches. The paper highlights a trade-off: the KKT-based method is preferred for noise-free, identifiable settings due to its computational efficiency and direct solution guarantees, while the BLO formulation is recommended for noisy, real-world imitation learning tasks due to its convergence properties and risk consistency. The authors note that while the current work focuses on LTI systems, the framework could potentially be extended to nonlinear systems via Koopman operator theory in future work.

The Magic Trick: The Fundamental Lemma

Two Ways to Solve the Puzzle

What This Paper Says "No" To

How Sure Are We?

The Bottom Line

More like this