⚡ electrical engineering

Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraints

This paper proposes a learning-based dual-ascent algorithm that reformulates infinite-horizon stochastic optimal control problems with joint chance constraints as unconstrained Markov decision processes via state augmentation, enabling the efficient computation of optimal and feasible deterministic policies for continuous state-input spaces.

Original authors: Francesco Cordiano, Kanghui He, Bart De Schutter

Published 2026-08-04

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Cordiano, Kanghui He, Bart De Schutter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating a dense asteroid field. Your mission is to reach a distant star while using as little fuel as possible. But there's a catch: you can't just avoid the asteroids you see right now; you have to guarantee that your entire journey, from launch to landing, stays safe with a very high probability. This is the heart of a field called stochastic optimal control. It's the science of making the best decisions when the future is fuzzy and full of surprises.

To understand the challenge, think of two ways to handle danger. The first is like checking your rearview mirror every second and saying, "Okay, I'm safe right now." This is called a "stagewise" check. The second, much harder approach, is like looking at your entire flight path on a map and saying, "I promise that every single point on this line will be clear of asteroids." This is a joint chance constraint. It's a "mission-wide" promise. The problem is that making this promise is incredibly difficult for computers because the future path depends on every single bump and turn that happened before, making the math explode in complexity. Usually, to make the math manageable, engineers have to be overly cautious, taking wide, slow detours that waste fuel, or they have to assume the universe stops being dangerous after a certain time.

This paper, written by Francesco Cordiano, Kanghui He, and Bart De Schutter, tackles the problem of how to navigate that infinite, dangerous path without being overly cautious or assuming the danger disappears. They propose a clever new way to teach a computer how to make these perfect, safe, and fuel-efficient decisions for systems that run forever, like a power grid or a self-driving car on a highway.

The Magic Trick: Turning a Memory Problem into a State Problem

The biggest headache with the "mission-wide" safety promise is that it's non-Markovian. In plain English, this means the computer needs to remember everything that has happened since the beginning of time to know if it's still safe. If you've never hit an asteroid, you're safe. If you hit one yesterday, you're already "failed." A standard computer brain (a Markov policy) usually only looks at where you are right now to decide what to do next. It doesn't have a long-term memory.

The authors' first breakthrough is a "magic trick" called state augmentation. They invent a new set of "virtual sensors" to attach to the spaceship.

The "All-Clear" Light (State $\xi$ ): This is a binary switch that stays "ON" (1) as long as the ship has never hit an asteroid. The moment it hits one, the switch flips "OFF" (0) and stays there forever.
The "First Hit" Alarm (State $\psi$ ): This is a special alarm that rings only at the exact moment the ship hits its first asteroid. If it rings, the system knows, "Ah, this is the moment we failed."
The "Time-Dial" (State $\phi$ ): Since the ship is trying to minimize fuel over an infinite future, the importance of future fuel usage changes over time. This dial tracks that changing importance.

By adding these three virtual sensors to the ship's actual position, the computer no longer needs to remember the whole history. It just needs to look at the current state of these sensors. If the "All-Clear" light is ON, it knows it's safe so far. If it's OFF, it knows it's already failed. This turns a complex, memory-heavy problem into a standard, manageable one that the computer can solve step-by-step.

The Balancing Act: The Price of Safety

Now that the problem is manageable, the next challenge is the "infinite horizon" part. The ship needs to stay safe forever, not just for the next 10 minutes. The authors use a mathematical concept called Lagrange duality to solve this.

Imagine you are hiring a robot to drive your car. You tell it, "Drive as fast as possible, but don't crash." The robot doesn't know how to balance speed and safety. So, you introduce a "Price of Safety." You say, "Every time you get close to crashing, you have to pay a fine."

If the fine is too low, the robot drives recklessly and crashes.
If the fine is too high, the robot drives so slowly it never gets anywhere.

The paper proposes an algorithm that acts like a smart negotiator. It starts with a low fine and lets the robot drive. If the robot crashes too often, the algorithm raises the fine. If the robot is driving too slowly and safely, it lowers the fine. The goal is to find the "Goldilocks" fine (called the dual variable, $\lambda$ ) where the robot drives as fast as possible while exactly meeting the safety requirement.

The authors prove that this negotiation works perfectly. They show that there is a specific price where the robot's "best speed" strategy is also the "safest" strategy. This allows them to turn the difficult "safety constraint" problem into a simpler "minimize cost plus fine" problem.

Teaching the Robot with Neural Networks

The final piece of the puzzle is that real-world systems (like robots or power grids) have infinite possibilities for where they can be and what they can do. You can't write down a rule for every single possibility. To handle this, the authors use machine learning.

They train a Neural Network (a type of computer brain inspired by the human brain) to learn the "value" of being in any situation.

First, they teach the network what happens if the safety rule is already broken. In this case, the robot just tries to get to the goal as fast as possible, ignoring safety.
Then, they teach the network the "All-Clear" situation. Here, the network learns to balance speed and the "Price of Safety" fine.

The training happens offline, meaning the computer does all the hard thinking before the robot ever starts moving. Once trained, the robot can make decisions in a split second (0.01 seconds in their tests) by just looking at its current state and the neural network's advice.

The Results: Faster, Safer, and Smarter

The authors tested their method on a simulation of a "unicycle" robot (a robot balancing on one wheel) trying to navigate a maze with a dangerous obstacle in the middle. They compared their method against a popular technique called Model Predictive Control (MPC), which is like a robot that plans its next few steps, checks if they are safe, and then re-plans.

The results were impressive:

Safety: The new method kept the robot safe with a violation rate of about 4.5%, which is well under the allowed 10% limit. The traditional MPC method, even with heavy adjustments, had a violation rate of 17%, failing the safety test.
Performance: The new method used significantly less "fuel" (cost) to reach the goal. The new method had a cost of 528.3, while the MPC method had a cost of 672.0. The new method was smarter about taking risks: if it did hit the obstacle (in the rare cases it did), it immediately switched to the fastest possible path to the goal, whereas the MPC method got stuck in a conservative loop.
Speed: This is the biggest win. The traditional MPC method took an average of 2.94 seconds to decide what to do at each step, and sometimes hit a 10-second limit, causing delays. The new method took only 0.01 seconds. It was nearly 300 times faster.

Why This Matters

This paper doesn't just say "we did it"; it provides a rigorous mathematical proof that their method works and converges to the best possible solution. It shows that you don't have to choose between being safe and being efficient. By using a clever state augmentation and a smart learning algorithm, you can build systems that are both incredibly fast and rigorously safe over an infinite future.

The authors admit that their method relies on simulations and that the learning process needs enough data to be accurate, especially near the edges of danger zones. However, they demonstrate that for complex, continuous systems, this approach is a massive leap forward. It turns a problem that was previously too hard to solve into one that a computer can solve in the blink of an eye, opening the door for safer, more efficient autonomous systems in the real world.

Technical Summary: Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraints

Problem Formulation
The paper addresses the challenge of solving infinite-horizon stochastic optimal control problems subject to joint chance constraints. Unlike individual (stagewise) chance constraints, which require safety at each step independently, joint chance constraints demand that the system state remains within a safe set $X$ for the entire trajectory with a probability of at least $1-\varepsilon$ . This formulation is critical for safety-critical systems but presents two major difficulties:

Non-Markovian Nature: The constraint depends on the entire history of the trajectory, rendering standard Markov Decision Process (MDP) formulations inapplicable.
Computational Intractability: Traditional methods often rely on conservative approximations (e.g., Boole's inequality) or finite-horizon assumptions, which either degrade performance or fail to address the infinite-horizon setting without restrictive assumptions on disturbance support.

Methodology
The authors propose a framework that transforms the original non-Markovian, constrained problem into an equivalent unconstrained Markovian problem solvable via learning-based dynamic programming. The methodology proceeds in three main stages:

State Augmentation and Reformulation:
The authors introduce an augmented state space $S = \mathbb{R}^n \times \{0, 1\}^2 \times [0, 1]$ . The original state $x_t$ is augmented with three auxiliary variables:
- $\xi_t$ : A binary indicator that is 1 if the trajectory has satisfied constraints up to time $t$ , and 0 otherwise.
- $\psi_t$ : A binary indicator that is 1 if the current state $x_t$ is the first state to violate the constraint, and 0 otherwise.
- $\phi_t$ : A scalar variable initialized to 1 and updated as $\phi_{t+1} = \gamma \phi_t$ to account for the discount factor in the cost function, eliminating temporal mismatch between the discounted cost and the non-discounted constraint.
This augmentation allows the joint chance constraint to be reformulated as an additive constraint on the expected sum of $\psi_t$ over the infinite horizon: $\mathbb{E}[\sum \psi_t] \leq \varepsilon$ .
Lagrange Dual Framework and Strong Duality:
The constrained problem is cast into a Lagrangian dual framework. The authors prove that strong duality holds for this formulation under Slater's condition. This allows the constrained problem to be reformulated as an equivalent unconstrained problem where the chance constraint is treated as a penalty term in the objective function. The resulting objective is:
$\inf_{\pi} \mathbb{E} \left[ \sum_{t=0}^{\infty} (\phi_t \ell(x_t, u_t) + \lambda \psi_t) \right] - \lambda \varepsilon$
where $\lambda \geq 0$ is the dual variable.
Dual-Ascent Algorithm and Learning:
To solve the unconstrained problem for continuous state-action spaces, the authors propose a dual-ascent algorithm:
- Primal Step: For a fixed $\lambda$ , the inner minimization is an unconstrained optimal control problem solved via dynamic programming (DP).
- Dual Step: The dual variable $\lambda$ is updated via projected gradient ascent based on the violation of the risk constraint.
To handle continuous spaces, the authors develop a learning-based framework using neural networks (NNs) to approximate the value function. They exploit the structural properties of the augmented state:
- If a violation has occurred ( $\xi=0$ ), the problem reduces to an unconstrained optimization of the cost, approximated by a network $\hat{W}$ .
- If no violation has occurred ( $\xi=1$ ), the value function $\hat{J}$ is learned, incorporating the probability of transitioning to the violation state.
  Theoretical analysis confirms that the value function is continuous, justifying the use of universal approximators like neural networks.

Key Contributions

Equivalent Markovian Formulation: The paper extends state augmentation techniques from finite-horizon to infinite-horizon settings, casting the problem as a constrained MDP without resorting to conservative approximations like Boole's inequality.
Strong Duality and Deterministic Policy: The authors prove strong duality for the infinite-horizon case. Crucially, they demonstrate that the proposed dual-ascent algorithm converges to a deterministic Markov policy that is both optimal and feasible. This contrasts with prior works that often rely on mixed policies or average feasibility over learning iterations.
Relaxed Feasibility Conditions: The paper shows that the assumption of bounded uncertainty support (common in infinite-horizon literature) is not strictly necessary; feasibility can be ensured if the probability of the first constraint violation decays sufficiently fast (e.g., via mean-square stability).
Learning-Based Solution: A dedicated offline training scheme is proposed to approximate the value function and the dual function, significantly reducing the computational complexity of the online control phase compared to Model Predictive Control (MPC).

Results
The approach was validated on a numerical example involving a unicycle navigating an obstacle field with a joint chance constraint ( $\varepsilon = 0.1$ ).

Performance: The learning-based approach achieved a lower closed-loop cost ( $L_{cl} = 528.3$ ) compared to a randomized MPC scheme ( $L_{cl} = 672.0$ ).
Safety: The empirical constraint violation rate for the proposed method was $0.045$, satisfying the $\varepsilon = 0.1$ requirement. In contrast, the MPC scheme with 25 samples yielded a violation rate of $0.17$, failing to meet the safety specification.
Efficiency: The online computation time for the proposed method was approximately 0.01 seconds per step (solving a one-step problem), whereas the MPC approach required an average of 2.94 seconds (with a 10-second limit), often struggling to find feasible solutions near the obstacle.
Behavior: The learned policy correctly identified that once a constraint violation occurs for a specific trajectory, the optimal strategy is to disregard the safety constraint and minimize cost, a behavior the MPC scheme failed to replicate efficiently.

Significance
The paper claims to provide a computationally efficient and theoretically rigorous solution to a class of problems that have remained open challenges: infinite-horizon stochastic control with joint chance constraints. By avoiding conservative approximations and leveraging strong duality, the method achieves better performance and safety guarantees than state-of-the-art MPC approaches while maintaining low online computational complexity. The work bridges the gap between constrained MDP theory and practical learning-based control for continuous systems.