⚡ electrical engineering

On the Value Function of Infinite-Horizon Optimal Control of Piecewise Affine Systems

This paper investigates the structure of the value function for constrained infinite-horizon optimal control of piecewise affine systems with $\ell_1$ or $\ell_\infty$ costs, demonstrating that the function can possess an infinite number of affine pieces and providing sufficient conditions to ensure it remains a proper piecewise affine function with a finite number of pieces.

Original authors: Francesco Cordiano, Kanghui He, Bart De Schutter

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Cordiano, Kanghui He, Bart De Schutter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of engineering, many machines do not move in a single, smooth way. Instead, they operate in distinct modes, switching between different rules depending on their current state or the commands they receive. Think of a thermostat that turns a heater on or off, or a battery system that charges and discharges under different constraints. These are known as piecewise affine systems, where the behavior of the machine is defined by a collection of simple, straight-line rules that apply to different regions of its operation. To make these machines work as well as possible, engineers use a method called optimal control, which involves calculating the perfect sequence of actions to reach a goal while minimizing cost, such as energy use or time. When the goal is to keep the machine running perfectly forever, rather than just for a short period, the math becomes incredibly complex. For decades, researchers have relied on a specific mathematical structure to solve these long-term problems, believing that the solution always breaks down into a manageable number of simple, straight-line pieces. This belief has been the foundation for modern techniques that use artificial intelligence to learn how to control these complex machines.

A team of researchers recently challenged this long-held assumption by asking a simple but profound question: is the solution to these infinite-time problems always made of a finite number of pieces, or could it be infinitely complex? They discovered that the answer depends entirely on how the machine is designed and how the costs are weighted. In a specific scenario involving a simple two-dimensional system, they showed that if the penalty for using control inputs is set too high, the optimal strategy does not settle into a neat, finite pattern. Instead, the solution develops an infinite number of distinct regions, even within a small, bounded area. This means the mathematical map of the best actions becomes infinitely detailed, with new, smaller straight-line sections appearing endlessly as you zoom in. This finding is critical because it reveals that the standard mathematical tools used to approximate these solutions might fail in certain cases, potentially leading to errors in the design of automated systems.

The researchers demonstrated this phenomenon using a counterexample, a specific setup designed to break the usual rules. They constructed a system where the machine's natural tendency is to stabilize itself, but the cost of applying any control force is so high that the machine is forced to rely on its own internal dynamics. In this situation, the optimal path to the target state involves a sequence of decisions that never repeats in a simple cycle. As the machine gets closer to its goal, the boundaries between different decision-making regions become finer and finer, creating a pattern that never settles. The researchers calculated the exact value of the cost for every possible starting point and found that the resulting map was not a simple collection of a few flat surfaces, but a complex structure with infinitely many facets. This result directly contradicts the idea that the solution is always a "proper" piecewise affine function, which by definition must have a finite number of pieces over any compact area.

However, the paper does not leave engineers without a way forward. After showing that the infinite complexity can occur, the authors derived a set of clear, verifiable conditions that guarantee the solution will remain simple and finite. They found that if the cost of using control inputs is kept within a specific range relative to the system's dynamics, the machine will be steered to its target in a predictable, finite number of steps. Under these conditions, the complex, infinite pattern collapses back into a manageable structure with a limited number of regions. The researchers proved that if the cost matrices are chosen correctly, the optimal strategy will always be a function with a finite number of straight-line pieces, ensuring that the mathematical models used in learning-based control schemes remain valid and reliable.

To test these theoretical findings, the team ran numerical simulations that visualized the behavior of the system. In one example, they showed a map where the regions of different behaviors were clearly defined and finite, confirming that their conditions worked as predicted. In another case, where the conditions were violated, the map showed the emergence of the infinite, fractal-like pattern they had predicted. These visualizations serve as a practical guide for engineers, showing exactly where the boundary lies between a solvable, finite problem and one that spirals into infinite complexity. The work clarifies the limits of current control theories and provides a safety net for the development of new, learning-based control systems. By identifying the precise conditions under which the value function remains well-behaved, the study ensures that the next generation of automated systems can be designed with confidence, knowing that their underlying mathematical foundations are solid and finite.

Technical Summary: On the Value Function of Infinite-Horizon Optimal Control of Piecewise Affine Systems

Problem Statement
This paper investigates the structural properties of the value function in constrained infinite-time optimal control (CITOC) problems for discrete-time piecewise affine (PWA) systems. The systems are governed by dynamics that are piecewise affine in the state and input, and the optimization utilizes stage costs defined by $\ell_1$ or $\ell_\infty$ norms. While existing literature (e.g., [1]) establishes that the value function for such problems is piecewise affine (PWA), these results do not rigorously address whether the resulting function constitutes a "proper" PWA function. A proper PWA function, as defined in the literature, must possess a finite number of affine pieces over compact sets. The central problem addressed is whether the infinite-horizon value function $J^\star_\infty$ necessarily satisfies this finiteness condition, or if it can exhibit an infinite number of affine pieces within a compact region.

Methodology
The authors employ a combination of theoretical analysis, counterexample construction, and dynamic programming (DP) iteration analysis.

Counterexample Construction: To challenge the assumption of finiteness, the authors construct a specific 2-dimensional linear system (a subclass of PWA systems) with $\ell_1$ stage costs. By selecting specific system matrices and sufficiently large input penalty weights, they derive a closed-form expression for the optimal value function.
Sufficient Conditions Derivation: To restore the property of a finite number of pieces, the paper introduces a set of sufficient conditions involving the system's controllability and the design of the cost matrices. This involves analyzing the DP iterations and proving that under specific conditions, the optimal policy can steer the system to the origin in a single step within a neighborhood of the origin.
Theoretical Proofs: The main theoretical result relies on two key lemmas:
- Lemma 1: Demonstrates that under specific assumptions on the cost matrices, a PWA optimizer exists that drives the system state to zero within a small neighborhood.
- Lemma 2: Establishes that for any initial state with a finite cost, the optimal trajectory enters this neighborhood in a finite number of steps.
  These lemmas are combined to prove that the value function converges to a PWA structure with a finite number of pieces.

Key Contributions

Demonstration of Infinite Pieces: The paper provides an explicit example where the value function of a CITOC problem for a stable and controllable linear system possesses infinitely many affine pieces on a compact set. This occurs when the input weights are large enough that the optimal policy is zero, yet the system does not reach the origin in finite time, leading to an accumulation of switching boundaries defined by hyperplanes $c_k x_1 + d_k x_2 = 0$ .
Rigorous Definition of Proper PWA: The authors highlight that the set of PWA functions is not closed; a sequence of PWA functions (from finite-horizon DP iterations) can converge to a function that is not PWA in the standard sense (i.e., having infinitely many pieces).
Sufficient Conditions for Finiteness: The paper derives explicit, easily verifiable sufficient conditions (Assumptions 1–4) that guarantee the value function is a proper PWA function. These conditions essentially require that the input penalty matrix $R$ is not too large relative to the state penalty $Q$ , ensuring the optimal policy can drive the state to the origin in one step within a neighborhood of the origin.
Numerical Validation: The theoretical findings are illustrated through numerical examples, including visualizations of the value function and the partition of the state space, confirming both the existence of infinite pieces in the counterexample and the restoration of finite pieces under the proposed sufficient conditions.

Results

The study confirms that for PWA systems with $\ell_1$ or $\ell_\infty$ costs, the infinite-horizon value function is not automatically a proper PWA function, even for linear systems with desirable properties like asymptotic stability and controllability.
The counterexample shows that if input penalties are high, the value function can have infinitely many regions defined by hyperplanes passing through the origin, with different slopes in each region.
Under the proposed sufficient conditions (specifically relating to the matrix $M$ where $R = M^\top Q B_j$ and $\|M\|_q \leq 1$ ), the value function is proven to be a proper PWA function with a finite number of pieces over any polyhedral subset of the set of states with finite cost.
The paper notes that while the derived conditions are sufficient for the DP iterations to terminate in two steps, they are not strictly necessary for the value function to have a finite number of pieces; in some cases, convergence may occur in more steps even if the strict condition is violated.

Significance and Claims
The paper claims to complement well-known results in the linear-quadratic case and existing literature on PWA systems. Its primary significance lies in clarifying a gap in the understanding of infinite-horizon optimal control for PWA systems: the distinction between a function being piecewise affine in a general sense and being a proper PWA function with a finite number of pieces.

The authors state that these findings are instrumental for recent learning-based control schemes (e.g., [9], [10]) that rely on the structure of the value function to design function approximators (such as PWA neural networks). By establishing conditions under which the value function has a finite number of pieces, the paper provides theoretical guarantees for the quality of such approximations. The work serves as a support for these learning-based approaches by ensuring that the target function to be approximated is structurally compatible with finite-parameter representations over compact sets.

More like this