Technical Summary: Feasible Action for Optimal Control (FAOC)

Problem Statement

Operating constrained dynamical systems requires controllers that can solve complex tasks while strictly enforcing recursive feasibility and safety constraints. While Reinforcement Learning (RL) has demonstrated the ability to solve complex control problems across various domains, it suffers from poor sample efficiency and an inherent inability to strictly guarantee constraint satisfaction. Conversely, Optimal Control (OC), particularly Model Predictive Control (MPC), offers rigorous safety guarantees through explicit constraint enforcement but struggles with computational tractability in nonconvex, long-horizon problems and often requires extensive tuning of action spaces.

Existing hybrid approaches that combine RL and OC often face a critical trade-off:

1. Feasibility Issues: When RL agents directly select parameters for an Optimal Control Problem (OCP), their actions may render the OCP infeasible, requiring slack variables or heuristic penalties that degrade performance.
2. Action Space Design: To ensure feasibility, prior works often employ static, heuristic action spaces (e.g., bounded hypercubes) that do not account for the state-dependent nature of the underlying OCP's feasible parameter set. This leads to the inclusion of infeasible or redundant actions, forcing the RL policy to implicitly learn complex feasibility boundaries, which hampers learning efficiency and final performance.

Methodology: Feasible Action for Optimal Control (FAOC)

The authors propose Feasible Action for Optimal Control (FAOC), a hierarchical framework that bridges RL and OC via a computationally efficient, optimization-based mapping algorithm. The core innovation is a bijective mapping that transforms an RL agent's output from a static, geometrically simple abstract action set into a state-dependent feasible parameter set of the OCP.

Framework Architecture

1. High-Level RL Policy: The RL agent operates within a static, compact, solid, and convex abstract action space $\bar{A}$ (e.g., a hyperbox). It outputs a raw action, which is transformed into an abstract action $\bar{a} \in \bar{A}$.
2. Mapping Algorithm ($M$): A novel algorithm maps $\bar{a}$ to a parameter $p$ belonging to the state-dependent feasible set $P(x)$ of the OCP. This mapping guarantees that $p$ is always feasible for the current system state $x$, ensuring the OCP remains solvable.
3. Low-Level OCP: The mapped parameter $p$ (e.g., a terminal state target) is fed into a parameterized OCP. The OCP solves for an optimal control trajectory subject to physical constraints, guaranteeing safety and recursive feasibility.

Key Algorithmic Components

The paper develops a family of mapping algorithms to handle the geometric transformation between the abstract set $\bar{A}$ and the state-dependent set $P(x)$:

• Topological Characterization: The authors establish mild geometric conditions (Lemma 1) ensuring that the feasible parameter set $P(x)$ of a generic OCP is compact, solid, and convex. This is explicitly demonstrated for linear MPC with terminal constraints (Corollary 1).
• Invertible Radial Mapping: The core mapping (Algorithm 1) is a radial scaling algorithm that bijectively transforms points from $\bar{A}$ to $P(x)$. It ensures invertibility, allowing any feasible parameter to be projected back to the abstract action space, preventing "action aliasing."
• Mitigation of Geometric Distortion:
  • 2D Area-Matching: For 2D spaces, a directional transformation is derived to match the marginal angular distributions of the sets' areas, preventing point accumulation in narrow regions of the target set (Propositions 2 & 3).
  • Linear Transformation (Arbitrary Dimensions): For higher dimensions, the authors propose using affine surrogates (specifically Maximum Volume Inscribed Ellipsoids) to approximate the geometric distortion. This allows for a scalable linear transformation that preserves distribution density without requiring explicit geometric representations of $P(x)$ (Proposition 4).
• Implicit Set Handling: A significant contribution is the ability to perform these mappings without explicit geometric representations of $P(x)$. By leveraging the structure of the OCP constraints, the authors derive robust formulations to compute interior points and shape matrices directly from the optimization constraints (Propositions 6–8), enabling real-time execution.

Key Contributions

The paper outlines five primary contributions:

1. Topological Characterization: Identification of geometric conditions ensuring the state-dependent parameter set of a generic OCP is compact, solid, and convex.
2. Invertible Feasible Action Mapping: A computationally efficient radial algorithm that bijectively maps abstract RL actions to guaranteed-feasible OCP parameters.
3. Geometric Distortion Mitigation: Development of area-matching (2D) and linear transformation (arbitrary dimensions) techniques to prevent point accumulation and accelerate learning.
4. Tractability for Implicit Sets: Derivation of robust formulations to compute necessary mapping components (interior points, shape matrices) directly from OCP constraints, avoiding computationally expensive explicit geometric representations.
5. Experimental Validation: Application to real-time motion planning for an 8-DoF robot table tennis system, demonstrating professional-level performance.

Experimental Results

The FAOC framework was evaluated on a real 8-DoF robot arm playing table tennis, a task requiring high-speed decision-making and strict adherence to kinematic constraints.

• Setup: The RL agent (using Soft Actor-Critic) selected 2D waypoints (position and velocity) for each joint. The FAOC mapper translated these into feasible terminal constraints for a parameterized OCP.
• Baselines: FAOC was compared against:
  • 1D Variants: Controllers where the RL agent selected only position, velocity, or acceleration (limited controllability).
  • 2Dsoft: A controller using a static, state-independent action space with soft terminal costs to handle infeasible targets.
• Performance:
  • Sample Efficiency: FAOC achieved the highest sample efficiency and final performance across all experiments, outperforming both 1D and 2Dsoft baselines.
  • Controllability: FAOC demonstrated superior controllability, particularly when the RL decision frequency was reduced (simulating higher latency). While other controllers degraded significantly at lower frequencies, FAOC maintained performance due to its state-dependent action space ensuring feasible trajectory segments.
  • Real-World Success: The framework enabled the robot to compete and win against professional-level human players in official ITTF matches.

Significance and Claims

The paper claims that FAOC resolves persistent feasibility and exploration challenges in combining RL and OC. By decoupling the RL agent from physical constraints, the framework allows the policy to focus solely on strategic decision-making while the OCP handles local kinematic constraints.

The authors emphasize that unlike prior work, FAOC does not require expert-designed action spaces or compromise the OCP formulation with infeasible actions. The framework effectively combines the predictable safety of OC with the flexibility of RL. The successful deployment on a real robot in a high-speed, competitive environment serves as a proof of concept that this approach can handle nonconvex strategic problems (handled by RL) while maintaining strict local feasibility (handled by OC). The mapping algorithm and OCP implementation are open-sourced to facilitate further research.