Learning-based control of a single-DOF Aero system

Technical Summary: Learning-Based Control of a Single-DOF Aero System

Problem Statement
Nonlinear mechatronic systems, such as aerospace platforms and robotic manipulators, face significant control challenges due to strong nonlinearities, parametric uncertainties, unmodeled dynamics, and external disturbances. While classical nonlinear control techniques like feedback linearization and sliding mode control offer formal stability guarantees, their performance is critically dependent on the availability of accurate system models. In practice, unavoidable modeling errors and unknown disturbances often degrade performance or compromise stability. Conversely, while Reinforcement Learning (RL) offers adaptability through interaction with the environment, standard RL algorithms frequently lack formal stability guarantees and require extensive exploration, making them difficult to deploy in safety-critical or highly nonlinear systems. The core problem addressed is the development of a control architecture that integrates the analytical stability guarantees of model-based control with the adaptability of learning-based methods to handle uncertainties without sacrificing safety.

Methodology
The authors propose a hybrid control framework applied to a single-degree-of-freedom (pitch) model of the Quanser AERO system. The methodology combines a Lyapunov-based feedback linearization controller with a policy-gradient reinforcement learning component.

1. Control Architecture:

   • Nominal Controller: A feedback linearization law is derived to linearize the system dynamics. This controller is designed using Lyapunov stability analysis to ensure closed-loop stability in the presence of modeling uncertainties.
   • Learning Component: An actor-critic reinforcement learning agent is employed to estimate and compensate for unmodeled dynamics and external disturbances online. The agent learns a disturbance compensator ($\hat{d}$) that is added to the nominal control law.
   • Algorithm: The learning module utilizes the REINFORCE-with-baseline algorithm. The policy (actor) is modeled as a Gaussian distribution, while a separate neural network (critic) estimates the state-value function. To stabilize training and reduce the variance of policy-gradient estimates, the authors employ Generalized Advantage Estimation (GAE).
   • Reward Function: The reward is defined based on the combined tracking error ($\eta$), specifically $r = -|\eta| \ln(1 + t)$. The logarithmic time-dependent term attenuates the influence of large transient errors early in the episode, shifting focus toward steady-state performance.

2. Stability Analysis:
   The paper provides a Lyapunov-based stability proof. It assumes the pitch angle and its derivative are measurable, the desired trajectory is continuously differentiable, and the disturbance is bounded. The analysis demonstrates that if the RL agent's output is bounded, the disturbance approximation error remains bounded. Consequently, the closed-loop system exhibits uniformly ultimately bounded (UUB) tracking errors. The stability of the system is primarily ensured by the Lyapunov-based feedback linearization, while the RL agent provides adaptive compensation within these stability bounds.

3. Simulation Setup:
   Numerical simulations were conducted in Python using the fourth-order Runge-Kutta method. The system was subjected to randomized initial conditions and time-varying desired trajectories. The RL agent was trained over 20,000 episodes using the Adam optimizer with a learning rate of $1 \times 10^{-4}$.

Key Results

• Algorithm Performance: Comparisons between vanilla REINFORCE and REINFORCE-with-baseline showed that the baseline-enhanced variant achieved significantly higher final returns and faster convergence. Vanilla REINFORCE exhibited unstable learning behavior and failed to converge at lower learning rates, whereas the baseline variant reached stable performance in approximately 2,500 episodes.
• Trajectory Tracking: In simulations involving 100 randomized initial conditions, the learning-based controller demonstrated superior trajectory tracking accuracy compared to the conventional feedback linearization controller alone. The proposed approach significantly reduced both steady-state and transient tracking errors.
• Robustness: The controller maintained high performance under unmodeled, time-varying disturbances (sinusoidal and cosinusoidal combinations) without retraining. The learned policy successfully adapted to capture the oscillatory structure of the injected disturbances, keeping the tracking error low and bounded.
• Control Effort: The improved tracking performance was achieved without introducing excessive control effort; the control voltage profiles remained similar between the conventional and learning-enhanced methods.

Significance and Claims
The paper claims that the proposed hybrid approach effectively combines the analytical guarantees of Lyapunov-based control with the adaptability of reinforcement learning. By using the model-based controller as a stable foundation and the learning module as an adaptive disturbance compensator, the system achieves robust trajectory tracking in uncertain environments. The authors emphasize that this architecture allows for "safe and stable learning" because the RL agent operates within the bounds of the conventional controller, ensuring that the system's stability conditions are not violated even as the agent adapts to unmodeled effects. The results suggest that incorporating a baseline in policy-gradient methods is crucial for improving sample efficiency and ensuring reliable policy learning in nonlinear mechatronic systems.