On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty

Technical Summary: Comparison of Reinforcement Learning and Adaptive Control for Linear Systems Under Packet Loss and Uncertainty

Problem Statement
This paper addresses the control of uncertain linear systems operating over communication channels subject to packet loss and input quantization. The specific challenge involves stabilizing a system that undergoes dynamic switching from a nominal unstable configuration to a more unstable one during operation. The study compares two distinct control paradigms in this setting: Adaptive Quantized Control (AQC), a model-based approach utilizing acknowledgment messages, and Deep Deterministic Policy Gradient (DDPG), a data-driven reinforcement learning (RL) approach trained on the nominal model without acknowledgments. The core problem is to evaluate the trade-off between the transient performance of learning-based methods and the robustness of adaptive control under model uncertainty and communication constraints.

Methodology
The study employs a discrete-time uncertain linear system model where the control signal is transmitted through a channel with a packet loss probability $\bar{p}$. The control input is subject to logarithmic quantization, modeled as a time-varying sector-bounded nonlinearity.

1. Adaptive Quantized Control (AQC):

   • Framework: Based on prior work [11, 12], the AQC utilizes a time-varying feedback gain $K(k)$ and a quantizer $q(k, u(k))$.
   • Mechanism: The controller relies on acknowledgment messages to determine if a packet was successfully received ($\eta(k)=1$) or lost ($\eta(k)=0$). This information is used to update the control law and the quantizer parameters.
   • Stability: The design is grounded in Lyapunov stability theory. A Riccati equation is solved to determine a stabilizing gain, and the adaptive law ensures that the system remains stable in probability even under packet loss, provided the loss bound satisfies a specific condition derived from the Lyapunov function.

2. Deep Deterministic Policy Gradient (DDPG):

   • Framework: An actor-critic architecture using feed-forward neural networks (FFNN). The actor maps states directly to continuous control actions, while the critic estimates the action-value function.
   • Training: The DDPG agent is trained exclusively on the nominal system model (without packet loss or dynamic switching) using a reward function based on tracking error.
   • Implementation: During evaluation, the trained policy is applied to the system with the same time-varying quantizer used in the AQC, but without access to acknowledgment messages. The agent must generalize its policy to handle packet loss and the switch to a more unstable system ($A_w$).

Key Contributions

• Comparative Analysis: The paper provides a direct numerical comparison between a rigorously designed adaptive controller and a modern deep reinforcement learning controller under identical conditions of input quantization, packet loss, and dynamic system switching.
• Robustness Evaluation: It investigates the performance of both methods when the system dynamics change from a nominal unstable state to a more unstable state, a scenario not present during the DDPG training phase.
• Quantization Integration: Both control strategies are implemented with a time-varying logarithmic quantizer, allowing for a fair comparison of how each method handles finite bandwidth constraints.

Numerical Results
Simulations were conducted on a second-order uncertain system with varying initial conditions and switching times ($t_d$).

• Training Environment Performance: In the absence of packet loss and within the nominal dynamic range, the DDPG controller demonstrated faster transient responses and improved damping compared to the AQC.
• Robustness under Uncertainty and Packet Loss:
  • AQC: Consistently stabilized the system across all tested scenarios, including high packet loss rates ($\bar{p} > 0.5$) and dynamic switching. The use of acknowledgment messages allowed the controller to compensate for lost packets, maintaining stability as guaranteed by Lyapunov theory.
  • DDPG: While effective for small deviations from the training environment, the DDPG controller failed to stabilize the system when subjected to significant model uncertainty (switching to highly unstable dynamics) or high packet loss rates. Without acknowledgment messages, the controller could not adapt to the loss of control signals, leading to instability.
• Stability Range: Table 1 in the paper indicates that while the DDPG controller remains stable only for parameter variations close to the training set, the AQC maintains stability over a significantly wider range of system parameters.

Significance and Claims
The paper concludes that there is a distinct trade-off between data-driven performance and model-based robustness.

• Performance vs. Robustness: DDPG offers superior transient performance when the operating environment closely matches the training distribution. However, AQC provides superior robustness and guaranteed stability in the presence of model uncertainty, packet loss, and dynamic switching due to its rigorous Lyapunov stability guarantees.
• Applicability: The results suggest that while reinforcement learning is promising for high-dimensional spaces, its applicability to networked uncertain systems is limited without explicit stability mechanisms or training that encompasses the full range of potential uncertainties.
• Future Directions: The authors note that future work will focus on extending these frameworks to networked multi-agent systems, specifically addressing communication delays, distributed interactions, and the development of RL algorithms with rigorous stability guarantees for cooperative control.