Technical Summary: Adaptive Outer-Loop Control of Quadrotors via Reinforcement Learning

Problem Statement

Autonomous quadrotor flight control in complex, real-world environments faces significant challenges due to unmodeled dynamics, such as severe aerodynamic disturbances, actuator wear, and sudden payload variations. Traditional control methods (PID, LQR, MPC) rely on explicit mathematical models and often degrade rapidly under such uncertainties. While Deep Reinforcement Learning (DRL) offers a model-free alternative, standard DRL controllers trained in simulation suffer from the "sim-to-real" gap. The prevailing solution, Domain Randomization (DR), produces robust policies but forces the controller to learn a single, overly conservative strategy that maximizes worst-case performance. This approach sacrifices agility and optimal performance in nominal conditions because the controller remains oblivious to the instantaneous state of the environment.

The core research question addressed is: How can a learning-based flight controller dynamically estimate unmodeled physical perturbations from past states and use this latent information to maintain optimal tracking performance while seamlessly bridging the sim-to-real gap?

Methodology

The authors propose a novel adaptive control architecture that moves beyond static, conservative policies by enabling the controller to actively perceive and react to instantaneous disturbances. The system is built upon a cascaded control structure where a high-level Reinforcement Learning (RL) policy acts as an outer-loop controller, outputting desired body rates and collective thrust, which are tracked by an onboard PID rate controller.

The methodology consists of three primary components:

1. Adaptive Controller with Oracle Baseline:
   To establish an upper bound on performance, the authors first train an RL policy alongside an "Oracle" that has access to ground-truth 6D environmental disturbances (forces and moments). This policy learns optimal responses to specific dynamic variations rather than a generalized compromise. However, since physical micro-UAVs lack the sensors to measure these forces in real-time, this Oracle cannot be directly deployed.

2. Residual Dynamics Predictor (RDP):
   To replace the Oracle during real-world deployment, the authors introduce a Recurrent Neural Network (RDP) designed as a nonlinear, implicit state estimator. The RDP estimates unobserved 6D external forces and moments using only a temporal history of vehicle states and low-level actuation signals.

   • Input: The RDP takes a window of vehicle kinematic states, previous outer-loop actions, and crucially, the 4-dimensional Pulse Width Modulation (PWM) signals commanded to the individual motors. The inclusion of PWM signals is vital for capturing transient actuator states and motor lag.
   • Architecture: The network utilizes Gated Recurrent Units (GRUs) to process a rolling historical window of standardized observations, outputting a predicted 6D perturbation vector.

3. Data-Efficient Sim-to-Real Calibration Bridge:
   To address the discrepancy between simulation-trained estimators and physical reality, the authors introduce a lightweight linear calibration layer. Instead of extensive retraining, they use a minimal dataset (three brief flight samples) to fit a multivariate linear regression model. This bridge corrects for steady-state offsets (e.g., due to mass distribution asymmetries or battery voltage curves). Additionally, an online thrust correction mechanism accumulates altitude tracking errors to adjust for constant vertical force offsets, ensuring the estimator's latent space aligns with physical reality.

Key Contributions

• Adaptive RL Controller: Development of an outer-loop controller trained with an Oracle to learn optimal responses to varying external forces, moving away from the conservative nature of DR.
• Residual Dynamics Predictor (RDP): Design of a recurrent neural network that implicitly predicts 6D physical perturbations using only state history and motor commands, effectively replacing the need for ground-truth disturbance sensors.
• Data-Efficient Sim-to-Real Bridge: Introduction of a computationally lightweight linear calibration layer and online thrust correction that aligns simulation-trained estimators with reality using mere seconds of flight data, eliminating the need for large-scale real-world datasets.

Experimental Results

The approach was validated on a Bitcraze Crazyflie 2.X micro-quadrotor in three real-world scenarios: central payload mass variation, asymmetric payload distribution, and dynamic trajectory tracking with a suspended load.

• Central Payload Variation: The adaptive controller with the learned RDP achieved a Root Mean Square Error (RMSE) of 0.024 m to 0.030 m across payload increases of 0% to 25%. This outperformed both the Base Controller (which degraded to 0.160 m at 25% load) and the Robust Controller (which exhibited severe oscillatory instability even at 0% load). Notably, the RDP-based controller performed slightly better than the Oracle-based version, suggesting that the learned latent representation implicitly compensates for modeling mismatches better than rigid physical inputs.
• Asymmetric Payload: In scenarios inducing coupled translational and rotational disturbances, the adaptive controller maintained significantly lower tracking errors compared to baselines. The RDP successfully estimated induced roll and pitch moments, allowing the controller to decouple and counteract these torques.
• Dynamic Trajectory Tracking: When tracking a figure-8 trajectory with a suspended payload, the system maintained stable flight even as trajectory aggressiveness increased (period T reduced from 15s to 3s). While tracking error increased with aggressiveness due to bandwidth limitations, the system remained stable without divergence. The RDP successfully captured the time-varying oscillatory dynamics of the suspended load.

Significance and Claims

The paper claims that active, real-time adaptation is fundamentally superior to passive robustness (DR) for maintaining stable and optimal flight tracking. Key insights include:

• Implicit Estimation Sufficiency: Complex 6D external forces and moments can be accurately reconstructed without specialized force-torque sensors, using only standard onboard kinematics and actuator signals.
• Control-Relevant vs. Exact Estimation: Precise reconstruction of ground-truth physical quantities is not strictly necessary for high performance. A bounded estimation error (e.g., 3%–7% in mass estimation) is sufficient, provided the network generates a consistent, control-relevant representation of disturbance trends.
• Efficient Sim-to-Real Transfer: The discrepancy between simulation and reality can be approximated by a low-order linear relationship. This allows for effective bridging of the sim-to-real gap using a lightweight linear calibration and online correction, avoiding complex nonlinear domain adaptation or extensive retraining.
• Decoupled Training Efficacy: Training an optimal policy with an Oracle before transitioning to a learned estimator circumvents the instability and sample inefficiency often associated with end-to-end RL in partially observable environments.

The authors conclude that this architecture enables micro-quadrotors to operate under severe dynamic uncertainties, such as mass variations and dynamic slung loads, with precision that significantly outperforms classical and robust RL baselines. Future work aims to extend this framework to turbulent wind gusts and larger platforms.