Technical Summary: A Heuristic Approach for Performance Tuning in RL-based Quadrotor Control

Problem Statement

While Reinforcement Learning (RL) has demonstrated impressive capabilities in high-speed, agile quadrotor tasks such as drone racing and navigation through cluttered environments, it often lacks the ability to provide tunable, precise, and controlled maneuvers required for applications like infrastructure inspection. Typical RL training setups are rigid, making it difficult to iteratively tune resulting behaviors to meet specific control-theoretic metrics (e.g., settling time, overshoot, steady-state error) without incurring prohibitive computational costs. Furthermore, unlike classical Proportional-Integral-Derivative (PID) controllers, which offer intuitive tuning heuristics, RL reward functions are often complex and difficult to map to specific transient and steady-state performance requirements.

Methodology

The authors propose a novel heuristic approach to achieve tunable performance in end-to-end RL-based quadrotor control by manipulating reward design and termination conditions. The system is modeled as a Partially Observable Markov Decision Process (POMDP) using the Proximal Policy Optimization (PPO) algorithm.

1. System and Observation

• Platform: The study utilizes a Crazyflie 2.1 quadrotor in an $X$ configuration.
• Dynamics: The system employs unit quaternions for orientation to avoid singularities, modeling rigid-body dynamics including thrust, torque, and gravity.
• Observation Space: The agent observes a 17-dimensional vector comprising position error, quaternion error, linear velocity, angular velocity, and the previous action. Gaussian noise is injected into all state components (except the previous action) to enhance robustness against real-time sensor noise.
• Action Space: The policy outputs normalized motor RPM values, mapping directly to the drone's four motors.

2. Reward Function Design

A multi-component reward function is designed to guide the policy toward stabilizing control:
$$R(t) = s_R + \lambda_1 e^R_{xy} + \lambda_2 e^R_{z} + \lambda_3 v_R + \lambda_4 \theta_R - \lambda_5 a^P_{\Delta}$$

• Survival Reward ($s_R$): Incentivizes the agent to avoid crashes or unsafe states.
• Position Error Rewards ($e^R_{xy}, e^R_z$): Utilizes dual-bandwidth exponential rewards. This structure splits the reward into coefficients influencing early response ($\alpha$) and steady-state accuracy ($\beta$).
• Velocity and Angle Rewards ($v_R, \theta_R$): Rewards the reduction of linear velocity and orientation error (measured via geodesic angle).
• Smoothness Penalty ($a^P_{\Delta}$): Penalizes the squared Euclidean norm of the difference between consecutive actions, acting as a damping term to encourage derivative-like control behavior.

3. Termination and Truncation Conditions

The authors define specific failure conditions (e.g., descending below 10 cm, excessive acceleration) and truncation horizons. These conditions are tuned to constrain the quadrotor's behavior, influencing agility and the critically damped nature of the response.

4. Heuristic Tuning Rules

The core contribution is a set of intuitive heuristics to adjust the reward weights, exponential coefficients, and termination bounds to shift performance between three distinct modes:

• Baseline: A critically damped response with low steady-state error.
• Acrobatic (Faster): Achieved by increasing the early response bandwidth of position errors and relaxing velocity constraints, allowing for faster settling times.
• Inspection (Slower): Achieved by shrinking velocity termination bounds, increasing the sharpness of position error rewards, and relaxing survival rewards to avoid local minima, resulting in smoother, slower transients.

Key Contributions

1. Novel Reward Structure: Introduction of dual-bandwidth exponential rewards that achieve a baseline critically damped response with low steady-state errors in setpoint tracking.
2. Tunable Heuristics: A framework of intuitive rules to adjust reward weights and termination conditions to produce "acrobatic-like" (fast) and "inspection-like" (slow) settling times while retaining the baseline steady-state error characteristics.
3. Sample Efficiency: Demonstration that the desired performance can be achieved in approximately 6 million time steps using PPO, without requiring complex hybrid architectures (e.g., RL cascaded with PID).
4. Statistical Validation: Validation across 100 trials with random initial conditions, demonstrating consistent performance across position and yaw tracking.

Results

The authors trained three distinct policies (Baseline, Acrobatic, Inspection) using identical PPO hyperparameters.

• Training Efficiency: All policies reached reward saturation at 6 million time steps. The variance across random seeds was low, indicating stability.
• Performance Metrics:
  • Settling Time: The Acrobatic policy consistently achieved shorter settling times, while the Inspection policy exhibited longer, smoother transients compared to the Baseline.
  • Overshoot: The x, y, and z position variables exhibited zero Interquartile Range (IQR) in overshoot, indicating that at least 75% of trials achieved the intended critically damped response. Yaw overshoot was larger, attributed to randomized initial conditions.
  • Steady-State Error: All three policies maintained steady-state errors within the prescribed 2% band for position and yaw.
• Motor Actuation: The Inspection policy required less motor actuation and settled faster in terms of RPM stabilization compared to the more aggressive Acrobatic policy.

Significance and Claims

The paper claims that the proposed heuristic approach constitutes a reliable and practical methodology for designing RL-based quadrotor controllers with tunable stabilizing performance. Unlike previous hybrid approaches that focus on training a single policy with fixed expectations, this work provides a mechanism to control the policy's performance characteristics (transient vs. steady-state) through intuitive parameter adjustments.

The authors emphasize that this approach bridges the gap between the flexibility of RL and the predictability of classical control theory, allowing for the generation of controllers suitable for diverse applications ranging from high-speed racing to precise infrastructure inspection. The work is modest in its scope, focusing on simulation-based validation with Gaussian noise, and explicitly identifies sim-to-real transfer, domain randomization, latency, and onboard computational limitations as necessary areas for future work.