Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstacles

1. Problem Statement

The paper addresses the challenge of autonomous drone racing in cluttered environments containing random obstacles. While autonomous quadrotors have demonstrated agility in obstacle-free settings, racing in dense environments introduces two conflicting objectives:

• Gate Traversal: Requires aggressive, precise flight to pass through gates at high speeds.
• Obstacle Avoidance: Requires conservative maneuvers to steer clear of obstacles.

Existing methods struggle with this duality. Traditional optimization-based approaches suffer from model mismatches and high computational costs, while standard Reinforcement Learning (RL) approaches often fail to generalize across unseen obstacle configurations or suffer from low success rates when transferring from simulation to the real world (sim-to-real gap). The core difficulty lies in designing a policy that balances these conflicting goals without overfitting to specific track layouts.

2. Methodology

The authors propose a Vision-based Curriculum Reinforcement Learning (CRL) framework. The system is an end-to-end controller that maps raw depth images and drone state directly to control commands (Collective Thrust and Body Rates - CTBR).

A. Network Architecture

• Input: A 17-dimensional state vector (position, velocity, orientation, relative gate positions) and a $64 \times 64$ depth map from an onboard camera.
• Processing:
  • State data is processed via a Multi-Layer Perceptron (MLP).
  • Depth images are processed via a Convolutional Neural Network (CNN).
  • Features are concatenated and fed into a Gated Recurrent Unit (GRU) to capture temporal dependencies.
• Output: An Actor-Critic network outputs CTBR commands ($T, \omega_x, \omega_y, \omega_z$) for direct onboard deployment on a Raspberry Pi.

B. Reward Function Design

To resolve the conflict between passing gates and avoiding obstacles, a composite reward function is designed:

• Progress & Alignment: Rewards moving toward the next gate and aligning the drone's yaw.
• Obstacle Avoidance ($r_{avoid}$): A continuous reward based on the distance to the nearest collision point, preventing the drone from treating gates purely as obstacles to be bypassed.
• Gate Passing & Crash: Binary rewards for successfully passing a gate or crashing.
• Control Smoothness: Penalties for large actions and excessive speed.

C. Training Strategy

The framework employs three key innovations to ensure robustness and efficiency:

1. Multi-Stage Curriculum Learning:
   • Level 1: Obstacle-free racing at low speeds.
   • Level 2: Introduction of random obstacles at low speeds.
   • Level 3: High-speed racing with dense, randomized obstacles.
     This progressive difficulty prevents the agent from falling into local optima.
2. Domain Randomization:
   • Randomizes obstacle positions, shapes, and densities within a safety margin.
   • Randomizes gate positions and drone initial states to ensure the policy learns a wide range of scenarios.
3. Multi-Scene Updating:
   • Unlike traditional parallel training where all agents update on the same scene, this method updates agents in groups across a variable number of scenes. This balances exploration (early stages) and exploitation (later stages), improving generalization and convergence speed.

3. Key Contributions

1. Systematic Generalization via Curriculum: A multi-stage learning strategy that enables agents to transition from simple navigation to complex, high-speed racing in unseen cluttered environments.
2. Multi-Scene Training Paradigm: A novel updating mechanism that dynamically adjusts the number of training scenes, significantly improving training efficiency and policy robustness against diverse obstacle configurations.
3. Conflict-Resolving Reward Design: A carefully structured reward function that balances the inherent trade-off between aggressive gate traversal and conservative obstacle avoidance, enabling large-angle maneuvers without compromising safety.
4. Real-World Validation: Successful zero-shot sim-to-real transfer of an end-to-end vision-based policy running on low-cost hardware (Raspberry Pi) at high speeds.

4. Experimental Results

The method was validated through simulation, Hardware-in-the-Loop (HITL), and real-world flights across three track types (S-shaped, 3D Circle, J-shaped).

• Success Rate (SR): The proposed method achieved 100% success rate across all three racetracks in both simulation and real-world experiments.
  • Comparison: Outperformed a baseline vision-based RL method (30-40% SR) and a state-based method (20-30% SR).
• Lap Time (LT): Achieved lap times comparable to state-based methods but significantly faster than baseline vision-based methods.
  • Real-world speeds: Up to 8 m/s.
  • HITL speeds: Up to 10 m/s.
• Ablation Studies: Removing any key component (Curriculum Learning, Obstacle Avoidance Reward, or GRU) led to significant performance degradation. Specifically, "one-step" training (no curriculum) resulted in 0% success rate.
• Robustness: The policy maintained high success rates (70-80%) even when obstacle density was increased to 5 obstacles per gate segment and gate positions were randomized by up to $\pm 1.0$ meters.

5. Significance

This work represents a significant advancement in autonomous drone racing by solving the "obstacle-rich" problem, which has historically been a bottleneck for RL applications.

• Practicality: It demonstrates that complex, high-speed flight control can be learned end-to-end using only visual inputs and deployed on lightweight, low-cost hardware (Raspberry Pi + Depth Camera).
• Robustness: The curriculum and multi-scene strategies effectively bridge the sim-to-real gap, allowing drones to handle dynamic, unpredictable environments without requiring retraining or manual tuning.
• Future Impact: The framework provides a scalable path toward fully autonomous flight in unstructured, real-world environments, moving beyond controlled laboratory tracks.