🤖 AI

Curriculum-based Sample Efficient Reinforcement Learning for Robust Stabilization of a Quadrotor

This paper proposes a novel, human-inspired three-stage curriculum learning framework that significantly improves the sample efficiency and convergence speed of end-to-end reinforcement learning for robust quadrotor stabilization, enabling the policy to simultaneously control position and yaw from random initial conditions while meeting strict performance specifications.

Original authors: Fausto Mauricio Lagos Suarez, Akshit Saradagi, Vidya Sumathy, Shruti Kotpaliwar, George Nikolakopoulos

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fausto Mauricio Lagos Suarez, Akshit Saradagi, Vidya Sumathy, Shruti Kotpaliwar, George Nikolakopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a toddler how to ride a bicycle.

If you just throw them on a bike, spin the wheels, and say, "Go! Don't fall!" they will likely crash immediately. They might get scared, give up, or learn the wrong habits. This is how traditional Reinforcement Learning (RL) works for complex robots like drones: you throw them into a chaotic environment and hope they figure it out through millions of crashes. It takes forever, costs a fortune in computer power, and often fails.

This paper proposes a smarter way: Curriculum Learning. Think of this as a "training camp" that breaks the impossible task into three manageable levels, just like a video game with easy, medium, and hard modes.

Here is the story of how they taught a tiny drone (a Crazyflie) to stabilize itself, even when thrown into the air with a spin.

The Goal: The "Perfect Hover"

The researchers wanted the drone to do something very specific:

Land perfectly at a specific spot in the air.
Face the right direction (like a camera pointing at a wall).
Do it quickly (under 5 seconds) and precisely (within the width of a fingernail).
Handle chaos: The drone might start in a weird spot, tilted sideways, or even moving fast in the wrong direction.

The Problem: The "One-Stage" Nightmare

In the past, researchers tried to teach the drone this all at once. They would let the drone fly around randomly, crashing and failing, hoping it would eventually "get it."

The Analogy: It's like trying to learn a complex language by being dropped in a foreign country with no dictionary, no teacher, and no idea what anyone is saying. You might eventually learn, but it would take a lifetime and you'd make a lot of embarrassing mistakes.
The Result: In this paper, the "one-stage" method failed completely. Even after running simulations for 23 hours (100 million attempts), the drone still couldn't learn to hover properly.

The Solution: The Three-Stage "Training Camp"

The authors decided to act like a patient human coach. They broke the training down into three stages, where the drone masters one skill before moving to the next.

Stage 1: The "Take-Off" (Hovering)

The Task: The drone starts on the ground, perfectly still. Its only job is to lift up and hover at a specific height.
The Analogy: This is like teaching a child to stand up on a balance beam without moving their feet. They just need to learn how to keep their balance.
Why it works: The drone learns the basic physics: "If I spin these motors, I go up." It gets a "gold star" (reward) for staying steady.

Stage 2: The "Dance" (Position & Orientation)

The Task: Now, the drone can start in a slightly different spot or tilted a little bit. It has to move to the target spot and turn to face the right way.
The Analogy: Now the child is on the balance beam and has to walk to the other end while turning around. It's harder because moving forward makes you tilt, and turning makes you wobble. The drone has to learn that "tilting forward" is how it "moves forward."
The Transfer: Because the drone already knows how to hover (Stage 1), it doesn't have to relearn the basics. It just learns how to combine hovering with moving.

Stage 3: The "Storm" (Robustness)

The Task: This is the final boss level. The drone starts in a random spot, tilted wildly, and is even pushed by an invisible wind (random velocity). It has to fight the wind, correct its tilt, and land perfectly.
The Analogy: Now the child is on the balance beam, but someone is pushing them sideways and spinning them around. They have to use everything they learned in Stages 1 and 2 to fight back and stay upright.
The Result: The drone learns to be "tough." It can handle being thrown into the air and still land perfectly.

The Secret Sauce: The "Scorecard"

To make sure the drone learns the right way, the researchers designed a special "Scorecard" (Reward Function).

If the drone gets close to the target, it gets points.
If it wobbles too much or flies too far away, it loses points.
If it crashes or spins out of control, the game ends immediately (like a "Game Over" screen).
Crucial Detail: They didn't just say "Good job." They said, "Good job, but you need to be this precise and this fast." This forced the drone to learn high-quality flying, not just "okay" flying.

The Results: A Victory Lap

When they tested the results:

Speed: The "Training Camp" (Curriculum Learning) took 11 hours to master the task. The "One-Stage" method took 23 hours and still failed.
Efficiency: The curriculum method used less than half the computer power (samples) to succeed.
Real-World Test: They tested the trained drone in a simulation where it had to inspect a wall. It flew from point A to B, turned to face the wall, and hovered perfectly, even when starting from a messy, chaotic position.

The Big Picture

This paper proves that how you teach a robot matters just as much as what you teach it. By mimicking how humans learn—starting simple and gradually adding difficulty—you can train complex robots much faster, cheaper, and more reliably.

Instead of throwing a robot into the deep end of the pool and hoping it doesn't drown, you teach it to kick, then float, then swim, and finally dive. The result? A robot that is ready for the real world.

1. Problem Statement

The paper addresses the challenge of training Reinforcement Learning (RL) agents to achieve robust stabilization of a quadrotor (specifically a Crazyflie 2.1) under complex conditions.

Objective: The agent must stabilize the drone's position ( $x, y, z$ ) and yaw orientation ( $\psi$ ) from random initial conditions, including non-zero linear and angular velocities, by directly controlling motor RPMs (end-to-end control).
Performance Constraints: The solution must meet strict control-theoretic metrics relevant to aerial inspection:
- Settling time: $< 5$ seconds.
- Positional steady-state error: $< 2.5$ cm.
- Yaw-tracking error: $< 2^\circ$ .
The Challenge: Conventional one-stage end-to-end RL training is computationally expensive and sample-inefficient. It often requires hundreds of millions of time steps and months of tuning to converge, frequently failing to meet the specific transient and steady-state requirements even with extensive training.

2. Methodology

The authors propose a three-stage Curriculum Learning (CL) framework to decompose the complex stabilization task into manageable sub-tasks, progressively transferring knowledge to improve sample efficiency.

A. Curriculum Structure

The learning process is divided into three sequential stages, expanding the difficulty of the initialization domain while maintaining a consistent reward structure:

Stage 1 ( $\Omega_1$ - Hovering):
- Task: Learn to take off and hover at a fixed target $(0,0,1)$ from a fixed start $(0,0,0)$ .
- Goal: Establish basic motor control (symmetric RPMs) and stability.
Stage 2 ( $\Omega_2$ - Coupling & Orientation):
- Task: Reach the target $(0,0,1)$ with $\psi=0$ from random initial positions (within a 2m radius cylinder) and random attitudes (roll/pitch $\in [-15^\circ, 15^\circ]$ , yaw $\in [-180^\circ, 180^\circ]$ ).
- Goal: Learn the coupling between translational degrees of freedom ( $x, y$ ) and rotational degrees of freedom (roll $\phi$ , pitch $\theta$ ).
Stage 3 ( $\Omega_3$ - Robust Stabilization):
- Task: Same as Stage 2 but with random initial linear and angular velocities ( $\in [-1, 1]$ m/s and rad/s).
- Goal: Learn to correct for momentum and disturbances, achieving the final robust stabilization objective.

B. Reward Function Design

A custom additive reward function $R(t)$ is designed to enforce stability and precision:
$R(t) = 25 - 20T_e - 100E + 20S - 18w_e$

Target Reward ( $T_e$ ): Penalizes distance to the target position and yaw error.
Exploration Reward ( $E$ ): Penalizes the agent if it drifts outside a defined cylindrical boundary (preventing uncontrolled exploration).
Stability Reward ( $S$ ): Provides a positive reward when the drone is within a tolerance sphere of the target and roll/pitch angles are near zero.
Navigation Reward ( $w_e$ ): Penalizes rapid changes in angular velocities to ensure smooth transitions.
Truncation: Episodes are truncated if roll/pitch exceed $\pm 15^\circ$ or if the drone leaves the safety cylinder, enforcing safety constraints during training.

C. RL Setup

Algorithm: Proximal Policy Optimization (PPO).
Environment: Gym-PyBullet-Drones (physics-based simulation).
Observation Space: 12 dimensions (position, Euler angles, linear velocities, angular velocities).
Action Space: 4 dimensions (RPM for each of the 4 motors).
Network Architecture: Multi-Layer Perceptron (MLP) with 4 layers (12 inputs $\to$ 64 hidden $\to$ 64 hidden $\to$ 4 outputs).

3. Key Contributions

Novel Three-Stage Curriculum: Unlike previous works that adjust reward weights, this approach expands the initialization domain (position, orientation, velocity) progressively. This respects the underactuated nature of quadrotors and the coupling between translational and rotational dynamics.
Sample Efficiency: The method significantly reduces the computational resources required. While one-stage training failed to converge even after 100 million steps, the CL approach achieved the goal in roughly 41.5 million steps.
Performance Guarantees: The reward function and truncation conditions are explicitly designed to meet pre-specified settling time and steady-state accuracy metrics required for real-world inspection missions.
Robust Validation: The policy was validated against 30 diverse random initial conditions and in a complex inspection pose-tracking scenario.

4. Results

Sample Efficiency Comparison:
- One-Stage Training: Failed to achieve the target task after 100 million time steps (23.4 hours of wall-clock time). The Episode Cumulative Reward (ECR) remained low.
- Curriculum Learning: Successfully achieved the target task in 41.5 million time steps (11 hours total).
  - Stage 1: 11.5M steps (3 hours).
  - Stage 2: 15M steps (4 hours).
  - Stage 3: 15M steps (4 hours).
Robustness Testing:
- The CL-trained policy successfully stabilized the drone from 30 randomized initial conditions (including high velocities and extreme orientations) in < 5 seconds in most cases.
- In extreme cases (high initial velocity), the drone exhibited large transients (dipping close to the ground) but successfully recovered and stabilized.
Inspection Scenario: The policy successfully tracked a sequence of inspection poses (positions with specific yaw angles) with errors within the defined limits ( $<2.5$ cm position, $<2^\circ$ yaw).

5. Significance and Conclusion

This work demonstrates that Curriculum Learning is a critical enabler for applying RL to complex, safety-critical aerial control tasks. By mimicking human learning (progressive difficulty), the authors achieved:

Drastic Reduction in Training Time: Cutting training time by more than half compared to a failed baseline.
Feasibility: Making sample-efficient RL training possible without requiring massive supercomputing clusters for months.
Reliability: Producing a policy that meets strict engineering specifications (settling time, error bounds) which standard RL often fails to guarantee.

Future Work: The authors plan to automate the curriculum setup and bridge the Sim2Real gap by deploying the trained policy on physical Crazyflie drones.