🤖 AI

Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulation

This paper presents a self-paced curriculum reinforcement learning framework that integrates Soft Actor-Critic with dynamic task generation to efficiently train an autonomous agent for stable and fast superbike racing in a physics-accurate simulation, outperforming standard SAC approaches.

Original authors: Luca Ghisi, Jacopo Essenziale, Carlo D'Eramo, Matteo Luperto

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Ghisi, Jacopo Essenziale, Carlo D'Eramo, Matteo Luperto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a toddler how to ride a bicycle. If you just throw them onto a busy highway and say, "Go!" they will likely fall over immediately. But if you start them in a quiet driveway, then move to a flat park, and finally introduce gentle hills, they learn much faster.

This paper is about teaching a computer "toddler" (an AI agent) how to race a high-speed superbike in a video game simulator called VRider SBK. The researchers found that the standard way of teaching AI wasn't working well for motorcycles, so they invented a smarter, step-by-step training method.

Here is the breakdown of their work using simple analogies:

1. The Big Challenge: Two Wheels vs. Four

Most AI racing research focuses on cars. Cars are like heavy trucks; they are stable. If you turn a corner too fast, the car might slide a bit, but it usually stays upright.

Motorcycles are different. They are like tightrope walkers.

Balance is key: To turn, a rider must lean. If they don't lean enough, they fall outward. If they lean too much, they fall inward.
The AI's struggle: The computer had to learn not just how to steer and hit the gas, but how to constantly balance the bike while leaning at extreme angles. If the AI messed up the balance, the bike would crash and fall over—a problem cars don't have.

2. The Solution: The "Self-Paced" Coach

The researchers used a standard AI trainer called SAC (Soft Actor-Critic). Think of SAC as a strict coach who throws the student into the deep end immediately.

They combined SAC with a new method called SPDL (Self-Paced Deep Reinforcement Learning). Think of SPDL as a smart, adaptive coach who watches the student and adjusts the difficulty in real-time:

The Start: The coach tells the AI, "Just follow the middle of the track. Don't worry about speed yet." This is the "easy" mode.
The Progress: As the AI gets better at staying upright, the coach slowly moves the goalposts. "Okay, now try to follow the 'ideal racing line' (the fastest path)."
The Finish: Eventually, the coach says, "Now go as fast as you can on that perfect line."

The paper claims this "self-paced" approach worked much better than the standard method. The AI learned to race faster, fell over less, and finished training sooner.

3. How the AI "Sees" the Track

To make the AI understand the bike, the researchers gave it a special "dashboard" of information (State Space):

Body Awareness: It knows how fast it's going, how much it's leaning, and how slippery the tires are.
Memory: Crucially, it remembers its recent leaning history. Just like a human rider feels the bike wobbling before they crash, the AI uses this history to stop itself from shaking too much.
The Map: It sees 60 points ahead on the track, like looking down a road to see the curves coming up.

4. The "Scorecard" (Rewards)

In AI training, the computer gets points (rewards) for good behavior and loses points for bad behavior. The researchers designed a very specific scorecard for motorcycles:

Good: Moving forward along the track.
Bad: Going off the track, hitting walls, or wobbling too much.
The "Lean" Penalty: They specifically punished the AI for shaking the bike back and forth (oscillating) while leaning. This forced the AI to learn smooth, controlled turns rather than jerky ones.

5. The Results: What Happened?

The researchers tested this on a famous race track called Barcelona and others.

The "Standard" AI (SAC alone): At the early stages of training, it couldn't even finish one lap. It kept crashing and falling over.
The "Self-Paced" AI (SPDL): It finished laps without falling. By the end of training, it was 0.57 seconds faster per lap than the standard AI.
The "Transfer" Test: They took the AI trained on a Ducati motorcycle and put it on a Kawasaki, a Honda, and a Yamaha without retraining it.
- Result: It worked! The AI could ride the new bikes reasonably well, proving it learned the general skill of balancing a bike, not just how to ride one specific model.
- Exception: It struggled a bit with the BMW, which is described as "aggressive and hard to drive," suggesting some bikes are just harder to learn than others.

6. What's Still Missing?

The authors are honest about the limitations.

The "Speed vs. Precision" Problem: Sometimes the AI gets so excited about going fast that it enters a corner too quickly and overshoots the turn (see Fig. 4 in the paper). It prioritizes speed over taking the perfect line.
One Track at a Time: Currently, they train a separate AI for each track. They hope to eventually train one AI that can handle any track it sees for the first time.

Summary

This paper presents the first successful attempt to teach an AI to race a motorcycle in a simulator using a "step-by-step" learning method. By letting the AI start on an easy path and gradually increasing the difficulty, they managed to create a rider that stays upright, learns faster, and can even switch between different bike models. It's a major step toward autonomous motorcycle racing, though the AI still needs to learn how to be a bit more precise when cornering at high speeds.

Technical Summary: Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulation

Problem Statement

Autonomous racing has achieved significant milestones using deep Reinforcement Learning (RL), particularly for four-wheeled vehicles in high-fidelity simulators like Gran Turismo. However, motorbike racing remains largely unexplored in RL literature due to fundamentally different physical and dynamical challenges. Unlike cars, racing motorbikes require the agent to actively manage lean angle to counteract centrifugal forces during cornering. Failure to control this angle results in a loss of balance and falling—a failure mode absent in four-wheeled vehicles. Additionally, racing bikes possess a higher power-to-weight ratio, subjecting them to more aggressive braking and acceleration dynamics. The core challenge is to train an agent that can simultaneously handle steering, throttle/brake control, and the complex balance required for two-wheeled stability at the physical limits of the vehicle.

Methodology

The authors propose a framework implemented in VRider SBK, a physics-accurate Unity-based simulator featuring real-world Superbike World Championship tracks and bike models. The system integrates Soft Actor-Critic (SAC) with Self-Paced Deep Reinforcement Learning (SPDL).

1. State and Action Space

State Space: The observation vector combines proprioceptive and global features, normalized to $[-1, 1]$ $[- 1, 1]$ .
- Proprioceptive: Linear/angular velocity, current/previous steering and lean angles, lean-angle history ( $\theta$ ), tire slip ratios, and previous control actions. The inclusion of lean-angle history is critical for anticipating and suppressing oscillatory behaviors.
- Global: 60 equally spaced 3D course points sampled along the current trajectory (center line and track limits) via Catmull-Rom spline interpolation, providing a speed-adapted look-ahead.
Action Space: Two continuous actions in $[-1, 1]$ $[- 1, 1]$ :
1. Steering (which also drives the bike's lean angle).
2. A combined throttle/brake signal (mutually exclusive).

2. Reward Design

The reward function is dense and multi-component, designed to minimize lap time while penalizing instability. It includes:

Progress: Displacement along the trajectory, negated if the bike travels backward or off-course.
Penalties: Off-track time, wall contact (kinetic-energy based), and specific stability penalties for large changes in steering, leaning, and angular velocity between steps.
History Penalties: Sigmoid-based penalties on accumulated angle deltas to discourage oscillatory behavior in steering and leaning.
Failure Conditions: Episodes reset if the bike falls, hits a wall, or exceeds penalty timers.

3. Self-Paced Curriculum (SPDL)

Instead of manual curriculum design, the system uses SPDL to dynamically generate tasks based on agent performance. The difficulty is defined by the trajectory the agent must follow:

Context Variables:
- $c_1$ : Distance from the median (centerline) trajectory.
- $c_2$ : Distance from the ideal racing line.
- $c_3$ : Maximum speed limit.
Mechanism: The algorithm maintains a context distribution $p(c|\nu)$ and optimizes a trade-off between reward maximization and convergence to a target distribution (the ideal line at full speed). A KL divergence constraint prevents abrupt difficulty jumps. As the agent exceeds a performance threshold $\delta$ , the curriculum automatically shifts from the easy median trajectory toward the ideal racing line.

Key Contributions

RL Architecture for Two-Wheeled Vehicles: A tailored state space including lean-angle history and a reward signal specifically designed to handle motorbike-specific instabilities (falling, oscillation).
Automated Curriculum Integration: The first integration of SPDL with Unity ML-Agents for motorbike racing, enabling dynamic, automatic task generation without manual intervention.
Baseline Establishment: Preliminary experimental results establishing the first RL baseline for autonomous motorbike racing, demonstrating superior training efficiency and stability compared to standard SAC.

Experimental Results

Experiments were conducted on the Barcelona circuit (primary) and generalized to Cremona and Phillip Island, using the Ducati Panigale V4 R and four other bike models.

Training Efficiency: At iteration 1150, a standard SAC agent failed to complete a single lap, repeatedly losing control. The SPDL agent completed all laps with zero falls and an average time of 1:32.85. By iteration 1900, SPDL achieved a lap time of 1:30.75, outperforming the trained SAC baseline (1:31.32) by 0.57 seconds.
Stability: SPDL agents demonstrated significantly fewer falls and off-track excursions compared to SAC.
Track Generalization: Agents trained with SPDL successfully completed laps on Cremona and Phillip Island with zero falls and zero collisions, demonstrating robustness across different track geometries.
Bike Generalization: A policy trained on the Ducati Panigale V4 R was transferred to other models (Kawasaki, Honda, Yamaha, BMW) without retraining.
- The Kawasaki Ninja ZX-10RR showed the best transfer, trailing the Ducati by only 0.87s on Barcelona.
- Honda and Yamaha models even outperformed the Ducati on Phillip Island due to better handling of specific corners.
- The BMW M1000 RR struggled on Phillip Island, failing to complete the lap, highlighting the difficulty of transferring to more aggressive, hard-to-drive bikes.

Significance and Limitations

The paper claims to establish the first RL baseline for autonomous motorbike racing, proving that combining SAC with self-paced curriculum learning is effective for managing the unique dynamics of two-wheeled vehicles. The framework successfully generalizes across multiple tracks and bike models, suggesting the learned policy captures generalizable control priors rather than just bike-specific dynamics.

However, the authors modestly note current limitations:

Trajectory Precision: The agent tends to prioritize speed over precision, occasionally overshooting corners (as seen in qualitative results).
Single-Track Training: Currently, a separate agent is trained per track; future work aims for multi-track generalization.
Sensing: The system currently relies on vector-based inputs; future iterations could incorporate vision-based inputs and asymmetric actor-critic architectures to improve sample efficiency and realism.

The work serves as a foundational step toward competitive autonomous superbike racing, with future directions including testing in competitive racing settings rather than just time-trial modes.