Track-Guided Hierarchical Reinforcement Learning for Autonomous Vehicle Drifting with Minimum-Lap-Time Planning

Technical Summary: Track-Guided Hierarchical Reinforcement Learning for Autonomous Vehicle Drifting with Minimum-Lap-Time Planning

Problem Statement
Autonomous vehicle drifting presents a complex dual-objective control challenge: stabilizing highly nonlinear drift dynamics while strictly minimizing lap time. Unlike Formula 1, where drivers optimize racing lines within tire grip limits, rally racing on loose or low-adhesion surfaces often requires intentionally breaking traction to drift. This maneuver aligns the vehicle for corner exits, reducing lap time but introducing significant control difficulties due to strong nonlinearities in tire and chassis dynamics. Existing Reinforcement Learning (RL) approaches face two primary hurdles: (1) a heavy reliance on costly, vehicle-specific expert demonstration datasets for action-level supervision, and (2) a lack of structured mechanisms to guide agents toward physically meaningful drift strategies during early training, often leading to suboptimal or unsafe policies.

Methodology
The paper proposes a Minimum-Lap-Time (MLT) Drift Planning and Track-guided Reinforcement Learning (TgRL) framework. This approach integrates a model-based trajectory planner with a deep RL controller to address the scarcity of expert data and the difficulty of learning extreme dynamics directly.

1. Minimum-Lap-Time (MLT) Drift Planning:

   • An Optimal Control Problem (OCP) is formulated using a high-fidelity, nonlinear 3-DoF vehicle model and the Magic Formula tire model to simulate combined slip conditions.
   • The objective is to minimize lap time by generating dynamically feasible, aggressive reference trajectories that include vehicle position, attitude, speed, sideslip angle, and yaw rate.
   • Constraints include road boundaries, state limits (e.g., sideslip angle $\beta \in [-\pi/6, \pi/6]$), and control input limits.
   • The resulting trajectories serve as high-quality prior data (reference paths) for the RL controller, eliminating the need for human expert demonstrations.

2. Track-Guided Reinforcement Learning (TgRL):

   • Algorithm: The authors employ the Soft Actor-Critic (SAC) algorithm, an off-policy, entropy-regularized actor-critic method suitable for continuous control.
   • Curriculum Learning: To overcome the difficulty of learning extreme dynamics with large sideslip angles directly, a progressive, three-stage training strategy is implemented:
     • Stage 1 (Basic Policy): Training on a simplified straight-line track to learn fundamental drift control (inducing tire slip and managing large sideslip angles).
     • Stage 2 (Corner Policy): Training on hairpin curves to learn drift cornering, focusing on initiating drift, maintaining slip angles, and executing smooth exits.
     • Stage 3 (Race Policy): Training on a full closed racing circuit (Lkarting) to integrate skills for minimizing lap time while maintaining stability.
   • State Space: Includes lateral/heading/sideslip errors, velocity errors, and future reference trajectory points.
   • Action Space: Continuous steering angle and throttle input, smoothed via an Adaptive Action Smoothing (AAS) strategy to prevent rollovers at high speeds.
   • Reward Function: A multi-objective reward combines:
     • Instant Rewards: Penalize errors in lateral deviation, heading, sideslip, and velocity. A speed-weighted term encourages high-speed cornering.
     • End Rewards: Discrete rewards issued at episode termination. Positive rewards are scaled based on how close the lap time is to the MLT planning optimum; negative rewards are applied for collisions or rollovers, inversely proportional to survival time.

Key Contributions

1. Novel Framework: The paper introduces a hierarchical planning-control framework (MLT-TgRL) that seamlessly integrates optimal control-based trajectory generation with deep RL, enabling autonomous agents to execute extreme drift maneuvers without expert data.
2. MLT Drift Planning: A specific OCP formulation is developed to generate aggressive, dynamically feasible drift trajectories that serve as priors for the RL agent, effectively replacing the need for high-cost human demonstrations.
3. Progressive Training Methodology: An innovative curriculum learning approach is introduced where the agent is trained step-by-step (from basic drift control to cornering, then to full racing). This is guided by track information and reinforced by a specialized reward function combining instant tracking terms and MLT-derived end rewards.

Results
The framework was validated using the CARLA simulator across three tracks: Lkarting, Willow Springs, and Nelson Ledges.

• Performance vs. Baselines: The proposed MLT+TgRL controller consistently achieved superior lap times compared to a standard SAC baseline and a previous RL-based drift controller (Cai et al.). On Lkarting, the MLT+TgRL framework achieved an average lap time of 45.4 s, outperforming the Cai_DRIFT baseline (51.2 s) by approximately 11%.
• Comparison with Planning: While the MLT planner achieved the theoretical lower bound (minimum lap time of 33.1 s on Lkarting), the TgRL controller achieved competitive results with a minimum lap time of 37.2 s and an average of 45.4 s, demonstrating the ability to approximate optimal trajectories in a real-time control setting.
• Stability and Dynamics: The agent successfully maintained large sideslip angles (averaging 15.9° on Lkarting, compared to 10.4° for a human driver and 16.3° for the baseline), confirming the ability to sustain drift states. The controller also maintained low yaw angle tracking errors and minimal lateral offset fluctuations.
• Ablation Studies: Removing the MLT guidance (using centerline references instead) or the progressive curriculum significantly degraded performance, confirming the necessity of both the model-based priors and the staged training approach. The inclusion of End Rewards and Adaptive Action Smoothing further improved lap time consistency and drift accuracy.

Significance and Claims
The paper claims that its primary significance lies in bridging the gap between theoretically optimal planning and practical drift execution. By leveraging model-based planners to provide dynamically feasible references and RL controllers to adapt to unmodeled dynamics and actuation delays, the method achieves aggressive drift maneuvers with improved stability and repeatability. The work establishes a foundation for using reinforcement learning in real-world autonomous racing systems by demonstrating that structured, curriculum-based training guided by optimal trajectories can overcome the data scarcity and exploration challenges inherent in learning extreme vehicle dynamics. The authors note that while the current approach is effective, future work will focus on generalizing the controller across different vehicle models, road conditions, and implementing hardware-in-the-loop (HIL) experiments.