Autonomous overtaking trajectory optimization using reinforcement learning and opponent pose estimation

1. Problem Statement

Autonomous racing is a high-stakes domain requiring vehicles to operate at dynamic limits while navigating adversarial conditions. A critical challenge in this environment is multi-agent overtaking, where an autonomous vehicle must not only follow an optimal racing line but also dynamically predict and react to an opponent's pose (position and orientation) to execute safe and efficient overtaking maneuvers.

Existing Reinforcement Learning (RL) approaches often focus on single-agent racing or treat opponents as static obstacles, leading to suboptimal performance or collisions in real-world, stochastic multi-agent scenarios. Furthermore, many state-of-the-art algorithms lack validation on physical hardware, relying solely on simulation. This paper addresses the gap by developing a system that combines robust opponent pose estimation with RL-based overtaking strategies, validated on a real-world F1TENTH platform.

2. Methodology

The proposed system consists of two primary components: a sensor fusion module for opponent pose estimation and a Multi-Agent Reinforcement Learning (MARL) algorithm for trajectory optimization.

A. Hardware and Platform

• Robot: F1TENTH scaled racing car.
• Compute: NVIDIA Jetson Xavier NX.
• Sensors:
  • Hokuyo 2D LiDAR (for 2D scanning).
  • Intel RealSense D435 Depth Camera (for RGB and depth data).
• Ground Truth: Optitrack motion capture system used for validation.

B. Opponent Pose Estimation (Sensor Fusion)

To accurately track the opponent, the system fuses data from the LiDAR and the depth camera using an Unscented Kalman Filter (UKF).

• LiDAR Processing: Uses a clustering and rectangle fitting algorithm to identify potential vehicle clusters. The most likely cluster is selected based on the previous UKF state estimate.
• Depth Camera Processing:
  • Object Detection: Utilizes a fine-tuned YOLOv8 model to detect the opponent's bounding box in RGB images.
  • Distance Estimation: Calculates distance using the bounding box height and a reciprocal function ($distance = \frac{a}{bbox\_height + b} + c$), calibrated experimentally.
  • Pose Logic: The yaw angle is not estimated via vision but derived from the vehicle's kinematic model (Ackermann steering), assuming the vehicle travels in the direction of its orientation.
• Fusion (UKF):
  • State Vector: $[x, y, v_x, v_y]$ (position and velocity).
  • Process Model: Constant velocity model.
  • Measurement Models: Two distinct models handle the different data formats and update rates of the LiDAR and camera.
  • Benefit: The fusion allows the system to run at 10 Hz (providing continuous trajectory updates) despite the sensors updating at 1–2 Hz. It prevents the LiDAR from tracking the wrong object by using the camera to identify the correct cluster.

C. Reinforcement Learning Algorithm

The overtaking behavior is learned using Proximal Policy Optimization (PPO).

• Environment: F1TENTH Gym (simulating a bicycle model, 2D occupancy grid, and LiDAR).
• Observation Space (Size 162):
  • Localized LiDAR data (108 points, downsampled from 1080).
  • Odometry of the agent and opponent.
  • Future waypoints (10 ahead) for both agent and opponent, derived from a pre-calculated minimum curvature racing line.
• Action Space: Continuous control outputs for steering angle ($\pm 0.34$ rad) and linear velocity ($0$ to $3$ m/s).
• Reward Function: A weighted sum of seven components:
  1. Velocity reward.
  2. Progress along the raceline.
  3. Overtaking reward: +1 if ahead, -1 if behind; linearly scaled when close.
  4. Raceline penalty (distance error).
  5. Heading penalty (yaw error).
  6. Collision penalty.
  7. Smoothness penalty (steering change).
• Training: Multi-agent setup where both agents use the same neural network policy, allowing the agent to learn from interactions with a similar opponent.

3. Key Contributions

1. Sensor Fusion for Pose Estimation: Implementation of a UKF to fuse 2D LiDAR and YOLO-based depth camera data, significantly improving opponent tracking robustness compared to single-sensor approaches.
2. Real-World Multi-Agent RL: Successful deployment of a PPO-based overtaking algorithm on a physical F1TENTH platform, moving beyond simulation-only validation.
3. Dynamic Overtaking Strategy: A reward formulation specifically designed to encourage overtaking maneuvers in adversarial environments rather than just obstacle avoidance.
4. Open-Source Stack: The code and stack are made available via the CRTA-Lab repository.

4. Results

• Pose Estimation Accuracy:
  • The UKF fusion outperformed individual sensors in the Y-direction (RMSE: 0.0531 m) and provided robust tracking in the X-direction (RMSE: 0.0816 m).
  • The fusion successfully mitigated the risk of the LiDAR tracking incorrect objects by leveraging the camera's object identification.
• Overtaking Performance:
  • The agent successfully executed overtaking maneuvers in multiple real-world runs without collisions.
  • Trajectories showed oscillations in the real world (amplified by noise and latency) compared to simulation, but the core maneuver was successful.
• Limitations:
  • The system struggled to generalize to tracks significantly different from the training environment (e.g., smaller looped tracks vs. wide fixed-width tracks).
  • Performance was limited by slow sensor update rates (1–2 Hz), necessitating lower operating speeds to prioritize estimation precision over reaction time.

5. Significance

This paper demonstrates a viable pathway for safe, high-speed autonomous racing by integrating perception and control. By treating the opponent as a dynamic entity with an estimated pose rather than a static obstacle, the system achieves more efficient and safer driving. The successful transition from simulation to a real-world F1TENTH platform validates the practical applicability of sensor-fused RL for adversarial driving tasks. Future work aims to improve trajectory smoothness and increase sensor update rates to allow for higher-speed operations.