💻 computer science

A Multi-Modal Perception Pipeline for Object Detection and Tracking in Autonomous Racing

This paper presents a robust multi-modal late-fusion perception pipeline that integrates camera, LiDAR, and RADAR data with a specialized tracking framework to achieve reliable object detection and tracking for autonomous racing under high-speed, adverse, and edge-case conditions.

Original authors: Davide Malvezzi, Michele Pestarino, Vittoria Cavicchioli, Valentina La Gamba, Silvia Severi, Fabio Bagni, Luca Bartoli, Massimiliano Bosi, Francesco Gatti, Micaela Verucchi, Ayoub Raji, Marko Bertogna

Published 2026-09-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Davide Malvezzi, Michele Pestarino, Vittoria Cavicchioli, Valentina La Gamba, Silvia Severi, Fabio Bagni, Luca Bartoli, Massimiliano Bosi, Francesco Gatti, Micaela Verucchi, Ayoub Raji, Marko Bertogna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where cars race at speeds that would make a Formula One driver sweat, yet no human is behind the wheel. In this extreme environment, the margin for error is measured in inches and milliseconds. At speeds approaching 80 meters per second, a delay of just one-tenth of a second means a car travels nearly eight meters without knowing where it is or what is in front of it. This is the reality of autonomous racing, a field that pushes perception technology to its absolute limit. To navigate this chaos, a vehicle must act as its own eyes and brain, constantly scanning the track to find other cars, predict their movements, and decide when to speed up, slow down, or swerve. The challenge is not just seeing, but seeing clearly through blurring motion, vibrating sensors, and the sudden appearance of obstacles, all while operating in conditions that would confuse even the most advanced human senses.

A team of researchers from the University of Modena and Reggio Emilia, working with HiPeRT Srl, has developed a new system designed to solve these problems for the 2025 Abu Dhabi Autonomous Racing League. Their work, presented in a recent paper, focuses on a "perception pipeline"—a complex chain of software and hardware that turns raw data from the car's sensors into a reliable map of the world. The team's approach is built on the idea that no single type of sensor is perfect. Cameras are excellent at recognizing shapes but struggle with distance in the dark or bad weather. LiDAR sensors, which use laser pulses to build a 3D map, are precise but can be overwhelmed by dust or rain. RADAR units are great at measuring speed but often lack detail. The researchers realized that to survive the high-speed chaos of a race, a car needs to combine all these senses into a single, unified view, a method they call "late fusion." This means the car processes each sensor's data independently first, then brings the results together to form a complete picture, rather than trying to merge raw data streams immediately.

The vehicle used in this study, the Dallara EAV-24, is equipped with a sophisticated array of tools: three LiDAR sensors, seven cameras, and four RADAR units, all synchronized to a precise clock. The system works by having each sensor type hunt for other cars on its own. The cameras use a neural network to spot the 2D outlines of vehicles in the video feed. The LiDARs scan the air with lasers to find 3D shapes, while the RADARs detect the speed and presence of objects based on radio waves. Once these independent detections are made, the system's "fusion" module acts as a matchmaker. It looks at the data from different sensors and asks, "Is this LiDAR blob the same car as that camera box?" If the timing and location align, the system merges them, creating a single, robust estimate of where the other car is and how fast it is moving. This process is critical because if the system relied on just one sensor, a momentary glitch or a blind spot could lead to a catastrophic failure. By combining them, the system ensures that if one sensor fails or is blocked, the others can keep the car safe.

However, seeing the car is only half the battle; knowing where it will be a fraction of a second from now is the other half. The researchers found that at racing speeds, even a tiny delay in processing can cause the car to think an obstacle is in one place when it has actually moved several meters. To fix this, their tracking system includes a "delay compensation" mechanism. It looks at the exact moment a sensor saw an object and calculates where that object must be right now, accounting for the time it took for the data to travel through the computer. Furthermore, the system doesn't just guess; it uses a deep understanding of how race cars behave. It knows that cars generally follow specific paths around the track, called racing lines, and that they slow down for corners and speed up on straights. By feeding this knowledge into its calculations, the system can predict a car's future position with much greater accuracy than a standard tracker that assumes cars move in straight lines at constant speeds.

The team tested this system in real-world conditions during the 2025 race event, pitting their autonomous car against others on the Yas Marina Circuit. They put the system through three specific, high-stress scenarios to see how it held up. In the first test, they simulated a car suddenly stopping on the track while the autonomous vehicle approached at high speed. The system detected the stopped car from about 80 meters away and, within a few hundred milliseconds, correctly estimated that it was not moving, giving the planning software enough time to brake or swerve safely. In the second scenario, they tested how the system handled a "blind spot" situation where one car blocked the view of another. When the blocking car moved, the system immediately spotted the newly revealed vehicle and began tracking it without hesitation, proving it could handle the sudden appearance of obstacles. Finally, they tested a side-by-side overtake, where two cars raced inches apart. While the system showed a slight bias in estimating the exact position of the other car depending on whether it was in front or behind, the overall accuracy was sufficient to complete the maneuver without a collision.

The results of these tests confirmed that combining multiple sensors is far superior to relying on just one. When the researchers removed the LiDAR sensors from the equation, the system's ability to judge distance dropped significantly. When they removed the RADAR, the system struggled to guess the speed of other cars accurately. Only when all three types of sensors worked together did the system achieve the best balance of precision and range. The study concludes that for autonomous vehicles to operate safely at extreme speeds, they must not only see the world through many eyes but also think about the world in a way that accounts for the physics of racing. While the system is not perfect—it still struggles slightly with the exact geometry of a car when viewed from the side—it represents a significant step forward. It proves that by fusing different types of data and understanding the context of the race, machines can learn to navigate the most dangerous and demanding environments on Earth.

Technical Summary: A Multi-Modal Perception Pipeline for Object Detection and Tracking in Autonomous Racing

Problem Statement
Autonomous racing presents an extreme operational domain for perception systems, characterized by speeds approaching 80 m/s, high vibrations, and minimal safety margins. In such environments, even minor latency (e.g., 100 ms) translates to significant spatial displacement errors (approx. 8 meters), severely impacting collision avoidance and overtaking safety. Existing literature often fails to simultaneously exploit all available sensor modalities (Cameras, LiDAR, RADAR) due to computational constraints or pipeline complexity. Furthermore, many state-of-the-art approaches lack explicit mechanisms to compensate for detection delays or fail to incorporate racing-specific prior knowledge (e.g., track layout and vehicle dynamics) into the tracking model. These gaps are critical in scenarios involving high relative velocities, sensor failures, and adverse conditions.

Methodology
The authors propose a unified, late-fusion perception pipeline deployed on the Dallara EAV-24 autonomous race car. The system integrates data from seven cameras, three LiDARs, and four RADARs.

Sensor-Specific Detection:
- Cameras: Utilize a YOLOv4 model, pre-trained on BDD100k and fine-tuned on a custom dataset of open-wheel race cars, optimized via tkDNN for an inference time of ~4.5 ms.
- LiDAR: Employs the PointPillars model, pre-trained on KITTI and fine-tuned on ~25,000 manually labeled race point clouds, achieving ~8.4 ms inference.
- RADAR: Uses a filtering and clustering approach based on Radar Cross-Section (RCS), Signal-to-Noise Ratio (SNR), and track boundaries, processing data in under 3 ms.
Late Fusion:
The system employs a late-fusion strategy where independent detections are temporally synchronized (Precision Time Protocol) and spatially aligned via offline calibration. A matching process associates detections from overlapping fields of view:
- Camera-LiDAR: Projects LiDAR points onto 2D bounding boxes to determine 3D position and distance via histogram analysis.
- Camera-RADAR & LiDAR-RADAR: Fuses bounding boxes with RADAR clusters to augment spatial data with radial velocity information.
Delay-Aware Multi-Object Tracking:
The core tracking module utilizes an Extended Kalman Filter (EKF) operating in Frenet coordinates relative to precomputed racing lines.
- State Representation: The state vector $[s, d, v_s, v_d]$ represents longitudinal/lateral position and velocity relative to the racing line.
- Dynamics Model: A modified constant-velocity model incorporates control inputs and time constants to converge toward a speed profile derived from the racing line, embedding prior knowledge of expected acceleration/deceleration.
- Delay Compensation: The system explicitly handles asynchronous detections by reverting tracked object states to the detection timestamp using a history buffer, associating new measurements, and then projecting states forward to the current time. This ensures temporal consistency despite latency.
- Lifecycle Management: New objects are initialized with ego-vehicle velocity and high covariance, requiring multiple confirmations (3–5) before being considered active. Duplicate objects are pruned based on proximity.

Key Contributions

Unified Multi-Modal Architecture: A complete pipeline that fuses Cameras, LiDAR, and RADAR in a late-fusion framework, addressing the lack of simultaneous multi-sensor exploitation in current racing stacks.
Explicit Delay Compensation: A tracking mechanism that mathematically corrects for detection latencies, a critical feature for high-speed scenarios where uncorrected delays cause meter-scale errors.
Dynamics-Informed Tracking: The integration of racing line priors and vehicle dynamics into the EKF process model, allowing the tracker to anticipate realistic speed profiles (e.g., braking for corners) rather than relying solely on constant-velocity assumptions.
Robustness to Sensor Failure: The late-fusion approach ensures continuity of tracking even if specific sensors fail, as remaining sensors can sustain object detection.

Experimental Results
The pipeline was evaluated using real-world data from the 2025 Abu Dhabi Autonomous Racing League (A2RL) at the Yas Marina Circuit, using shared ground-truth localization data for benchmarking.

Ablation Study: A 315-second high-speed chase sequence demonstrated that the full multi-modal configuration (Camera + LiDAR + RADAR) provided the best overall balance. While LiDAR alone offered high spatial precision and RADAR alone improved velocity estimation, the fused system achieved the lowest velocity RMSE (0.77 m/s), highest lateral accuracy (0.41 m), and longest tracking duration (313.1 s). The study confirmed that LiDAR is critical for depth, RADAR for velocity, and Cameras for range continuity.
High Relative Speed Overtaking: In a scenario where the ego-vehicle approached a stationary object at ~60 m/s, the system detected the object at ~80 m. The tracker converged to a near-zero velocity estimate within 350 ms, providing the planning module sufficient time (Time-to-Collision of 1 s at 57 m) to execute a safe evasive maneuver.
Occlusion Handling: During an overtaking maneuver where a previously occluded vehicle suddenly appeared, the system initialized tracking within two iterations, demonstrating low latency in reacting to newly visible obstacles.
Side-by-Side Overtaking: In a close-proximity pass (1.75 m lateral distance), the system maintained sufficient accuracy to complete the maneuver. However, the study noted a longitudinal bias (up to 0.84 m) dependent on the relative position of the opponent (underestimating when ahead, overestimating when behind) due to the lack of semantic awareness regarding which part of the vehicle is observed.

Significance and Claims
The paper claims that the proposed pipeline validates the necessity of combining multi-modal sensing with delay-aware tracking and dynamics-informed priors for reliable perception in extreme-speed environments. The authors assert that their approach successfully supports safe and adaptive planning decisions in critical racing scenarios, including high-speed overtaking and obstacle avoidance.

The authors remain modest regarding limitations, acknowledging that estimation accuracy is influenced by the relative position of the tracked object and the current lack of semantic understanding of vehicle geometry (e.g., distinguishing front vs. rear). They conclude that while the system is effective for high-speed planning, future work must address these biases by incorporating semantic-aware vehicle geometry models to refine state estimation. The work positions autonomous racing as a valuable experimental field for developing robust perception systems applicable to broader autonomous driving challenges, particularly regarding adverse conditions and sensor failures.

More like this