🤖 machine learning

RadarGen: Automotive Radar Point Cloud Generation from Cameras

RadarGen is a diffusion-based framework that synthesizes realistic automotive radar point clouds from multi-view camera imagery by leveraging BEV-aligned visual, semantic, and motion cues to bridge the domain gap for multimodal generative simulation.

Original authors: Tomer Borreda, Fangqiang Ding, Sanja Fidler, Shengyu Huang, Or Litany

Published 2026-08-14

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomer Borreda, Fangqiang Ding, Sanja Fidler, Shengyu Huang, Or Litany

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. You can show it a million hours of video, and it will learn to recognize stop signs and pedestrians just like a human. But a self-driving car doesn't just "see" with eyes; it also "feels" the world with invisible waves. These waves, called radar, bounce off objects to tell the car how far away they are, how fast they are moving, and even how hard they are made of. It's like the car is using sonar to build a 3D map of the road, but instead of sound, it uses radio waves.

The problem is that while we have endless videos of the world, we don't have enough of these "radar feelings" to teach the robot properly. Recording real radar data is slow, expensive, and messy. It's like trying to teach someone to play the piano by only letting them listen to the keys being pressed, but never letting them hear the music. Scientists have been trying to use computers to imagine what the radar data should look like based on the video, but so far, the computer's imagination has been a bit fuzzy and inaccurate. It's like trying to guess the texture of a rock just by looking at a photo of it; you might get the shape right, but you'll miss the roughness or the speed of a rolling stone.

This is where a new team of researchers steps in with a tool called RadarGen. Think of RadarGen as a magical "radar translator." It takes a set of normal camera photos of a street scene and uses a special type of AI (called a diffusion model) to dream up what the radar data should look like for that exact moment. It doesn't just guess the location of objects; it also invents realistic details about how fast they are moving and how much of the radar signal they bounce back.

The researchers found that by teaching their AI to look at the world from a "bird's-eye view" (a top-down map) and by using other smart AI tools to understand depth and motion, they could create radar data that is surprisingly close to the real thing. When they tested this fake radar data on a car's navigation system, the system could actually "see" and react to the objects, just as if the data were real. It's not perfect yet—the computer still struggles a bit in very dark or tricky situations—but it suggests that we might soon be able to generate unlimited, realistic radar training data just by editing videos, saving us from the need to record every single scenario on the road.

The Magic of RadarGen

The Problem: The Missing Sense
Self-driving cars are like superheroes with multiple senses. They have cameras (eyes) to see colors and shapes, and they have radar (a special kind of touch) to sense distance and speed, even in the dark or rain. But here's the catch: while we have mountains of video data to train the "eyes," we have very little radar data. Why? Because recording real radar is hard. It requires special, expensive cars to drive around and capture the invisible radio waves bouncing off the world. Plus, the raw data is so huge and complicated that most companies only save the final, processed version, which loses a lot of the fine details.

Because of this shortage, the "radar sense" of our self-driving cars is often undertrained. It's like trying to teach a pianist to play a concerto using only sheet music but never letting them hear the actual sound. The car might know where a car is, but it might not know how fast it's going or if it's a slippery truck or a bouncy car.

The Solution: Dreaming Up Radar
The authors of this paper, RadarGen, decided to fix this by teaching a computer to imagine the radar data. They built a system that looks at standard camera photos and says, "Okay, based on what I see here, what would the radar 'feel' be?"

To do this, they didn't just look at the photos; they used a clever trick. They turned the radar data into a format that looks like a map, specifically a Bird's-Eye-View (BEV) map. Imagine looking down at a city from a helicopter. Instead of seeing the cars as 3D objects, you see them as dots on a flat grid.

The Map: The system creates three layers on this map:
1. Where the dots are: The location of the objects.
2. How "loud" they are: This is called Radar Cross Section (RCS), which tells us how reflective the object is (a big truck reflects more than a small car).
3. How fast they are moving: This is the Doppler value, which tells us the speed relative to the car.

How It Works: The "Dreaming" Machine
The team used a powerful AI model called a diffusion model. You can think of this like a sculptor who starts with a block of noise (static) and slowly chips away the noise to reveal a statue.

The Input: The system takes camera photos from the front, back, and sides of the car.
The Clues: Before it starts "dreaming," it uses other smart AI tools to figure out the depth (how far away things are), the type of object (is it a car or a tree?), and the motion (is it moving?). It projects all these clues onto that same bird's-eye-view map.
The Generation: The diffusion model takes these clues and starts "denoising" a blank map. It slowly fills in the map with dots that look like real radar returns. Because it's a probabilistic model, it doesn't just guess one answer; it creates a variety of realistic possibilities, just like real radar sometimes has "noise" or missing dots.
The Recovery: The result is a smooth, blurry map. The system then uses a mathematical trick (called deconvolution) to sharpen this map back into specific, sharp dots, creating the final point cloud.

What They Found
The team tested RadarGen on a dataset of real truck driving scenes (the MAN TruckScenes dataset). They compared their generated radar data against real radar data and against a simpler, older method (a baseline).

Better Accuracy: RadarGen was much better at guessing where the objects were and how fast they were moving. In their tests, the "geometric fidelity" (how close the shape was to reality) was significantly higher than the baseline.
Realistic Stats: The fake radar data looked statistically similar to real radar data. The distribution of speeds and reflectivity matched the real world much better than previous attempts.
It Works for Driving: The biggest test was whether a real self-driving car could use this fake data. They trained a detector (a program that finds cars) on real radar data and then asked it to find cars in the generated radar data. The detector found 66% of the cars in the generated data (a metric called Hit Rate), whereas the baseline detector barely found anything. This suggests that the generated data is good enough to actually help train self-driving cars.

The "What If" and the "What Not"
The paper also showed that this system is flexible. If you take a photo of a street and use an image editor to swap a small car for a giant truck, RadarGen updates the radar data instantly. It removes the radar dots where the small car used to be and adds new dots for the truck, even handling the fact that the truck might block (occlude) things behind it. This proves the system understands the 3D world, not just the 2D picture.

However, the paper is careful to note what this isn't.

It's not magic: The system relies on the quality of the camera photos and the other AI tools (like the depth estimator). If the camera is in the dark or the other AI gets confused, RadarGen will make mistakes.
It's not perfect: While it's better than before, the generated data still isn't exactly the same as real data. The paper notes that the detection quality on generated data is still lower than on real data, suggesting there are subtle details the AI hasn't fully captured yet.
No raw signals: The system generates the final "dots" (point clouds), not the raw radio waves. The authors explain that generating the raw waves would require massive amounts of raw data that isn't publicly available, so they stuck to the processed point clouds.

Why It Matters
The main takeaway is that RadarGen offers a scalable way to create radar data. Instead of needing to drive thousands of miles in every possible weather condition to record radar, we might soon be able to generate that data from videos. This could speed up the development of safer self-driving cars, allowing them to "practice" in millions of virtual scenarios without ever leaving the lab. It's a step toward a future where self-driving cars have a complete, multi-sensory understanding of the world, even when the real world is too dangerous or expensive to explore.

Technical Summary: RadarGen

Problem Statement

Autonomous driving simulation has advanced significantly in the visual and LiDAR domains, yet radar remains an underexplored frontier in generative modeling. While radar is ubiquitous in production vehicles due to its low cost and weather resilience, current neural simulators cannot reproduce its distinctive sensing characteristics, such as signal sparsity, Radar Cross Section (RCS), and Doppler velocity.

Generating radar data from vision presents unique challenges:

Stochasticity: Radar measurements are highly stochastic due to multipath reflections, interference, and material-dependent scattering, making deterministic predictions insufficient.
Data Representation: Unlike LiDAR, which can be reshaped into dense range images, radar detections are sparse, unordered 3D point sets derived from peak-based extraction (e.g., CFAR). They lack the dense, grid-aligned structure required by standard image-based diffusion models.
Data Availability: Large-scale datasets typically provide only processed point clouds (after proprietary signal processing like range-Doppler transforms and beamforming), discarding raw phase information. Consequently, models must learn to generate sparse point clouds with RCS and Doppler attributes directly from camera imagery.

Methodology: RadarGen

RadarGen is a probabilistic diffusion framework designed to synthesize realistic automotive radar point clouds from multi-view camera inputs. The method consists of three primary stages:

1. Radar Representation as Bird's-Eye-View (BEV) Images

To leverage efficient image-latent diffusion models, RadarGen converts sparse radar point clouds into dense, image-like BEV representations.

Projection: Radar points are projected onto the BEV plane, discarding elevation (which carries little discriminative information in automotive setups).
Map Construction: Three single-channel BEV maps are constructed:
- Point Density Map ( $M_p$ ): Created by convolving the sparse point map with a Gaussian kernel ( $K_\sigma$ ) to produce a smooth density estimate.
- RCS Map ( $M_r$ ) and Doppler Map ( $M_d$ ): Constructed via Voronoi tessellation, where each pixel inherits the attribute value (RCS or Doppler) of the nearest radar detection.
Encoding: These maps are replicated across three channels to fit the RGB input space of a pretrained autoencoder, generating latent representations ( $z_p, z_r, z_d$ ) for the diffusion model.

2. Conditional Generation with Foundation Models

RadarGen utilizes SANA, an efficient image-latent diffusion model based on a Diffusion Transformer (DiT) with linear attention, to model the conditional distribution $p(z_p, z_r, z_d \mid c)$ .

Conditioning Cues: To guide the stochastic generation toward physically plausible patterns, the model is conditioned on BEV-aligned cues extracted from pretrained foundation models:
- Depth: Metric depth estimation (e.g., UniDepthV2) provides geometric structure.
- Semantics: Semantic segmentation (e.g., Mask2Former) provides categorical context.
- Motion: Optical flow (e.g., UniFlow) is backprojected to estimate radial velocity, serving as a proxy for Doppler cues.
Architecture: The conditioning maps are concatenated with the radar latent tokens. Learnable modality identifiers are added to preserve the distinct statistics of the density, RCS, and Doppler channels while allowing cross-modal correlation.

3. Sparse Point Cloud Recovery

The diffusion model outputs dense latent BEV maps. To recover the final sparse point cloud:

Deconvolution: The generated Point Density Map is treated as a blurred version of the underlying sparse point map. An explicit inverse problem is solved using an L1-regularized non-negative deconvolution (LASSO) via an Iteratively Reweighted L1 (IRL1) solver with a FISTA optimizer.
Attribute Retrieval: Once the sparse 2D coordinates are recovered, the corresponding RCS and Doppler values are retrieved from the generated $M_r$ and $M_d$ maps at those locations.

Key Contributions

First Probabilistic Radar Diffusion Framework: RadarGen is the first framework to generate realistic automotive radar point clouds (including location, RCS, and Doppler) from multi-view camera inputs using a probabilistic diffusion approach.
BEV Latent Diffusion Methodology: The paper introduces a novel methodology that trains on a BEV image representation of sparse radar attributes, conditioned by BEV-aligned visual priors (depth, semantics, motion) from foundation models.
Comprehensive Evaluation Metrics: The authors establish a dedicated suite of metrics for radar point cloud generation, assessing geometric fidelity (Chamfer Distance, IoU), radar attribute fidelity (Distance-Attribute Recall/Precision/F1), and distribution similarity (MMD).
Scene Editing Capability: The framework supports controllable generation, allowing for scene editing (e.g., object removal or replacement) where the radar response updates semantically and geometrically consistent with the edited visual scene.

Experimental Results

The method was evaluated on the MAN TruckScenes dataset (and nuScenes in supplementary materials), comparing RadarGen against a baseline feedforward model (RGB2Point adapted for radar).

Quantitative Performance: RadarGen broadly outperforms the baseline across all metrics. Notable improvements include:
- Geometric Fidelity: Lower Chamfer Distance (CD Loc: 1.68 vs 1.84) and higher IoU@1m (0.31 vs 0.23).
- Attribute Fidelity: Higher Distance-Attribute (DA) F1 scores (0.24 vs 0.14).
- Distribution Similarity: Significantly lower Maximum Mean Discrepancy (MMD) for location, RCS, and Doppler, indicating the generated data better matches the statistical distribution of real radar.
Downstream Perception: A VoxelNeXt detector trained on real data achieved an NDS of 0.30 on RadarGen's synthetic data, compared to nearly zero on the baseline's output. This demonstrates that the generated radar points are interpretable by perception models trained on real data.
Ablation Studies: Removing any of the conditioning maps (Appearance, Semantic, or Velocity) degraded performance, with the Semantic map being most critical for geometric fidelity and RCS distribution. The BEV conditioning approach was found to be more geometrically accurate and computationally efficient (2 days training vs. 9 days for a direct multi-view conditioning baseline) than conditioning directly on raw multi-view images.

Significance and Claims

The paper positions RadarGen as a step toward unified generative simulation across sensing modalities. By bridging the gap between visual inputs and radar outputs, the method offers a scalable direction for multimodal generative simulation.

Scalability: Conditioning on images allows the approach to leverage existing large-scale visual datasets and simulation frameworks, enriching them with realistic radar data without the need for costly re-recording or manual setup.
Controllability: The ability to modify radar responses via image editing (e.g., replacing a car with a truck) enables the creation of diverse training scenarios for perception systems, including handling occlusions and novel object geometries.
Limitations: The authors modestly note that performance is tied to the capabilities of upstream foundation models (struggling in low-light or heavily occluded scenes) and that the model can "hallucinate" points in regions not visible to cameras. Furthermore, the method does not model explicit physical radar mechanisms (like raw waveforms) due to the lack of large-scale raw data, focusing instead on the statistical properties of processed point clouds.