4DLidarOpen: An Open 4D FMCW Lidar Dataset for Motion-Aware Autonomous Driving

Technical Summary of 4DLidarOpen: An Open 4D FMCW Lidar Dataset for Motion-Aware Autonomous Driving

Problem Statement
As autonomous driving systems evolve toward Level 3 and beyond, the field is shifting toward end-to-end (E2E) learning paradigms that integrate perception, prediction, and planning. However, existing large-scale datasets (e.g., KITTI, nuScenes, Waymo) primarily rely on conventional Time-of-Flight (ToF) Lidar and cameras. These sensors provide strong geometric measurements but lack direct, instantaneous observability of object motion. Consequently, current systems must infer motion indirectly through frame-to-frame association or heuristic feature aggregation, which often dilutes temporal consistency and degrades performance in complex, high-speed, or densely interactive scenarios. There is a critical gap in public benchmarks for 4D scene understanding that leverages direct radial velocity measurements to support motion-aware perception and planning.

Methodology and Dataset Construction
To address this gap, the authors introduce 4DLidarOpen, a large-scale, open multi-modal dataset centered on 4D Frequency-Modulated Continuous-Wave (FMCW) Lidar sensing. The dataset was collected in complex urban environments in Beijing, covering diverse scenarios including dense pedestrian interactions, congested traffic, high-speed driving, and unprotected maneuvers.

• Sensor Configuration: The data collection vehicle is equipped with a synchronized suite of five Lidars and five surround-view cameras. The core sensor is a forward-facing 4D FMCW Lidar that provides point-wise $(x, y, z, v)$ data, where $v$ represents instantaneous radial velocity. This is complemented by a 360° rotating OT Lidar (for high-quality geometric reference), a solid-state AT Lidar (for high-density forward scanning), and two ATX blind-spot Lidars. All sensors are hardware-synchronized with sub-millisecond temporal alignment (within $\pm 1.39$ ms) and jointly calibrated.
• Annotation Strategy: The dataset employs a hybrid annotation approach. It includes 225 human-annotated scenarios (167 training, 58 validation) with 3D bounding boxes and persistent track IDs across five object categories (Car, Van, Pedestrian, Cyclist, Traffic Cone). To support scalable self-supervised learning, the authors also release three tiers of auto-labeled data containing up to 2,000 sequences. The annotation pipeline utilizes an auto-labeling system refined by human experts, ensuring high fidelity for objects within 150 meters.
• Privacy: All faces and license plates are automatically blurred to comply with privacy regulations.

Key Contributions

1. Dataset Release: The authors release 4DLidarOpen, the first large-scale corpus coupling 4D FMCW Lidar data with high-level perception and E2E driving annotations. It integrates heterogeneous Lidar streams (FMCW, rotating, solid-state, blind-spot) with synchronized cameras and 6-DOF ego-poses.
2. Unified Benchmarks: The paper establishes standardized benchmarks for three critical tasks: 3D object detection, Bird's-Eye View (BEV) segmentation and flow prediction, and motion forecasting coupled with path planning.
3. Empirical Analysis of Velocity: The authors provide a systematic evaluation quantifying the role of point-wise radial velocity. They demonstrate that while 4D FMCW Lidar may have lower spatial resolution than scanning Lidars for static geometry, its direct velocity measurements offer complementary motion cues that significantly enhance dynamic scene understanding.

Experimental Results
The authors evaluated the dataset using state-of-the-art models (e.g., CenterPoint, Sparse4D, MotionNet, PriorMotion, SparseDrive, DiffusionDrive) across different sensor configurations.

• 3D Object Detection: In terms of raw detection metrics (mAP/NDS), the solid-state AT128 Lidar achieved the highest performance, followed by the rotating OT128, with the 4D FMCW Lidar showing slightly lower raw detection scores. However, ablation studies revealed that incorporating the radial velocity channel ($v$) into the 4D FMCW input significantly improved detection performance, particularly for dynamic vulnerable road users (e.g., pedestrian AP increased from 17.59% to 48.28% when velocity was added).
• BEV Segmentation and Flow Prediction: Models utilizing 4D FMCW data with velocity fusion (e.g., PriorMotion) achieved the lowest motion prediction errors across static, slow, and fast regimes. The velocity channel notably reduced static cell errors by an order of magnitude, indicating superior ego-motion compensation and background stabilization.
• Motion Forecasting and Planning: In a counter-intuitive finding, the 4D FMCW Lidar configuration outperformed both the AT128 and OT128 in end-to-end motion forecasting and planning tasks, despite its lower raw detection accuracy. The FMCW configuration achieved the lowest Average Displacement Error (ADE) and Final Displacement Error (FDE). The authors attribute this to the direct velocity measurements enabling more robust motion initialization and reducing multi-modal ambiguity in trajectory prediction, which is critical for long-horizon planning.

Significance and Claims
The paper claims that 4DLidarOpen provides a reproducible foundation for advancing motion-aware autonomous driving. The authors argue that while traditional scanning Lidars excel at dense geometric reconstruction, 4D FMCW Lidar offers a distinct advantage in dynamic-scene understanding by providing direct, instantaneous motion cues. This capability allows autonomous agents to detect dynamic objects earlier (e.g., detecting a rolling cone or darting pedestrian in the first frame rather than after several frames of accumulation) and plan more robustly.

The authors conclude that 4D FMCW Lidar is a promising sensing modality that complements existing sensors. They suggest that future systems should consider co-designing sensing modalities with motion-aware architectures to fully leverage the unique capabilities of direct velocity measurement, moving beyond static scene parsing toward true 4D scene understanding. The dataset and evaluation toolkit are publicly released to facilitate research in 4D scene understanding, multi-Lidar fusion, and velocity-aware perception and planning.