Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models

Autonomous vehicles rely on a constant, high-speed stream of data to understand the world around them. While cameras capture the visual richness of a scene, they can struggle in poor weather or low light. To compensate, many self-driving systems use LiDAR, a sensor that fires rapid pulses of laser light to measure distances and build a three-dimensional map of the environment. This map appears as a cloud of millions of tiny points, each representing a surface the laser hit. To make this data useful for navigation, computers must learn to identify what those points belong to—is that cluster a pedestrian, a car, or a tree? Furthermore, because the vehicle is moving, the system must track these objects over time, connecting the dots from one moment to the next to understand motion and intent. This process, known as four-dimensional segmentation, is the backbone of safe autonomous driving, but it faces a massive hurdle: the data required to teach these systems is incredibly difficult and expensive to create.

For a computer to learn how to recognize objects in a LiDAR cloud, humans must first manually label thousands of frames, drawing outlines around every car and person in every single point cloud. This is a slow, tedious, and error-prone task. Because the points are sparse and scattered in three-dimensional space, labeling them is far less intuitive than drawing on a flat photograph. As a result, the progress of these systems is often bottlenecked not by the intelligence of the algorithms, but by the sheer lack of high-quality, labeled data. Researchers have long wondered if there is a way to generate this training data automatically, perhaps by borrowing the visual understanding that computers have already learned from billions of images and videos.

A team of researchers from KAIST and Chung-Ang University has taken a significant step toward answering that question with a new framework called LiDAR-SAM2. Their work demonstrates that it is possible to create a powerful labeling tool for 3D LiDAR data without a single human drawing a line on a point cloud. Instead of relying on manual annotation, they built a system that translates the capabilities of a video-based artificial intelligence model into the language of 3D laser scans. The core idea is to use a foundation model, a type of AI trained on vast amounts of video data to track objects across time, and adapt it to understand the unique geometry of LiDAR sensors. By doing so, they turned a 2D video expert into a 4D LiDAR supervisor.

The process begins with a synchronized set of data: a sequence of 3D LiDAR scans paired with standard video footage from cameras mounted on the same vehicle. The researchers first feed the video footage into the video foundation model, which is excellent at identifying objects and following their movement from frame to frame. When a user clicks on a car in the first video frame, the model automatically tracks that car through the entire video sequence, creating a mask that outlines the vehicle in every subsequent frame. The researchers then take these 2D outlines and project them onto the corresponding 3D LiDAR points. Because the camera and the laser sensor are calibrated to know exactly how they relate to each other, the system can determine which laser points correspond to the pixels in the video mask.

However, simply projecting 2D masks onto 3D points is not enough. A single camera cannot see the entire vehicle; it might see the side but miss the back, or the view might be blocked by another object. To solve this, the researchers developed a method to combine the views from multiple cameras and stitch them together over time. They use the known movement of the vehicle to project what was seen in the previous moment into the current moment, filling in the gaps where the camera view was limited. This creates a dense, consistent set of labels for the 3D points, effectively generating a high-quality training dataset automatically. These generated labels, which the researchers call pseudo-labels, are then used to train a new model specifically designed to understand LiDAR data.

The resulting system, LiDAR-SAM2, acts as an interactive tool. A human operator needs only to click on an object in the first frame of a LiDAR sequence, and the system automatically generates a consistent, high-quality track for that object throughout the entire video. The researchers tested this approach on a standard dataset called SemanticKITTI, which contains complex urban driving scenes. They found that the labels generated by their system were remarkably accurate, achieving high precision and recall rates. When they used these automatically generated labels to train standard 3D segmentation models, the models performed nearly as well as those trained on data that had been painstakingly labeled by human experts. In fact, for semantic segmentation tasks, the models trained on the automatic labels achieved performance levels that approached the full human-annotated ground truth, despite the fact that no human ever manually labeled the LiDAR points used for training.

The study also explored the quality of the labels themselves. By visualizing the output, the researchers showed that the system produces clean boundaries around objects and maintains consistent identities for cars and pedestrians as they move through the scene. This consistency is crucial for autonomous driving, where a system must know that the car it saw ten seconds ago is the same car it sees now. The researchers demonstrated that this level of quality could be achieved with minimal human input, requiring only a few clicks per object to initialize the tracking. This suggests that the heavy burden of data annotation for 3D and 4D scene understanding can be significantly reduced, potentially allowing for the rapid development of safer and more capable autonomous systems without the need for massive teams of human labelers.

The work explicitly challenges the notion that high-quality 3D training data must always come from human labor. While previous attempts at interactive 4D segmentation relied on human-annotated datasets to teach the system how to track objects, this new approach minimizes that dependency by requiring only minimal prompts. The researchers argue that by leveraging the temporal understanding already present in video foundation models, it is possible to bootstrap a 3D labeling system from scratch. They showed that their method is not just a theoretical possibility but a practical tool that produces usable, high-fidelity data. The results indicate that the gap between automated and human-labeled data is closing rapidly, offering a scalable path forward for the next generation of perception systems.

In the broader context of artificial intelligence, this research highlights a shift toward using one type of powerful model to teach another. By treating a 2D video model as a source of supervision for 3D data, the team bypassed the traditional limitations of the 3D domain. They did not invent a new way to see the world; instead, they found a way to translate what a computer already knows about video into the language of laser scans. The findings suggest that the future of autonomous driving data may not lie in hiring more annotators, but in building smarter bridges between different types of visual data. As the technology matures, it could enable the creation of vast, diverse datasets for self-driving cars, making the technology more robust and accessible. The study concludes that with the right framework, the dream of automatic, high-quality 3D annotation is no longer a distant goal, but a present reality.