💻 computer science

Investigating Vision-Language Model for Point Cloud-based Vehicle Classification

This study proposes a novel framework that adapts vision-language models for point cloud-based heavy-duty truck classification by integrating real-world LiDAR data with specialized preprocessing and few-shot in-context learning, thereby reducing annotation costs while enhancing safety in cooperative autonomous driving.

Original authors: Yiqiao Li, Jie Wei, Camille Kamga

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiqiao Li, Jie Wei, Camille Kamga

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

On the road, the largest vehicles are often the most dangerous. Heavy-duty trucks carry immense weight and occupy significant space, yet they cannot stop as quickly as a car and have large areas around them that a driver simply cannot see. For the future of safe, automated driving to work, the computers guiding these vehicles need to understand exactly what kind of truck they are facing and how that truck is moving. Traditionally, teaching a computer to recognize these vehicles has required a slow, expensive process where humans manually label thousands of images or sensor scans. However, a new approach is emerging that borrows from the way large language models have learned to understand text and images, aiming to teach machines to recognize trucks with far less human effort.

Researchers at the City College of New York have developed a method to help computers identify heavy-duty trucks using data from roadside sensors. These sensors, known as LiDAR, work by sending out laser pulses to measure distances, creating a cloud of points that represents the shape of objects in the real world. While this technology is excellent for seeing the physical world, the data it produces looks very different from the photographs that most modern artificial intelligence systems are trained to understand. The new study bridges this gap by taking the raw, scattered points from the laser sensors and transforming them into a format that a vision-language model can read. These models are advanced computer programs capable of understanding both pictures and words, but they usually expect to see standard photographs, not laser point clouds.

To make the laser data usable, the team first combined multiple snapshots of a truck as it passed the sensor. A single snapshot is often too sparse to show clear details, so the researchers aligned several frames together to build a denser, more complete picture of the vehicle. They then smoothed out the image, removing small errors and noise to create a clean outline of the truck. Once the data was prepared, they used a technique called "few-shot prompting." Instead of spending months teaching the computer with thousands of labeled examples, they showed the model just a handful of examples—sometimes as few as three—along with a description of what to look for. This allowed the model to learn the task quickly by observing the pattern in the few examples provided, rather than requiring a massive, pre-built database of every possible truck type.

The researchers tested this system using real-world data collected from a major highway ramp in Southern California, where trucks travel between Northern and Southern California. The sensors captured vehicles moving at speeds ranging from zero to fifty miles per hour under both free-flowing and congested traffic conditions. The goal was to see if the system could correctly identify twelve different categories of vehicles, including various types of semi-trailers, tankers, and passenger cars. When they compared the results of their processed images against the raw, unprocessed data, the improvement was clear. The system performed best when it was shown three examples before being asked to classify a new truck. With this small amount of guidance and the cleaned-up images, the model achieved a high level of accuracy, correctly identifying the vehicle type in more than half of the cases on average.

The study found that while the method worked well for many truck types, it faced challenges with vehicles that look very different from one another, such as flatbed trucks that might be empty or carrying oddly shaped loads. These variations made it harder for the model to find a single pattern to follow. Nevertheless, the results suggest that this approach could significantly reduce the time and cost required to train safety systems for autonomous driving. By using the powerful reasoning capabilities of modern language models and adapting them to work with laser sensor data, the researchers demonstrated that accurate vehicle classification is possible without the massive datasets that have traditionally been required. This work represents a preliminary step toward making roads safer by giving automated systems a better, more efficient way to understand the heavy vehicles sharing the highway.

Technical Summary: Investigating Vision-Language Model for Point Cloud-based Vehicle Classification

Problem Statement

Heavy-duty trucks present significant safety challenges in cooperative autonomous driving due to their large size, weight, and limited maneuverability. Traditional LiDAR-based classification methods for these vehicles rely on hand-crafted feature extraction and extensive manual annotations, making them labor-intensive, costly, and often lacking generalizability across different road environments. While Vision-Language Models (VLMs) have demonstrated remarkable few-shot learning capabilities in image-based tasks, they are primarily trained on 2D image datasets. Consequently, applying these models directly to sparse and scattered 3D LiDAR point cloud data poses a significant technical challenge.

Methodology

This study proposes a novel framework that integrates roadside LiDAR point cloud data with VLMs to facilitate efficient vehicle classification. The methodology consists of three primary stages:

1. Point Cloud Data Processing

To address the sparsity of single-frame LiDAR data, the study employs a multi-step processing pipeline:

Segmentation and Tracking: Vehicle point clouds are segmented using background subtraction and DBSCAN clustering. Cross-frame tracking is achieved using the SORT algorithm with Kalman filtering and the Hungarian algorithm for assignment.
Point Cloud Registration: To generate dense 3D renderings suitable for VLMs (which expect RGB-like inputs), a probabilistic-based pairwise point cloud registration approach is applied. This aligns vehicle objects between consecutive frames by minimizing point-to-point distances, refined by a point-to-plane strategy to enhance precision as vehicles approach the sensor.
Image Optimization: The reconstructed point clouds undergo statistical outlier removal and mathematical morphological operations (opening: erosion followed by dilation). These techniques smooth contours, eliminate noise, and preserve the overall vehicle structure, resulting in a cleaner 2D representation.

2. Few-Shot Prompting with In-Context Learning (ICL)

Instead of fine-tuning model parameters, the study utilizes In-Context Learning (ICL) with few-shot prompting. The VLM is guided by embedding task demonstrations (image examples paired with class labels) directly into the input prompt. This allows the pre-trained model to infer the desired classification task without costly parameter updates. The approach frames the problem as predicting the class with the highest probability score given a query image and a set of $k$ demonstration examples.

3. Experimental Setup

The framework was evaluated using the Gemini 1.5 VLM. Experiments were conducted on a dataset collected from the I-5S freeway entrance ramp in Southern California, featuring 12 fine-grained vehicle classes (11 truck categories and 1 passenger vehicle category). The study compared the performance of the proposed processed images against original projected 2D point cloud images across varying shot settings (1-shot to 9-shot).

Key Contributions

The paper outlines three specific innovations:

Real-World Data Utilization: The use of roadside LiDAR sensor data to capture detailed point cloud representations, ensuring practical applicability in real-world scenarios.
Systematic Adaptation Pipeline: A proposed method to adapt point cloud data for VLM input, combining point cloud registration for dense rendering and morphological techniques for feature enhancement.
Few-Shot Classification Strategy: The introduction of a few-shot prompting approach that enables VLMs to classify heavy-duty trucks with minimally labeled training data, significantly reducing the need for extensive manual annotation.

Experimental Results

The study evaluated performance using F1 scores (the harmonic mean of Recall and Precision) across 12 vehicle classes:

Impact of Processing: The proposed image processing method consistently outperformed the use of original images. The best overall performance was achieved with a 3-shot setting on processed images, yielding an average F1 score of 0.53. This represents a 15% improvement over the 3-shot performance on original images (0.46).
Class-Specific Performance: Four vehicle classes (Bobtail, Tank Tank, Enclosed Van (SU), and Passenger Vehicle) achieved F1 scores greater than 0.60 with the 3-shot processed approach.
Challenges: Classes with high intraclass variability, such as "Platform (SU)," "Low Boy Platform," and "Pickup/Utility/Service," exhibited lower F1 scores. This is attributed to the diverse shapes of commodities and load configurations, which make distinctions difficult for few-shot learning techniques to capture.
Efficiency: Unlike traditional deep learning approaches (e.g., PointNet) that require large training datasets, this method achieved an F1 score greater than 0.50 with only 3 shots, demonstrating a significant reduction in labeling costs.

Significance and Claims

The authors position this work as a preliminary effort to transfer the representational power of VLMs directly to LiDAR-based images for heavy-duty truck classification. They claim this is the first study of its kind to apply VLMs in this specific domain, observing encouraging results on a real-world dataset.

The paper modestly frames these findings as a starting point for practical utilities. It suggests that while the current results are promising, future work will involve:

Comparing this approach with traditional few-shot implementations.
Integrating visual deep nets (e.g., YOLO) and retrieval-augmented generation (RAG) systems.
Utilizing Low-Rank Adaptation (LoRA) for fine-tuning.
Exploring Agentic workflows and Chain-of-Thought prompting to further leverage VLM capabilities for traffic safety monitoring and content understanding.