⚡ electrical engineering

An Open-Source Two-Stage Computer Vision Pipeline for Fine-Grained Vehicle Classification using Vision Transformers

This paper introduces an open-source, two-stage computer vision pipeline combining RT-DETR and Vision Transformers to accurately classify six fine-grained vehicle body types relevant to cyclist safety, featuring a confidence-based abstention mechanism that maintains high robustness across different recording sites while preventing silent misclassifications.

Original authors: Gandhimathi Padmanaban, Fred Feng

Published 2026-06-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gandhimathi Padmanaban, Fred Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to sort a massive pile of mail. Some letters are clearly marked "Newspaper," some are "Magazine," and some are "Junk Mail." But you also have thousands of envelopes that look like they could be anything: a birthday card, a bill, a flyer, or a letter from a lawyer. If you just guess, you might make mistakes.

This paper describes a new, open-source "robot sorter" designed to look at video footage of streets and sort vehicles into very specific categories, not just the broad ones we usually see.

Here is how the system works, broken down into simple steps:

1. The Problem: Why We Need a Better Sorter

Currently, most traffic cameras and safety studies only know the difference between a "car," a "truck," or a "bus." But for cyclist safety, this isn't enough.

The Analogy: Imagine a cyclist getting hit. If a small sedan hits them, it's bad. But if a tall SUV or a massive commercial truck hits them, the injury is often much worse because the front of the vehicle is higher up and hits the cyclist differently.
The Gap: We need to know exactly what kind of vehicle is passing a cyclist (e.g., is it a Minivan or a Large Van? Is it a Pickup or a Commercial Truck?). Existing tools can't tell the difference between these specific types in real-world, messy video.

2. The Solution: A Two-Stage "Detective Team"

The authors built a two-step system, like a team of two detectives working together.

Detective #1: The Fast Scout (RT-DETR)

Job: This detective scans the video quickly. It doesn't need to know the exact model of the car; it just needs to say, "Hey, there's a vehicle there!" and draw a box around it.
Specialty: It's very good at spotting anything that looks like a car, truck, bus, or motorcycle. It uses a pre-trained brain, so it didn't need to be taught how to spot cars from scratch.
Filter: If it sees a bus or motorcycle, it stops there. If it sees a "car" or "truck," it passes the job to Detective #2.

Detective #2: The Expert Classifier (Vision Transformer)

Job: This detective takes the specific "crop" (the picture inside the box) from Detective #1 and looks at it very closely.
Specialty: This is the fine-grained expert. It sorts the vehicle into one of six specific categories:
1. Passenger Car
2. SUV
3. Pickup Truck
4. Minivan
5. Large Van
6. Commercial Truck
The "I Don't Know" Button: This is the most important part. If the picture is blurry, the angle is weird, or the vehicle is partially hidden, this detective might feel unsure. Instead of guessing and potentially making a mistake, it has a rule: "If I'm less than 60% sure, I will say 'Unknown'."
- Why this matters: In safety research, it's better to say "I don't know" than to confidently say "That's a pickup truck" when it's actually a minivan. This prevents "silent errors."

3. The Training: How They Taught the Robot

Teaching a robot to tell the difference between a "Large Van" and a "Commercial Truck" is hard because there are very few pictures of those specific trucks in standard datasets.

The Mix: The researchers built a custom training library by mixing three things:
1. Studio Photos: High-quality photos of cars from a famous dataset (Stanford Cars).
2. Web Scraping: They gathered pictures of vans and trucks from the internet.
3. Real Street Cuts: They took actual video clips from Ann Arbor, Michigan, and cut out the vehicles to show the robot what real, messy street conditions look like (blurry, sideways, partially hidden).
The Imbalance: The robot saw way more pictures of regular cars and SUVs than big trucks. To fix this, the training process was tweaked to pay extra attention to the rare truck pictures so the robot wouldn't ignore them.

4. The Results: How Well Did It Work?

The team tested this robot on two different sets of video.

Test 1: The "Home Court" (In-Distribution)

The Setup: They tested it on video from the same street where they took the training clips (Ann Arbor).
The Score: It got 94% correct.
The Details: It was amazing at spotting SUVs (97% accuracy). It was also very good at passenger cars and pickups. The only time it struggled was when the vehicle was hard to see, and it correctly used its "I don't know" button instead of guessing wrong.

Test 2: The "Away Game" (Out-of-Distribution)

The Setup: They tested it on video from a completely different location, taken by a special research bicycle with different cameras and lighting. They did not retrain the robot for this new location.
The Score: It got 89% correct.
The Details: This is a very strong result for a system that wasn't retrained for a new place.
- SUVs and Pickups stayed very accurate.
- Minivans got a bit trickier. The accuracy dropped, but mostly because the robot became more cautious. It started saying "Unknown" more often (25% of the time) because the minivans looked different in the new video. It didn't start guessing wildly; it just admitted uncertainty.

5. Why This Matters

It's Open Source: The authors released all the code, the trained robot brain, and the data for free. Anyone can download it and use it to analyze their own street videos.
Safety First: By distinguishing between a small car and a large truck, city planners and safety researchers can finally understand exactly which types of vehicles are putting cyclists at risk.
No Manual Work: Before this, humans had to watch hours of video to count these vehicles. Now, this tool can do it automatically.

In short: The paper presents a smart, free tool that acts like a two-step filter. First, it finds the cars. Second, it carefully sorts them into specific types. If it's not sure, it admits it, ensuring that safety data isn't corrupted by bad guesses. It works well even on new streets without needing to be retrained.

Technical Summary: An Open-Source Two-Stage Computer Vision Pipeline for Fine-Grained Vehicle Classification

Problem Statement
The paper addresses a critical gap in cyclist safety research: the lack of automated tools capable of classifying vehicles into injury-risk-relevant categories from naturalistic roadway video. While standard object detection benchmarks (e.g., COCO) provide coarse labels (car, truck, bus, motorcycle), these are insufficient for safety analyses that require distinguishing between specific body types, such as passenger cars versus SUVs or commercial trucks. Existing fine-grained recognition systems are typically trained on controlled, frontal imagery (e.g., manufacturer photos) and have not been rigorously evaluated for robustness across different recording sites or naturalistic conditions involving motion blur, occlusion, and variable lighting. Furthermore, no open-source, end-to-end pipeline exists that integrates real-time detection with fine-grained classification for deployment on existing roadside video archives without custom infrastructure.

Methodology
The authors propose a modular, two-stage computer vision pipeline designed for deployment on standard roadside video:

Stage 1: Coarse Vehicle Localization
- Model: A pre-trained RT-DETR (Real-Time DEtection TRansformer) detector, specifically the rtdetr_r50vd_coco_o365 checkpoint.
- Function: Scans full-resolution video frames to detect all vehicles and assigns coarse COCO category labels (car, truck, bus, motorcycle).
- Filtering: The system retains only "car" and "truck" detections for further processing. Bus and motorcycle detections are retained with their coarse labels.
- Constraints: A confidence threshold of 0.35 filters out low-confidence detections. A size gate excludes bounding boxes where the shorter side is <5.5% of the frame height or area is <0.04% of the frame, preventing the second stage from processing vehicles too distant for meaningful texture analysis.
Stage 2: Fine-Grained Classification
- Model: A fine-tuned Vision Transformer (ViT-Base/16), pre-trained on ImageNet-21k.
- Input: Cropped images of vehicles detected in Stage 1.
- Output: One of six specific body-type categories: passenger car, SUV, pickup truck, minivan, large van, and commercial truck.
- Training Strategy: The model was trained on a composite dataset (16,581 images) combining Stanford Cars (mapped to 4 classes), web-scraped imagery (for underrepresented classes), and field crops from the Ann Arbor study site. To address severe class imbalance (e.g., commercial trucks represent <1% of data), the authors employed focal loss with inverse frequency weighting and a weighted random sampler.
- Abstention Mechanism: A confidence-based gating mechanism is applied to the Stage 2 softmax output. If the maximum predicted probability falls below a class-specific threshold of 0.60, the system outputs an "unknown" label rather than a low-confidence prediction. This is designed to prevent silent misclassifications in ambiguous scenarios.

Key Contributions

Novel Vocabulary and Pipeline: The first open-source system explicitly trained and evaluated on a six-category body-type taxonomy relevant to injury risk, moving beyond coarse COCO labels or make-and-model identification.
Confidence-Based Abstention: The implementation of a thresholding mechanism that converts uncertain predictions into "unknown" labels, providing a calibrated measure of model uncertainty rather than forcing erroneous classifications.
Rigorous Evaluation: The study provides a large-scale evaluation on 3,805 annotated overtaking events from a bicycle-lane corridor (in-distribution) and an independent out-of-distribution (OOD) evaluation on 311 events from a separate instrumented bicycle dataset, without retraining.
Open-Source Release: The full pipeline, including inference scripts, training code, evaluation utilities, and fine-tuned model weights, is released to support reproducibility.

Results

In-Distribution Performance: On the Ann Arbor dataset (3,805 events), the pipeline achieved an overall accuracy of 0.94. Per-class F1 scores ranged from 0.91 (minivan) to 0.97 (SUV). The dominant errors involved confusion between visually similar categories (e.g., passenger car vs. SUV).
Out-of-Distribution Performance: On the independent instrumented bicycle dataset (311 events), the pipeline achieved an overall accuracy of 0.89 without any retraining or threshold adjustment.
- Three of the four well-represented categories (SUV, pickup truck, passenger car) maintained F1 scores $\ge$ 0.90.
- Minivan Degradation: The most significant drop occurred in the minivan category (F1: 0.91 $\to$ 0.72). The authors attribute this primarily to a rise in the abstention rate (from 2.4% to 25.0%) rather than active misclassification, indicating the confidence mechanism successfully propagated model uncertainty.
Minority Classes: Commercial trucks and large vans showed high uncertainty due to sparse sample sizes in both training and evaluation sets, though the model demonstrated the ability to classify them when present.

Significance and Claims
The paper claims that a modular, coarse-to-fine architecture combining a general-purpose detector with a fine-tuned Vision Transformer can achieve robust fine-grained vehicle classification on naturalistic roadway video without requiring bounding-box annotations for the detection stage or site-specific retraining for deployment.

The significance of this work lies in its application to cyclist safety research. By enabling automated, scalable annotation of vehicle body types, the pipeline allows researchers to quantify exposure to specific vehicle types (e.g., SUVs vs. passenger cars) that are disproportionately associated with severe injuries. This granularity is currently absent in standard traffic monitoring and crash databases. Additionally, the confidence-based abstention mechanism offers a safety-critical feature for automated monitoring systems, allowing them to degrade gracefully by flagging uncertain events for human review rather than producing silent errors. The open-source nature of the tool is intended to lower barriers for researchers studying vulnerable road user exposure and traffic composition.