Multi-View Camera Perception System for Variant-Aware Automated Vehicle Inspection and Defect Detection

Technical Summary: Multi-View Camera Perception System for Variant-Aware Automated Vehicle Inspection

Problem Statement
End-of-line quality control in automotive manufacturing faces a dual challenge: the need to inspect a high volume of vehicles within strict "tact time" constraints (seconds per vehicle) while ensuring accuracy across a wide variety of configurable variants. A single vehicle platform may produce dozens of variants differing in features such as roof rails, antennas, wheel types, badging, and powertrains (ICE vs. EV). Traditional single-camera stations and manual visual inspections struggle to simultaneously achieve high throughput, full variant coverage, and consistency. Furthermore, existing automated systems often treat inspection tasks in isolation (e.g., detecting defects or identifying wheel types separately) and lack the ability to condition their decisions on a specific vehicle's expected configuration (VIN-derived manifest). This creates a "perception-reasoning gap" where detectors report what is visible but cannot determine if the visible features match what should be there for that specific vehicle.

Methodology: The AVI System
The paper presents the Automated Vehicle Inspection (AVI) system, an engineering integration designed to bridge this gap. The system is not defined by a novel neural network architecture but by a specific integration design comprising a fixed multi-view rig, task-specific routing, and a deterministic decision engine.

1. Hardware Setup: The system utilizes 11 synchronized cameras capturing a full 360-degree sweep of each vehicle at 3840x2160 resolution.
2. Task-to-View Routing: Instead of processing all views through all models, the system employs a static routing policy where specific cameras feed dedicated perception modules:
   • Vehicle Identification & Branding: Cameras T1 (Front) and T3 (Rear) feed a YOLOv8 detector to localize logos, mascots, and grilles. The mascot region is processed by Gemini 1.5 Flash for OCR to extract variant names, while a classical routine checks logo alignment.
   • Powertrain Classification: The front grille crop (T1) is classified as Internal Combustion Engine (ICE) or Electric Vehicle (EV) using an EfficientNet model.
   • Variant Feature Detection: A second YOLOv8 model detects configurable features (roof rails, antennas, rear wipers, wheel types). Crucially, this model only processes views where targets are physically visible (e.g., T2/T3 for roof items; L1/R1 for wheels), reducing inference cost and false positives.
   • Surface Defect Segmentation: A YOLOv8-Seg model processes all eight side-facing views (L1–L4, R1–R4) to generate pixel-level masks for scratches, dents, and paint deformities.
3. Evidence Standardization & Decision Logic:
   • Standardization Layer: Heterogeneous outputs (bounding boxes, OCR text, class probabilities, segmentation masks) are converted into a common format: boolean feature flags with confidence scores.
   • View-Aware Aggregation: For each task, the system applies a max-pooling strategy over the assigned views to determine feature presence, avoiding signal dilution from irrelevant views.
   • Deterministic Rule Engine: The system retrieves the expected feature manifest from the vehicle's VIN. It performs a set-theoretic comparison between the detected features and the expected manifest to identify missing or extra items.
   • Verdict: A final Pass/Fail decision is emitted if the set of missing features, extra features, and detected surface anomalies (above an area threshold) are all empty. This process is designed to be auditable and independent of live database queries once the vehicle enters the bay.

Key Contributions

• Targeted View Assignment: A routing policy that restricts detectors to informative views, reducing false positives and computational cost.
• Evidence Standardization and Rule-Based Decision Layer: A fusion scheme that converts diverse perception outputs into comparable boolean flags, enabling a deterministic comparison against a VIN-derived manifest.
• Data-Efficient Defect Module: A segmentation branch tailored for limited-label scenarios, enabling reliable anomaly localization on side panels.
• Comparative Benchmarking: An evaluation across 40 vehicle models using ablation studies to establish the necessity of multi-view designs and dedicated segmentation branches.

Results
The system was evaluated on a hybrid corpus: a private OEM dataset (40 models, 1250 vehicles in pilot deployment) and public scratch/dent datasets.

• Performance: The full AVI system achieved 95% end-to-end verification accuracy and 86.5% defect detection recall (specifically for scratches and dents).
• Throughput: The system processes vehicles at a rate of 3.3 vehicles/minute (approx. 18 seconds per vehicle), with a per-vehicle processing latency of 285 ms, well within the production tact time.
• Ablation Findings:
  • Single-view configurations (e.g., Front-only T1) dropped verification accuracy to 88–93% and feature coverage to ~28–43%, confirming the necessity of multi-view capture.
  • Removing the segmentation branch reduced defect detection recall from 86.5% to 71.4%, despite a slight latency reduction, proving the segmentation branch is an operational requirement.
• Model Selection: Comparative testing indicated YOLOv8 and YOLOv8-Seg as the strongest off-the-shelf backbones for the respective tasks, outperforming Faster R-CNN, YOLOv5, and YOLOv7 in precision, recall, and mAP/mIoU.

Significance and Claims
The paper claims its primary significance lies in the integration design rather than algorithmic novelty. By decoupling probabilistic perception from deterministic reasoning, the system achieves an auditable, industrial-grade pipeline.

• Bridging the Gap: It addresses the "is it correct" problem by validating visual evidence against a specific vehicle manifest, distinguishing between a valid missing feature (base trim) and a defect (premium trim missing a feature).
• Deployability: The architecture allows perception modules to be retrained or swapped without altering the decision logic, making the system robust to variant evolution and maintenance.
• Operational Reality: The system demonstrates that for industrial deployment, the bottleneck is often the integration of multi-modal evidence rather than raw detector precision. The use of established backbones (YOLOv8, EfficientNet) simplifies retraining and maintenance compared to heavy monolithic transformers.

The authors explicitly note limitations, including the static nature of the view-to-task mapping (which does not adapt to temporary occlusions) and the reliance on a closed manifest (which cannot detect unknown anomalies). Future work is suggested to address adaptive view attention and open-world anomaly detection.