Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment

1. Problem Statement

General-purpose Vision-Language Models (VLMs) have demonstrated strong capabilities in everyday domains but fail when applied to specialized technical fields like civil infrastructure. Specifically, they struggle with:

• Domain-Specific Terminology: They lack the precise engineering vocabulary required for pavement distress classification (e.g., distinguishing between "alligator" and "longitudinal" cracks).
• Structured Reasoning: They cannot adhere to strict engineering standards (such as ASTM D6433) for severity quantification and Pavement Condition Index (PCI) estimation.
• Spatial Grounding: They often fail to provide precise bounding box coordinates for distress localization.
• Workflow Integration: Existing pavement datasets (e.g., CrackSeg9k, RDD2022) are designed for unimodal tasks (detection/segmentation) and lack the instruction-response pairs necessary for multimodal reasoning, conversational assessment, and maintenance decision-making.

The core challenge is bridging the gap between general VLMs and the rigorous, standardized requirements of pavement engineering without relying on proprietary "black box" models.

2. Methodology

The authors propose a two-pronged solution: the creation of a massive, domain-specific instruction dataset (PaveInstruct) and the training of a specialized foundation model (PaveGPT).

A. PaveInstruct Dataset Construction

The dataset was created by unifying annotations from nine heterogeneous pavement datasets (including PID, PaveTrack, RDD2022, SVRDD, DSPS23/24, and PCIer), resulting in 278,889 image-instruction-response pairs.

• Integration Pipeline:
  1. Format Unification: Converted diverse annotation schemas (YOLO, Pascal VOC, color-coded folders, CSV) into a unified schema preserving spatial coordinates, semantic classification, and condition ratings.
  2. Coordinate Harmonization: Normalized bounding boxes across varying image resolutions and aspect ratios to ensure geometric consistency.
  3. Instruction Generation: Utilized a Large Language Model (LLM) guided by a Task Taxonomy covering five categories:
     • Spatial Reasoning: Object grounding, counting, ranking, and relationship analysis.
     • Condition Assessment: Severity classification, PCI estimation, and distress identification.
     • Professional Workflow: Maintenance recommendations, safety analysis, and checklist completion.
     • Reasoning: Chain-of-thought (CoT) analysis, error correction, and comparative analysis.
     • Multi-Modal Interaction: Multi-turn dialogues and multi-length captioning.
• Quality Assurance: A multi-stage validation process involving automated consistency checks and review by licensed pavement engineers to ensure adherence to ASTM D6433 standards.

B. PaveGPT Model Architecture

PaveGPT is a domain-specialized VLM built upon the Qwen2.5-VL architecture.

• Components:
  • Vision Encoder: A frozen Vision Transformer (ViT) with 2D-RoPE and window attention mechanisms to preserve spatial relationships.
  • Cross-Modal Projection: A learnable MLP that aligns visual tokens with the language model's embedding space.
  • Language Backbone: A decoder-only transformer (initialized from Qwen2.5-VL) trained via Supervised Fine-Tuning (SFT) on PaveInstruct.
• Training Strategy: The vision encoder remains frozen to preserve general visual features, while the projection layer and language backbone are fine-tuned. The model is trained to generate diverse output modalities (text, bounding boxes, numeric PCI scores) as token sequences.

3. Key Contributions

1. PaveInstruct Dataset: The first comprehensive, instruction-following dataset for pavement assessment, containing ~279k pairs across 32 task types. It bridges the gap between raw computer vision data and engineering decision-making workflows.
2. Systematic Integration Pipeline: A novel framework for unifying heterogeneous datasets with different annotation formats, coordinate systems, and task emphases into a single, semantically rich resource.
3. PaveGPT Foundation Model: A specialized VLM capable of end-to-end pavement assessment, from distress detection to maintenance recommendation, producing outputs compliant with ASTM D6433.
4. Empirical Validation: Comprehensive evidence demonstrating that domain-specific instruction tuning transforms general-purpose VLMs into expert-level tools, outperforming zero-shot baselines significantly.

4. Results and Evaluation

The authors evaluated PaveGPT and other fine-tuned baselines (LLaVA, MiniCPM, InternVL) against state-of-the-art models using metrics for spatial grounding, reasoning, and generation.

• Performance Gains: Instruction tuning resulted in improvements exceeding 20% in spatial grounding, reasoning, and generation tasks compared to zero-shot settings.
• Spatial Grounding: Fine-tuned models achieved significant increases in Mean Intersection over Union (mIoU). For example, InternVL's mIoU jumped from ~8.9% (zero-shot) to 30.5% (fine-tuned).
• PCI Estimation: Fine-tuned models achieved a Mean Absolute Error (MAE) of 13.1 (MiniCPM) and 19.3 (PaveGPT), a massive improvement over zero-shot baselines which often produced nonsensical values or failed to generate outputs.
• Reasoning and Compliance: Fine-tuned models successfully generated step-by-step Chain-of-Thought reasoning for PCI calculations and provided maintenance recommendations aligned with engineering standards.
• Qualitative Analysis:
  • Zero-shot models often hallucinated distress types, missed critical defects, or failed to follow formatting constraints.
  • Fine-tuned models correctly identified distress types, localized them with precise bounding boxes, and provided context-aware maintenance advice (e.g., distinguishing between "preventive treatment" and "corrective repair" based on pothole severity).
• Efficiency: PaveGPT demonstrated a reasonable trade-off between accuracy and computational cost, achieving a throughput of 39.7 tokens/sec with 16.81 GB memory usage on an NVIDIA A6000 GPU, making it viable for practical deployment.

5. Significance and Impact

• Unified Workflow: This approach allows transportation agencies to replace multiple specialized systems (one for detection, one for PCI, one for reporting) with a single conversational AI tool.
• Democratization of Expertise: By encoding engineering standards into the model, it reduces the reliance on highly specialized human inspectors for preliminary assessments, potentially lowering costs and speeding up infrastructure monitoring.
• Scalability: The methodology establishes a pathway for applying instruction-driven AI to other infrastructure domains, such as bridge inspection, railway maintenance, and building condition assessment.
• Standardization: The work demonstrates that AI can be trained to strictly adhere to regulatory standards (ASTM D6433), a critical requirement for adoption in public infrastructure management.

Limitations: The current dataset reflects specific geographic and climatic conditions, necessitating local fine-tuning for different regions. Additionally, the model is designed as a decision-support tool for screening rather than a replacement for final regulatory ratings, and it currently lacks temporal tracking capabilities for forecasting degradation.