Technical Summary: Rethinking Infrastructure Inspection as Image Difference Classification

Problem Statement

Road infrastructure maintenance traditionally relies on manual visual inspections, a process that the Road Digital Twin (DT) framework aims to digitalize. However, the adoption of AI-driven inspection is significantly hindered by the scarcity of annotated data, a common challenge in Architecture, Engineering, and Construction (AEC) fields. Existing approaches for traffic sign maintenance often suffer from two critical limitations:

1. Dataset Limitations: Many existing datasets contain only close-ups of sign faces (lacking posts and backgrounds) or utilize coarse condition labeling (e.g., binary damaged/undamaged or single-defect types). None currently support condition tracking over time using paired imagery.
2. Task Modeling Limitations: Current methods typically treat defect identification as single-image classification tasks. They fail to model the relational nature of continuous asset monitoring, where an inspection image is compared against a historical reference of the same asset. Furthermore, they often ignore the multi-class, multi-label reality where multiple defects occur simultaneously.

Methodology

This work reformulates defect identification as an Image Difference Classification (IDC) task. Instead of classifying a single image in isolation, the system compares an inspection image against a reference image of the same asset taken previously. This approach leverages existing asset images in road asset management systems (AMS) as references, reducing the need for additional annotation.

Dataset

The authors curated a new, high-quality dataset from the UK National Highways AMS, addressing previous gaps:

• Scope: Contains 970 image pairs (reference + inspection) capturing the full asset context (sign face, post, and background).
• Granularity: Features fine-grained, multi-label annotations across nine condition categories (e.g., tilted post, deformed face, vandalism, dirt) plus a "no defect" flag.
• Balance: While class imbalance exists among specific defect types, the dataset maintains a balance between damaged and undamaged signs (~4:6).

Experimental Setup

The study evaluated two tasks: binary defect presence detection and multi-class multi-label defect classification.

• Low-Resource Simulation: Performance was tested under few-shot learning conditions (0 to 8 examples per class) to simulate data-constrained environments.
• Pipeline Architectures:
  • Encoder-Based Pipelines: Four variations using vision encoders (MetaCLIP2 as the backbone). These included linear classifiers on concatenated embeddings, cross-attention mechanisms between images, and image-text fusion architectures.
  • Instruction-Based Pipelines: Utilized Vision Language Models (VLMs), specifically Qwen3-VL-8B. Inspection and reference images were inserted into user prompts with instructions to use the reference as a guide, while textual class descriptions were provided in system prompts.
• Control: To ensure fair comparison, all pipelines used the same training images. Reference images in the training set were treated as pseudo "no defect" examples for all models, ensuring performance gains were due to the context of the reference during inference, not an artificial increase in training data volume.

Key Results

1. Defect Presence Detection

• Performance: Most pipelines achieved high F1 scores (~0.9) even with very few examples, as binary detection is a simpler task.
• Reference Utility: Encoder-based pipelines showed inconsistent results when adding reference images, often performing worse than single-image baselines. In contrast, the instruction-based pipeline consistently outperformed all encoder-based models, achieving an F1 > 0.9 with just 1-shot.
• Calibration: Untrained instruction-based models (0-shot) performed poorly (F1 ~0.007), classifying almost all images as defective. However, fine-tuning on just one example per class "calibrated" the model, raising performance to 0.905.

2. Defect Classification (Multi-class Multi-label)

• Complexity: Scores were lower overall due to task complexity. Encoder-based pipelines struggled to exceed an F1 of 0.5, while the instruction-based pipeline reached just over 0.6.
• Reference Utility: The instruction-based pipeline consistently gained from the inclusion of reference images across all shot settings (gains ranging from +0.008 to +0.038). Notably, its 1-shot performance exceeded the 8-shot performance of encoder-based pipelines.
• Statistical Significance: A paired t-test confirmed that the gains from reference images in the instruction-based pipeline were statistically significant at 1-shot and 8-shot settings (p < 0.01), though less significant at 2 and 4 shots due to few-shot instability.

3. Ablation Studies

• Model Size: Increasing the encoder backbone size from 2B (MetaCLIP2) to 7B (DINOv3) did not bridge the performance gap. The instruction-based model (8B) still outperformed the larger encoder, suggesting parameter count is not the sole determinant of success.
• Pseudo Defects: Removing the "pseudo no defect" training strategy from the instruction-based pipeline still resulted in performance gains over single-image baselines, confirming the robustness of the IDC approach.

Significance and Claims

The paper claims that Image Difference Classification (IDC) is an effective task modeling strategy for continuous infrastructure inspection and Road DT asset condition updating. The primary contributions are:

1. Data Efficiency: IDC allows for the utilization of existing historical asset images as references, reducing reliance on large, newly annotated datasets.
2. Task Formulation: It successfully addresses the multi-class, multi-label nature of real-world defects and the temporal aspect of condition tracking.
3. Model Architecture: The study demonstrates that instruction-based VLMs are superior to encoder-based models for this specific task, provided they undergo a minimal "calibration" fine-tuning step (as few as one example per class).

The authors conclude that while few-shot learning introduces some instability, the ability of instruction-based classifiers to leverage reference images for grounded classification makes IDC a viable solution for data-constrained infrastructure monitoring. Future work is suggested to test larger datasets to stabilize few-shot fluctuations and explore the use of multiple reference images per sign.