Technical Summary: From UAV Images to Semantically Annotated 3D Models

Problem Statement

Unmanned Aerial Vehicle (UAV) photogrammetry is a standard method for infrastructure inspection, generating high-overlap imagery (70–90%) that is converted into 3D models. However, translating these massive image sets into actionable engineering findings faces three primary challenges:

1. Limited Semantics: Traditional deep learning detectors operate under closed-set assumptions, recognizing only pre-trained defect categories. They lack the ability to provide open-vocabulary, component-level semantic understanding (e.g., distinguishing a crack on a load-bearing girder from one on a parapet).
2. VLM Hallucinations and Redundancy: While Vision-Language Models (VLMs) offer open-vocabulary capabilities, applying them to thousands of redundant, high-overlap UAV images individually leads to "hallucinations" (fabricating non-existent defects) and prohibitive computational costs (token consumption).
3. Lack of Structural Context: General-purpose models often fail to anchor findings to specific structural components, which is critical for accurate condition assessment and maintenance decision-making.

Methodology

The paper proposes a Keypoint-Guided VLM Framework that converts high-overlap UAV imagery into an interactive, semantically annotated 3D model. The workflow consists of four stages:

1. Overview and Inputs

The system ingests a high-overlap UAV image set ($I$) and a set of expert-specified keypoints ($K$) marking inspection-critical locations on a reconstructed 3D model ($G$). The output is a semantically annotated 3D model ($M_{sem}$) containing structured annotations.

2. Keypoint-Guided Image Selection

Instead of analyzing all images, the framework selects a compact, multi-view cluster for each keypoint based on flight geometry:

• Coordinate Transformation: Camera positions and keypoints are transformed into a common metric reference frame (UTM).
• Ray-Sphere Intersection: Each image is abstracted as a viewing ray. For a keypoint $k_j$, a sphere of radius $\tau$ is defined. An image is selected if its viewing ray intersects this sphere.
• Adaptive Radius: The radius $\tau$ is derived from the flight's spatial sampling density ($\bar{\delta}$) rather than fixed empirically ($\tau = \alpha \bar{\delta}$), ensuring the selection adapts to flight speed and capture rate.
• Result: Only the most informative views observing a specific keypoint from multiple angles are retained, filtering out redundant or irrelevant imagery.

3. Multi-View VLM Analysis with Structured Prompting

The selected image clusters are analyzed jointly by a general-purpose VLM (e.g., GPT-5.5, Qwen3.7-plus) via API.

• Joint Reasoning: All images in a cluster are fed into a single query, allowing the VLM to perceive structural context and reason about the defect's location relative to the component.
• Structured Prompting: The prompt includes:
  • Role specification (infrastructure inspection expert).
  • Domain knowledge (explicitly listing false positives like water stains or shadows to reduce hallucinations).
  • Output format constraints (machine-readable JSON).
• Output: The VLM returns a tuple containing the defect name, description, component-level location, severity, maintenance suggestion, and image-space localizations of the defect instances.

4. Semantic Annotation and Interactive Visualization

The VLM outputs are fused with the georeferenced 3D model. Annotations are anchored to the 3D coordinates of the keypoints. An interactive viewer allows inspectors to navigate the model, click on keypoints to view structured reports, and retrieve the specific evidence images used for the analysis.

Key Contributions

1. Training-Free Framework: A VLM-based inspection pipeline that requires no task-specific training or fine-tuning. Adaptation to new scenarios (e.g., bridges to pagodas) is achieved solely by modifying the prompt content.
2. Keypoint-Guided Selection Mechanism: A geometric method to restrict VLM reasoning to informative multi-view clusters. This approach significantly reduces token consumption and hallucination risks compared to exhaustive analysis while improving detection performance.
3. Component-Level Semantic Understanding: The framework identifies not only the defect type but also the specific structural component it resides on, generating actionable maintenance suggestions grounded in the 3D spatial context.

Experimental Results

The framework was validated through three experiments:

• Case Study (Lixinsha Bridge): On a dataset of 1,209 images, the framework reduced the input to 44 images (3.6%) grouped around 5 keypoints. The VLM successfully identified defects (e.g., spalling on concrete piers), localized them to specific components, and provided maintenance suggestions.
• Cross-Scenario Generalization (Longfeng Tower): The framework was applied to a historic pagoda with different materials and structural forms. By modifying only the prompt (changing component vocabulary and false-positive examples), the system achieved zero-shot transfer without retraining or code changes.
• Quantitative Evaluation (1,581-image Bridge Dataset):
  • Efficiency: Keypoint-guided selection reduced the number of analyzed images by 95.00% (from 1,581 to 79) and estimated token consumption by 95.00%.
  • Performance: Precision increased from 63.72% to 86.17%, and recall (computed over the selected subset) increased from 53.59% to 79.27%.
  • Hallucination Suppression: The reduction in false predictions (from 115 to 13) indicates that removing redundant, low-information views effectively suppresses VLM hallucinations.

Significance and Claims

The paper claims that this framework offers a more accurate and efficient route to condition assessment for aging infrastructure by:

• Elevating UAV photogrammetry from purely geometric models to carriers of component-level engineering semantics.
• Providing a human-in-the-loop decision support tool that delivers not just "where" a defect is, but "what" it is, its severity, and recommended actions.
• Demonstrating that redundancy removal in high-overlap datasets is critical for VLM performance; analyzing fewer, higher-quality views yields better accuracy and lower costs than exhaustive analysis.

Limitations Acknowledged by the Authors:

• The framework relies on pre-existing, high-quality keypoints (from as-built docs or expert surveys); it does not automatically detect keypoints in unfamiliar environments.
• Coverage is confined to keypoint neighborhoods, meaning defects outside these areas will be missed.
• The reasoning process of the VLM is not fully traceable, necessitating confirmation by qualified engineers before maintenance decisions are made.