Technical Summary: Robust Structure from Motion for Aerial-Ground Images via Detector-Free Feature Matching and Multi-View Track Refinement

1. Problem Statement

The integration of aerial and ground images is critical for generating high-fidelity, comprehensive 3D urban models. However, this task faces severe challenges due to massive variations in viewpoint, scale, rotation, and illumination between the two platforms. Traditional hand-crafted feature methods (e.g., SIFT) often fail to establish effective matches under these extreme cross-view conditions, leading to the failure of joint Structure from Motion (ISfM) reconstruction.

While deep learning-based detector-free methods (e.g., LoFTR) have shown promise in texture-poor and cross-view scenes by generating dense pixel-level correspondences, they exhibit two critical bottlenecks when applied to aerial-ground ISfM:

1. Lack of Rotational Robustness: Existing detector-free networks rely on CNN-based backbones with translation-invariant square kernels, which struggle to capture features under arbitrary rotations common in aerial-ground pairs.
2. Poor Multi-View Compatibility: These methods typically process image pairs independently, resulting in low feature repeatability across multiple views. This leads to fragmented feature tracks and disconnected sub-scenes, preventing the formation of a coherent global 3D model.

2. Methodology

The authors propose a two-part framework: a rotation-robust detector-free feature matching network and a multi-view track refinement strategy tailored for ISfM.

2.1. Rotation-Robust Detector-Free Matching Network

The network replaces traditional convolutional backbones with four key modules:

• Rotation-Robust Feature Extraction (RRFE): This module utilizes an Omnidirectional State Space Block (OSS Block). Instead of standard convolutions, it employs an Omnidirectional Selective Scan Module (OSSM) that scans features across eight symmetrical directions (horizontal, vertical, diagonal, anti-diagonal, and their inverses). This allows the network to model long-range spatial dependencies and synthesize rotation-invariant feature maps, addressing the orientation discrepancies between aerial and ground views.
• Multi-Scale Attention Transformer: To handle high-resolution images efficiently without the quadratic complexity of standard Transformers, the authors employ Quadtree Attention. This creates a hierarchical token pyramid where attention is calculated from coarse to fine scales. It isolates high-association token regions and discards irrelevant areas, capturing long-range context with linear computational complexity while preserving fine-grained details.
• Coarse Matching: A symmetric coarse-to-fine scheme is used. Coarse alignment computes dual-direction Softmax confidence matrices under mutual nearest neighbor (MNN) constraints to establish initial correspondences.
• Bidirectional Fine Matching: To overcome the ambiguity of unidirectional dependency, the network uses a bidirectional fine matching module. It crops local windows around coarse seeds, performs bidirectional self- and cross-attention, and regresses sub-pixel coordinate offsets using a Multi-Layer Perceptron (MLP). This ensures symmetric optimization and sub-pixel localization accuracy.

2.2. Multi-View Track Refinement for ISfM

To bridge the gap between pair-wise matching and global reconstruction, the authors introduce a Fine Feature Tracks Connection strategy:

• Integrated Indexing: Matching results from multiple associated image pairs are indexed by "image-feature point-confidence."
• Spatial Proximity & Confidence Selection: For feature points in an intermediate image, a spatial proximity threshold is set. If multiple points fall within this neighborhood, the one with the highest matching confidence is selected as the "anchor."
• Track Linking: Lower-confidence matches in the neighborhood are linked to the high-confidence anchor. This connects disjoint sub-tracks into continuous, multi-view tracks, ensuring stable feature repeatability without sacrificing the localization accuracy achieved by the detector-free network.

Finally, a Coarse Model Refinement stage iteratively optimizes the initial ISfM reconstruction using bundle adjustment with specific reprojection error thresholds to remove geometric outliers.

3. Key Contributions

1. Novel Network Architecture: The proposal of a detector-free network integrating an Omnidirectional State Space Block for rotation-invariant extraction and Quadtree Attention for efficient multi-scale modeling.
2. Bidirectional Refinement: The introduction of a bidirectional fine matching module that regresses sub-pixel offsets, solving the localization ambiguity inherent in traditional one-way refinement.
3. ISfM-Specific Track Refinement: A confidence-driven track connection strategy that links fragmented sub-tracks to high-confidence anchors, enabling detector-free methods to function effectively within an ISfM pipeline.
4. Comprehensive Validation: Extensive testing on five real-world aerial-ground datasets, demonstrating significant improvements over state-of-the-art methods.

4. Experimental Results

The proposed method was evaluated on five datasets (including Center, Zeche, SWJTU-LIB, SWJTU-BLD, and ShaoGuan) against benchmarks like LoFTR, ASpanFormer, SIFT, and SuperPoint.

• Pose Estimation: The proposed method improved the Area Under the Curve (AUC) at 5° pose error by 93.9% compared to LoFTR (0.4591 vs. 0.2366).
• Feature Matching: The network generated an order of magnitude more initial matches and inliers than LoFTR across all datasets. For rotated image pairs (Dataset 5), LoFTR failed completely (0 matches), while the proposed method successfully established matches.
• ISfM Reconstruction:
  • Completeness: The proposed strategy achieved complete image registration across all datasets, whereas standard detector-free methods failed to register more than 71 images in some cases due to track fragmentation.
  • Density: The method generated nearly 10 times more 3D points than SIFT-based approaches.
  • Accuracy: The final reconstruction accuracy improved by 27.6% to 32.7% compared to baselines. The mean distance error to ground-truth LiDAR was reduced significantly, with the proposed method outperforming SIFT+MNN by 33.33% in mean distance.
  • Refinement: The iterative refinement process successfully reduced reprojection errors from ~0.8 pixels to ~0.35 pixels while retaining rich scene details.

5. Significance and Conclusion

The paper claims that the proposed workflow provides a reliable solution for the integrated 3D reconstruction of aerial-ground images. By addressing the specific limitations of rotation sensitivity and multi-view track fragmentation, the method enables the use of detector-free networks in complex urban ISfM pipelines. The authors conclude that their approach achieves high-precision reconstruction without the precision loss inherent in grid quantization strategies used by previous detector-free SfM methods.

Future work suggested by the authors includes developing lightweight backbones for ultra-high-resolution images, creating end-to-end architectures that jointly optimize matching and reconstruction, and incorporating multi-modal priors (depth, semantics) to handle extreme lighting and occlusions.