PolyBuild: An End-to-End Method for Polygonal Building Contour Extraction from High-Resolution Remote Sensing Images

Technical Summary: PolyBuild

Problem Statement
Extracting vectorized building polygon contours from high-resolution remote sensing images is critical for mapping, urban modeling, and reconstruction. However, this task faces significant challenges due to varying imaging conditions, complex building structures, and obstructions like trees and shadows. Mainstream approaches typically rely on pixel-level semantic segmentation followed by post-processing vectorization. These methods often produce rough, jagged contours, struggle with occlusions, and fail to provide direct vector information. While contour-based methods offer a more direct approach by regressing contour vertices, existing solutions suffer from limitations: some use overly simplistic initial contours (e.g., bounding boxes or ellipses) that are far from the true target, while others rely on limited center-point features that lead to error accumulation as regression distances increase. Furthermore, many methods lack the ability to capture global spatial relationships between contour vertices, resulting in inaccurate regression offsets.

Methodology
The paper proposes PolyBuild, an end-to-end framework that directly extracts building vector polygons without requiring post-processing. The architecture consists of two primary modules:

1. Initial Contour Generation Module (ICGM):

   • Instead of relying on a single building center point, the ICGM first predicts a bounding box (bbox) for each building instance and divides it into four sub-regions (top-left, top-right, bottom-left, bottom-right).
   • It extracts features from the centers of these four sub-regions and concatenates them.
   • These coupled features are used to regress a set of vertex offsets. The offsets are added to the respective sub-region center points to generate the initial contour.
   • This approach leverages richer building information and reduces the regression distance compared to single-center methods, resulting in a more stable and precise initial outline.
   • The module utilizes a DLA-34 backbone, a center prediction head (trained with Focal Loss), a bbox parameter prediction head (trained with Smooth L1 Loss), and an offsets prediction head.

2. Contour Optimization Module (COM):

   • The initial contours generated by the ICGM are often rough and require refinement. The COM employs a hybrid CNN-Transformer architecture to iteratively optimize the contour over three iterations.
   • Coarse Adjustment: A preliminary global adjustment is performed using concatenated features of contour vertices and sub-region centers.
   • Fine Optimization: The core of the COM uses a Transformer decoder. It takes local contour features (extracted via circular convolution) and vertex position information as input. The Transformer's self-attention mechanism allows the model to query global vertices and establish dependencies between vertices at different positions, capturing both local and global spatial relationships.
   • Loss Functions: The first two optimization iterations are supervised using Smooth L1 loss. The third iteration utilizes Dynamic Matching Loss (DML), which aligns predicted vertices with the nearest ground truth boundary points and specifically targets corner vertices to ensure topological accuracy.

Key Contributions

• End-to-End Framework: PolyBuild enables the parallel implementation of building detection and contour extraction, eliminating the need for separate post-processing steps.
• Novel Initial Contour Generation: The paper introduces the ICGM, which utilizes coupled sub-region center features rather than a single center point. This strategy provides richer contour information and significantly reduces the complexity of subsequent optimization.
• CNN-Transformer Integration: The Contour Optimization Module (COM) innovatively integrates CNNs for local feature extraction and Transformers for global dependency modeling. This allows the network to automatically query and index associated vertices, correcting contour errors more accurately than methods relying solely on fixed convolution kernels or RNNs.
• State-of-the-Art Performance: Extensive experiments demonstrate that PolyBuild outperforms existing mask-based and contour-based methods across multiple datasets.

Experimental Results
The authors evaluated PolyBuild on three datasets: the WHU aerial building dataset, the WHU-Mix (Vector) building dataset, and the Crowd AI dataset.

• WHU Aerial Dataset: PolyBuild achieved an Average Precision (AP) of 74.0% and Average Recall (AR) of 80.7%, surpassing the second-best contour-based method (Line2Poly, 73.8% AP) and significantly outperforming mask-based methods like Mask R-CNN (65.3% AP).
• WHU-Mix Dataset: On the challenging, diverse WHU-Mix dataset, PolyBuild achieved 53.5 mAP and 62.2 mAR, outperforming the best mask-based method by 0.6 AP and the best contour-based method by 0.9 mAP. It maintained high accuracy on Test Set II (geographically disjoint from training), demonstrating strong generalization.
• Crowd AI Dataset: PolyBuild achieved 60.6% AP, outperforming other methods including PolyMapper (55.7% AP).
• Ablation Studies: Results confirmed that the ICGM strategy (concatenating four sub-region features) yields the most accurate initial contours. Furthermore, the inclusion of the COM module significantly improved final contour accuracy, with the full model showing a 2.0% increase in AP over the baseline.
• Efficiency: Despite the complex architecture, PolyBuild has a parameter count of only 30.3M and achieves an inference speed of 30.46 FPS on 512×512 images using an RTX 2080 Ti GPU.

Significance and Claims
The paper claims that PolyBuild addresses the fundamental limitations of current building extraction methods by providing a direct, end-to-end solution for vector polygon generation. By moving away from pixel-level segmentation and single-point regression, the method achieves higher precision and robustness, particularly in handling complex building shapes and occlusions. The authors highlight that the integration of CNN and Transformer architectures allows for a better understanding of vertex positions and their interdependencies, leading to high-quality boundary delineation. The method is presented as a significant advancement for downstream surveying and mapping applications requiring accurate, instance-level building contours. The authors acknowledge that while PolyBuild is robust, its performance is still dependent on the initial object detection stage, and future work may explore integrating RNNs with Transformers to further improve the reconstruction of heavily occluded regions with complex topologies.