ArchSym: Detecting 3D-Grounded Architectural Symmetries in the Wild

Technical Summary: ArchSym: Detecting 3D-Grounded Architectural Symmetries in the Wild

1. Problem Statement

Symmetry is a critical geometric prior for computer vision tasks such as 3D reconstruction, pose estimation, and scene completion. However, existing learning-based methods for detecting 3D reflectional symmetries from single RGB images face two major hurdles:

• Dataset Gap: Most current models are trained on object-centric, synthetic datasets (e.g., ShapeNet), which lack the complexity, occlusions, and varying illumination found in real-world "in-the-wild" scenes.
• Scale Ambiguity: Because monocular input lacks absolute depth information, localizing a 3D symmetry plane (determining its exact position/offset) is an ill-posed problem. Most existing works only predict the plane's orientation (normal vector) rather than its full 3D localization.

2. Methodology

The authors propose a two-pronged approach: a novel automated data annotation pipeline and a new single-view symmetry detector.

A. Automated Annotation Pipeline (ArchSym Dataset):
To overcome the lack of real-world data, the authors leverage the "doppelganger" problem in Structure-from-Motion (SfM) reconstructions—where image matchers incorrectly match visually similar but physically distinct parts of a scene.

• Cross-View Matching: Instead of just matching an image to its own reflection (within-view), the pipeline matches an image against the reflected versions of other visually similar images from the same scene (cross-view). This allows the recovery of symmetries not fully visible from a single viewpoint.
• Geometric Extraction: The pipeline uses a dense matcher (MASt3R), unprojects 2D correspondences into 3D points using SfM depth/camera data, and fits a symmetry plane by minimizing the distance between matched 3D points and their reflected counterparts.
• Aggregation: Candidate planes are clustered using DBSCAN, and high-confidence clusters are used as ground-truth annotations.

B. Single-View Symmetry Detector:
The model is built upon a frozen 3D foundation model (VGGT) to leverage its pre-trained understanding of scene geometry.

• Symmetry as Signed Distance Maps (SDM): To resolve scale ambiguity, the model does not regress plane parameters directly. Instead, it predicts a dense Signed Distance Map. For every pixel, the model predicts the distance from that pixel's predicted 3D point to the ground-truth symmetry plane. This grounds the symmetry in the model's own predicted geometry, ensuring scale consistency.
• Two-Stage Architecture:
  1. Symmetry Identification Module: A transformer decoder uses learnable instance queries to identify potential symmetry planes at a global level.
  2. Dense Map Synthesis Module: A Dense Prediction Transformer (DPT) head, conditioned by the instance queries via FiLM (Feature-wise Linear Modulation) layers, generates the actual dense SDMs and confidence maps.
• Inference: At test time, the model uses the predicted SDMs and the VGGT point map to solve a constrained least-squares optimization problem, extracting the final 3D plane parameters (normal and offset).

3. Key Contributions

• ArchSym Dataset: The first large-scale, automated dataset of in-the-wild architectural symmetries (93 landmark scenes, ~34k images).
• Scale-Consistent Parameterization: A novel method of predicting symmetries via signed distance maps relative to predicted geometry, solving the monocular scale ambiguity problem.
• End-to-End Framework: The first framework capable of detecting 3D-grounded symmetries in complex, real-world environments.

4. Results

• Quantitative Performance: The model significantly outperforms state-of-the-art baselines (REFLECT3D and a direct regression baseline). It achieves superior Exactness (angular error of normals), Completeness (detecting visible symmetries), and Dense Symmetry Error (full-plane alignment).
• Qualitative Robustness: The model demonstrates strong generalization across different architectural eras, extreme viewpoints, and diverse cultural styles (beyond Western landmarks). It is robust to occlusions and varying lighting.
• Downstream Application: The authors demonstrate that the detected symmetries can be used for single-view point cloud completion, effectively "hallucinating" the occluded or unseen sides of a building by reflecting the visible geometry across the detected symmetry plane.

5. Significance

This work bridges the gap between synthetic object-level symmetry detection and real-world scene understanding. By providing both a massive new benchmark (ArchSym) and a robust detection architecture, the paper enables the use of symmetry as a reliable geometric prior for large-scale 3D reconstruction and augmented reality applications in complex, unconstrained environments.