🤖 machine learning

Beyond Segmentation: Structurally Informed Facade Parsing from Imperfect Images

This paper proposes a structurally informed facade parsing method that enhances YOLOv8 with a custom alignment loss to enforce grid-consistent bounding box arrangements, thereby improving structural coherence for procedural reconstruction while maintaining detection accuracy on imperfect images.

Original authors: Maciej Janicki, Aleksander Plocharski, Przemyslaw Musialski

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maciej Janicki, Aleksander Plocharski, Przemyslaw Musialski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a beautiful, old apartment building. It has rows and rows of windows, all supposed to be perfectly aligned, like soldiers standing in formation.

Now, imagine you are a robot trying to draw a map of these windows based on a single photo. The problem is, the photo isn't perfect. Maybe a tree is blocking some windows, maybe the photo was taken from an angle (making the building look tilted), or maybe the camera lens is a bit warped.

The Problem: The "Messy" Robot
Standard AI detectors (like the ones currently used in self-driving cars or security cameras) are like very fast, but slightly careless, students. They look at each window individually.

"I see a window here!"
"I see another one there!"
"Oh, that one is half-hidden by a branch, I'll guess where it is."

Because they look at each window in isolation, they often make small mistakes. One window might be drawn slightly too high, another slightly too wide, or a whole row might look like a wobbly line instead of a straight one. If you try to use this messy drawing to build a 3D model of the building later, the computer gets confused and the result looks glitchy.

The Solution: The "Strict Architect" Loss
The authors of this paper decided to teach the AI a new trick. They didn't want to replace the robot; they just wanted to give it a strict architect to whisper in its ear while it learns.

They added a special rule (called a "loss term") to the AI's training process. Think of it like a teacher correcting a student's homework:

The Old Way: The teacher just checks if the student found the right windows.
The New Way: The teacher says, "Okay, you found the windows, but look at this row. If Window A is at height 10, and Window B is right next to it, Window B must also be at height 10. If you draw them crooked, you lose points."

This rule forces the AI to care about the group as much as the individual. It encourages the windows to snap into a perfect grid, even if the photo makes them look a little messy.

How It Works (The Analogy)
Imagine you are trying to arrange a set of books on a shelf, but the shelf is slightly tilted and some books are covered by a blanket.

Standard AI: It places the books where it thinks they are based on what it can see. The result is a slightly messy pile.
This New Method: It adds a magnetic force between the books. Even if the blanket hides a book, the magnetic pull from the books next to it says, "You belong right here, in line with us!" The AI learns to ignore the visual noise (the tilt, the blanket) and reconstruct the logical truth of the building.

The Trade-Off: Perfection vs. Reality
The paper admits there is a catch. If you make the "architect" too strict, the AI might start forcing windows into a perfect line even when they shouldn't be there, or it might miss a tiny window because it doesn't fit the pattern.

The authors found a "sweet spot." By adjusting a dial (a weight parameter), they can tell the AI: "Be 80% focused on finding every single window, and 20% focused on making them look like a perfect grid." This gives them the best of both worlds: a map that is accurate enough to be real, but structured enough to be useful for building 3D models.

Why Does This Matter?
This isn't just about drawing pretty pictures. It's about Digital Twins and Construction.

If you want to turn a photo of a building into a 3D video game asset or a CAD drawing for renovation, the computer needs to know that the windows are in a grid.
If the input is messy, the 3D model breaks.
This method cleans up the input before the 3D model is built, making the whole process smoother and more reliable.

In a Nutshell
The paper introduces a simple but powerful "nudge" for AI. It teaches the AI to stop looking at architectural elements as isolated objects and start seeing them as part of a structured, repeating pattern. It's like teaching a robot to understand that a building isn't just a collection of random holes; it's a symphony of aligned windows, and the robot should play the music correctly, even if the sheet music (the photo) is a bit torn.

1. Problem Statement

Reconstructing architectural facades from single images is a critical task for urban modeling, digital twins, and procedural generation. While recent neuro-symbolic methods (e.g., FaçAID) can convert facade segmentations into editable procedural programs, they are heavily bottlenecked by the quality of the visual input.

The Core Issue: Standard object detectors (like YOLOv8 or Mask R-CNN) treat architectural elements (windows, doors) as independent objects optimized for local appearance.
The Consequence: This independence leads to outputs that are visually plausible but structurally inconsistent. Common artifacts include:
- Small boundary jitter across repeated elements.
- Gaps created by occlusion (e.g., trees or balconies).
- Systematic drifts caused by perspective distortion or relief effects.
Impact: When these inconsistent detections are fed into downstream procedural reconstruction models, the ambiguity increases, making it difficult to infer the logical grid structure of the building.

2. Methodology

The authors propose a detector-side training strategy that injects geometric priors directly into the detection pipeline without altering the inference process.

Base Model

The method utilizes YOLOv8, a state-of-the-art single-stage, anchor-free detector. The baseline training objective includes:

Complete Intersection over Union (CIoU) loss for bounding boxes.
Binary Cross Entropy (BCE) for classification.
Distribution Focal Loss (DFL) for precise box regression.

The Proposed Solution: Alignment Loss

The authors introduce a custom, lightweight pairwise alignment regularizer added to the standard YOLOv8 loss function. This term encourages bounding boxes of the same class to form a consistent grid structure.

Mechanism:
1. The loss operates on the set of positive bounding boxes selected by the YOLO task-aligned assigner during training.
2. Pair Selection: A pair of boxes $(i, j)$ $(i, j)$ is considered for alignment if:
  - They belong to the same class.
  - They do not overlap (IoU $\approx$ 0).
  - The absolute difference in their $x_1$ (left) and $x_2$ (right) coordinates is below a threshold $T$ (for x-axis alignment; analogous logic applies to y-axis).
3. Loss Calculation: If conditions are met, the loss penalizes the misalignment:
  $L_{ij} = |x_{i1} - x_{j1}| + |x_{i2} - x_{j2}|$
4. Total Loss: The final alignment loss ( $L_{align}$ ) is the normalized sum of losses over all valid pairs in both axes, weighted by a hyperparameter $W$ .
Key Design Choice: The method does not impose an explicit grammar or change the inference pipeline (e.g., Non-Maximum Suppression remains standard). It simply biases the training process toward grid-consistent detections.

3. Key Contributions

Geometric Regularity Injection: A novel training objective that injects facade-specific geometric priors (alignment and repetition) into bounding box predictions via a pairwise alignment term.
Seamless Integration: The method integrates into standard YOLOv8 training without modifying the inference procedure, ensuring compatibility with existing workflows.
Controllable Trade-off: The introduction of a weight parameter ( $W$ ) allows users to explicitly control the balance between standard detection accuracy (mAP) and structural regularity.

4. Experimental Results

The method was evaluated on the CMP Facade Dataset (689 cropped facade images).

Metrics

mAP@0.5: Standard detection accuracy.
SVD-based Regularity Metric: A novel metric based on Singular Value Decomposition. It measures how well the detected window masks can be approximated by low-rank matrices. A lower score indicates higher structural regularity (fewer components needed to describe the grid).

Quantitative Findings

Regularity vs. Accuracy: Increasing the alignment weight $W$ significantly lowers the SVD metric (improving regularity) but causes a gradual decline in mAP.
Optimal Balance: Moderate values of $W$ allow for a significant improvement in structural regularity with minimal impact on mAP.
Threshold Sensitivity: The threshold parameter $T$ (defining which boxes are candidates for alignment) is critical. Values that are too small (e.g., $T=6, 7$ ) or too large can degrade performance. An optimal range (e.g., $T=9, 10, 12$ ) was identified where the trade-off is most favorable.
Over-constraining: Very high weights ( $W=1.0$ ) with small thresholds can cause the SVD metric to spike (worsen) due to over-constraining the model.

Qualitative Findings

Occlusion Handling: The model successfully reconstructs windows partially or fully obstructed by trees or balconies, inferring their correct size and position based on the grid.
Perspective Correction: The model corrects systematic drifts caused by perspective, aligning misaligned rows/columns that standard detectors miss.
Missing Elements: In some cases, the model infers and detects windows in "missing" parts of the image (e.g., black triangles) that the ground truth or baseline missed.
Failure Modes: If $W$ is too high, the model may miss detections of very large or very small windows that deviate from the dominant grid pattern.

5. Significance and Limitations

Significance:
This work bridges the gap between visual detection and procedural modeling. By ensuring that the output of a detector is structurally coherent, it stabilizes downstream inverse procedural modeling (IPM) tasks. It offers a lightweight, efficient way to produce "grid-ready" data without complex post-processing or explicit grammar rules.

Limitations:

Accuracy vs. Regularity Trade-off: Prioritizing alignment can degrade detection accuracy, particularly for outliers (unique window sizes) or in cases where the model prioritizes the grid over pixel-level evidence.
Dependency on Pairs: The method relies on the detector finding at least two elements to form a pair. Singleton classes or sparse elements receive little benefit.
Perspective Constraints: The method assumes perspective distortion is relatively small. Severe perspective or structural misalignment indistinguishable from distortion cannot be corrected.
Resolution Dependence: The threshold $T$ is defined in absolute pixels, making the method sensitive to image resolution.
Occlusion Recovery: If the detector fails to propose any candidates in a region (severe occlusion), the alignment loss cannot recover the missing structure as it only operates on positive predictions.

Conclusion

The paper presents a robust strategy to enhance facade parsing by embedding geometric priors directly into the training loss of modern object detectors. It successfully demonstrates that structural regularity can be improved for procedural reconstruction tasks while maintaining a controllable balance with standard detection performance.