🤖 machine learning

Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generation

This paper presents Pro-DG, a novel ControlNet pipeline that leverages hierarchical procedural rules to generate photo-realistic architectural facades with precise structural edits, such as floor duplication and window rearrangement, while maintaining local appearance fidelity and architectural identity.

Original authors: Aleksander Plocharski, Jan Swidzinski, Przemyslaw Musialski

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aleksander Plocharski, Jan Swidzinski, Przemyslaw Musialski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photograph of a beautiful, old building. Now, imagine you want to change its design—maybe add a whole extra floor, rearrange the windows, or turn a single door into a triple-door entrance—without losing the building's original "personality" or making it look like a completely different, fake structure.

This is the problem the paper Pro-DG tries to solve. The authors created a smart tool that acts like a digital architect who understands both the "rules" of how buildings are built and the "art" of how they look.

Here is how it works, broken down into simple steps:

1. The "Reverse Engineering" (Reading the Blueprint)

First, the system looks at your photo of the building. Instead of just seeing pixels (dots of color), it tries to figure out the hidden blueprint or "recipe" behind the image.

The Analogy: Think of it like looking at a finished cake and trying to figure out the exact recipe used to bake it (how many layers, how the frosting was piped, where the sprinkles go).
The Tech: It uses a special AI to turn the photo into a hierarchical tree. This tree breaks the building down into logical parts: "The whole building" splits into "Floors," which split into "Walls," which split into "Windows." This is called a procedural grammar.

2. The "Architect's Edit" (Changing the Recipe)

Once the system has the blueprint, you (the user) tell it what to change.

The Analogy: You tell the chef, "I want to double the number of windows on the second floor," or "Move the balcony to the third floor."
The Magic: Because the system understands the rules of the building (the tree structure), it can make these changes logically. It doesn't just stretch a window; it understands that if you add a floor, you need to add a whole new row of windows to match.

3. The "Matchmaker" (Connecting Old to New)

This is the most clever part. The system needs to know which part of the old building corresponds to which part of the new design.

The Analogy: Imagine you are rearranging furniture in a room. You need to know that the "old sofa" moves to the "new sofa spot." If you just threw the furniture around randomly, the room would look messy.
The Tech: The system uses a special math trick (comparing shapes and patterns) to pair up the old windows with the new window spots. It ensures that the "brick texture" from the old wall goes to the new wall, and the "sky" stays the sky.

4. The "Painter" (Generating the New Photo)

Finally, the system uses a powerful image generator (called a Diffusion Model) to paint the new picture.

The Analogy: Imagine a painter who is blindfolded but is being guided by two people. One person holds a wireframe of the new building (the structure), and the other holds a photo of the old building (the style). The painter uses the wireframe to know where to paint and the photo to know what the paint should look like.
The Result: The AI generates a brand-new, realistic photo of the building with your changes. It looks like a real photograph, not a cartoon, and it keeps the original building's unique style (the lighting, the brick color, the weathering).

Why is this special?

Most AI image tools are like a magic wand that can only change small things (like removing a trash can or changing the color of a door). If you try to add a whole new floor with a magic wand, the AI usually gets confused and makes the building look weird or broken.

Pro-DG is different because it treats the building like a structured puzzle. It understands that buildings have rules. This allows it to make big, structural changes (like adding floors or rearranging entire sections) while keeping the building looking like the same building, just edited.

What the Authors Tested

The team tested this on 50 real building photos. They compared their method to:

Standard AI tools: Which often failed to keep the building's identity or structure correct.
Manual editing (Photoshop): Which takes a human hours to do manually.

The Verdict:

Better than standard AI: People preferred their results because the buildings looked more realistic and kept their original "vibe."
Better than manual editing: It was much faster and more consistent than trying to cut and paste windows in Photoshop.
The Catch: If you try to add something that didn't exist in the original photo at all (like adding a door to a wall that had no doors), the system sometimes gets confused because it has no "old door" to copy the style from. Also, if the original photo has trees or signs blocking the view, the system might get tripped up.

In short, Pro-DG is a tool that lets you redesign the skeleton of a building while keeping its skin and soul intact, all done automatically by a computer that understands both math and art.

Technical Summary: Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generation

Problem Statement

The generation of realistic architectural facade variations that adhere to structural principles while preserving the core identity of an original design remains a significant challenge. Existing methods in computer graphics and vision typically focus on either procedural modeling (often requiring substantial artist intervention for photorealism) or diffusion-based image synthesis (which struggles with precise structural control and global consistency). While recent neuro-symbolic approaches have attempted to bridge this gap, there is a lack of frameworks that effectively combine symbolic, hierarchical procedural structures with diffusion models to enable controllable, large-scale structural edits (e.g., floor duplication, window rearrangement) without losing the visual fidelity of the source image. Current diffusion-based editing methods often focus on localized object manipulation rather than restructuring the entire image space.

Methodology

The proposed framework, Pro-DG, unifies symbolic facade grammars with diffusion-based image synthesis through a five-stage pipeline:

Inverse Procedural Reconstruction:
The system begins by extracting a procedural representation from an input facade image ( $F_{in}$ ) and its segmentation. It utilizes FaçAID, a transformer-based neuro-symbolic method, to infer a hierarchical split grammar derivation tree ( $P_{in}$ ). This tree divides the image space into semantic regions (e.g., floors, walls, windows) and non-terminals, capturing the facade's structural hierarchy.
Structure Editing:
Users define edits by modifying the procedural representation to create a target structure ( $P_{out}$ ). Edits can involve adjusting parameters (e.g., number of floors) or altering the hierarchy (e.g., deleting specific elements). The pair $(P_{in}, P_{out})$ defines the desired transformation.
Hierarchical Matching:
A novel matching algorithm creates a correspondence between the source ( $P_{in}$ ) and target ( $P_{out}$ ) structures. This step pairs symbols from the target tree to the source tree based on two custom metrics:
- SVD Metric: Uses Singular Value Decomposition to compare structural complexity and symmetry of image regions, treating regions with different element counts but similar structural patterns as identical.
- Histogram Metric: Uses Hellinger distance on intensity histograms to ensure content awareness (e.g., distinguishing between a wall and a window).
  The algorithm enforces parent-child consistency, ensuring that matched nodes respect the hierarchical constraints of the grammar.
Diffusion Reconstruction (Inversion):
The system performs Null-Text Inversion on the input image to recover the initial noise vector and network activations ( $\Psi_{in}$ ) required to reconstruct $F_{in}$ . Simultaneously, Canny edge maps ( $C_{in}$ ) are extracted. This step captures the "identity" of the original facade.
Guided Inference:
The core synthesis step guides a diffusion model (Stable Diffusion 1.5 with a Canny-edge ControlNet) to generate the output ( $F_{out}$ ).
- Control Inputs: The hierarchical matching is used to construct a new Canny edge map ( $C_{out}$ ) and target network activations ( $\Psi_{out}$ ) by copying and resizing segments from the source based on the established correspondences.
- Optimization: The diffusion process is guided by $C_{out}$ via ControlNet to enforce the target structure. Additionally, an optimization objective minimizes the L2 distance between the current latent activations and $\Psi_{out}$ , ensuring the preservation of local appearance and texture.

Key Contributions

The paper presents three primary contributions:

Hierarchy-Aware Matching Algorithm: A novel metric combining SVD-based structural analysis and histogram-based content analysis to enforce parent-child consistency while aligning structural and visual cues between procedural trees.
Unified ControlNet Pipeline: A system that synthesizes hierarchy-derived control signals (Canny edges) and activation targets to guide all steps of the diffusion process, enabling global image-space restructuring rather than localized edits.
Comprehensive Evaluation: An evaluation across 50 diverse facade images demonstrating consistent gains in structural fidelity and visual coherence compared to state-of-the-art baselines.

Results and Evaluation

The method was evaluated on a dataset of 50 real-world facade images, with a subset of 20 used for qualitative analysis and user studies.

Qualitative Results: The system successfully generates edited facades that preserve the core identity of the original design while adhering to complex structural edits. The outputs exhibit photorealism comparable to the underlying Stable Diffusion 1.5 model.
Baselines: The authors compared their method against:
- ControlNet Baseline: A standard segmentation-based ControlNet approach using Null-Text Inversion.
- Photoshop Baseline: A semi-automatic collage-based inpainting approach using Adobe Fireﬂy.
- Synthetic Renderings: 3D renderings of modified facade models used as a proxy for ground truth.
User Study: In a study involving 105 users and 3,045 evaluations, Pro-DG was significantly preferred over the ControlNet baseline for realism and appearance preservation. It also outperformed the Photoshop baseline in edit adherence and identity preservation, despite the Photoshop baseline using a more advanced generative fill model (Fireﬂy).
Quantitative Metrics:
- Sliced Wasserstein Distance (SWD): Pro-DG achieved the lowest SWD (0.1185), indicating superior preservation of local appearance and texture compared to the ControlNet (0.1844) and Photoshop (0.124) baselines.
- CLIP Cosine Distance (CCD): Pro-DG showed strong semantic similarity (0.118), outperforming the ControlNet baseline (0.2146). While the Photoshop baseline had a slightly lower CCD (0.0936), this is attributed to the baseline's direct copying of objects, which the metric favors.
Ablation Studies: Experiments confirmed the necessity of both ControlNet and activation guidance; removing either led to hallucinations or blurry outputs. The study also highlighted that hierarchical matching for terminal symbols (e.g., walls) is critical for maintaining texture fidelity.

Significance and Limitations

The paper claims that Pro-DG addresses the unexplored gap between symbolic procedural structures and diffusion models, offering a framework for high-quality, controllable image generation that maintains architectural integrity. Unlike most diffusion-based editing methods that transform specific objects, Pro-DG enables the restructuring of the entire image space while retaining the original design's identity.

Limitations acknowledged by the authors include:

New Element Generation: The method struggles when the target structure introduces elements not present in the original design (e.g., adding doors to a doorless facade), as the hierarchical pairing lacks valid correspondences.
Grammar Expressiveness: If the underlying grammar (FaçAID) cannot express the complexity of a specific facade, the inverse reconstruction fails, leading to mismatches.
Visual Noise: Input images with obstructions (trees, signs) can introduce non-structural noise into the Canny maps and activations, resulting in incoherent outputs.
Model Constraints: The current implementation relies on Stable Diffusion 1.5 and its ControlNet, which limits the achievable fidelity compared to newer, specialized diffusion backbones.

The authors conclude that the synergy between symbolic procedural representations and data-driven synthesis holds promise for applications in facade restoration and interactive architectural design, demonstrating the potential of neuro-symbolic integration in structured image editing.