💻 computer science

S23DR 2026: End-to-End 3D Wireframe Prediction via DETR-Style Set Prediction with Contrastive Denoising

This paper presents WireframeDETR, a novel approach for the S23DR 2026 challenge that directly predicts 3D building wireframes from multi-view point clouds using a DETR-style set prediction framework enhanced by contrastive denoising, multi-scale encoding, and progressive auxiliary loss weighting to achieve state-of-the-art performance.

Original authors: Nitiz Khanal

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitiz Khanal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are handed a giant, messy cloud of colored dust floating in 3D space. This dust represents a building's roof, but it's incomplete, scattered, and missing huge chunks. Your job? To look at this chaotic dust and instantly draw the perfect blueprints (the wireframe) of the roof, connecting the dots to show exactly where the walls, ridges, and edges go.

This is the challenge the authors of this paper tackled for the S23DR 2026 Challenge. They built a system called WireframeDETR to solve this puzzle. Here is how they did it, explained without the jargon.

The Problem: Why Previous Attempts Failed

Before building their new system, the team tried two other ways, and both hit a wall:

The "Over-Thinker" (Path A): They tried to tweak an existing, complex AI model (called Perceiver). It was like trying to teach a master chef a new recipe by shouting at them while they were already cooking. The model got confused, forgot what it knew, and its performance crashed.
The "Two-Step" (Path B): They tried a two-step process: first, find all the "corners" (vertices) of the roof, and then, try to guess which corners connect to make "edges." It was like trying to build a house by first finding every single brick and then guessing which bricks touch. They got really good at finding the bricks (corners), but terrible at guessing which ones connected (edges). The "connection" part was the weak link.

The Solution: WireframeDETR (The "Direct Draw" Approach)

The team decided to stop guessing corners first. Instead, they taught the AI to look at the whole cloud of dust and directly draw the lines (edges) it sees.

Think of it like this: Instead of asking, "Where are the dots?" and then "Do these dots connect?", the AI is asked, "Draw me a line here, and a line there." It predicts the entire shape as a set of lines all at once.

Here are the three "secret sauces" that made this work:

1. The "Training Wheels" (Contrastive Denoising)

When the AI first starts learning, it is very confused. It tries to match its drawn lines to the real roof lines, but it keeps making mistakes, getting frustrated, and learning the wrong things.

The Fix: The authors gave the AI "training wheels." They took the correct answer (the real roof lines), shook them up a little bit to make them slightly messy, and showed them to the AI as a "hint."
The Result: Because the AI knew exactly which messy line came from which real line, it learned much faster and more steadily. Once it got confident, they removed the training wheels. This stopped the AI from getting confused in the early days of learning.

2. The "Multi-Layered Memory" (Multi-Scale Encoder)

Usually, when an AI looks at an image or a point cloud, it processes it through several layers of "thinking." By the time it reaches the final layer, it has a great big-picture view but has forgotten the small details.

The Fix: The authors built a "memory bank" that saves the thoughts from the last three layers of thinking. They mixed the small details (from the earlier layers) with the big picture (from the final layer) using a special "volume knob" (learned weights) to decide how much of each to use.
The Result: The AI could see both the fine details of the roof edges and the overall shape of the building simultaneously, leading to a much sharper blueprint.

3. The "Gradual Coach" (Progressive Auxiliary Loss)

The AI has several layers of "decoders" (think of them as students in a classroom). The first student is a beginner, and the last student is an expert.

The Fix: Instead of treating every student the same, the authors acted like a smart coach. They gave the beginner students a little bit of credit for their early guesses, but as the students got closer to the final answer (the later layers), the coach demanded more perfection and gave them more "points" (weight) for their work.
The Result: This ensured that the AI didn't ignore the early layers, but also didn't let the early, messy guesses drag down the performance of the final, expert guesses.

The Results

The system worked incredibly well.

The Score: In the competition, their method scored a 0.575 (a metric called HSS).
The Comparison: This was much higher than the official baseline (0.350) and the previous "two-step" method (0.442).
The Trade-off: The system is a bit slower to train (about twice as slow as the baseline) because it has to do extra math to keep those "training wheels" and "memory banks" running. However, the accuracy gain was worth it.

In a Nutshell

The team stopped trying to build the roof piece-by-piece (finding corners, then edges). Instead, they built an AI that looks at the messy cloud of 3D points and directly "draws" the connecting lines, using special training tricks to keep it focused, a memory system to see both details and the big picture, and a smart grading system to help it learn efficiently. The result is a highly accurate 3D blueprint of a building's roof, generated directly from a sparse cloud of points.

Technical Summary: WireframeDETR for S23DR 2026

Problem Definition

The S23DR 2026 Challenge focuses on 3D building wireframe reconstruction from sparse, multi-view COLMAP point clouds. The input data includes per-point RGB colors, Gestalt (73-class) and ADE20K (107-class) semantic segmentation maps, and monocular depth estimates. The goal is to predict a set of 3D vertices and connecting edges representing the building rooftop structure.

Performance is evaluated using the Hybrid Structure Score (HSS), defined as the harmonic mean of the vertex F1 score and the edge volumetric IoU:
$\text{HSS} = \text{harmonic mean}(\text{F1}(v_{gt}, \hat{v}), \text{IoU}(e_{gt}, \hat{e}))$

Key challenges identified include the sparsity of COLMAP reconstructions on textureless surfaces, inconsistent camera parameters causing pose-annotation misalignment, and the difficulty of fusing 3D point clouds with 2D semantic labels.

Methodology: WireframeDETR

The authors propose WireframeDETR, an end-to-end model that reformulates wireframe prediction as a direct set prediction problem over 3D point clouds, eliminating intermediate vertex detection stages and post-hoc Non-Maximum Suppression (NMS).

Architecture Overview

The model processes an unordered set of $N$ 3D points (sampled from the point cloud) through a Transformer-based pipeline:

Input Processing: 3D RGB features are projected to an embedding dimension $d=384$ . Sinusoidal 3D positional encodings are added to each token.
Encoder: A 4-layer Transformer encoder processes the tokens.
Multi-Scale Memory: Instead of passing only the final layer's output, the model aggregates the last $K=3$ encoder layer outputs using learned scalar weights. This allows the decoder to access both local geometry and abstract structural features.
Decoder: A 5-layer Transformer decoder utilizes $Q=128$ learned edge queries. Each query predicts a 6D coordinate pair $(x_1, y_1, z_1, x_2, y_2, z_2)$ representing an edge.
Head: Separate classification and coordinate regression heads output the final edge set.

Key Technical Contributions

The paper identifies three specific technical innovations that stabilize training and improve performance:

Contrastive Denoising (CDN) Training:
Addressing the instability of Hungarian matching in early training epochs (where queries may be matched to different ground-truth targets across iterations), the authors adapt the CDN approach from DN-DETR.
- Mechanism: For each ground-truth edge, the model generates positive denoising queries (perturbed GT coordinates) and negative queries (background).
- Constraint: An attention mask ensures denoising queries and learned queries attend only within their own groups, keeping gradient paths independent.
- Effect: This provides a clean gradient signal from epoch 0, stabilizing convergence.
Multi-Scale Encoder Aggregation:
The model computes a learned weighted sum of the last three encoder layer outputs ( $H_{L-2}, H_{L-1}, H_L$ ). This enables the decoder to leverage intermediate representations that capture local geometric details alongside high-level structural features, with minimal parameter overhead.
Progressive Auxiliary Loss Weighting:
Auxiliary outputs are generated from intermediate decoder layers. The authors observe that prediction quality improves as the decoder depth increases. Consequently, they apply a linearly increasing weight to the auxiliary losses ( $w_{aux}$ ), ranging from 0.6 for the first auxiliary layer to 1.0 for the pre-final layer. This concentrates gradient signals on the layers that benefit most from supervision.

Data and Training Strategy

Gestalt-Guided Sampling: Points are sampled based on structural importance. Vertex classes (e.g., apex, eave) are weighted 100x, edge classes 10x, and background 1x. This concentrates samples near structural features without encoding semantic labels directly into the model features.
Coordinate Normalization: Coordinates are normalized to $[0, 1]^3$ per scene using the 5–95% percentile range.
Training Setup: Trained for 200 epochs on a single NVIDIA A100 GPU using AdamW with a one-cycle learning rate schedule. Gradient clipping ( $\ell_2$ norm < 0.1) is used to stabilize CDN training.
Inference: Predicted edges are filtered by classification score (>0.9). Vertices are de-duplicated by merging points within 0.5m. Optional test-time augmentation (4 Y-axis rotations) is available but disabled by default in the submission.

Experimental Results

The model was evaluated on the S23DR 2026 dataset (approx. 22,000 scenes).

Performance: WireframeDETR achieved a public test HSS of 0.575 (Vertex F1 = 0.664, Edge IoU = 0.516). On the cleaned validation split, it achieved a best HSS of 0.534.
Comparison:
- Outperformed the official Perceiver baseline (HSS 0.350) and the handcrafted baseline (HSS 0.391).
- Significantly outperformed a two-stage PointNet pipeline (Path B), which struggled with edge precision (F1 = 0.289) despite high vertex recall.
- Path A (Perceiver fine-tuning) resulted in catastrophic forgetting, dropping HSS below 0.26.
Training Dynamics: The CDN coordinate loss decreased steadily from ~2.07 to ~1.25 over the first 15 epochs, confirming that the denoising path converges reliably and provides cleaner gradients than standard Hungarian matching.

Significance and Claims

The paper claims that WireframeDETR successfully demonstrates that DETR-style set prediction can be applied directly to sparse 3D point clouds for structured geometry reconstruction, bypassing the need for cascaded error propagation found in two-stage pipelines.

The authors emphasize that their approach handles variable-cardinality wireframes without post-hoc NMS and stabilizes training in the presence of noisy matching via contrastive denoising. While the method is approximately twice as slow to train as the Perceiver baseline (due to variable sequence lengths preventing torch.compile), the substantial gain in HSS validates the end-to-end set prediction paradigm for outdoor building wireframes.

The work builds upon the success of RoomFormer [11] in indoor floor plans, extending the paradigm to the more complex and sparse domain of outdoor building rooftops. The authors note that future work could explore joint vertex-edge prediction with contrastive vertex losses and feature-guided vertex merging.