💻 computer science

SiZeUp: Fast 3D Proxy from Aerial Images via Depth Ordinal Loss

SiZeUp is a fast and scalable method for generating large-scale 3D urban proxy models from aerial images by optimizing building heights via a novel ordinal depth consistency loss that leverages relative depth ordering from monocular priors, achieving a 23–52× speedup over state-of-the-art pipelines while maintaining high coverage and volume consistency.

Original authors: Wenjun Zhou, Yunshan Li, Qiaoyu Zhu, Weidan Xiong, Hao Zhang, Daniel Cohen-Or, Hui Huang

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenjun Zhou, Yunshan Li, Qiaoyu Zhu, Weidan Xiong, Hao Zhang, Daniel Cohen-Or, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine looking down at a city from a drone. You see a complex tapestry of rooftops, streets, and shadows, a three-dimensional world flattened into a two-dimensional photograph. For computers, making sense of this flat image to understand the actual height and shape of every building is a notoriously difficult puzzle. This challenge sits at the heart of digital mapping and urban planning, where creating accurate 3D models of cities is essential for everything from simulating emergency responses to designing new infrastructure. Traditionally, the path to these models has been long and computationally heavy, requiring the computer to first build a dense cloud of millions of points to represent the city's surface before it can even begin to simplify that data into useful shapes. It is a process that demands vast amounts of time and processing power, often making it impractical for rapid, large-scale updates.

A team of researchers has now introduced a faster, more direct way to solve this problem, bypassing the need for those heavy, point-by-point reconstructions. Their method, called SiZeUp, takes calibrated aerial photographs and instantly transforms them into simplified 3D building models. Instead of trying to reconstruct every brick and window, the system focuses on the most critical structural information: the footprint of the building on the ground and its total height. By treating each building as a simple vertical block rising from its base, the researchers reduced a complex 3D geometry problem into a much simpler task of estimating a single number for each structure. This approach allows them to generate large-scale urban models with remarkable speed, achieving results that are dozens of times faster than previous methods while maintaining a high level of structural accuracy.

The core of this innovation lies in how the system "sees" depth. In standard photography, a computer struggles to know exactly how far away an object is because a flat image lacks depth information. While some artificial intelligence models can guess the relative distance of objects in a single photo, these guesses are often unreliable regarding exact measurements. The researchers realized they did not need perfect measurements; they only needed to know the correct order of things. They developed a technique that checks whether the computer's 3D model agrees with the relative ordering of heights seen in the photos. For instance, if a building appears taller than a tree in the image, the computer's 3D model must reflect that same relationship. By constantly adjusting the building heights until this relative ordering matches the visual evidence across many different camera angles, the system converges on an accurate solution without needing to calculate precise distances.

To make this work, the process begins by identifying the exact outline of each building on the ground, a step known as footprint extraction. The researchers trained a specialized tool to recognize these outlines even in complex, angled aerial views where buildings might be partially hidden or distorted. Once the ground outlines are locked in, the system selects only the most useful photographs from the entire set to guide the height estimation. It does not waste time processing every single image; instead, it picks a small, diverse group of views that offer the best clues about how the buildings rise into the sky. Using a differentiable renderer—a tool that allows the computer to mathematically trace how changes in height would alter the appearance of the image—the system iteratively adjusts the height of each building block. It compares its own rendered view of the city against the depth hints provided by a pre-trained AI model, correcting any mismatches in the relative ordering of surfaces until the model aligns perfectly with the visual data.

The results of this approach are striking in their efficiency. When tested on real-world urban scenes, the method produced 3D proxy models in a matter of seconds, a speedup of between 23 and 52 times compared to the best existing techniques that rely on dense point clouds. While the resulting models are simplified blocks rather than detailed meshes with every architectural feature, they capture the essential volume and spatial arrangement of the city with high fidelity. The researchers found that these simplified models were just as effective as the more complex ones for tasks requiring structural consistency and coverage, such as planning drone flights or analyzing urban density. The system proved robust even in challenging conditions, handling scenes with varying building heights and complex layouts, though it does face limitations with buildings that have stepped roofs or multiple distinct levels, which a single block cannot fully represent.

This work suggests a shift in how we approach urban modeling, prioritizing the specific needs of the task over the pursuit of unnecessary geometric detail. By focusing on the fundamental degrees of freedom—the footprint and the height—the researchers demonstrated that a streamlined, task-specific formulation can outperform general, computationally expensive methods. The success of SiZeUp indicates that for many applications in smart cities and digital twins, the most valuable representation is not the most detailed one, but the one that is fastest to generate and most reliable in its structural logic. This approach opens the door to updating digital city models in near real-time, a capability that could transform how we monitor, plan, and interact with our built environment.

Technical Summary: SiZeUp

Problem Statement

Automated reconstruction of large-scale 3D urban environments from aerial imagery is a critical task for planning, simulation, and digital twins. However, existing approaches for generating "3D proxies"—simplified, structurally faithful geometric representations of buildings—often rely on indirect pipelines. Conventional methods typically require dense 3D reconstruction (via Structure-from-Motion and Multi-View Stereo) to generate point clouds or meshes, which are then abstracted into proxies. These workflows are computationally expensive, data-intensive, and highly dependent on the quality of intermediate 3D representations, often requiring high-overlap imagery. Furthermore, many downstream applications prioritize structural correctness and scalability over visual realism, suggesting that full dense reconstruction is unnecessary and inefficient for proxy generation.

Methodology

The authors propose SiZeUp, a fast and scalable approach that constructs large-scale 3D urban proxy models directly from calibrated oblique aerial imagery, bypassing the need for dense point cloud reconstruction. The method reduces 3D building abstraction to a low-dimensional optimization problem based on a height-from-footprint representation, where each building is modeled as a 2D footprint extruded vertically by a single height parameter.

The pipeline consists of three primary stages:

Footprint Extraction (FPSAM):
- The system first identifies near-nadir views (optical axis within 15° of vertical) to extract 2D building footprints.
- It introduces FPSAM, a novel extractor that fuses texture, semantic, and depth-aware geometric cues. It leverages two frozen foundation models: SAM 3 (for semantic and texture features prompted with "building") and Depth Anything 3 (DA3) (for coarse geometric structure).
- These features are projected, concatenated, and decoded by a trainable U-Net to produce ground-contact footprint masks, which are then vectorized into closed polygons.
Dynamic View Selection (DVS):
- To ensure efficiency, the method does not use all available oblique images. Instead, it employs a Dynamic View Selection strategy to choose a small, informative subset of views for height optimization.
- The selection criteria include: (i) high coverage of building surfaces, (ii) spatial diversity in viewing angles, and (iii) ensuring every building is observed by at least $k$ views.
- This heuristic selection reduces the number of views involved in optimization to 10–27% of the total available images, significantly lowering computational costs while maintaining multi-view constraints.
Building Height Estimation via Ordinal Depth Loss:
- The core innovation is a differentiable renderer that maps parametric building proxies (footprints + heights) into multi-view depth images.
- Instead of relying on metric depth values—which are prone to scale ambiguity in monocular depth models—the method introduces an Ordinal Depth Consistency Loss.
- This loss enforces agreement between the relative depth ordering of pixels in the rendered proxy depth and the depth priors predicted by DA3.
- Specifically, for sampled pixel pairs, the method penalizes cases where the rendered depth reverses or weakly respects the ordering given by the prior, using a hinge loss with a margin. This allows gradients to propagate from the depth supervision to the building height parameters without requiring explicit feature matching or dense surface reconstruction.
- The optimization minimizes the aggregate ordinal violation across the selected views, solving for the building heights $\{h_i\}$ .

Key Contributions

Direct Image-to-Proxy Pipeline: SiZeUp bypasses the computationally heavy SfM/MVS reconstruction steps, directly optimizing building proxies in image space using calibrated camera poses.
Ordinal Depth Consistency Loss: The authors propose a novel loss function that leverages the relative depth ordering from monocular depth priors rather than absolute metric values. This provides a robust signal that is invariant to scale ambiguity and effective even with noisy depth estimates.
FPSAM Footprint Extractor: A specialized extractor combining semantic (SAM 3) and geometric (DA3) cues to accurately segment ground-contact footprints from oblique imagery, outperforming existing near-orthographic and offset-based methods.
Efficiency via Dynamic View Selection: A strategy that selects a minimal subset of informative views, enabling the system to achieve significant speedups while maintaining geometric consistency.

Experimental Results

The authors evaluated SiZeUp on real-world urban scenes and compared it against state-of-the-art proxy reconstruction baselines (including ConvexHull, ArcPro, and ProxyRecon) that rely on point cloud inputs.

Speed: SiZeUp achieves a 23–52× speedup over existing proxy reconstruction pipelines. While point-cloud-based methods take hundreds to thousands of seconds (including SfM front-end), SiZeUp completes height estimation in tens of seconds.
Accuracy:
- Footprint Extraction: On the new VertiFP benchmark, FPSAM achieved the highest IoU (70.89%), Precision (81.49%), and F1-Score (81.90%), significantly outperforming baselines like PolyWorld and Pix2Poly.
- Proxy Quality: While SiZeUp exhibits higher point-level surface error (Err.) compared to low-poly mesh baselines due to its coarse LOD1 (Level of Detail 1) representation (vertical extrusion), it maintains comparable proxy-level coverage (Comp.) and volume consistency (IoU). In many metrics, it ranks among the top two methods.
Robustness: The method performs well across various urban layouts and capture patterns, including non-grid flight trajectories, demonstrating that the ordinal optimization is driven by multi-view relative depth constraints rather than specific acquisition geometries.

Significance and Claims

The paper argues that for many urban modeling tasks, full geometric reconstruction is unnecessary. By deliberately constraining the representation to a low-dimensional space (footprints + heights) and aligning the supervision (ordinal depth) and optimization with the specific requirements of the task, SiZeUp achieves a superior efficiency-quality tradeoff.

The authors claim that SiZeUp offers a practical and viable alternative to conventional point-cloud-based pipelines, enabling rapid generation of structurally coherent 3D proxies for large-scale urban scenes. They emphasize that their approach highlights the importance of task-specific formulations over defaulting to increasingly complex and computationally expensive models. The method is particularly suited for applications where structural correctness and scalability are prioritized over visual realism, such as drone path planning, real estate analytics, and smart city simulations.

Limitations: The authors acknowledge that the vertical extrusion model cannot fully capture complex building geometries such as stepped roofs or multi-layer structures, which may result in less accurate proxies for such buildings. They suggest that decomposing complex buildings into finer footprint components could mitigate this issue.