💻 computer science

Feed-Forward Gaussian Splatting from Sparse Aerial Views

This paper introduces AnyCity, a feed-forward generative framework that reconstructs large-scale urban scenes from sparse aerial views by combining observation-grounded geometry with scaffold-conditioned diffusion priors to resolve evidence imbalances and eliminate artifacts like ghosting and melted facades.

Original authors: Dongli Wu, Zhuoxiao Li, Tongyan Hua, Yinrui Ren, Xiaobao Wei, Rongjun Qin, Wufan Zhao

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongli Wu, Zhuoxiao Li, Tongyan Hua, Yinrui Ren, Xiaobao Wei, Rongjun Qin, Wufan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a whole city, but you only have a few blurry photos taken from a drone flying high up.

The Problem: The "Roof vs. Wall" Blind Spot
When a drone flies straight down or at a shallow angle, it sees the tops of buildings (roofs) and open streets very clearly. It sees them over and over again. But the sides of the buildings (facades), the back of distant skyscrapers, and anything hidden behind other structures are almost invisible.

If you try to build a 3D city from just these few photos, standard computer programs get confused. They might turn a solid brick wall into a melted puddle, stretch a building's texture like taffy, or leave gaps where the walls should be. This happens because the computer is trying to guess the missing parts based on very little evidence.

The Solution: AnyCity
The researchers created a new tool called AnyCity. Think of it as a smart construction crew that knows exactly which parts of the city are real and which parts need a little creative help.

Here is how it works, step-by-step:

1. The "Solid Scaffold" (The Observation-Supported Part)

First, AnyCity looks at the photos and builds a "scaffold" or a skeleton using only the parts it can clearly see.

Analogy: Imagine a carpenter building a house frame. They nail down the beams and walls that are clearly visible in the blueprints. They don't guess where the windows go yet; they just secure the parts they are 100% sure of.
Result: This creates a stable, reliable base for the roofs and open areas, ensuring the computer doesn't "hallucinate" fake structures where it has good data.

2. The "Creative Fill-in" (The Completion Tokens)

Once the solid scaffold is built, AnyCity turns its attention to the missing parts—the invisible walls and distant buildings.

Analogy: Now, the carpenter brings in a "ghostwriter" or a creative architect. This expert looks at the solid frame and says, "Okay, I know there's a wall here because that's how cities usually look, even though I can't see it in the photo."
How it works: The system uses a "video prior" (a brain trained on thousands of hours of city videos) to guess what the missing textures and shapes should look like. But crucially, it doesn't just replace the whole building with a guess. It only adds a "residual update"—a gentle correction to the weak spots.

3. The "Safety Gate" (Observation Preservation)

This is the most important trick. The system has a strict rule: "Do not change what you can already see."

Analogy: Imagine the creative architect is trying to fix the invisible wall, but they are wearing a helmet that locks their hands if they try to touch the solid beams the carpenter already built. They can only work on the empty spaces.
Result: This prevents the "ghostwriter" from accidentally melting the real walls or changing the shape of the roof just because it thinks it knows better. It keeps the real data real and only fills in the blanks.

4. The "Teacher-Student" Training

To teach this system, the researchers used a clever training method.

The Teacher: A computer that gets to see many photos of the city (dense views). It knows the city perfectly.
The Student: The AnyCity model, which only sees the few photos (sparse views).
The Lesson: The Teacher shows the Student what the city looks like when fully observed. The Student tries to guess the missing parts to match the Teacher's view, but it must keep the parts it can already see exactly the same. This teaches the Student how to fill in the blanks without messing up the known facts.

The Final Result

When you give AnyCity just a few drone photos, it spits out a complete, coherent 3D city model in seconds.

The Good: The roofs and roads look exactly like the photos. The invisible walls and distant buildings look realistic and consistent, not melted or stretched.
The Speed: Unlike older methods that take hours to calculate, AnyCity does this in a single "feed-forward" pass (like reading a sentence from start to finish without stopping to think).

In Summary:
AnyCity is like a master builder who builds a house based on a few photos. It nails down the parts it can see with absolute precision, then uses its knowledge of how cities usually look to gently fill in the invisible parts, all while making sure it never accidentally changes the parts it already built. This results in a realistic 3D city that is both accurate to the photos and complete in its structure.

Technical Summary: Feed-Forward Gaussian Splatting from Sparse Aerial Views (AnyCity)

Problem Statement
Reconstructing large-scale urban scenes from sparse aerial views is a critical challenge for digital twins and urban planning. However, sparse aerial captures suffer from a severe evidence imbalance due to biased top-down and shallow-oblique camera poses. While roofs and open regions are repeatedly observed, vertical facades, distant buildings, and occluded structures receive little multi-view support. Existing feed-forward 3D Gaussian Splatting (3DGS) methods, which regress deterministic representations directly from sparse inputs, often fail in this setting, producing ghosting, melted facades, and stretched textures. Conversely, recent generative methods that rely heavily on priors often lack a clear separation between observed geometry and generated content, leading to plausible but geometrically inconsistent structures.

Methodology: AnyCity
The authors propose AnyCity, an observation-grounded generative reconstruction framework designed to address this imbalance. The core insight is to treat sparse aerial reconstruction as a controlled generative refinement problem, where generative priors assist only weakly constrained content while preserving observation-supported geometry. The pipeline operates in a single feed-forward pass without per-scene optimization.

Observation-Supported Scaffold:
The model first predicts a geometry latent ( $Z_{geo}$ ) from sparse unposed aerial images using a feed-forward multi-view backbone. This latent anchors reliable structures directly supported by input observations. During the first training stage, this pathway is optimized using photometric and lightweight geometric losses (depth alignment and normal flattening) to suppress floating Gaussians and facade melting.
Aerial Completion Tokens:
To handle weakly constrained content (e.g., facades, distant buildings), the model introduces aerial completion tokens. These tokens read context from the observation-supported scaffold and predict a gated residual update ( $\Delta Z$ ) to the geometry latent. This ensures the scaffold remains responsible for reliable geometry, while the completion branch handles ambiguous regions.
Gated Video Prior Injection:
To provide fine-grained urban appearance cues, AnyCity utilizes a frozen video diffusion backbone (Wan 2.2) adapted for the aerial domain via LoRA. The video prior features are projected into the completion-token space. Crucially, a token-wise gate modulates how strongly each completion token attends to the video prior. This allows the generative prior to refine weakly constrained content without directly modifying the main geometry stream or altering reliable observations.
Training Strategy:
The model employs a two-stage progressive training approach:
- Stage I: Stabilizes the observation-supported scaffold using geometric losses.
- Stage II: Activates the completion branch, video prior, and gated conditioning. This stage utilizes dense-to-sparse distillation, where a "teacher" model trained on dense views guides the "student" (sparse input) on structures missed by sparse observations. An observation-preserving loss ensures that the residual refinement does not alter geometry already supported by the sparse inputs.

Key Contributions

AnyCity Framework: A feed-forward generative reconstruction framework specifically targeting sparse, unposed aerial urban views, capable of handling limited overlap, weak parallax, and uneven scene evidence.
Aerial-Adapted Video Prior: The introduction of a video diffusion prior routed through scaffold-conditioned aerial completion tokens. This design allows generative cues to refine weakly constrained content before Gaussian decoding, rather than forming the scene directly, thereby maintaining geometric consistency.
Progressive Training & Observation Preservation: A training strategy that first stabilizes the observation-supported scaffold and then supervises video-prior refinement using dense-to-sparse distillation and observation-preserving objectives. This prevents the model from hallucinating structures where reliable evidence exists.
Empirical Validation: Extensive evaluation on synthetic, aerial-domain, UAV-textured, and real-world scenes demonstrates consistent improvements over existing feed-forward baselines, particularly in weakly constrained regions.

Results
Experiments show that AnyCity achieves superior novel-view synthesis quality compared to generalizable feed-forward baselines (e.g., NoPoSplat, FLARE, AnySplat) across various input view counts (2, 4, and 16 views).

Quantitative: On the GoogleEarth and CityNerf datasets, AnyCity outperforms baselines in PSNR, SSIM, and LPIPS. For instance, with 4 views on GoogleEarth, it achieves a PSNR of 18.466 compared to 17.256 for AnySplat.
Qualitative: The method produces more photorealistic renderings with better facade continuity and fewer floating artifacts.
Efficiency: Unlike per-scene optimization methods (e.g., CityGaussianV2) which require minutes to hours, AnyCity reconstructs the final 3DGS in seconds (e.g., ~3.4s for 32 views) without requiring precise camera poses or SfM initialization.
Diagnostic: Region-wise analysis confirms that the method significantly improves weakly constrained regions (facades, distant buildings) while maintaining high fidelity in observation-supported regions.

Significance and Claims
The paper claims that AnyCity successfully bridges the gap between the efficiency of feed-forward reconstruction and the generative power of diffusion models, specifically tailored for the unique challenges of aerial imagery. By decoupling reliable observation-supported geometry from generative refinement, the method avoids the "plausible but inconsistent" pitfalls of pure generative approaches. The authors position AnyCity as a step toward scalable, pose-free 3D urban reconstruction that can operate effectively under the evidence-imbalanced conditions typical of aerial data collection. The work highlights that generative priors are most effective when used as latent conditioners for residual updates rather than as direct scene generators.