💻 computer science

Seeing Across Skies and Streets: Feedforward 3D Reconstruction from Satellite, Drone, and Ground Images

This paper introduces Cross3R, a feed-forward model that leverages an intermediate UAV image to overcome the limitations of nadir satellite views, enabling the simultaneous 6-DoF pose estimation and 3D reconstruction of ground, drone, and satellite imagery without requiring known relative poses.

Original authors: Qiwei Wang, Zhongyao Tuo, Xianghui Ze, Yujiao Shi

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiwei Wang, Zhongyao Tuo, Xianghui Ze, Yujiao Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Flat Map" Trap

Imagine you are trying to find a specific street corner using only a bird's-eye view map (like Google Maps) and a photo taken by a person standing on that corner.

For a long time, computers have been bad at this. Why? Because the map is a flat, top-down picture, and the photo is a 3D perspective looking forward. The computer gets confused. It can usually guess where you are on the map (left/right, forward/backward) and which way you are facing (North/South), but it has no idea about height or tilt.

It's like trying to guess how a person is standing in a photo just by looking at their shadow on a flat floor. If the person is leaning against a wall, standing on a ramp, or holding a camera at a weird angle, the "flat map" method fails. It assumes everything is perfectly flat and level, which is rarely true in the real world.

The Secret Ingredient: The "Drone Middleman"

The authors of this paper realized that to solve this puzzle, you need a third perspective. They introduced a Drone (UAV) image as a "middleman."

Think of it like a conversation between three people:

The Satellite: Looking straight down from space (sees the layout, but no height).
The Ground Camera: Looking straight ahead from the street (sees the 3D world, but not the big picture).
The Drone: Flying in the middle. It sees the street from an angle.

The drone is the bridge. It sees the 3D buildings and slopes that the satellite misses, but it also sees the overall layout that the ground camera misses. By adding this one drone photo, the computer can finally figure out the tilt, the pitch, and the exact height of the ground camera. It's like the drone handing the satellite a 3D model of the street so the satellite can finally understand the ground photo.

The Solution: Cross3R (The "All-in-One" Solver)

The paper introduces a new AI model called Cross3R.

How it works: You feed it three pictures at once: the satellite map, the ground photo, and the drone photo.
The Magic: In a single "blink" (one forward pass), it doesn't just guess the location. It rebuilds the entire 3D world of that scene. It creates a cloud of 3D points, figures out exactly where every camera was standing, and tells you precisely where the ground camera is located on the satellite map.
No Training Required: Unlike older methods that needed to be taught specifically for every new city, this model is "feed-forward." It's like a master chef who can cook a new dish immediately after tasting the ingredients, without needing a recipe book for that specific meal.

The New Playground: CrossGeo Dataset

To teach this model, the researchers couldn't use existing data because no one had ever collected all three types of photos (Satellite, Drone, Ground) together with perfect 3D measurements.

So, they built their own giant playground called CrossGeo.

The Scale: They gathered data from 85 different scenes all over the world (cities, suburbs, rural areas, hills) on every continent except Antarctica.
The Collection: They used Google Maps and Google Earth to simulate the drone and satellite views, and Google Street View for the ground views.
The Result: A massive library of 278,000 images where the computer knows exactly how high the drone was, how tilted the ground camera was, and how the satellite sees the world.

The Results: Why It Matters

When they tested Cross3R:

It built better 3D maps: It created 3D point clouds that were much more accurate and had fewer holes than previous methods.
It found locations better: It could pinpoint the ground camera's location on the satellite map with much higher precision, even on slopes and ramps where old methods failed.
It worked on new data: Even when tested on a dataset it had never seen before (the KITTI dataset, which is just ground and satellite photos), it outperformed models that were specifically trained on that data.

The Bottom Line

This paper says: "If you want to know exactly where a ground camera is and how it's tilted, don't just look at the ground photo and the map. Bring in a drone."

By adding that one middle view, the AI can finally understand the 3D shape of the world, fixing the "flat map" problem and allowing for accurate localization even on bumpy, tilted, or complex terrain.

Technical Summary: Cross3R – Feedforward 3D Reconstruction from Satellite, Drone, and Ground Images

Problem Statement
Cross-view localization traditionally seeks to determine the pose of a ground-level camera relative to a nadir satellite image. Existing methods are fundamentally limited to 3-DoF (Degrees of Freedom) estimation—recovering only the $(x, y)$ position and yaw angle. This limitation arises because a single nadir satellite tile provides no direct cues for roll, pitch, or altitude, forcing models to rely on planar-motion and zero-tilt assumptions. These assumptions fail in real-world environments featuring slopes, ramps, uneven terrain, or tilted camera mounts, leading to sharp degradations in accuracy and robustness. Furthermore, standard feed-forward 3D reconstruction models are designed for image collections captured at roughly similar altitudes and struggle to align point clouds and poses across the extreme viewpoint gap between satellite (top-down), UAV (oblique), and ground (perspective) views.

Methodology: Cross3R
The authors propose Cross3R, a flexible feed-forward foundation model designed to jointly recover a cross-view 3D point cloud, the 6-DoF poses of all input cameras, and the on-tile localization of perspective cameras in a single forward pass. The model accepts a satellite tile paired with one or two perspective views (UAV and/or ground).

Core Insight: The introduction of a single UAV image acts as a geometric bridge. It reveals 3D structure invisible from the nadir view and provides the necessary cues for roll, pitch, and altitude that satellite imagery alone cannot supply. Crucially, this approach requires only spatial overlap between the UAV and ground cameras, not known relative poses.
Architecture: Building upon the $\pi3$ $π 3$ model, Cross3R introduces three key modifications to handle the tri-view disparity:
1. Encoder: Injects Fourier positional embeddings of the satellite's normalized pixel grid directly into satellite tokens to encode the orthographic grid structure from the first layer. It also doubles the size of the register-token bank to disentangle satellite tokens from perspective tokens.
2. Alternating Attention: Fuses hidden states across layers using softmax-weighted gates, allowing the model to access both fine local geometry (early layers) and coarse global layout (late layers) simultaneously.
3. Decoding Heads:
  - Satellite Branch: Enforces an explicit orthographic geometric prior. A learned scalar $\rho$ ties the per-pixel scale of the satellite tile to the scene. Instead of predicting $(x, y)$ , the model predicts pixel coordinates $(u, v)$ scaled by $\rho$ , ensuring internal scale consistency.
  - Perspective Branches (UAV/Ground): Satellite features are compressed and broadcast-added into perspective tokens before the shared point and camera heads, providing explicit guidance from the satellite coordinate frame.
Training Objective: The model is trained with a weighted sum of four loss terms: geometric depth loss ( $L_{geo}$ ), surface normal loss ( $L_{norm}$ ), confidence map loss ( $L_{conf}$ ), and camera pose loss ( $L_{cam}$ ).
Per-Sample Altitude Redefinition: To stabilize training, the authors re-anchor altitudes locally. Ground cameras are set to $y=0$ , UAVs to relative altitudes ( $30\text{--}120$ m), and the satellite camera is redefined to a constant $150$ m (just above the max UAV altitude) rather than its true $\sim5,726$ m altitude. This prevents gradient instability caused by massive translation vectors.

Key Contributions

Cross3R Model: A feed-forward model that breaks the 3-DoF ceiling of prior cross-view methods, enabling full 6-DoF pose estimation and dense 3D reconstruction from satellite, UAV, and ground images.
CrossGeo Dataset: A large-scale, tri-view dataset comprising 277,812 images (46,302 samples) across 85 globally distributed scenes. It is the first public benchmark providing paired satellite, UAV, and ground views with full 6-DoF poses and dense metric depth. Data is sourced from Google Maps, Google Earth, and Google Street View.
Performance Gains: The model demonstrates that adding a UAV view significantly improves ground-to-satellite localization, effectively bridging the viewpoint gap and recovering vertical structure.

Experimental Results

CrossGeo Evaluation: Cross3R consistently outperforms feed-forward 3D baselines (including $\pi3$ , VGGT, and AerialMD) in point-cloud reconstruction (Acc-mean, $\delta@\tau$ ) and 6-DoF camera pose estimation (RRA, RTA, AUC). It also achieves superior cross-view localization accuracy, halving the translation error of the best baseline ( $\pi3^*$ ) and nearly quadrupling the PCK@2m score.
KITTI Zero-Shot: Despite being trained only on CrossGeo (with no KITTI data), Cross3R outperforms specialized cross-view methods trained directly on KITTI (LM, SliceMatch, CCVPE) on most metrics, including lateral recall and orientation recall.
AnyVisLoc (OOD): On a real-world out-of-distribution test set of UAV photographs, Cross3R achieves the best ground and UAV translation errors and the best ground yaw error, demonstrating robustness to real aerial photography despite being trained on rendered UAV data.
Ablation Studies:
- The UAV view is shown to be critical for ground localization, particularly when the UAV is at moderate altitude with a near-nadir viewpoint.
- Removing the orthographic head ( $\rho$ ) causes cross-view localization to collapse, confirming its role as the essential link for scale consistency.
- Using the raw satellite altitude ( $\sim5,700$ m) leads to training divergence, validating the necessity of the altitude redefinition strategy.
- Data scaling experiments show monotonic performance improvements with increased data, suggesting the model has significant headroom for further scaling.

Significance and Claims
The paper claims that Cross3R successfully recovers satellite, drone, and ground point clouds, camera poses, and cross-view localization in a single shot, overcoming the limitations of planar assumptions in real-world terrain. By leveraging a UAV image as an intermediate viewpoint, the method supplies the missing vertical cues required for 6-DoF estimation. The authors emphasize that while the current model is limited to three images per forward pass and pinhole inputs, the proposed acquisition pipeline and network design establish a new paradigm for scalable, data-driven cross-view 3D reconstruction. The work highlights the potential for applications in autonomous driving on rough roads, robot navigation, and search-and-rescue, while acknowledging privacy constraints by releasing only metadata (pano_ids) rather than raw street-level images.