💻 computer science

EOGS++: Earth Observation Gaussian Splatting with Internal Camera Refinement and Direct Panchromatic Rendering

EOGS++ advances Earth Observation Gaussian Splatting by enabling direct panchromatic rendering from raw satellite imagery, integrating internal bundle adjustment via optical flow for refined camera poses, and incorporating optimization techniques to achieve state-of-the-art reconstruction quality and geometric accuracy on benchmark datasets.

Original authors: Pierrick Bournez, Luca Savant Aira, Thibaud Ehret, Gabriele Facciolo

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pierrick Bournez, Luca Savant Aira, Thibaud Ehret, Gabriele Facciolo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive collection of satellite photos taken from space. Your goal is to build a perfect, 3D digital twin of the Earth's surface—complete with buildings, roads, and terrain—using only these flat pictures.

For a long time, scientists used two main ways to do this:

The Old Way: Like a human surveyor, they would manually measure and stitch photos together. It was slow and needed perfect conditions.
The "NeRF" Way: A newer, AI-based method that learns the 3D shape by "dreaming" about the scene. It's very accurate but takes forever to train (like cooking a slow-simmered stew).

Then, 3D Gaussian Splatting arrived. Think of this as a "fast-food" version of 3D modeling. Instead of slow cooking, it uses millions of tiny, fuzzy 3D "paint splatters" (Gaussians) that are quickly arranged to look like a 3D scene. It's incredibly fast and looks great.

The Problem:
The original version of this fast method for satellites (called EOGS) had a few annoying prerequisites. It was like trying to bake a cake but being told you must buy a pre-mixed, pre-chopped, pre-sifted flour kit first.

It needed Pansharpening: A complex, separate step to mix a blurry color photo with a sharp black-and-white photo to make a sharp color photo. If this step was done poorly, the cake tasted bad.
It needed External Bundle Adjustment: A separate, heavy-duty math tool to fix the camera's position before training could even start. If the camera was slightly off, the whole model was crooked.
It only worked with RGB (Color) images, ignoring the super-sharp black-and-white (Panchromatic) images that satellites actually take.

The Solution: EOGS++
The authors of this paper built EOGS++. Think of this as upgrading the kitchen so you can bake the cake from scratch, using the raw ingredients, and fixing the oven while the cake is baking.

Here is what they changed, using simple analogies:

1. The "Self-Correcting GPS" (Internal Bundle Adjustment)

Old Way: You had to hire a surveyor to measure your location before you started driving.
EOGS++ Way: The car has a built-in GPS that watches the road (the image) and says, "Wait, we are 5 meters to the left," and instantly steers itself back on track while you drive.
How it works: They use a technique called Optical Flow (which tracks how pixels move between images). As the AI tries to build the 3D model, it constantly checks: "Does my 3D model look like the photo?" If not, it nudges the camera's position slightly to match. It fixes the camera errors during the training, so no external surveyor is needed.

2. The "Raw Ingredient" Strategy (Direct Panchromatic Rendering)

Old Way: You had to mix a blurry color photo with a sharp black-and-white photo to get a sharp color photo before you could start. This mixing process often introduced weird artifacts (like a glitchy filter).
EOGS++ Way: They realized, "Why mix them? Let's just use the super-sharp black-and-white photo!"
How it works: Satellites take high-res black-and-white (Panchromatic) photos and lower-res color photos. EOGS++ ignores the lower-res color photos entirely. It takes the sharp black-and-white photo, copies it three times to make a fake "RGB" image, and trains the model on that. It turns out, the sharpness of the black-and-white photo is so good that the model builds a better 3D structure without the messy mixing step.

3. The "Stop-When-Perfect" Mechanism (Early Stopping & Opacity Reset)

The Problem: Sometimes, the AI gets too excited. It keeps adding more and more "paint splatters" (Gaussians) to the scene. Eventually, it starts adding "ghosts" or "floaters"—tiny, floating specks of 3D noise that don't belong to any real building.
The Fix:
- Opacity Reset: Imagine the AI is painting a wall. Every few minutes, it wipes the wall clean and starts painting again, but this time it paints more carefully. This prevents the "ghosts" from piling up.
- Early Stopping: The AI watches its own progress. It knows that when the picture looks perfect, if it keeps training, the picture actually gets worse (it starts adding noise). So, the moment the quality dips, the system hits the "Stop" button. It stops exactly at the peak of perfection.

4. The "3D Puzzle Solver" (TSDF Post-Processing)

The Problem: The raw 3D model is a cloud of fuzzy splatters. It looks good from the front, but if you try to measure the height of a building, it might be a bit jagged or have holes.
The Fix: After the AI finishes training, they take all the different views of the 3D model and fuse them together using a mathematical technique called TSDF. Imagine taking a 3D scan from the front, the side, and the top, and merging them into one solid, smooth block. This removes the jagged edges and fills in the holes, resulting in a clean, measurable Digital Surface Model (DSM).

The Result

By doing all this, EOGS++ is faster, more accurate, and requires less human setup than previous methods.

It builds 3D models of cities that are sharper and more accurate.
It makes fewer mistakes on building heights (reducing errors from 1.33 meters down to 1.19 meters).
It doesn't need the messy "pre-cooking" steps that other methods require.

In a nutshell: EOGS++ is like taking a high-tech 3D printer and giving it the ability to calibrate its own sensors, use raw materials instead of pre-mixed kits, and know exactly when to stop printing so the final product is perfect. It's a major step forward for mapping our world from space.

1. Problem Statement

The paper addresses the challenges of 3D reconstruction from satellite imagery, specifically for generating Digital Surface Models (DSMs). While traditional photogrammetry requires precise calibrations and temporally consistent acquisitions, and Neural Radiance Fields (NeRF) offer high quality but suffer from high computational costs, 3D Gaussian Splatting (3DGS) has emerged as a faster alternative.

However, the existing Earth Observation Gaussian Splatting (EOGS) framework has two major limitations:

Dependency on Preprocessing: It requires external Bundle Adjustment (BA) to correct camera poses and Pansharpening to fuse low-resolution multispectral (MSI) data with high-resolution panchromatic (PAN) data into RGB images. These steps are computationally expensive and can introduce artifacts.
Data Limitations: It is restricted to RGB data, ignoring the native high-resolution PAN data available in satellite imagery.

The goal of this work is to create a framework that operates directly on raw satellite data, eliminates external preprocessing dependencies, and improves geometric accuracy.

2. Methodology: EOGS++

The proposed EOGS++ framework extends the original EOGS by integrating refinement steps directly into the training pipeline and modifying the rendering strategy.

A. Internal Bundle Adjustment via Optical Flow

Instead of relying on external BA software (like SIFT-based tools), EOGS++ performs internal bundle adjustment during training:

Mechanism: It renders an image from the current 3D Gaussians and compares it to the ground-truth training image.
Optical Flow Alignment: An off-the-shelf optical flow estimator (RAFT) computes a displacement field between the rendered and training images.
Pose Refinement: Since satellite camera errors often manifest as constant pixel offsets, the system calculates a constant averaged displacement vector ( $\bar{\Delta}$ ). This vector is used to bilinearly resample the rendered image to align with the training image.
Optimization: The displacement is treated as a learnable parameter (with gradients blocked for the flow estimator itself) to refine the camera pose (specifically the affine translation terms) without external tools.

B. Direct Panchromatic Rendering (3-PAN)

EOGS++ eliminates the need for pansharpening by operating natively on high-resolution Panchromatic (PAN) data:

Strategy: Instead of fusing PAN and MSI data, the single-channel PAN images are replicated into a 3-channel RGB representation.
Training: The model renders 3-channel outputs and compares them directly to the replicated PAN training images.
Rationale: Experiments show that fusing MSI and PAN data (via linear combination or classical pansharpening) often degrades performance or introduces artifacts. Using raw PAN data preserves the highest spatial resolution available.

C. Training Improvements: Opacity Reset & Early Stopping

To address issues of "floaters" (floating Gaussian artifacts) and lack of sharpness:

Opacity Reset: Every 3,000 iterations, the opacity of all Gaussians is cyclically reset ( $\alpha_k \leftarrow \min(\alpha_k, 0.05)$ ). This forces the optimizer to re-evaluate the necessity of Gaussians, removing spurious ones.
Early Stopping: The photometric loss ( $L_1 + L_{D-SSIM}$ ) exhibits non-monotonic behavior during opacity resets (it decreases, then increases). The training is halted immediately when the loss starts to increase, ensuring the model stops at the point of highest fidelity before overfitting or degrading. This typically occurs between 10,000 and 20,000 iterations.

D. Post-Processing: TSDF Fusion

To generate the final DSM:

TSDF Fusion: Instead of relying on a single virtual nadir view, the method aggregates depth maps from all training views using a Truncated Signed Distance Function (TSDF).
Mesh Extraction: The fused TSDF is converted to a mesh using Marching Cubes.
Cleaning: Isolated voxels (floaters) are removed, and holes are filled to produce a clean, geometrically consistent surface model.

3. Key Contributions

Internal Camera Refinement: Introduces an optical flow-based bundle adjustment integrated directly into the training loop, removing the need for external pose optimization tools.
Raw Panchromatic Processing: Demonstrates that high-quality 3D reconstruction can be achieved using only raw high-resolution PAN data, eliminating the need for pansharpening and MSI data fusion.
Enhanced Stability and Sharpness: Implements opacity reset and early stopping mechanisms to prevent Gaussian floaters and improve geometric sharpness.
TSDF Post-Processing: Adds a robust post-processing step to fuse multi-view depth maps into a coherent DSM, reducing artifacts from single-view rendering.

4. Experimental Results

The method was evaluated on the IARPA 2016 and DFC 2019 datasets (WorldView-3 satellite imagery).

Accuracy (MAE):
- EOGS++ achieved a Mean Absolute Error (MAE) of 1.19 meters on building reconstruction, improving upon the original EOGS (1.33m) and other NeRF-based methods (e.g., EO-NeRF at 1.29m).
- The internal optical flow bundle adjustment achieved results comparable to external BA (within 3cm), validating the internal refinement approach.
Data Efficiency:
- The 3-PAN strategy (using only PAN data) outperformed strategies that attempted to fuse MSI and PAN data (Linear Combination), which showed a performance degradation of ~0.5m.
Computational Efficiency:
- Training time was significantly reduced compared to NeRF methods (e.g., ~25 minutes for EOGS++ vs. ~900 minutes for EO-NeRF).
- The method maintains the speed advantages of Gaussian Splatting while achieving state-of-the-art (SOTA) accuracy.

5. Significance

Scalability: By removing the dependency on external preprocessing (pansharpening and external BA), EOGS++ offers a streamlined, end-to-end pipeline suitable for large-scale remote sensing applications.
Robustness: The ability to work directly with raw PAN data makes the system more adaptable to varying sensor conditions and reduces the risk of artifacts introduced by fusion algorithms.
Geometric Fidelity: The combination of internal pose refinement and TSDF post-processing significantly improves the geometric accuracy of reconstructed structures (buildings), which is critical for urban planning and disaster response.
Trade-off: While the method excels at reconstructing permanent structures (buildings), it acknowledges that high-frequency transient details (like vegetation) are sometimes suppressed due to the regularization mechanisms (early stopping/opacity reset), which is acceptable given the primary focus on structural geometry.

In conclusion, EOGS++ represents a significant step forward in satellite 3D reconstruction, proving that Gaussian Splatting can be adapted to handle raw, unprocessed satellite data with high efficiency and superior geometric accuracy compared to existing NeRF and traditional methods.