AeroDGS: Physically Consistent Dynamic Gaussian Splatting for Single-Sequence Aerial 4D Reconstruction

This paper presents AeroDGS, a physics-guided 4D Gaussian splatting framework that leverages a monocular geometry lifting module and physics-based optimization priors to achieve robust, high-fidelity dynamic reconstruction from single-view aerial UAV videos, addressing the inherent depth ambiguity and motion instability of such scenarios.

The Gist

Imagine you are flying a drone over a busy city. You're holding a camera, recording a video of cars zooming by, people walking, and buildings standing still. Now, imagine you want to take that single video and turn it into a 3D time machine. You want to be able to fly your virtual drone anywhere, look at the cars from any angle, and watch them move smoothly through time, just like you were there.

This is exactly what the paper AeroDGS tries to do. But there's a huge catch: Your drone only has one eye (a single camera).

The Problem: The "One-Eye" Illusion

When you look at the world with two eyes, your brain easily figures out how far away things are because of the slight difference in what each eye sees (parallax). But a drone with one camera? It's like trying to judge the distance of a car while closing one eye.

In aerial videos, this gets even harder because:

1. Everything is far away: The cars look tiny.
2. They move fast: A car might zip across the screen in a second.
3. The drone moves too: The camera is shaking and changing angles constantly.

Because of this, standard computer programs get confused. They might think a car is floating in the sky, or that it's spinning upside down, or that it suddenly teleported. The math becomes "ill-posed," which is a fancy way of saying, "There are too many wrong answers, and the computer doesn't know which one is right."

The Solution: AeroDGS (The Physics Detective)

The authors created AeroDGS, a smart system that acts like a physics detective. Instead of just guessing where things are based on blurry pixels, it asks: "Does this make sense in the real world?"

Here is how it works, broken down into simple steps:

1. The "Lifting" Module (Building the Skeleton)

First, the system looks at the video and tries to build a rough 3D skeleton of the scene. It separates the static stuff (buildings, roads) from the dynamic stuff (cars, people).

• Analogy: Imagine you are looking at a photo of a street. You know the road is flat and the buildings are tall. The system uses this common sense to "lift" the flat 2D video into a 3D structure, even without a second camera.

2. The "Gaussian Splatting" (The Magic Dust)

Instead of building a rigid 3D model made of triangles (like a video game character), AeroDGS uses Gaussian Splatting.

• Analogy: Think of the scene as being made of millions of tiny, glowing, fuzzy clouds (Gaussians). Some clouds are stuck to the ground (the buildings), and some are attached to the cars.
• These clouds are special because they can change color, size, and position instantly. This allows the system to render the scene super fast and look incredibly realistic, like a high-definition photo.

3. The "Physics-Guided" Rules (The Strict Teacher)

This is the secret sauce. Since the drone only has one eye, the system needs rules to stop the cars from doing impossible things. It applies three "laws of physics" to the fuzzy clouds:

• Rule 1: Ground Support (The "No Flying Cars" Rule)

  • The Problem: Without a second camera, the computer might think a car is hovering 10 feet in the air.
  • The Fix: The system forces the bottom of every car to stay glued to the ground plane. If the math says the car is floating, the system pushes it down until it touches the road.
  • Analogy: It's like a strict teacher telling a student, "You can't sit on the ceiling. Your feet must be on the floor."

• Rule 2: Upright Stability (The "No Tipping" Rule)

  • The Problem: The computer might think a car is doing a backflip or leaning sideways at a 90-degree angle.
  • The Fix: The system forces cars to stay upright. They can turn left or right, but they can't tilt over like a falling tree.
  • Analogy: Imagine a toy car on a table. If you push it, it rolls. It doesn't suddenly stand on its roof. The system enforces this common sense.

• Rule 3: Trajectory Smoothness (The "No Teleporting" Rule)

  • The Problem: Because the video is shaky, the computer might think a car jumped 50 feet in one frame.
  • The Fix: The system ensures that if a car is moving, it moves smoothly. It can't teleport; it has to accelerate and decelerate naturally.
  • Analogy: Think of a smooth rollercoaster ride versus a bumpy one where you suddenly jerk forward. The system smooths out the ride so the motion looks natural.

The Result: A New Dataset

The authors realized there wasn't enough real-world data to teach these systems how to handle aerial views. So, they built a new dataset called Aero4D.

• Analogy: It's like a training gym for AI. They flew drones over real cities, recorded the footage, and carefully labeled where every car and building was. This helps other researchers test their own "physics detectives."

Why Does This Matter?

Before AeroDGS, if you wanted to create a 3D map of a city from a drone video, the moving cars would look like blurry, floating ghosts.

• With AeroDGS: You get a photorealistic 3D world where cars drive on the road, stay upright, and move smoothly.
• Real-world use: This is huge for autonomous drones (so they can navigate better), digital twins (creating perfect 3D copies of cities for planning), and emergency response (visualizing a disaster zone in 3D from a single video).

In a Nutshell

AeroDGS is a smart system that takes a shaky, single-camera drone video and turns it into a perfect 3D movie. It does this by using a magical "fuzzy cloud" representation and forcing the computer to obey the laws of physics: Cars must stay on the ground, stay upright, and move smoothly. It's like teaching a computer to "think" like a human driver, ensuring the 3D world it builds actually makes sense.

Technical Summary

1. Problem Statement

The paper addresses the challenge of monocular aerial 4D reconstruction (reconstructing dynamic 3D scenes from a single moving UAV camera). While 4D reconstruction has advanced in indoor and ground-based settings, aerial scenarios present unique, severe difficulties:

• Ill-posed Geometry: UAVs typically use lightweight monocular cameras with narrow baselines and limited parallax.
• Depth Ambiguity: High flight altitudes create large depth variations, while dynamic objects (e.g., cars) appear small with limited spatial footprints, making depth estimation highly ambiguous.
• Motion Disparity: Objects move rapidly and undergo significant illumination changes, leading to unstable motion estimation.
• Lack of Data: Existing datasets are either synthetic (lacking temporal continuity) or ground-based, failing to capture realistic aerial dynamics.

Current state-of-the-art (SOTA) methods often rely on multi-view inputs, LiDAR supervision, or ground-level priors, making them unsuitable for single-view aerial videos where geometric cues are sparse and ambiguous.

2. Methodology: AeroDGS

The authors propose AeroDGS, a physics-guided 4D Gaussian Splatting framework designed to reconstruct physically consistent 4D scenes from a single monocular UAV sequence. The framework consists of three core components:

A. Monocular Geometry Lifting Module

To overcome the lack of geometric priors, this module reconstructs reliable static and dynamic geometry from raw video:

• Initialization: Uses zero-shot 2D foundation models (depth estimation, segmentation, tracking) to generate coarse semantic and structural priors.
• 3D Reconstruction: Triangulates long-term background features to estimate camera poses and scale-consistent keypoints.
• Dynamic Object Handling: Groups pixels belonging to moving instances into object-level point sets. It fits oriented bounding boxes (via PCA) and predicts object height using a pretrained MLP (since depth cannot be inferred from single-view geometry alone).
• Refinement: Resolves ID switches and occlusions by grounding objects on physically plausible positions along camera rays, providing a dense geometric foundation for optimization.

B. Scene Representation

The scene is represented using explicit 3D Gaussian primitives:

• Unified Representation: Both static backgrounds and dynamic foregrounds are modeled as Gaussians.
• Dynamic Encoding: Moving objects are modeled as continuous 6-DoF trajectories in the Lie group $SE(3)$. Each object has a canonical space, and its motion is defined by a time-dependent transformation $T_{o,t}$.
• Appearance Modeling: To handle varying sunlight and reflections, the appearance of each Gaussian is modeled as a continuous field conditioned on spatial position, viewing direction, and time, using hash grids and spherical harmonics.

C. Physics-Guided Optimization Module

This is the core innovation that resolves the inherent ambiguity of monocular aerial views. Instead of relying solely on photometric loss, the framework incorporates differentiable physics-based constraints into the optimization loss function:

1. Ground Support ($L_{support}$): Enforces that dynamic objects maintain contact with the local ground plane. It penalizes deviations between the object's bottom and the inferred ground plane along the camera ray.
2. Upright Stability ($L_{upright}$): Constrains the vertical axis of objects (e.g., vehicles) to align with the gravity vector or ground normal, preventing unrealistic tilting or floating.
3. Trajectory Smoothness ($L_{traj}$): A second-order smoothness constraint that suppresses high-frequency jitter and enforces continuous acceleration, ensuring objects move naturally out of the frame rather than stopping abruptly.

The total loss combines photometric supervision ($L_{photo}$) with these three regularization terms, weighted to balance appearance fidelity and physical consistency.

3. Key Contributions

1. AeroDGS Framework: A novel 4D Gaussian Splatting framework specifically designed for single-sequence aerial dynamic reconstruction, integrating object decomposition, temporal association, and physics-guided optimization.
2. Physics-Guided Regularization: A new paradigm for monocular reconstruction that embeds physical priors (ground support, upright stability, trajectory smoothness) into differentiable optimization, enabling stable recovery of moving objects without multi-view constraints.
3. Aero4D Dataset: The construction of a real-world UAV dataset containing diverse urban scenes with varying altitudes, illumination, and motion patterns. It includes pixel-level instance masks, 3D bounding boxes, and 6-DoF trajectories, serving as a benchmark for aerial 4D reconstruction.

4. Experimental Results

The authors evaluated AeroDGS on both synthetic (UAV3D) and real-world (Aero4D) datasets, comparing it against SOTA methods like 4DGS, BézierGS, and 4DGF.

• Quantitative Performance: AeroDGS achieved state-of-the-art results across all metrics (PSNR, SSIM, LPIPS, and Dyn-PSNR).
  • On the real-world Aero4D dataset, it outperformed the best baseline (4DGF) by significant margins (e.g., 34.84 vs. 32.35 PSNR and 21.75 vs. 17.26 Dyn-PSNR on the Intersection-Day scene).
  • On the synthetic UAV3D dataset, it achieved 33.61 PSNR and 15.60 Dyn-PSNR, significantly outperforming other methods in dynamic regions.
• Qualitative Performance: Visual comparisons showed that AeroDGS produces sharper structures, more consistent appearances, and fewer artifacts (blurring, floating objects) compared to baselines, which often struggled with wide-baseline viewpoints and small moving objects.
• Ablation Studies: Removing any of the physics-guided constraints (ground support, upright stability, or trajectory smoothness) resulted in a noticeable drop in reconstruction fidelity and temporal coherence, validating the necessity of each component.

5. Significance and Impact

• Solving the Monocular Aerial Challenge: AeroDGS demonstrates that high-fidelity 4D reconstruction is possible from a single UAV camera by leveraging physical laws to resolve geometric ambiguities, removing the need for expensive multi-sensor setups (LiDAR) or multi-view captures.
• Real-World Applicability: The method is robust to varying altitudes, lighting conditions, and complex urban dynamics, making it suitable for applications like autonomous navigation, digital twin construction, and large-scale city monitoring.
• Dataset Contribution: The release of the Aero4D dataset fills a critical gap in the community, providing the first real-world benchmark for dynamic aerial reconstruction under realistic monocular conditions.

Limitations: The current method uses a motion threshold to separate static/dynamic objects, which may misclassify very slow-moving objects. Additionally, it does not currently reconstruct pedestrians due to their extremely small pixel footprint in high-altitude views.