💻 computer science

Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls

The paper introduces **Aero-World**, a method that adapts pretrained image-to-video diffusion models to generate controllable aerial videos from inertial control signals by injecting action tokens and utilizing a frozen latent-space Physics Probe for supervision, alongside the **AeroBench** benchmark to evaluate action alignment and physical consistency.

Original authors: Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-talented artist who can paint incredibly realistic videos of the world just by reading a story description. They are great at making things look pretty, but if you ask them to "fly a drone forward while spinning left," they might just guess. They might make the drone fly forward, but the spinning could be too slow, too fast, or happen at the wrong time. In the real world, if a drone pilot makes a tiny mistake in their controls, the drone could crash or drift wildly off course.

The paper introduces Aero-World, a new way to teach these "artist" AI models to become precise drone pilots. Instead of just guessing, Aero-World lets the AI "feel" the exact controls a human pilot would use.

Here is how they did it, using some simple analogies:

1. The Problem: The "Guessing Game"

Current video AI models are like actors who have memorized thousands of movies. If you say, "Show me a car chase," they can make a great one. But if you say, "Accelerate exactly 5 meters per second squared, then turn left," they don't really understand the physics. They just try to make it look like a turn. For a real drone, which flies in 3D space with six different ways to move (forward/back, up/down, left/right, and three types of spinning), this "guessing" leads to unstable, unrealistic videos.

2. The Solution: The "Flight Simulator" Approach

The researchers wanted to take a pre-trained video model (the talented artist) and teach it to follow IMU signals. Think of an IMU (Inertial Measurement Unit) as the "black box" in a plane or drone that records exactly how the vehicle is accelerating and spinning at every millisecond.

They didn't want to retrain the whole artist from scratch (which is expensive and slow). Instead, they used a clever two-step trick:

Step A: The "Action Token" Stream: They gave the AI a new language. Instead of just reading a story, the AI now receives a stream of numbers representing the exact push and spin commands (like a pilot's joystick movements).
Step B: The "Frozen Physics Probe" (The Strict Coach): This is the most creative part. Imagine you are learning to juggle. You have a coach who watches you and yells, "Too high!" or "Too slow!"
- In this paper, the "coach" is a small, separate AI model called the Physics Probe.
- This coach was trained first on real drone footage and its actual flight data. It knows exactly what a video should look like if the drone is doing a specific spin or acceleration.
- Crucially, this coach is frozen. It doesn't learn; it just watches. During the training of the main video artist, the coach constantly checks: "Does this generated video actually match the spin command you were given?"
- If the video looks like a spin but the math says it's too slow, the coach gives a "penalty" (a signal to correct the artist).

3. The Result: A "Controlled" Drone

By using this "Strict Coach," the main video model learned to generate aerial videos that aren't just pretty, but physically honest.

If you tell it to surge forward, the video shows the ground rushing by at the right speed.
If you tell it to spin aggressively, the video blurs and tilts exactly as a real drone would.

They tested this on a new benchmark called AeroBench (like a driving test for AI drones). They found that their method made the AI much better at following the controls compared to other methods, without making the video look blurry or weird.

Summary Analogy

Think of the original video AI as a tourist taking a selfie video. They can walk around and film nice scenery, but they don't know how to drive a race car.

Aero-World is like giving that tourist a race car simulator and a strict driving instructor (the Physics Probe). The instructor watches the tourist's hands on the wheel and the car's speedometer. If the tourist tries to turn too sharply without slowing down, the instructor corrects them immediately. The result? The tourist can now film a video that looks exactly like a professional race car driver, even though they are just following the instructor's strict rules.

What the paper claims:

They created a method to make video generators follow precise drone controls (acceleration and spinning).
They used a "frozen" helper model (the Physics Probe) to check if the video matches the controls without needing to decode the video into full color every time (saving computer power).
They proved this works better than just telling the AI to "try harder" with the controls, resulting in videos that are both high-quality and physically accurate.

Technical Summary: Aero-World

Problem Statement

Foundation video models have achieved impressive perceptual realism but remain limited in their application to embodied AI, particularly for aerial flight. Current models are primarily trained on natural language rather than low-level control signals, leading to a lack of action-faithfulness and physical consistency. This gap is critical for aerial agents because drones operate in an unconstrained 6-degree-of-freedom (6-DoF) space where translational and rotational dynamics interact non-linearly. Small errors in ego-motion (e.g., roll or pitch) can induce large spatial drifts, and aggressive maneuvers often amplify temporal inconsistencies.

Existing approaches to controllable video generation often rely on coarse text prompts, camera parameters, or intermediate annotations (e.g., optical flow, depth) that are either unavailable to embodied agents during training or do not match the agent's native control interface (6-DoF inertial signals). Furthermore, scaling data and model size has not been shown to correlate well with action adherence, and obtaining paired video-IMU annotations is costly. Consequently, there is a need for a method to adapt pretrained video generators to generate aerial videos that strictly follow fine-grained inertial action signals (linear acceleration and angular velocity) without requiring expensive retraining from scratch.

Methodology: Aero-World

The authors propose Aero-World, a framework that converts a pretrained image-to-video diffusion model into a controllable aerial video generator conditioned on dense 6-DoF inertial commands. The approach leverages the visual priors of existing large-scale models while introducing two key mechanisms to enforce action-motion consistency:

1. Action-Token Injection

The model accepts a sequence of 6-DoF inertial actions ( $a_{1:T}$ ), where each action vector contains translational acceleration and angular velocity. These actions are embedded via a lightweight MLP and injected into the diffusion transformer blocks through a cross-attention mechanism, creating a separate action-token stream alongside visual and timestep conditioning. The authors employ causal action alignment, where the command $a_{t+1}$ conditions the transition to frame $\hat{I}_{t+1}$ , ensuring the model learns the relationship between motion changes and visual transitions rather than static appearances.

2. Frozen Latent-Space Physics Probe

To enforce physical consistency without computationally expensive video decoding, the authors introduce a Physics Probe.

Training: The probe ( $\Phi$ ) is trained independently on real synchronized video-IMU pairs. It operates in the latent space of a frozen 3D VAE, mapping video latents to discretized bins of inertial states (linear acceleration and angular velocity) for each of the six axes.
Architecture: It consists of lightweight 3D convolutional layers, global spatial pooling, and an MLP classifier. The use of discretized bins (quantization) addresses the heavy-tailed distribution of IMU data in aggressive flight and focuses on recovering the correct motion regime (e.g., accelerating vs. turning) rather than precise sensor-grade reconstruction.
Supervision: During the LoRA finetuning of the generator, the Physics Probe remains frozen. It evaluates the denoised latent prediction ( $\hat{z}_0$ ) and provides a differentiable loss signal ( $L_{physics}$ ) that penalizes discrepancies between the probe-inferred motion and the commanded action sequence.
Inference: The probe is discarded at inference time; the model generates videos using only the initial frame and the IMU action sequence.

Training Strategy

The framework utilizes Low-Rank Adaptation (LoRA) to finetune the pretrained diffusion backbone. The training objective combines the standard denoising loss ( $L_{diff}$ ) with the physics consistency loss ( $L_{physics}$ ):
$L_{total} = L_{diff} + \lambda_{phy}L_{physics}$
A warm-up stage is employed where $\lambda_{phy} = 0$ initially to allow the action pathway to interact with the pretrained backbone before enabling the physics supervision.

Key Contributions

Inertia-Guided Adaptation: A method to adapt pretrained diffusion video backbones for IMU-conditioned aerial video generation using latent-space inertial supervision, avoiding the need for training from scratch.
Physics Probe Supervision: A frozen, differentiable probe that provides action-motion consistency signals during lightweight LoRA finetuning, enabling the model to respect 6-DoF motion constraints while preserving visual priors.
AeroBench and Metrics: The introduction of a new benchmark and two specific metrics for evaluating action-conditioned aerial generation:
- Action Alignment Score (AAS): Measures bin-level agreement between commanded IMU actions and probe-inferred motion.
- Physical Consistency Rate (PCR): Measures frame-to-frame oscillatory instability (temporal stability) based on the derivatives of probe-predicted bins.

Experimental Results

The authors evaluated Aero-World on AeroBench using datasets from UZH FPV drone racing and TII drone racing.

Visual Fidelity vs. Control Trade-off: Aero-World achieves a superior trade-off compared to baselines like AirScape and action-only finetuning. It maintains lower Fréchet Video Distance (FVD) of 596.5 (vs. 1058.6 for AirScape) and higher Structural Similarity (SSIM) of 0.595 (vs. 0.505), indicating better perceptual quality and structural similarity.
Action Alignment: The method improves the mean Action Alignment Score (AAS) from 57.7 (action-only finetuning) to 63.6.
Temporal Stability: Physics supervision slightly reduces the Physical Consistency Rate (PCR) from 0.033 to 0.029, indicating smoother frame-to-frame dynamics.
Independent Validation: Using an independent Flow-IMU evaluator (trained on RGB flow to IMU regression, distinct from the latent probe), Aero-World achieved the highest mean correlation (0.44) compared to Action FT (0.30) and AirScape (0.20). Notably, Aero-World showed positive correlations across all six axes, whereas AirScape showed negative correlations on several.
Qualitative Performance: The model successfully generates stable, action-faithful motion in both in-distribution maneuvers and out-of-distribution scenes, preserving ego-motion even when the scene distribution differs from training data.

Significance and Claims

The paper claims that frozen Physics Probe supervision is a practical and effective mechanism for adapting pretrained video generators to be more responsive to low-level aerial motion controls. By operating in latent space, the method avoids the computational cost of decoding videos during training while still enforcing physical consistency.

The authors position Aero-World as a solution to the "action-faithfulness" gap in foundation video models, enabling the safe and scalable training of robust control policies for aerial agents entirely in software. The work highlights that while visual realism can be achieved through scaling, achieving physical consistency requires explicit, data-driven supervision mechanisms like the proposed Physics Probe. The authors modestly note that a fidelity-control trade-off remains, suggesting this is an area for future work.