🤖 machine learning

SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

This paper introduces SkyJEPA, a JEPA-style latent dynamics model with a physics-inspired prober and automated data pipeline that enables accurate long-horizon prediction and robust zero-shot sim-to-real control for agile quadrotors on embedded hardware.

Original authors: Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Drone to "Dream" Before It Flies

Imagine you are teaching a child how to ride a bicycle.

The Old Way: You tell them, "Pedal, then turn the handlebars, then pedal again." If they make a tiny mistake (like wobbling slightly), and you keep giving instructions based on that wobble, the instructions get worse and worse until they crash. This is how most current robot controllers work. They try to predict the exact future step-by-step, and small errors pile up like a snowball rolling down a hill.
The SkyJEPA Way: Instead of predicting every tiny wobble, you teach the child to understand the feeling of balance. You teach them to recognize the "shape" of a smooth ride versus a crash in their mind. Even if they wobble a bit, their internal sense of balance keeps them on track.

This paper introduces SkyJEPA, a new way to teach drones (quadrotors) to fly. It allows a drone to learn from a video game (simulation) and then fly perfectly in the real world without needing to be re-taught or having its sensors cleaned up first.

The Three Main Problems They Solved

1. The "Whisper Game" Problem (Long-Horizon Prediction)

The Issue: Most AI models try to predict the future by guessing the next second, then using that guess to guess the second after, and so on. It's like the "Whisper Game" (Telephone): Person A whispers a message to Person B, who whispers it to Person C. By the time it reaches Person Z, the message is garbled.
The SkyJEPA Solution: Instead of whispering the exact message (the exact position of the drone), SkyJEPA teaches the drone to predict the gist or the vibe of the future in a compressed "dream space" (latent space).

Analogy: Imagine predicting a movie. Instead of trying to describe every single frame (which gets messy), you predict the plot summary. Even if you miss a detail in the summary, the overall story stays on track. This stops the errors from piling up over long flights.

2. The "Black Box" Problem (Interpretability)

The Issue: Deep learning models are often "black boxes." They know how to fly, but they can't tell you why or give you numbers like "I am moving at 5 meters per second." You can't put a "black box" inside a safety controller because the controller needs real numbers to know if it's about to hit a wall.
The SkyJEPA Solution: They added a special translator called a "Physics-Inspired Prober."

Analogy: Think of the drone's brain as a chef who speaks only "Flavor Language" (the abstract dream space). The "Prober" is a translator who knows the rules of physics (like gravity and thrust). The translator takes the chef's "Flavor Language" and converts it into a recipe with exact measurements (meters, degrees, speed). This lets the drone's safety system understand exactly what is happening.

3. The "Dangerous Field Trip" Problem (Data Collection)

The Issue: To teach a robot to fly, you usually need to crash it a lot in the real world to learn from mistakes. This is expensive, dangerous, and wastes hardware.
The SkyJEPA Solution: They built a "Magic Simulator" that creates thousands of different flying scenarios automatically.

Analogy: Instead of taking a student driver out on a busy highway with real traffic (where they might crash), you put them in a driving simulator. But this simulator is special: it randomly changes the car's engine, the road conditions, the wind, and the weight of the car for every single lesson.
By training on this chaotic, randomized simulator, the drone learns to handle any real-world situation. When it finally flies outside, it's like a veteran driver who has already experienced every possible weather condition and car malfunction in the simulator.

How It Works in Real Life

The researchers tested this on a real drone in an outdoor field. Here is what happened:

Zero-Shot Transfer: They trained the drone only in the computer simulation. They never showed it a single piece of real-world data. When they took the drone outside, it flew perfectly immediately. No retraining, no fine-tuning.
The "What If" Scenarios: They tested the drone in three tricky situations:
- Normal Flight: It followed complex paths like figure-eights.
- Payload Change: They hung a heavy weight on the drone (changing its weight). The drone adjusted instantly.
- Propeller Switch: They swapped the propellers for different ones (changing how the motors work). The drone still flew smoothly.
The Result: The SkyJEPA drone was much more accurate and stable than other drones using older methods. It didn't crash, and it didn't get confused when the drone's weight or parts changed.

The Takeaway

SkyJEPA is like giving a drone a "gut feeling" for physics.

It learns by dreaming in a compressed, error-resistant way (Latent Space).
It has a translator that turns those dreams into real-world numbers (The Prober).
It trains in a chaotic, randomized video game so it's ready for anything in the real world.

The result is a drone that can fly agilely, safely, and accurately in the real world, even if it has never seen the real world before, simply because it learned the "rules of the game" perfectly in the simulation.

Technical Summary: SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

Problem Statement

Accurate dynamics models are essential for informed decision-making in agile aerial vehicles, particularly for model-based control which requires reasoning about future system evolution. However, existing neural network dynamics models face significant limitations:

Long-Horizon Instability: Most approaches use autoregressive encoder-decoder architectures that recursively feed predictions back as inputs. This mechanism amplifies small one-step errors over time, leading to drift, instability, and physically implausible trajectories as the prediction horizon increases.
Lack of Interpretability: Many learned models operate in abstract latent spaces or reconstruct full observations, failing to expose physically meaningful quantities (e.g., position, velocity, attitude) required for enforcing constraints and safety bounds in controllers.
Data Dependency and Generalization: Collecting diverse, informative real-world data is expensive, risky, and labor-intensive. Furthermore, models trained on specific platforms or tasks often fail to generalize to new configurations (e.g., payload changes, propeller swaps) without retraining.
Real-Time Constraints: High-frequency control on resource-constrained embedded hardware requires models that can be evaluated repeatedly with low latency, a challenge for complex reconstruction-based models.

The paper addresses the central question: Can we learn an efficient dynamic model for aerial robots that provides accurate long-horizon prediction, physically interpretable rollouts, real-time control capability, and zero-shot generalization across tasks, while reducing dependence on risky real-world data collection?

Methodology

The authors propose SkyJEPA, a framework based on Joint Embedding Predictive Architectures (JEPAs) tailored for real-time quadrotor control. The approach consists of four main components:

JEPA-Style Latent Dynamics Model:
Instead of autoregressively predicting future states or observations, the model learns to predict future representations in a compact latent space. It takes a history of states ( $X_t$ ) and actions ( $A_t$ ) over a window $H$ , encodes them into latent representations ( $s_t, z_t$ ), and predicts the next latent state ( $\tilde{s}_{t+1}$ ) conditioned on future actions. This formulation avoids the compounding error inherent in autoregressive state prediction by operating in a structured latent space.
Training Objective (SIGReg):
The model is trained using a two-term loss:
- Predictive Consistency ( $L_{pred}$ ): Minimizes the distance between the rolled-out latent predictions and the encoded future states.
- Anti-Collapse Regularization (SIGReg): Uses Sketched Isotropic Gaussian Regularization to prevent representation collapse. It encourages the latent embeddings to match an isotropic Gaussian distribution by comparing random one-dimensional projections of the latent distribution to a standard Gaussian. This promotes diversity without requiring complex stop-gradient mechanisms or multiple loss terms.
Physics-Inspired Prober:
To make the latent predictions usable for control, a "prober" network maps the frozen latent rollouts to physically meaningful metric states. This is a two-stage process:
- The latent dynamics model is trained first.
- The encoders and predictor are frozen, and a lightweight probing network is trained to predict residual correction terms (translational and angular acceleration) based on the latent states.
- These residuals are integrated into a differentiable kinematic model (using SO(3) exponential maps for attitude) to reconstruct the full state trajectory. This ensures the output is interpretable and respects physical constraints.
Automated Data Synthesis & Domain Randomization:
To eliminate the need for extensive real-world data collection, the authors propose a pipeline for generating diverse training data entirely in simulation:
- Reference Generation: Gaussian processes generate smooth, randomized reference trajectories that cover a broad flight envelope.
- Closed-Loop Tracking: A tracking controller (combining NMPC and MPPI) executes these references in simulation to produce physically feasible state-action pairs.
- Domain Randomization: Each rollout is simulated with randomized physical parameters (mass, inertia, drag, motor time constants, etc.) to expose the model to a family of systems rather than a single nominal model.
Sampling-Based Control (MPPI):
The learned model is integrated into a Model Predictive Path Integral (MPPI) controller. The controller samples action sequences, rolls them out using the latent dynamics model and prober, evaluates trajectory costs, and updates the control input. This runs entirely on embedded hardware (NVIDIA Orin NX) at high frequencies.

Key Contributions

JEPA for Real-Time Control: Introduction of the first JEPA-style latent dynamics model trained end-to-end for real-time quadrotor control, moving beyond high-level kinematic planning.
Physics-Inspired Probing: A novel mechanism that maps frozen latent representations to interpretable metric states via a structured kinematic integrator, enabling physically grounded long-horizon prediction.
Zero-Shot Sim-to-Real Transfer: Demonstration that a model trained entirely on domain-randomized simulation data can achieve robust control on real hardware without task-specific fine-tuning.
Automated Dataset Pipeline: A principled procedure for automated dataset generation using Gaussian processes and domain randomization, reducing reliance on unsafe real-world data collection.

Results

The framework was evaluated through open-loop prediction and outdoor closed-loop experiments on a 1.3-kg quadrotor.

Long-Horizon Accuracy: SkyJEPA significantly outperformed autoregressive predictive baselines in open-loop rollouts. It exhibited a lower compounding ratio (error growth relative to teacher-forced prediction) and lower error growth rates over long horizons (up to 60 steps).
Interpretability: The physics-inspired prober drastically improved metric-state recovery. Compared to a standard MLP prober, the PI Prober reduced position RMSE from ~5.56m to ~1.43m and attitude error from ~40° to ~4.7° in open-loop tests.
Zero-Shot Sim-to-Real Transfer: In real-world experiments, the proposed controller achieved lower position and attitude tracking errors than baselines (MPPI with predictive models) across diverse trajectories (circles, figure-8s, lemniscates). For example, on a circle trajectory, position RMSE was reduced from 0.39m (baseline) to 0.24m.
Robustness: The system maintained performance under significant platform variations without retraining, including payload additions (300g) and propeller switching. It consistently outperformed baselines in these non-nominal conditions.
Data Quality: Analysis using a Trajectory Distribution Quality (TDQ) score showed a strong inverse correlation between dataset diversity/coverage and prediction error, validating the domain-randomized data synthesis approach.
Real-Time Performance: The lightweight model (~9K parameters) and optimized inference allowed for real-time execution on embedded hardware within the 10ms control budget.

Significance

The paper claims that SkyJEPA satisfies the four key properties required for a useful world model in aerial robotics: accurate long-horizon prediction, interpretability, real-time inference, and zero-shot task generalization.

The primary significance lies in demonstrating that simulation-only training with domain randomization is sufficient for robust real-world control when paired with an appropriate learning framework. By avoiding autoregressive error accumulation through latent prediction and grounding the output with physical structure, the approach achieves a level of generalization and stability that purely predictive models cannot. The authors suggest this framework offers a scalable alternative to costly real-world data collection and manual system identification, enabling agile aerial robots to operate reliably in uncertain and changing environments. Future work is proposed to extend this to high-dimensional visual inputs and incorporate explicit safety constraints.