⚡ electrical engineering

Ground-JEPA: Learning Physically Grounded Latent World Models for Zero-Shot Dynamics Generalization of Legged Robots

Ground-JEPA introduces a minimal, physics-grounded latent world model that achieves zero-shot dynamics generalization and superior control performance for legged robots in high-DoF environments without relying on task-specific rewards, thereby challenging the assumption that reward-free learning is inherently inferior for complex robotic systems.

Original authors: Yu-Xiang Wu, Yuyan Wu

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu-Xiang Wu, Yuyan Wu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Robot's Inner Compass

Imagine trying to teach a robot to walk across a rocky field. You could give it a strict set of rules, like "lift your foot exactly 10 centimeters," but the real world is messy; rocks shift, the ground is slippery, and the robot's legs might be slightly heavier than expected. This is the challenge of legged robot control: getting machines to move smoothly and safely in unpredictable environments. For a long time, scientists have tried to solve this by building "dynamics models"—essentially, a mental map inside the robot that predicts what will happen next if it moves a certain way.

Traditionally, these mental maps were built by rewarding the robot for doing good things (like staying upright) and punishing it for bad things (like falling). This is called reward-based learning. However, there's a catch: if the robot makes a tiny mistake in its prediction, that mistake can pile up over time, like a snowball rolling down a hill, until the robot's mental map is completely wrong and it crashes. Another approach, called Joint-Embedding Predictive Architectures (JEPAs), tries to skip the messy details of the real world. Instead of predicting exactly where every pixel of the robot's camera will be, it predicts a compressed "summary" or "latent" version of the future. Think of it like a musician predicting the next note in a song by feeling the rhythm, rather than calculating the exact vibration of the air. The big question scientists have been asking is: Can a robot learn to walk perfectly just by understanding this internal rhythm, without needing a teacher to constantly tell it "good job" or "try again"?

Ground-JEPA: The Robot That Learned to Walk by Feeling the Beat

A team of researchers has introduced a new system called Ground-JEPA, which suggests that robots might not need constant praise or punishment to learn complex movements. Instead, they can learn by simply trying to keep their internal "feeling" of the future consistent with reality.

Imagine a robot learning to walk not by being told "don't fall," but by trying to make sure its internal sense of "where I am going" matches its actual movement. In this new system, the robot doesn't try to reconstruct a perfect video of the future. Instead, it compresses its experience into a tiny, efficient "latent space"—a kind of abstract mental shorthand. The core trick here is a physics-inspired probe. While the robot's internal map is abstract, this probe acts like a translator, converting those abstract thoughts back into real-world physics like position, speed, and orientation. It's like having a robot that thinks in pure concepts but has a built-in translator that ensures those concepts still obey the laws of gravity and momentum.

The researchers found that the secret to making this work wasn't just the translation, but the planning horizon. Previous attempts failed because the robot was only looking a few steps ahead—like trying to walk by looking only at your toes. The Ground-JEPA system forces the robot to look much further ahead, specifically 12 steps into the future. This is crucial because a full walking cycle (a gait) takes about that long. By looking at the whole cycle, the robot can see the "big picture" of its movement, rather than just optimizing the next tiny step. This allows the system to learn stable walking patterns purely through self-supervision, meaning it learns by checking if its internal predictions are consistent, without needing any external rewards.

The results from their simulations are quite surprising. On a simulated four-legged robot (quadruped), this reward-free system achieved a performance score of 510 ± 68, which was actually better than the traditional reward-based system that scored 450 ± 234. Even more impressively, they successfully applied this to a simulated humanoid robot (a robot with a human-like body) for the first time, achieving a score of 350 ± 50 without any reward signals. The traditional method failed completely on the humanoid in this setup.

One of the most exciting findings is how well this system handles "extreme" changes. The researchers tested the robot in scenarios where the physics were drastically altered: the robot's mass was changed by ±30%, the ground friction was multiplied by 2.5 or reduced to 0.4, and there was a 50 ms delay in the motors. In a compound scenario where all these changes happened at once, the Ground-JEPA model retained 78% of its original performance. In contrast, a traditional reward-based model, even when trained with random variations, only kept 21% of its performance. This suggests that by learning the fundamental "manifold" or shape of movement, the robot becomes much more robust to changes in the world.

The team also discovered that the size of the model didn't need to be massive. This entire system runs on a laptop GPU and uses only about 1.3 million parameters, which is tiny compared to other massive AI models that require billions of parameters. They showed that the "collapse" (where the robot stops learning) seen in previous studies wasn't because the method was flawed, but because the robot was using the wrong type of internal map (specifically, a normalized map called SimNorm) and looking too far ahead without the right structure. By using raw latent representations and the correct planning horizon, the system works beautifully.

However, the authors are careful to note that these results are currently based on simulations. While the robot learned to walk in a virtual world with extreme changes, the system has not yet been tested on a physical robot in the real world. Additionally, while they solved the problem of a humanoid standing still, getting a humanoid to walk dynamically on two legs remains a challenge due to the complex physics of balancing on one foot.

Ultimately, Ground-JEPA suggests a new path for robotics: instead of spending months engineering perfect reward functions to teach a robot how to walk, we might just need to teach it to keep its internal rhythm consistent with the laws of physics. This could democratize advanced robot control, allowing smaller labs with standard laptops to build robots that can adapt to the real world without needing massive supercomputers or endless trial-and-error rewards.

Technical Summary: Ground-JEPA

Problem Statement

Accurate dynamics models are essential for the model-based control of legged robots, particularly for tasks requiring long-horizon planning to ensure stability and prevent falls. However, existing neural network approaches often rely on autoregressive prediction in state space, leading to significant error accumulation over time. Furthermore, prior attempts to apply Joint-Embedding Predictive Architectures (JEPAs) to robotics have faced two major hurdles:

Scalability: Successful JEPA implementations (e.g., V-JEPA) have required massive pre-training on foundation-model scales (billions of parameters), making them inaccessible to resource-constrained labs.
Reward-Free Limitations: Previous research suggested that reward-free learning via latent consistency degrades severely in high-degree-of-freedom (DoF) environments (e.g., quadrupeds and humanoids), leading to the hypothesis that latent consistency is fundamentally underconstrained for complex dynamics.

Methodology

The authors introduce Ground-JEPA, a minimal (~1.3 million parameters) latent world model designed to control legged robots entirely through latent consistency, without relying on task-specific reward functions. The system comprises three core components:

Encoder ( $E_\phi$ ): A 3-layer MLP that maps proprioceptive observations ( $o_t$ ) to a latent representation ( $z_t$ ).
Predictor ( $P_\psi$ ): An action-conditioned predictor that forecasts future latent states ( $\hat{z}_{t+1}$ $\overset{z}{^}_{t + 1}$ ).
- For high-DoF tasks (quadrupeds, humanoids), the authors employ a lightweight Transformer Decoder with cross-attention between latent and action embeddings to capture long-range temporal dependencies essential for gait coordination.
- For lower-DoF tasks (cheetah), a 2-layer GRU is used.
Physics-Inspired Prober: A novel component that maps frozen latent representations to interpretable physical states (position, velocity, orientation, angular velocity). Unlike a standard decoder, this prober incorporates kinematic inductive biases (e.g., $p_{t+1} = p_t + v_t \Delta t + \frac{1}{2}a_{comp}\Delta t^2$ ) to bridge abstract latent dynamics with physical constraints.
Planner: A Model Predictive Path Integral (MPPI) planner optimizes action sequences in the latent space. Crucially, the planning horizon ( $H$ ) is extended to 12 steps to capture full gait cycles, addressing the limitations of shorter horizons found in prior work.

Training Objective:
The model is trained using a JEPA objective that minimizes the squared error between the predicted latent state and the target latent state (derived from an EMA target encoder), without a value head or reward signal:
$L = \mathbb{E}[\| \text{sg}(E_\phi(o_{t+1})) - P_\psi(E_\phi(o_t), a_t) \|^2]$

Key Contributions

Laptop-Scale JEPA Control: Demonstrates that JEPA-based control for legged robots can match or exceed reward-grounded baselines using a model with only ~1.3M parameters, trained in under 20 minutes on a laptop GPU, eliminating the need for billion-parameter foundation models.
Physics-Inspired Probing: Introduces a probing mechanism that enforces physical interpretability and structure on latent representations, reducing position and orientation prediction errors significantly compared to linear or standard MLP probes.
Re-evaluation of Reward-Free Learning: Challenges the prevailing hypothesis that reward-free learning fails in high-DoF systems. The authors identify that prior failures were artifacts of insufficient planning horizons (e.g., $H=3$ ) and inappropriate latent parameterization (e.g., SimNorm), rather than a fundamental flaw in latent consistency.
Zero-Shot Generalization: Establishes that self-supervised latent consistency provides a robust signal for sim-to-real transfer, achieving high performance retention under extreme compound dynamics shifts without fine-tuning.

Results

Performance on Control Tasks

Ground-JEPA was evaluated on the dm_control suite. The reward-free model (trained without task rewards) outperformed or matched reward-grounded baselines:

Cheetah-Run (6-DoF): 530 ± 45 (Reward-Free) vs. 522 ± 139 (Reward-Grounded).
Quadruped-Walk (12-DoF): 510 ± 68 (Reward-Free) vs. 450 ± 234 (Reward-Grounded). The reward-free model showed significantly lower variance.
Humanoid-Stand (21-DoF): 350 ± 50 (Reward-Free). This marks the first successful demonstration of reward-free JEPA control on a humanoid platform, where reward-grounded baselines failed.

Impact of Planning Horizon

Ablation studies on the Quadruped-Walk task revealed that the planning horizon is critical:

H=3 (Prior Work): 184 ± 141 (Chaotic gait, no phase coordination).
H=12 (Ours): 510 ± 68 (Stable trot gait, phase-locked).
H=20: 420 ± 85 (Performance degraded due to noise accumulation).
The results indicate that a horizon covering a full gait cycle is necessary to provide sufficient structural signal for the consistency objective.

Latent Parameterization

The authors found that "collapse" in reward-free learning is an artifact of latent parameterization, not the absence of rewards.

Raw Latents: Reward-free training achieved 496 ± 31, matching reward-grounded performance.
SimNorm Latents: Reward-free training collapsed completely (return $\to$ 4), confirming that raw latent spaces preserve richer representational structure for self-supervised learning.

Zero-Shot Generalization

Under extreme compound dynamics shifts (Mass ±30%, Friction 2.5×, Actuator Latency +50ms), Ground-JEPA retained 78% of its nominal performance. In contrast, a reward-grounded baseline trained with domain randomization retained only 21%, as its reward function became misaligned with the shifted dynamics.

Significance and Claims

The paper claims to fundamentally redefine the role of rewards in model-based control for legged robots. The authors argue that in structured dynamical systems, inherent physics (kinematics and contacts) serves as a stronger inductive bias than sparse task rewards, provided the planning horizon is sufficiently long.

Key implications highlighted by the authors include:

Democratization: State-of-the-art model-based control is accessible to resource-constrained labs without requiring massive pre-training or foundation models.
Algorithmic vs. Scale: Success is driven by algorithmic breakthroughs (physics-inspired probing, long-horizon consistency) rather than model scale.
Robustness: Self-supervised learning naturally captures the fundamental manifold of legged motion, leading to superior generalization to unseen dynamics compared to reward-engineered approaches.

Limitations: The authors note that all experiments were conducted in simulation. Physical robot validation is deferred to future work. Additionally, while "humanoid-stand" was achieved, dynamic bipedal walking remains an open challenge due to the ill-conditioning of underactuated ankles, which the authors suggest may require hierarchical extensions.