🤖 AI

Reinforcement Learning from Cross-domain Videos with Video Prediction Model

The paper introduces XIPER, a novel reward model that enables reinforcement learning from expert videos across visually distinct domains by training a cross-domain video prediction model to map agent observations into the expert domain and utilizing the prediction likelihood as a reward signal, thereby effectively overcoming challenges related to agent appearance differences and sim-to-real gaps.

Original authors: Zhao Yang, Xinrui Zu, Jacob E. Kooi, Thomas Delliaux, He Liu, Shujian Yu, Kevin Sebastian Luck, Vincent François-Lavet

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhao Yang, Xinrui Zu, Jacob E. Kooi, Thomas Delliaux, He Liu, Shujian Yu, Kevin Sebastian Luck, Vincent François-Lavet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to play a complex video game, like a high-speed race or a difficult puzzle. Usually, you need a teacher who can tell you exactly what to do and give you points (rewards) when you do it right. But what if your teacher is in a completely different world?

Maybe your teacher is a bright orange robot in a cartoon world, and you are a gray, heavy-duty robot in a realistic simulation. Or maybe your teacher is a real-life robot arm, and you are a digital simulation. You can see what the teacher does, but you can't talk to them, you don't know their score, and you look nothing like them. This is the problem the paper XIPER tries to solve.

Here is how XIPER works, broken down into simple concepts:

The Core Idea: "The Crystal Ball"

Instead of trying to force the two worlds to look the same (which is hard), XIPER uses a clever trick involving a Crystal Ball (a video prediction model).

The Translator (The Magic Lens): First, XIPER has a special "translator" that looks at what you (the gray robot) are doing and instantly reimagines it as if you were the orange robot. It takes your gray, heavy movements and paints them into the orange, cartoon style.
The Crystal Ball (The Predictor): Next, XIPER has a "Crystal Ball" that has watched thousands of hours of the orange robot's expert videos. This ball is really good at guessing: "If the orange robot does X right now, what will it do next?"
The Score (The Reward): Here is the magic part. XIPER feeds your "translated" orange-self into the Crystal Ball.
- If the Crystal Ball says, "Oh, I've seen this exact move before! I know exactly what happens next!" (High confidence), you get a high score.
- If the Crystal Ball is confused and says, "I have no idea what happens next; this looks weird," (Low confidence), you get a low score.

Basically, the robot learns by trying to make the Crystal Ball feel confident. If the robot's actions look like something the expert would do (even after being translated), the Crystal Ball is happy, and the robot gets a reward.

The Experiments: Proving It Works

The researchers tested this idea in two main ways:

The "Color" Test: They had agents learn tasks where the only difference was color (orange vs. gray). XIPER was great at this, beating other methods that tried to use "adversarial" (fighting) techniques to learn.
The "Body Shape" Test: They made the difference even harder by changing the agent's body shape (making it thicker). This is like a human trying to learn to walk by watching a video of a person with much wider legs. Even here, XIPER succeeded where others failed.
The "Real vs. Fake" Test: They tried to use videos of a simulated robot to teach a real physical robot arm. They didn't train the real robot to move yet, but they checked if XIPER could look at the real robot's movements and say, "Yes, this looks like the expert simulation." It worked! The scores XIPER gave the real robot matched up with what a human would consider a "good" move.

Why This Matters

Most previous methods tried to force the learner and the teacher to speak the same "visual language" or used complicated fighting algorithms that often broke down. XIPER is different because it doesn't try to change the learner; it just translates the learner's actions into the teacher's language and asks, "Does this look like something the teacher would do?"

The Limitations (What the Paper Says)

The authors are honest about two things:

Random Practice: To teach the "Translator," they used random, messy movements. If a task requires very precise, long sequences of moves, random practice might not be enough to teach the translator perfectly.
Simulation to Reality: While they showed the system could give good scores to a real robot, they haven't yet run a full training session where the real robot learns to move on its own using this method. That is the next step.

In short: XIPER is a system that lets a robot learn from a video of a totally different-looking expert by translating its own actions into the expert's style and checking if a "future-predicting" model recognizes the behavior as expert-like. If the future looks familiar, the robot gets a reward.

Technical Summary: XIPER – Reinforcement Learning from Cross-domain Videos with Video Prediction Model

1. Problem Statement

The paper addresses the challenge of training Reinforcement Learning (RL) agents to learn from expert visual demonstrations when the agent and the demonstrator exist in visually distinct domains. This "cross-domain" setting is characterized by significant domain gaps, such as differences in agent color, morphology (e.g., body shape), or the sim-to-real gap, where the agent's appearance differs fundamentally from the expert's.

Crucially, the agent has no access to:

Ground-truth reward signals.
Low-dimensional state information.
Action labels from the expert.
Paired data between the agent and expert domains.

The agent only receives raw visual observations from its own domain and a set of unlabeled expert videos from a different domain. The core difficulty lies in deriving a meaningful reward signal that guides the agent to reproduce expert-like behaviors despite these visual and dynamic discrepancies, without relying on unstable adversarial training or domain-invariant representations that often fail to scale.

2. Methodology: XIPER

The authors propose XIPER (Cross-domain Video Prediction Reward), a framework that bypasses adversarial training by using cross-domain video prediction likelihood as a reward signal. The system operates on the premise that if an agent's behavior is consistent with expert demonstrations, the agent's observations (once translated to the expert's visual style) should be highly predictable by a model trained on expert videos.

The methodology consists of two main components, both pre-trained offline and frozen during the RL phase:

Domain Translation Model ( $T_\phi$ ):
- This component maps agent observations from the agent domain ( $A$ ) to the expert domain ( $E$ ).
- Since paired data is unavailable, it is trained as an unpaired image-to-image translation problem using the Neural Optimal Transport (NOT) objective. This minimizes the distance between the distribution of translated agent images and the distribution of expert images while preserving content.
- It is trained on pre-collected random observations from both domains.
Expert Domain Video Prediction Model ( $p_\theta$ ):
- Based on VideoGPT, this generative model learns the distribution of expert behaviors by predicting the next expert frame given a sequence of past expert frames.
- It is trained on expert trajectories using a maximum likelihood objective over a context window of $k=16$ frames.

The Reward Signal:
During RL training, the agent's recent sequence of observations is first translated into the expert domain using $T_\phi$ . The video prediction model then calculates the log-likelihood of the current translated observation given the translated context. This likelihood serves as the reward ( $r_t^{xiper}$ ):
$r_t^{xiper} = \ln p_\theta(T_\phi(o_t) | T_\phi(o_{t-k:t-1}))$
Higher likelihoods indicate that the agent's behavior aligns with the expert distribution. To encourage exploration, an uncertainty-based exploration bonus is added.

The RL agent (using DreamerV3 as the backbone) optimizes this reward signal to maximize cumulative return, effectively learning to act in a way that makes its translated observations predictable by the expert video model.

3. Key Contributions

Novel Reward Formulation: Introduction of XIPER, a cross-domain reward model that enables RL agents to learn from unlabeled expert videos across different visual domains using only pixel observations, eliminating the need for ground-truth rewards, state information, or adversarial training.
Empirical Superiority: Demonstration that XIPER consistently outperforms competitive baselines (including same-domain video prediction, adversarial imitation learning, and domain-confusion methods) across the DMC Color Suite (8 tasks with color gaps) and DMC Body Suite (3 tasks with morphological gaps).
Sim-to-Real Applicability: Analysis showing that XIPER can generate meaningful reward signals for real-robot observations using only simulated expert videos, successfully bridging the sim-to-real gap in reward generation without manual annotation.
Ablation Studies: Systematic analysis of design choices, confirming that high-quality translation (low FID) is critical for precision tasks and that a specific exploration coefficient ( $\beta=1$ ) yields the most robust performance.

4. Experimental Results

The authors evaluated XIPER on 11 tasks from the DeepMind Control (DMC) suite and a Sim-to-Real dataset (Lynx Robot).

Performance: XIPER achieved higher returns than all baselines in 9 out of 12 DMC tasks.
- In the Color Suite, XIPER outperformed baselines, while the same-domain baseline (VIPER) showed some generalization but was less robust.
- In the Body Suite (morphological changes), XIPER significantly outperformed all baselines. Notably, adversarial baselines (XAIL, TPIL-Dv3) suffered from unstable learning curves and poor performance, highlighting the instability of adversarial methods even with strong RL backbones.
Reward Quality: The likelihood-based rewards correlated strongly with ground-truth task rewards on both random and expert trajectories.
Translation Generalization: The domain translation model, trained solely on random data, successfully generalized to out-of-distribution expert trajectories, accurately translating agent colors and morphologies to match the expert domain.
Sim-to-Real: In the Lynx Robot experiment, the translation model successfully filtered real-world visual noise (cabling, hardware) to produce simulation-style images. The resulting XIPER rewards on real-robot episodes correlated well with ground-truth task rewards, validating the potential for sim-to-real transfer.

5. Significance and Claims

The paper claims that XIPER offers a scalable and stable alternative to adversarial imitation learning for cross-domain visual RL. By leveraging generative video prediction and unpaired translation, it removes the need for domain-invariant representations and the instability associated with GAN-based training.

The authors emphasize that their approach unlocks the vast pool of online videos as a source of expert demonstrations, removing the need for laborious reward engineering and costly domain-specific data collection. While the current work validates the reward signal's quality and the agent's ability to learn in simulation, the authors modestly note that closing the loop with online RL training on physical robots remains a necessary future step. They also acknowledge that training the translation model solely on random data may be a bottleneck for tasks requiring precise, long-horizon behaviors, suggesting future work could explore advanced exploration strategies or joint optimization.