💻 computer science

Why Latent Actions Fail, and How to Prevent It

This paper analytically demonstrates that standard latent action models fail because they inadvertently encode exogenous background changes, and proves that focusing on endogenous components or using auxiliary objectives like action-supervision effectively mitigates this interference.

Original authors: Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to "See" Actions

Imagine you want to teach a robot how to cook by showing it thousands of videos of people cooking. However, you don't have labels telling the robot what the person is doing (e.g., "chopping," "stirring"). You just have the raw video.

To solve this, scientists use Latent Action Models (LAMs). Think of a LAM as a student trying to guess the "action" just by looking at the difference between two video frames. If the video shows a hand moving a spoon from a bowl to a pot, the model should learn that this movement equals "stirring."

The Problem: The "Distracted Student"

The paper argues that when these models try to learn from real-world videos ("in-the-wild"), they often fail. Why? Because real videos are messy.

Imagine you are trying to learn how to juggle by watching a video.

The Good Stuff (Endogenous): The juggler's hands moving the balls. This is the action you want to learn.
The Bad Stuff (Exogenous): The background crowd cheering, a camera shaking, or a bird flying past. This is "noise."

The paper shows that standard LAMs are like distracted students. Instead of focusing on the juggler's hands, they accidentally memorize the background noise. If the camera shakes in the video, the model thinks, "Ah, the 'shaking' is the action!" It learns the wrong thing.

Why Does This Happen? (The "Future Leak")

The authors explain this with a clever trick called Future Leakage.

Imagine the model is trying to predict the next frame of the video based on the current frame and a guessed "action."

The Shortcut: It's hard to predict exactly how the background will change. But it's easy to just copy the background from the next frame if you have access to it.
The Mistake: The model realizes, "Hey, if I hide the background information inside my 'action' guess, I can just copy it to the next frame and get a perfect score!"

So, the model starts stuffing the "action" label with information about the background (like the camera angle or the crowd) because it helps it cheat on the test. It stops learning the actual robot movement and starts learning the camera movement.

The Solution: Two Ways to Fix the Student

The paper proposes two main ways to stop the model from cheating and force it to focus on the real action.

1. The "Multi-View" Trick (Cross-Exogenous Reconstruction)

Imagine you are watching the juggler from two different cameras at the same time.

Camera A sees the juggler with a red background.
Camera B sees the juggler with a blue background.
The Action: The juggler throws a ball. This action is the same in both videos.

The paper suggests training the model to look at both videos. If the model tries to guess the action based on the red background, it will fail when looking at the blue background. The only thing that stays consistent across both views is the juggler's hand movement.

By forcing the model to find the "common denominator" between different views (or different backgrounds), it learns to ignore the background noise and focus only on the action.

2. The "Teacher's Answer Key" (Exogenous-Robust Supervision)

Sometimes, you can't get two cameras, but you might have a tiny bit of help. Maybe you know the type of movement (like "optical flow" or a few labeled actions) but not the full story.

The paper suggests giving the model a "target" that is immune to the background.

If you ask the model to predict "how much the background changed," it will fail.
If you ask the model to predict "how much the hand moved," it will succeed, because the hand movement is the same whether the background is red or blue.

By training the model to predict these "background-proof" targets, you force the model to align its "action" guesses with the real physical movement, ignoring the noise.

The Proof: It Works in Theory and Practice

The authors didn't just guess; they did two things:

Math: They built a simplified, mathematical version of the problem (like a cartoon version of a robot) and proved that without these fixes, the model will learn the background. They also proved that the two fixes above mathematically force the model to ignore the background.
Experiments: They tested this on both simple math models and complex, real-world-like AI models (using neural networks).
- Result: When they used the "Multi-View" trick or the "Teacher's Answer Key," the models stopped learning the background noise. They became much better at identifying the actual actions, even when the videos were full of distractions.

Summary

The Problem: AI models trying to learn actions from videos get distracted by background noise (camera shakes, moving crowds) and learn the wrong things.
The Cause: The models cheat by hiding background details inside their "action" guesses to make predictions easier.
The Fix: Force the model to find what stays the same across different backgrounds (Multi-View) or train it to predict things that don't change with the background (Robust Targets).
The Outcome: The models stop cheating, ignore the noise, and actually learn the robot's movements.

Technical Summary: Why Latent Actions Fail, and How to Prevent It

Problem Statement

Latent Action Models (LAMs) aim to learn action-relevant representations from unlabeled videos by compressing frame-to-frame changes, offering a solution to the scarcity of action-labeled datasets in real-world robotics. However, in "in-the-wild" videos, observations contain not only the agent's endogenous state but also exogenous states (e.g., background clutter, camera motion, lighting changes). These exogenous factors introduce frame transitions unrelated to the agent's actions.

Prior works have observed that LAMs trained with standard reconstruction objectives often fail to align with ground-truth actions in such environments. Instead of learning action dynamics, latent actions spuriously encode exogenous noise. While empirical remedies exist (e.g., auxiliary supervision, multi-view data), there has been no unified theoretical analysis explaining why standard objectives fail in the presence of exogenous noise or why specific auxiliary objectives mitigate this failure.

Methodology

The authors propose a unified theoretical framework based on the Exogenous Block Markov Decision Process (Ex-BMDP) to analyze latent action learning.

1. Theoretical Framework: Ex-BMDP

The authors extend the standard Block MDP (BMDP) to explicitly model exogenous noise. In this formulation:

The latent state $x$ is decomposed into an endogenous state $s$ (controllable by the agent) and an exogenous state $\xi$ (independent of the agent's actions).
The transition functions and initial distributions are decoupled: $T(x'|x, a) = T_s(s'|s, a)T_\xi(\xi'|\xi)$ .
Observations are generated via a deterministic emission map $o = h(s, \xi)$ , which includes a vision encoder $f$ .

2. Linear LAM Analysis

To ensure theoretical tractability, the authors first analyze a Linear LAM setting where:

Observation transitions are additive: $o' = o + \phi_\xi(a) + \epsilon$ , where $\phi_\xi(a)$ is the action effect and $\epsilon$ is exogenous noise arising from state changes ( $\xi \to \xi'$ ).
The inverse dynamics model ( $\psi_{IDM}$ ) and forward dynamics model ( $\psi_{FDM}$ ) are linear maps.
The standard objective is minimizing the reconstruction error $L_{LAM} = \mathbb{E}[\|o' - \hat{o}'\|_2^2]$ .

3. Auxiliary Objectives

The authors reformulate two families of existing auxiliary objectives within the Ex-BMDP framework:

Cross-Exogenous Reconstruction ( $L_{X-exo}$ ): Uses paired observations from different exogenous states ( $\xi, \tilde{\xi}$ ) sharing the same endogenous state to enforce consistency.
Exogenous-Robust Target Prediction ( $L_{\xi-robust}$ ): Supervises the latent action with a target $y$ (e.g., ground-truth actions or optical flow) that is robust to exogenous variations (i.e., $y$ remains similar regardless of $\xi$ ).

Key Contributions

1. Identification of "Future Leakage"

The authors prove that minimizing the standard reconstruction objective $L_{LAM}$ in the presence of exogenous noise causes future leakage.

Mechanism: Since the forward model $\psi_{FDM}$ must reconstruct the future observation $o'$ (which contains the future exogenous state $\xi'$ ), and $\xi'$ is not available as an input, the latent action $z$ is forced to encode information about $\xi'$ .
Result: Proposition 4.2 shows that no global minimizer of $L_{LAM}$ admits a latent action independent of $\xi'$ . Consequently, $z$ absorbs exogenous information from the future frame, reducing the capacity available for encoding the actual action dynamics.

2. Role of Vision Encoder Sensitivity

The analysis reveals that the energy of the exogenous noise in the latent space is governed by the sensitivity of the vision encoder to exogenous changes.

The noise energy is proportional to the frequency of exogenous state switches and the sensitivity of the emission function $h$ to $\xi$ .
Empirical results show that encoders like DINOv2, which are object-centric and robust to background variations, significantly reduce this sensitivity compared to raw pixel inputs, leading to better action alignment.

3. Theoretical Justification of Auxiliary Objectives

The paper provides proofs showing how auxiliary objectives mitigate exogenous interference by enforcing consistency across exogenous states:

Cross-Exogenous Reconstruction: Under linear assumptions, minimizing $L_{X-exo}$ is equivalent to Canonical Correlation Analysis (CCA). This suppresses transition components that vary with specific exogenous states while preserving components that persist across them (i.e., action-driven changes).
Exogenous-Robust Prediction: Proposition 4.8 demonstrates that predicting a target $y$ that is $\eta$ -robust to exogenous states provides an explicit bound on the mismatch of latent actions across different exogenous states. Minimizing this loss forces the latent action to be consistent regardless of the background.

Experimental Results

Linear LAM Experiments

Using synthetic data with controlled exogenous switching probabilities ( $p_{switch}$ ):

Future Leakage Validation: Experiments confirmed that latent actions encode future exogenous states. When the exogenous state of the next frame was swapped during inference, the reconstruction output shifted to match the swapped frame, confirming the "shortcut" behavior.
Noise Sensitivity: As the sensitivity of the observation to exogenous changes ( $\alpha$ ) increased, the Normalized Mean Squared Error (NMSE) for action prediction worsened, validating the theoretical link between encoder sensitivity and failure.
Auxiliary Efficacy: Adding $L_{X-exo}$ or $L_{\xi-robust}$ to the standard loss consistently improved action alignment and reduced the variance of latent actions across exogenous states.

Nonlinear (Practical) LAM Experiments

To test generalizability, the authors trained a nonlinear LAM (VQ-VAE with CNN/UNet) on a grid-world dataset with multiplicative exogenous noise (violating the linear assumptions):

Vanilla LAM: As noise intensity increased, the model encoded exogenous regions, leading to high consistency loss and low exogenous region MSE (meaning it predicted the noise).
Augmented LAM: Models trained with $L_{X-exo}$ or $L_{\xi-robust}$ maintained high exogenous region MSE (ignoring the noise) and low consistency loss across all noise levels.
Conclusion: The theoretical insights regarding future leakage and the benefits of consistency-enforcing objectives hold even when the linearity and additive assumptions are violated.

Significance and Claims

The paper claims to provide a unified theoretical analysis of latent action learning failures in the presence of exogenous noise. Its primary significance lies in:

Explaining Failure Modes: It formally identifies "future leakage" as the mechanism by which standard reconstruction objectives degrade action learning, rather than just observing the phenomenon empirically.
Unifying Remedies: It demonstrates that diverse auxiliary objectives (multi-view, action supervision, optical flow) share a common theoretical mechanism: they enforce consistency of latent actions across exogenous states.
Guiding Design: It highlights the critical importance of the vision encoder's sensitivity to exogenous factors, suggesting that robust feature extractors are a prerequisite for effective LAMs.

The authors conclude that while their theoretical proofs rely on linear assumptions, the empirical validation on nonlinear models suggests these principles extend to practical, real-world latent action learning scenarios.