💻 computer science

MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction

The paper proposes MVP-LAM, a multi-viewpoint model that learns action-centric latent actions through cross-viewpoint reconstruction to improve VLA pretraining and downstream robotic manipulation performance.

Original authors: Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, Jungwoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Robots Need to Learn, But Videos Lie

Imagine you want to teach a robot how to make a sandwich. The best way to learn is usually by watching a human do it. But here's the catch: videos don't come with a "remote control" attached.

When you watch a video of someone making a sandwich, you see the result (the bread moves, the knife cuts), but you don't see the exact muscle movements (the "actions") the human used. Robots usually need those exact muscle movements to learn.

To get around this, scientists have been trying to teach computers to "guess" the actions by watching how the video changes from one frame to the next. They call these guesses "Latent Actions." Think of these as a secret code the computer invents to describe "what happened between these two pictures."

The Trap:
The problem is that videos are noisy. If a person moves their hand to grab a cup, the video changes. But if the camera also moves, the video changes too.

The Mistake: A computer might look at the video and think, "Ah, the cup moved because the camera spun around!" instead of "The cup moved because the hand pushed it."
The Result: The robot learns the wrong lessons. It learns to react to camera movements rather than actual hand movements. This is like a student studying for a test by memorizing the font size of the questions instead of the answers.

The Solution: MVP-LAM (The "Two-Camera" Trick)

The authors propose a new method called MVP-LAM (Multi-ViewPoint Latent Action Model). Their secret sauce is using two cameras (or multiple viewpoints) recording the same event at the same time.

Here is the analogy:
Imagine you are watching a magician perform a trick.

Camera A is standing on the left.
Camera B is standing on the right.

If the magician moves their hand, both cameras see the hand move.
If the magician doesn't move, but Camera A gets bumped and shakes, only Camera A sees the shake. Camera B sees a still image.

How MVP-LAM Works:
Instead of just watching one camera and guessing the action, MVP-LAM plays a game of "cross-checking":

It looks at the action from Camera A.
It tries to predict what Camera B will see in the next second.
The Rule: If the "secret code" (the latent action) is just about Camera A moving, it won't help predict what Camera B sees. The only thing that both cameras agree on is the actual movement of the objects.

By forcing the computer to explain the future of View B using the action from View A, the computer is forced to ignore the camera movement and focus only on the real action (the hand moving the cup). It's like asking two people to describe a secret; if they only agree on the parts that are actually true, you know you've found the truth.

What They Found

The paper tested this on a dataset called Bridge V2 (a collection of robot videos) and compared it to other methods.

Better "Secret Codes": The codes MVP-LAM invented were much better at describing the actual robot movements. They contained 62% more useful information about the real actions than previous methods.
Robustness: Even if the camera angle changed slightly (like if the robot's head tilted), the system still knew what the robot was doing. It didn't get confused by the new angle.
Better Robot Performance: When they used these "better codes" to train a robot to do tasks (like stacking blocks or picking up objects), the robot learned faster and did better.
- The Result: A robot trained with MVP-LAM could solve complex puzzles (in a simulation) using 3 times less training data than other robots. It was like a student who could pass the exam after studying for 1 hour, while others needed 3 hours.

Why This Matters (According to the Paper)

The paper claims that by using videos from multiple angles, we can teach robots to understand "cause and effect" (I moved my hand -> the cup moved) without needing expensive human labels telling them exactly how to move.

The Analogy: It's the difference between a student who memorizes the specific lighting in a classroom (and fails when the lights change) versus a student who understands the concept of the lesson (and can pass the test in any room).

Summary

The Issue: Robots get confused by camera movements when learning from videos.
The Fix: Use two cameras and force the AI to explain one camera's view using the other's action.
The Outcome: The AI learns "pure" actions, ignores camera noise, and teaches robots to do tasks faster and with less data.

The paper concludes that this method is a significant step toward creating "universal" robot brains that can learn from the vast amount of human videos already on the internet, without needing a human to label every single move.

Technical Summary: MVP-LAM

Problem Statement

Training generalist robot policies is bottlenecked by the high cost and scarcity of real-world, action-labeled demonstrations. While large-scale human manipulation videos offer a promising alternative for learning transferable priors, they lack ground-truth action labels, preventing standard supervised imitation learning. Recent approaches attempt to address this by learning Latent Action Models (LAMs), which encode frame-to-frame transitions into compact representations used as pseudo-action labels for Vision-Language-Action (VLA) pretraining.

However, a critical challenge remains: latent actions learned from single-view videos often fail to remain informative about the underlying ground-truth actions. They tend to overfit to exogenous noise, particularly viewpoint variation. In human videos (often egocentric), camera motion is entangled with agent actions. Consequently, standard LAMs may encode viewpoint-specific cues (e.g., background movement or camera shifts) rather than the agent's actual control signals, reducing their utility as pseudo-labels for downstream control.

Methodology: MVP-LAM

The authors propose Multi-ViewPoint Latent Action Model (MVP-LAM), an unsupervised framework designed to learn action-centric latent actions by leveraging time-synchronized multi-view videos.

Core Objective

The goal is to maximize the mutual information $I(Z_t; A_t)$ between the latent action $Z_t$ and the ground-truth action $A_t$ , while minimizing the information $Z_t$ holds about the viewpoint $V_t$ . Theoretically, under a fixed-capacity bottleneck (e.g., vector quantization), encoding viewpoint information reduces the capacity available for action-relevant dynamics.

Architecture and Training

MVP-LAM is built upon a Vector Quantized Variational Autoencoder (VQ-VAE) architecture, similar to prior works like UniVLA, but introduces a novel cross-viewpoint reconstruction objective:

Input: Time-synchronized multi-view video pairs $(I^{v_1}_t, I^{v_2}_t)$ .
Feature Extraction: Visual features $o^v_t$ are extracted using a frozen encoder (DINOv2).
Latent Action Inference: An encoder $E_\theta$ predicts a continuous latent $e^v_t$ from consecutive features $(o^v_t, o^v_{t+1})$ of a specific view $v$ , which is then vector-quantized into a discrete token $z^v_t$ .
Reconstruction Losses: The model is trained with two complementary objectives:
- Self-Viewpoint Reconstruction ( $L_{self}$ ): Reconstructs the next frame $o^v_{t+1}$ using the current frame and the latent token from the same view ( $z^v_t$ ).
- Cross-Viewpoint Reconstruction ( $L_{cross}$ ): Reconstructs the next frame $o^v_{t+1}$ using the current frame and the latent token from a different, synchronized view ( $z^{\tilde{v}}_t$ where $v \neq \tilde{v}$ ).

Mechanism of Action-Centricity:
The cross-viewpoint objective forces the latent token $z^v_t$ to explain the future state of a different camera view. Since viewpoint-specific factors (e.g., background motion unique to camera $v$ ) cannot explain the transition in view $\tilde{v}$ , the model is discouraged from encoding them. Instead, $z^v_t$ must capture the shared, action-driven dynamics that are consistent across views. This effectively filters out exogenous noise without requiring action labels.

Key Contributions

MVP-LAM Framework: Introduction of an unsupervised learning framework that utilizes multi-view videos and a cross-viewpoint reconstruction objective to learn discrete latent actions highly aligned with ground-truth actions.
Improved Action-Centricity: Demonstration that MVP-LAM achieves significantly higher mutual information with ground-truth actions and better linear action prediction accuracy compared to single-view baselines (UniVLA, LAPA, Moto) on the Bridge V2 dataset.
Robustness to Viewpoint Perturbation: Evidence that MVP-LAM latent actions preserve transition-relevant information even when inferred from viewpoint-perturbed transitions, maintaining consistent dynamics across views.
Efficient VLA Pretraining: Validation that VLAs pretrained with MVP-LAM latent actions outperform baselines on manipulation benchmarks (SIMPLER and LIBERO) while requiring substantially less pretraining data (e.g., 3 $\times$ smaller dataset than OXE-based baselines).

Experimental Results

The authors evaluated MVP-LAM on the Bridge V2 dataset for latent action learning and the SIMPLER and LIBERO suites for downstream VLA performance.

Action-Centricity (Bridge V2):
- Mutual Information (MI): MVP-LAM achieved the highest estimated MI ( $I(Z; A)$ ) across KSG, BA, and MINE estimators compared to UniVLA, LAPA, and Moto.
- Linear Probing: A simple linear layer trained to predict ground-truth actions from latent tokens achieved the lowest Normalized Mean Squared Error (NMSE) for MVP-LAM, indicating superior action information retention.
Downstream Manipulation (SIMPLER & LIBERO):
- SIMPLER: MVP-LAM pretrained VLAs achieved a 60.4% average success rate, significantly outperforming UniVLA (39.6%) and other baselines across all four tasks.
- LIBERO: MVP-LAM achieved a 94.1% average success rate, outperforming UniVLA (92.5%) and remaining competitive with state-of-the-art models like $\pi_0$ (94.2%), despite being pretrained on a much smaller dataset (Bridge V2 only, $\le$ 60k trajectories vs. OXE's $\ge$ 970k).
- Out-of-Distribution (OOD): MVP-LAM showed robust performance on OOD tasks (Spatial, Object, Goal, Long), particularly excelling on the challenging LIBERO-Long suite.
Robustness and Scaling:
- Viewpoint Perturbation: When tested on novel-view synthesized transitions, MVP-LAM maintained high MI and low NMSE, whereas baselines like UniVLA showed significant degradation in attention focus and prediction accuracy.
- Synchronization Error: The model remained robust to synthetic synchronization lags ( $\ell \in \{0, 2, 4\}$ frames).
- Scaling: Performance improved consistently with increased numbers of viewpoints, dataset ratios, and model size.

Significance and Claims

The paper claims that MVP-LAM represents a step toward a universal latent action model by solving the problem of exogenous noise in video-based pretraining without requiring action supervision.

Data Efficiency: The primary significance lies in the ability to learn high-quality, action-centric representations from web-scale, multi-view human videos, enabling effective VLA pretraining with orders of magnitude less data than previous methods relying on massive robot datasets.
Generalization: By decoupling action dynamics from viewpoint-specific cues, MVP-LAM produces representations that generalize better to novel viewpoints and environments.
Practicality: The method leverages existing multi-view datasets (e.g., EgoExo4D, Open X-Embodiment) and integrates naturally into current embodied AI pipelines, offering a scalable path to generalist robot learning.

The authors note limitations, including the reliance on synchronized multi-view data (which is harder to collect than single-view) and the current evaluation being limited to simulation, though they highlight that SIMPLER correlates with real-world performance. Future work is suggested to explore weakly synchronized or pseudo-paired multi-view data to further relax instrumentation requirements.