💻 computer science

Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies

This paper proposes a cross-view action consistency method that regularizes flow-based Vision-Language-Action policies to achieve robustness against scene-camera viewpoint changes using only RGB images and proprioception, significantly improving performance on both simulated and real-world robot tasks without requiring camera labels or depth inputs.

Original authors: Bingqi Huang, Bingchuan Wei, Xuan Wang, Yingkai Cai, Zhaokui Wang

Published 2026-09-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingqi Huang, Bingchuan Wei, Xuan Wang, Yingkai Cai, Zhaokui Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that learn from human demonstrations are becoming increasingly capable, yet they often suffer from a peculiar fragility: they are tied to the specific viewpoint from which they were taught. Imagine a robot trained to pick up a cup while a camera sits on a shelf above the table. If that camera is bumped, moved to the side, or raised higher, the robot may suddenly fail, even though the cup, the table, and the robot's own body have not moved at all. This happens because the robot's "brain," a type of artificial intelligence that connects what it sees with what it should do, has learned to recognize the task based on the exact angle of the original camera. In the real world, cameras get bumped, robots move, and lighting changes, so a system that cannot adapt to a shifted viewpoint is not truly useful. Researchers have tried to solve this by giving robots more complex sensors, like depth cameras that see in 3D, or by teaching them to understand the geometry of the room, but these solutions often require expensive hardware or complex calibration that is difficult to maintain.

A team of researchers at Tsinghua University and the University of Amsterdam has found a way to make these robotic policies robust to camera movement without adding any new sensors or changing how the robot operates in the real world. They focused on a specific kind of robot controller that learns by predicting a "flow" of actions, similar to how water flows toward a destination, rather than just guessing the next move. The core of their discovery is a training method that forces the robot to agree with itself. They created pairs of training images showing the exact same physical scene from two different angles: one from the original camera position and one from a slightly moved position. Crucially, they ensured that the robot was told to perform the exact same action for both images. By training the robot to predict the same flow of movement for both views, they taught it to ignore the change in the camera's position and focus only on the task at hand.

The researchers tested this idea in a simulated environment using a benchmark called LIBERO-Plus, which is designed specifically to see how well robots handle camera shifts. They compared their new method against standard training techniques. When the camera was moved, a robot trained with standard methods succeeded in only about 17% of attempts. A robot trained on many different camera angles without their special consistency rule improved to about 75% success. However, the robot trained with their new cross-view consistency rule reached a success rate of 87.2%. This improvement was significant and consistent across different random starting points for the training. The researchers also proved that the success depended entirely on the fact that the two images showed the same physical state. When they scrambled the training data so that the robot tried to match a view of a cup with an action meant for a different state, performance collapsed to just 25.8%, showing that the robot was not just smoothing out its answers but was truly learning to link different views of the same reality to the same action.

To ensure this was not just a result of the computer simulation, the team took their method to a real robot arm in a laboratory. They collected training data using three synchronized cameras watching the same table, allowing them to create the necessary pairs of views from the real world. They then tested the robot using a single camera placed in a position it had never seen during training. The robot trained with their new method succeeded in 74.4% of these unseen camera tests, while the standard method succeeded in only 53.3%. The improvement was most dramatic in difficult tasks, such as removing headphones from a stand, where the standard method failed completely in the new camera positions, but the new method succeeded in nearly half of the attempts. The study confirms that by teaching a robot to see the consistency of its own actions across different angles, it can become much more reliable in the real world, all without needing extra cameras, depth sensors, or complex geometric maps. The robot simply learns that the task remains the same, even if the picture changes.

Technical Summary: Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies

Problem Statement

Vision-Language-Action (VLA) policies, particularly those fine-tuned from demonstrations collected with a fixed scene camera, often fail when the camera is moved, bumped, or re-mounted, even if the physical task state, language instruction, and robot proprioception remain unchanged. Existing solutions to this robustness issue typically alter the inference contract by requiring depth sensors, point clouds, calibrated multi-view inputs, camera labels, or geometry-aware architectures. These approaches introduce hardware dependencies or computational overheads at deployment.

This paper addresses the challenge of achieving scene-camera viewpoint robustness for flow-based VLAs using only a single scene RGB image, language instructions, and proprioceptive state at inference. The goal is to prevent performance degradation when the camera pose changes, without relying on depth data, camera extrinsics, or novel-view rendering during deployment.

Methodology: Cross-View Action-Flow Consistency

The proposed method introduces a training-time objective that regularizes the action-flow velocity field directly, rather than intermediate feature representations. The core insight is that for flow-based policies, actions are generated by integrating a learned velocity field $v_\theta$ . If two different camera views observe the same physical state, the policy should predict consistent velocity vectors at the same flow coordinates, leading to consistent action chunks.

1. Action-Equivalent Pairs

The method relies on constructing training pairs $(o_0, o_p, l, q, a)$ where:

$o_0$ and $o_p$ are RGB observations of the same physical state $s$ rendered from a nominal camera $c_0$ and a perturbed camera $c_p$ .
$l$ (language), $q$ (proprioception), and $a$ (demonstrated action chunk) are identical for both views.
In simulation, these pairs are generated by resetting the MuJoCo environment to stored states and re-rendering. On real robots, they are collected using synchronized multi-camera setups observing the same timestamp.
Crucially, the wrist-camera stream is masked during both training and inference to prevent the policy from using an unperturbed visual shortcut.

2. Loss Function

For a pair of views, the policy predicts velocity fields $v_0$ and $v_p$ at sampled flow coordinates $(t_k, \epsilon_k)$ . The total loss $\mathcal{L}$ combines two terms:

Standard Flow Matching ( $\mathcal{L}_{FM}$ ): Both views are supervised to match the demonstrated action target $u_{t_k}$ .
$\mathcal{L}_{FM} = \frac{1}{2} (\|v_0 - u_{t_k}\|^2 + \|v_p - u_{t_k}\|^2)$
Cross-View Consistency ( $\mathcal{L}_{CV}$ ): The predicted velocities for the two views are encouraged to agree on the active action dimensions.
$\mathcal{L}_{CV} = \|v_p[\cdot, : A_{act}] - v_0[\cdot, : A_{act}]\|^2$

The total objective is $\mathcal{L} = \frac{1}{K}\sum \mathcal{L}_{FM} + \lambda_{CV} \frac{1}{K}\sum \mathcal{L}_{CV}$ .

3. Theoretical Justification

The paper argues that feature-level invariance does not guarantee action-level invariance. By regularizing the velocity field itself, the method directly bounds the divergence of the integrated action chunks. Under a local Lipschitz assumption, the expected squared difference between action chunks generated from nominal and perturbed views is bounded by the expected squared difference of their velocity predictions along the integration trajectory.

4. Control Mechanisms

To verify that the gain comes from action-equivalent pairing rather than generic smoothing, the authors employ a shuffled-pair control. In this ablation, the cross-view loss compares velocities from different physical states (breaking action equivalence) while keeping flow-matching labels correct. This control causes performance to collapse, confirming that the method relies on true state correspondence.

Key Contributions

Formulation: The paper formulates scene-camera robustness for flow-based VLAs as a local action-flow consistency problem across action-equivalent views.
Objective: It proposes a cross-view action-flow consistency objective that regularizes the velocity field directly. This approach adds no sensors, calibration, or camera-specific inputs at inference time.
Empirical Validation: Through controlled ablations, the authors demonstrate that:
- The improvement depends on action-equivalent pairing (shuffled pairs fail).
- The benefit extends to camera poses (azimuth, distance, rotation) that were not covered by the training pairs (extrapolation).
- The method improves robustness without sacrificing performance on the nominal camera view.

Experimental Results

Simulation (LIBERO-Plus)

Evaluated on the LIBERO-Plus camera-perturbation benchmark (C1: distance/scale, C2: spherical position, C3: endpoint orientation):

Proposed Method: Achieved 87.2 ± 0.4% success on the camera-perturbation track.
Baselines:
- +7.4 percentage points (pp) over a "same-data" Flow Matching (FM) only control (79.8%), which used the exact same paired data but without the cross-view loss.
- +12.5 pp over standard mixed-camera Supervised Fine-Tuning (SFT) (74.7%).
- +70.4 pp over a nominal-only baseline (16.8%).
Nominal Performance: The method maintained high success on the standard nominal camera benchmark (94.2% vs. 93.6% for the FM-only control), demonstrating no trade-off in nominal performance.
Ablations: Shuffled pairs collapsed performance to 25.8%, proving the necessity of action-equivalent correspondence.

Real-Robot Evaluation

Evaluated on three tabletop tasks (Pick & Place, Contact Rich, Dexterous Operation) with synchronized multi-camera training and held-out camera deployment:

Held-out Camera Success: Improved from 53.3% (FM-only) to 74.4% (Proposed).
Task Specifics: The Laptop lid task achieved 100% success on held-out cameras with the proposed method. The Headphone stand task showed the largest relative gain (from 16.7% to 46.7%), though absolute performance remained limited by observability issues (occlusion of thin objects).
Generalization: Gains were most pronounced for geometrically aggressive held-out placements (e.g., higher, top-down views).

Significance and Claims

The paper claims that action-level consistency is a more effective target for camera robustness than feature-level invariance. By regularizing the velocity field that directly generates actions, the method ensures that different views of the same state yield the same intended behavior.

The significance of this work lies in its ability to improve robustness without altering the inference contract. Unlike methods requiring depth sensors, camera labels, or 3D reconstruction, this approach allows a VLA to be deployed with a single RGB camera while remaining robust to viewpoint shifts. The authors emphasize that the method is a training-time recipe that leverages observed action-equivalent views (via simulation reset or synchronized cameras) to teach the policy to ignore nuisance viewpoint changes, making it applicable to existing flow-based VLA architectures like $\pi_0.5$ .

The paper remains modest regarding its limitations: it does not recover information occluded by the camera (e.g., depth cues near contact for thin objects) and requires true action-equivalent pairs for training, which excludes standard single-camera datasets. However, it establishes a new baseline for camera-robust VLA training that is compatible with standard RGB-only inference.