💻 computer science

Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA

The paper introduces Cloak, a training method that enables Vision-Language-Action models to achieve zero-shot cross-embodiment manipulation by masking the end-effector from wrist camera views during training, thereby allowing a model trained on a single robot to generalize to unseen hardware without collecting new data.

Original authors: Michael Piseno, Guy Tevet, C. Karen Liu

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Piseno, Guy Tevet, C. Karen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to pick up a cup. You show it a video from the robot's own "eye" (a camera on its wrist). The problem is, in that video, the robot's own hand (the gripper) takes up a huge chunk of the screen.

If you train a robot using a specific type of hand—say, a simple two-fingered pincer—it learns to recognize that specific shape. If you then swap that robot's hand for a completely different one, like a human-like hand with five fingers, the robot gets confused. It's like if you taught someone to drive a car by only showing them the steering wheel of a Ford. If you suddenly put them in a Tesla with a different steering wheel, they might panic because the "visual map" in their brain doesn't match what they see.

The Problem: The "Hand" Distraction
In the world of robotics, collecting data is expensive and slow. We have huge libraries of data showing robots with two-fingered grippers doing tasks. But the future of robotics is moving toward complex, multi-fingered hands. We want to use our old data to teach these new hands, but the old robot's hand looks nothing like the new one. The robot's brain is too focused on the specific shape of the old hand to understand the new one.

The Solution: "Cloak"
The paper introduces a clever trick called Cloak. Think of it as a digital "blindfold" or a "muzzle" for the robot's vision.

Hiding the Hand: Instead of letting the robot see its own hand in the camera feed, Cloak digitally paints over the hand with a mask. It's like putting a piece of tape over the part of the camera lens where the hand appears.
Learning the Scene, Not the Hand: By hiding the hand, the robot is forced to look at the rest of the world—the table, the cup, the obstacles. It learns the logic of the task (e.g., "move forward until I hit the cup") without getting distracted by the specific shape of its own fingers.
The "Chameleon" Training: To make sure the robot doesn't just learn to ignore a specific shape of tape, the researchers randomly change the shape of the "blindfold" during training. Sometimes the mask is big, sometimes small, sometimes weirdly shaped. This teaches the robot that "my hand might look different tomorrow, so I shouldn't rely on seeing it."

How It Works in Real Life
When the robot is actually doing the job:

The Eyes: The camera feed still has the hand, but the software instantly covers the hand with a digital mask before the robot's brain sees it.
The Brain: The robot's brain (a Vision-Language-Action model) sees a masked image and a text command like "pick up the cup." It figures out the movement based on the environment.
The Body: Once the brain decides "move my fingertips to this spot," a translator (called tip-pose retargeting) converts that instruction into the specific muscle movements needed for the new hand. If the new hand has five fingers, the system figures out how to move those five fingers to match the two-fingered plan.

The Results
The researchers tested this by training a robot on a simple two-fingered gripper using existing data. Then, they tried to use that same trained brain on three completely different robots it had never seen before:

A different two-fingered gripper (different color and shape).
A different robot arm entirely.
A complex, five-fingered human-like hand.

The Outcome:

Without Cloak: The robot failed miserably on the new hands. It was confused because the visual "map" didn't match.
With Cloak: The robot performed almost as well on the new hands as it did on the original one. It successfully transferred its skills zero-shot (meaning it didn't need any new training data for the new hands).

The Big Takeaway
Cloak proves that you can separate a robot's "brain" (the skill of doing a task) from its "body" (the specific hardware). By hiding the body from the brain's view during learning, the data collected on one robot can be reused on completely different robots in the future. It's like teaching a person to swim by having them wear a blindfold so they focus on the water's movement rather than their own limbs; once they learn the rhythm of the water, they can swim with any kind of fins or flippers.

Limitations
The paper notes this works best for tasks that can be done by placing two points (like two fingertips) on an object. It doesn't yet solve complex tasks that require intricate in-hand manipulation (like juggling a ball inside a hand), as those rely heavily on the specific feel and shape of the fingers.

Technical Summary: Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA

1. Problem Statement

Robotic manipulation research is currently fragmented across diverse end-effectors, ranging from simple parallel-jaw grippers to complex, anthropomorphic multi-fingered hands. While Vision-Language-Action (VLA) models trained on specific "source" embodiments achieve strong performance on those specific robots, their capabilities degrade significantly when deployed on "target" embodiments with different morphologies.

The primary barrier to zero-shot cross-embodiment transfer is the visual gap. In standard setups with wrist-mounted cameras, the end-effector occupies a large, consistent region of the visual field. A policy trained on a source robot (e.g., a parallel-jaw gripper) learns features heavily dependent on that specific end-effector's appearance. When deployed on a target robot with a different appearance (e.g., a five-fingered hand), the visual observation shifts out of distribution, causing performance collapse. Existing solutions often rely on collecting new data for every target or using generative models to "repaint" the robot in the image, which introduces latency, hallucination risks, and computational overhead.

2. Methodology: The Cloak Recipe

The authors propose Cloak, a training recipe that enables a VLA to generalize to unseen embodiments by explicitly masking the end-effector from its own wrist camera view during training. This forces the model to rely on scene context and language instructions rather than the specific geometry of the source robot's hand.

Core Components

Geometric Masking (Training & Inference):
- Instead of using learned segmentation or generative inpainting, Cloak renders a geometric mask of the end-effector in real-time using the robot's known kinematics (forward kinematics), link meshes, and camera intrinsics.
- This mask is projected onto the wrist camera image, identifying the pixels occupied by the robot's own body.
- Masked Attention: The mask is downsampled to the vision encoder's patch grid (e.g., ViT patches). During the attention computation in the vision encoder, patches falling within the mask are assigned an attention bias of $-\infty$ . This effectively removes the end-effector's visual information from the policy's perception while maintaining the sequence length.
Mask Augmentation (Training Only):
- To prevent the model from overfitting to the specific silhouette of the source gripper, the authors augment the mask during training.
- The augmentation involves randomly adding "capsules" to the mask's contour and subtracting "disks" from its interior.
- To fill the resulting holes in the image (which would otherwise reveal the source robot), the authors use a simple, non-generative inpainting strategy: they roll a copy of the same image onto the masked region. This ensures the pixel distribution remains realistic without hallucinating new content.
Tip-Pose Retargeting (Kinematic Bridging):
- While the visual gap is addressed by masking, the kinematic gap (different joint spaces) is addressed via Forward Kinematics (FK) and Inverse Kinematics (IK).
- Input Retargeting: The target robot's state is converted to the source robot's joint space by computing the FK of the target's designated "tips" (e.g., fingertips or gripper jaws) and then solving for the source robot's joint configuration that matches those tip poses.
- Output Retargeting: The VLA outputs actions in the source robot's joint space. These are converted to the target robot's actions by computing the FK of the source tips, then solving IK for the target robot to achieve the same tip poses.
- For the gripper signal, a linear interpolation between predefined open/closed states is used for the target hand.

3. Key Contributions

Cloak-VLA: A VLA policy trained on a single parallel-jaw gripper dataset (DROID) that transfers zero-shot to unseen embodiments, including a different gripper, a different arm, and a five-fingered dexterous hand, without collecting any new data.
Geometry-Based Masking: A novel approach to closing the visual gap by rendering masks from known geometry rather than relying on generative models or learned segmentation, thereby avoiding hallucination and ensuring real-time performance.
Mask Augmentation Strategy: A technique to randomize the end-effector silhouette during training, ensuring the model learns embodiment-agnostic reasoning rather than memorizing a specific shape.
Decoupling Data from Hardware: A demonstration that manipulation skills can be decoupled from the specific body that executes them, allowing data collected on one robot to "outlive" the hardware.

4. Experimental Results

The authors evaluated Cloak-VLA on the DROID dataset (source: Franka Panda with Robotiq 2F-85 gripper) and tested zero-shot transfer on three unseen embodiments:

UMI Gripper: A different parallel-jaw gripper (different color, shape, size).
YAM Arm: A different arm platform with a gripper.
Sharpa Hand: A five-fingered dexterous humanoid hand.

Performance Metrics (Task Progression Rate):

Source Embodiment: Cloak-VLA performed on par with the unmasked baseline ( $\pi0.5$ -droid), showing that masking the wrist view incurs negligible cost on the source robot.
Unseen Embodiments:
- Cloak-VLA maintained high performance across all unseen targets (e.g., 81.8% on the Sharpa hand, 85.1% on the UMI gripper).
- Baselines ( $\pi0.5$ -droid and LAP-VLA) suffered significant performance drops on unseen embodiments, particularly on the five-fingered hand (dropping to 54.4% and 60.2%, respectively).
- Ablation Studies: Removing mask augmentation (Cloak-VLA-NoAug) or using a fixed target mask overlay (Cloak-VLA-Overlap) resulted in lower performance, confirming that augmenting the source silhouette is critical for generalization.

5. Significance and Claims

The paper claims that Cloak represents a significant step toward body-agnostic manipulation. By cloaking the end-effector, the model learns to reason about the task and the environment without being biased by the specific appearance of the robot's hand.

Data Longevity: The primary significance is the ability to reuse large-scale datasets (like DROID) for new, emerging hardware without retraining or new data collection.
Robustness: The method avoids the pitfalls of generative models (hallucination, speed) and learned segmentation (unreliability on thin structures).
Scope: The authors modestly note that the current approach is limited to tasks expressible via two fingertips (grasping/placing) and does not yet cover complex in-hand reorientation or skills requiring rich contact strategies. They suggest that the resulting body-agnostic model serves as a strong initialization for future fine-tuning on new embodiments.