UMI-Underwater: Learning Underwater Manipulation without Underwater Teleoperation

The UMI-Underwater system enables robust underwater robotic grasping by combining a self-supervised data collection pipeline with a depth-based affordance representation that transfers knowledge from on-land demonstrations to the underwater domain, achieving zero-shot generalization and outperforming RGB-only baselines.

The Gist

Imagine trying to teach a robot to pick up a specific seashell from the bottom of a murky, dark ocean. Now, imagine you can't go underwater yourself to show the robot how to do it because it's too expensive, dangerous, or just too hard to control a robot remotely in that environment.

That is the problem the UMI-Underwater team solved. They built a system that lets a robot learn to grab things underwater without needing a human to steer it underwater, and without needing to show it thousands of underwater videos first.

Here is how they did it, broken down into simple concepts:

1. The "Self-Teaching" Robot (The Autonomous Collector)

Usually, to teach a robot, humans have to hold a controller and manually guide the robot arm thousands of times to show it what "success" looks like. Underwater, this is a nightmare.

The Solution: The team gave the robot a "try-and-learn" mindset.

• The Analogy: Imagine a toddler learning to pick up a toy. They don't get a lecture; they just reach, maybe miss, grab the wrong thing, drop it, and try again. Eventually, they get it right.
• How it works: The robot dives down, picks a random object, and tries to grab it using a simple set of rules (like "move forward until it looks big"). If it grabs it and the object doesn't slip away when the robot pulls back, the computer says, "Good job!" and saves that moment as a lesson. If it fails, the robot just backs up and tries again.
• The Result: The robot collected hundreds of successful "grasping" videos all by itself, with almost no human help.

2. The "Land-to-Sea" Translator (The Affordance Map)

Even with the robot's self-collected videos, there's a big problem: Underwater is weird. The light is blue, things look blurry, and colors are distorted. A robot trained on clear, sunny land videos would be completely confused underwater.

The Solution: They created a "Universal Translator" called an Affordance Heatmap.

• The Analogy: Imagine you are trying to find a specific person in a crowded room. If you look at their face (which might look different in the dark), it's hard. But if you have a glowing red dot on their chest that says "GRAB HERE," you don't care what the lighting looks like. You just follow the dot.
• How it works:
  1. The Land Training: The team went to a park (on land) and used a handheld gripper with a camera to grab various objects (rocks, ducks, cans). They didn't need a robot for this; a human just held the gripper.
  2. The Magic Trick: Instead of teaching the robot to recognize the color or shape of the object (which changes underwater), they taught it to recognize where to grab based on depth (how far away things are).
  3. The Transfer: They took this "grab here" knowledge from the land and applied it underwater. Because depth doesn't change much between land and water (a rock is still a certain distance away), the robot can look at the underwater scene, see the "depth map," and instantly know, "Ah, the red dot is on that rock, I should grab there."

3. The "Brain" (The Diffusion Policy)

Once the robot knows where to grab (thanks to the depth map), it needs to know how to move its arm to get there.

The Solution: They used a "Diffusion Policy."

• The Analogy: Think of this like a GPS that doesn't just give you a destination, but simulates the whole drive. It takes the "grab here" map and the robot's current position, and it runs a simulation in its head: "If I move left, then down, then close my gripper, I will succeed." It generates a smooth, perfect path to the target.
• Why it's special: This "brain" was trained on the robot's own self-collected underwater videos, so it knows exactly how to move in the water's resistance.

Why This is a Big Deal

• No More "Human-in-the-Loop": You don't need a diver or a pilot to spend hours underwater teaching the robot. The robot teaches itself.
• Zero-Shot Transfer: You can train the robot on land (where it's cheap and easy) and it works perfectly underwater immediately, without needing to re-train it on underwater data.
• Robustness: If you change the background (like putting a new wallpaper on the pool wall), a normal robot would get confused. This robot ignores the background colors and focuses on the "depth map," so it keeps working even when the scenery changes.

The Bottom Line

The team built a system where a robot learns to swim and grab on its own, using a land-based "cheat sheet" (the depth map) to understand what to grab, even when the underwater world looks totally different from the land world. It's like teaching a fish to catch a fly by showing it a picture of a fly in a book, and then letting it figure out the rest in the water.

Technical Summary

1. Problem Statement

Underwater robotic manipulation faces two primary bottlenecks that hinder the development of robust autonomous systems:

• Perceptual Degradation: Underwater environments suffer from severe visual challenges, including wavelength-dependent attenuation, scattering, turbidity, and rapidly changing illumination. These factors cause significant domain shifts between training data (often collected on land) and deployment environments, making end-to-end visuomotor policies brittle.
• Data Collection Burden: Collecting diverse, high-quality underwater demonstrations is expensive, time-consuming, and dangerous. Most systems rely on human teleoperation, which limits the scale of data collection and the diversity of scenarios a robot can learn from.

The paper aims to solve these issues by creating a system that autonomously collects underwater data and transfers grasp knowledge from land to water without requiring mixed training or fine-tuning.

2. Methodology

The proposed system, UMI-Underwater, consists of three core components: a self-supervised underwater data collection pipeline, a cross-domain affordance predictor, and an affordance-conditioned diffusion policy.

A. Self-Supervised Underwater Data Collection

To eliminate reliance on teleoperation, the authors developed an autonomous pipeline using a tethered ROV (Remotely Operated Vehicle):

• Heuristic Controller: The robot uses a PD controller to execute a staged grasping routine: (1) Yaw alignment, (2) Forward approach, (3) Depth adjustment, (4) Close-range approach using monocular depth estimation, and (5) Grasp execution.
• Success Verification: Instead of external sensors, the system uses a "drag verification" method. After a grasp attempt, the robot retreats for 3 seconds; if the object remains in the gripper, the episode is labeled as a success.
• Recovery Behaviors: The system includes automatic recovery strategies for failed attempts, such as "Regrasp" (backing up and retrying with a lateral offset) and "Overshoot recovery" (retreating if the object moves out of view due to hydrodynamic forces).
• Scale: This pipeline autonomously collected 536 grasp episodes (233 successful) with minimal human intervention (only for resetting tangled tethers).

B. UMI-Aquatic and Cross-Domain Affordance Transfer

To bridge the land-to-water gap, the authors introduced UMI-Aquatic, a handheld interface derived from the Universal Manipulation Interface (UMI).

• Data Collection: A handheld gripper equipped with an iPhone camera and AprilTags was used to collect 800 diverse grasping demonstrations on land.
• Geometric Alignment: To align the iPhone camera view with the underwater robot's camera, the system uses a "plane-at-depth" warping technique. This projects the on-land RGB images into the underwater camera's geometry, correcting for intrinsic and distortion differences.
• Depth-Based Affordance: Instead of using RGB, the system trains a goal-conditioned affordance predictor on depth maps. This is crucial because depth is invariant to the lighting and color shifts that plague underwater RGB imagery.
• Zero-Shot Transfer: The affordance model is trained exclusively on on-land data. At deployment, it takes an on-land goal image (converted to depth) and the current underwater depth observation to predict a grasp heatmap. No fine-tuning or mixed training is required.

C. Affordance-Conditioned Diffusion Policy

The final control layer is a diffusion policy that generates action trajectories:

• Inputs: The policy conditions on three modalities:
  1. The predicted affordance heatmap (indicating where to grasp).
  2. Monocular depth (providing geometric context).
  3. Robot proprioception (compass, pitch, depth).
• Architecture: A 1D UNet denoiser predicts a 16-step action horizon (yaw, translation, gripper control) based on the inputs.
• Training: The policy is trained via behavior cloning on the successful episodes collected by the autonomous underwater pipeline.

3. Key Contributions

1. Self-Supervised Underwater Data Pipeline: A practical system that autonomously gathers successful underwater grasp demonstrations using heuristic control and automatic success filtering, drastically reducing human teleoperation time.
2. UMI-Aquatic & Zero-Shot Transfer: A novel interface and methodology that leverages cheap, portable on-land demonstrations to train a depth-based affordance model. This model transfers directly to underwater environments without fine-tuning, effectively bridging the land-to-water domain gap.
3. Robustness via Affordance: Demonstrating that using affordance heatmaps as a spatial goal signal, conditioned on depth rather than RGB, significantly improves generalization to unseen backgrounds and novel objects.

4. Experimental Results

The system was evaluated in a swimming pool under three settings:

• In-Distribution (ID): On seen objects with the training background.
  • Result: The proposed method (DP + Aff + Depth) achieved 85% success, outperforming the RGB-only baseline (65%). The affordance signal helped the robot select the correct target in multi-object scenes, whereas the RGB baseline often confused distractors.
• OOD Visual Generalization (Background Shift): On seen objects with unseen background wallpapers (introducing strong texture/color shifts).
  • Result: The RGB-only baseline failed completely (0% success). The proposed method maintained 80% success. Notably, adding RGB to the proposed method (DP + RGB + Aff + Depth) also caused performance to collapse to 0%, proving that RGB inputs can actively degrade robustness under domain shift even when other robust signals are present.
• OOD Novel-Object Generalization: On objects seen only in the on-land dataset (e.g., power drills, pitchers) but never in the underwater training set.
  • Result: The proposed method achieved 75% success, significantly outperforming the RGB baseline (50%). This confirms that the affordance model trained on land successfully generalized to guide grasping of novel objects underwater.

5. Significance and Impact

• Scalability: By removing the need for teleoperation and enabling zero-shot transfer from land to water, this approach makes it feasible to scale underwater manipulation training to large, diverse datasets.
• Robustness: The paper establishes that depth-based affordances are a superior intermediate representation for underwater tasks compared to raw RGB, as they are invariant to the severe lighting and color degradation inherent to the underwater domain.
• Generalization: The ability to deploy policies on novel objects without retraining or fine-tuning represents a significant step toward truly autonomous underwater robots capable of handling diverse real-world tasks like debris removal and infrastructure inspection.

Limitations & Future Work: The authors note that relying solely on depth can discard useful texture/color cues (e.g., material properties). Future work aims to fuse RGB and depth more effectively using domain adaptation. Additionally, improving low-level control (e.g., replacing PID with Model Predictive Control) could reduce overshoot-induced failures.