💻 computer science

Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts

The paper proposes BRIDGE, a state-gated mixture of diffusion experts that effectively combines scalable handheld data with targeted teleoperated demonstrations to significantly improve success rates in contact-rich manipulation tasks by dynamically routing between action types based on the robot's current state.

Original authors: Vidullan Surendran, Neehar Peri, David Watkins

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vidullan Surendran, Neehar Peri, David Watkins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to perform delicate tasks, like threading a needle or screwing a battery into a tight spring-loaded slot. You have two main ways to teach it, but both have a major flaw.

The Two Flawed Teachers

The "Handheld" Teacher (The Fast Tourist):
Imagine a human holding a controller that looks like a robot hand, walking around a room, and showing the robot what to do. This is fast and easy. You can collect hundreds of examples quickly.
- The Problem: When the robot is just moving through empty air, this works great. But the moment the robot needs to touch something (like pushing a button or inserting a pipe), the human holding the controller can't feel the resistance perfectly. The human's hand might wiggle or push too hard because they are trying to "see" the path, not "feel" the physics. If the robot tries to copy this shaky movement exactly, it might smash the object or break the robot's arm.
The "Teleoperation" Teacher (The Slow Surgeon):
Imagine a human sitting at a high-tech console, controlling the robot directly with perfect precision, feeling every bit of resistance.
- The Problem: This is perfect for the delicate parts, but it is incredibly slow and exhausting. It takes forever to collect enough data to teach the robot the whole task.

The Paper's Big Idea: "The Hybrid Coach"

The authors of this paper realized that you don't need a perfect teacher for every second of the task. You only need a perfect teacher for the tricky parts.

They came up with a system called BRIDGE (which stands for Bi-modal Routing for Imitation Data via Gated Experts). Think of it like a sports team with two specialized players and a smart coach:

Player A (The Handheld Expert): This player is great at running fast and moving through open space. They learn from the "Fast Tourist" data. They handle the boring, easy parts of the job.
Player B (The Teleop Expert): This player is a master of delicate, high-pressure situations. They learn from the "Slow Surgeon" data. They only step in when things get sticky.
The Coach (The Router): This is the magic part. The Coach watches the robot's current situation.
- If the robot is just moving through the air? The Coach yells, "Player A, go!"
- If the robot is about to touch a surface or push a spring? The Coach yells, "Switch to Player B immediately!"

Why "Naive Mixing" Fails

The paper tested what happens if you just throw all the data into a single pot and hope the robot learns the difference. It's like trying to teach a student by giving them a textbook on "how to run" and a textbook on "how to perform surgery" and telling them to just "figure it out." The robot gets confused. It tries to run with surgical precision (too slow) or perform surgery with running speed (too dangerous). The paper found that simply mixing the data actually made the robot worse than if it just used the handheld data alone.

The Results: Speed Meets Precision

By using their "Coach" to switch between the two players at the right moment, the robot achieved amazing results:

It learned the general shape of the task from the fast, cheap handheld data.
It only needed a tiny amount of slow, expensive "surgeon" data (just the tricky parts) to fix the mistakes.
The Outcome: The robot became up to 36.7% more successful at difficult tasks compared to using only the fast handheld data. It managed to get almost as good as the robot trained entirely by the slow surgeon, but it took a fraction of the time and effort to collect the data.

In a Nutshell

The paper solves a trade-off between speed and precision. Instead of choosing one or the other, they built a system that uses the "fast and dirty" data for the easy parts and the "slow and perfect" data only for the critical, contact-heavy moments. It's like hiring a tour guide for the walking tour but calling in a specialist mechanic only when the car breaks down.

Technical Summary: Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts

Problem Statement

The paper addresses the fundamental trade-off between scalability and fidelity in robot data collection for contact-rich manipulation tasks.

Handheld Data (e.g., UMI): Systems like the Universal Manipulation Interface (UMI) enable scalable, "in-the-wild" data collection by decoupling the interface from specific robot hardware. However, they capture only observed actions (the physical state of the handheld device). In contact-sensitive phases, these observed actions often deviate from the desired actions (the reference trajectory sent to the controller) due to finite stiffness, filtering, and tracking errors. Naively tracking these observed trajectories with high stiffness generates unsafe, large contact forces.
Teleoperated Data: Systems like GELLO or ALOHA provide high-fidelity, on-robot data capturing desired actions directly. However, collecting full-task teleoperated demonstrations is prohibitively time-consuming and lacks the diversity of handheld collection.

The core problem is that while handheld data is valid for error-tolerant, free-space motion, it fails to provide dynamic feasibility during contact-sensitive phases. Conversely, mixing observed and desired actions naively into a single policy degrades performance because the supervision signals are incompatible.

Methodology: BRIDGE

The authors propose BRIDGE (Bi-modal Routing for Imitation Data via Gated Experts), a mixture-of-experts diffusion policy architecture designed to leverage the complementary strengths of both data sources.

1. Dual-Mode Data Collection (DM-UMI)

The authors utilize a modified UMI device capable of dual-mode operation:

Handheld Mode: Collects a large-scale base dataset of observed actions across diverse environments to learn the general task scaffold.
Teleoperated Mode: Collects a sparse support dataset of desired actions, targeted specifically at task segments where the base policy fails (contact bottlenecks).

2. Architecture

The BRIDGE model consists of three sequential training stages:

Stage 1: Base Policy Training ( $\pi_b$ ): A diffusion policy is trained on the handheld base dataset. The architecture uses a DINOv2 vision encoder retaining full spatial patch tokens (processed via a Perceiver IO block) rather than just the CLS token, to better reason about spatial relationships in contact-rich tasks. This policy learns to predict observed actions.
Stage 2: Support Head Training ( $\pi_s$ ): With the base policy and vision encoder frozen, a new latent adapter and diffusion head are trained on the teleoperated support dataset. This expert learns to predict desired actions (the reference trajectory) specifically for contact-sensitive regimes.
Stage 3: Router Training ( $G_\psi$ ): A lightweight MLP gate is trained to dynamically route observations to either the base or support expert. The router is trained using a k-Nearest Neighbor (k-NN) pseudo-labeling strategy on latent representations. It learns to identify when the current state falls within the "support manifold" (contact-sensitive phases) and switches to the desired-action expert.

3. Inference

During execution, the router performs a hard-switch based on the current robot state. If the state is identified as belonging to a contact-sensitive phase, the system routes to the support expert to execute the desired action trajectory. Otherwise, it uses the base expert for free-space motion. This avoids averaging incompatible action distributions.

Key Contributions

Formalization of Action Validity: The paper demonstrates that observed actions are sufficient for error-tolerant, free-space motion but fail during force-sensitive phases where desired actions are necessary for precise, compliant execution.
BRIDGE Architecture: The introduction of a gated mixture-of-experts diffusion policy that jointly learns from heterogeneous supervision sources (observed vs. desired actions) by routing based on local action validity.
Efficient Hybrid Strategy: Empirical evidence that augmenting scalable handheld data with a small number of targeted teleoperated demonstrations significantly improves success rates in contact-rich tasks without the cost of full-task teleoperation.

Experimental Results

The method was evaluated on three contact-rich manipulation tasks using a Franka FR3 arm: NIST Pulley Routing, Pipe Insertion, and Spring-Loaded Battery Insertion.

Performance Gains: BRIDGE consistently outperformed handheld-only baselines, improving success rates by up to 36.7% across the three tasks.
- NIST Pulley: Improved from 44.0% (Handheld) to 76.0% (BRIDGE), approaching the 84.0% of full teleoperation.
- Pipe Insertion: Improved from 13.3% (Handheld) to 50.0% (BRIDGE), closing the gap to the 63.3% teleoperation upper bound.
- Battery Insertion: Improved from 10.0% (Handheld) to 33.3% (BRIDGE).
Failure of Naive Mixing: A "Naive Mix" baseline, which simply combined observed and desired labels in a single policy, performed worse than the handheld-only baseline (e.g., 0% success in Pulley Routing), confirming the incompatibility of the supervision signals without gating.
Data Efficiency: The targeted support data covered less than 15% of the end-effector path length in terms of spatial distance, yet recovered up to 79% of the performance gap to the full teleoperation upper bound.
Router Performance: The router achieved 99.0% recall and 69.0% precision on the pipe insertion task, effectively identifying contact phases without false negatives.

Significance and Claims

The paper claims that BRIDGE provides a practical mechanism for specializing robot behavior across task phases. By separating observed and desired action heads, the approach allows for the use of scalable, low-cost handheld data to learn the general task structure while reserving expensive, high-fidelity teleoperation for the specific bottlenecks where contact dynamics require precise control.

The authors emphasize that this approach bridges the "embodiment gap" between handheld data collection and on-robot execution. They note that while full teleoperation remains the in-domain upper bound, it is often impractical due to cognitive and physical burdens. BRIDGE offers a middle ground, enabling high-performance contact-rich manipulation with significantly reduced data collection effort compared to full teleoperation, while avoiding the performance degradation associated with naive data mixing.

The paper remains modest regarding its limitations, acknowledging that the current approach relies on empirical identification of failure modes, that hard-switching can introduce control discontinuities, and that the method's scalability to long-horizon tasks with multiple contact regimes requires further investigation.