Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA

This paper proposes an integrated framework combining RL-augmented teleoperation via the IMCopilot assistant and a Mixture-of-Dexterous-Experts VLA (MoDE-VLA) architecture to overcome data and learning bottlenecks, enabling robust human-like, contact-rich bimanual in-hand manipulation with significantly improved success rates.

The Gist

Imagine teaching a robot to do something as tricky as peeling an apple. It sounds simple to us, but for a robot, it's like trying to thread a needle while riding a unicycle on a tightrope. The robot needs to hold the fruit, rotate it with its fingers, and slice the skin off without squishing the fruit or dropping it.

This paper introduces a new system that helps robots learn these "human-like" skills. Think of it as a three-part recipe for robot mastery: Better Training Wheels, A Smart Assistant, and A Super-Brain.

Here is how it works, broken down into simple concepts:

1. The Problem: Robots Are "Clumsy" with Their Fingers

Most robots today are great at picking things up and putting them down (like a forklift). But they struggle with dexterous manipulation—using fingers to rotate, twist, and feel objects.

• The Data Gap: To teach a robot, we usually let a human control it remotely (teleoperation). But controlling a robot with 63 moving parts (two arms, two hands, fingers) is incredibly hard. It's like trying to play a piano with 63 keys using only your elbows. Even experts drop the "apple" or slip the "peeler" because they can't feel the pressure.
• The Sensory Gap: Robots usually just "see" with cameras. But peeling an apple requires "feeling." You need to know if the skin is slipping or if you're pressing too hard. Current robot brains don't know how to mix "sight" with "touch" and "pressure" effectively.

2. The Solution: The "IMCopilot" (The Training Wheels & The Assistant)

The authors created a tool called IMCopilot. Think of this as a smart autopilot for the robot's hands.

• During Training (The Training Wheels): When humans are teaching the robot, they can't perfectly control the fingers to rotate an apple. So, they use foot pedals to say, "Hey, just hold the apple steady and spin it for me." The IMCopilot takes over the hard finger work, while the human just moves the arms. This makes collecting training data much faster and less frustrating.
• During Execution (The Assistant): Once the robot is working alone, the main brain (the VLA) can say, "I need to rotate the apple now," and it triggers the IMCopilot to do the actual spinning. It's like a conductor (the main brain) telling a virtuoso violinist (IMCopilot) to play a specific solo.

3. The Solution: The "MoDE-VLA" (The Super-Brain)

The second part is the robot's brain, called MoDE-VLA.

• The Old Way: Imagine trying to read a book while someone is shouting numbers in your ear. If you just mix the text and the numbers together, you get confused. That's what happens when robots try to mix camera images with force sensors.
• The New Way (MoDE): This system is like a specialized team of experts.
  • It has a main brain that knows how to move based on what it sees and hears (Vision-Language-Action).
  • It has a special "Touch Team" that only looks at force and tactile data.
  • The Magic: Instead of forcing the touch data to change the whole brain, it acts like a fine-tuning knob. It says, "The main brain thinks the arm should move here, but my touch sensors say the apple is slippery, so let's nudge the movement slightly to the left." It adds a "residual correction"—a small, smart adjustment based on feeling, without messing up the robot's general knowledge.

4. The Results: From "Fumbling" to "Peeling"

The team tested this on four difficult tasks:

1. Gear Assembling: Pushing gears onto a shaft (needs precise pressure).
2. Charger Plugging: Finding the hole and pushing the plug in (needs to feel the "click").
3. Test Tube Rearranging: Moving tubes between hands without dropping them.
4. Apple Peeling: The ultimate test. Holding a peeler in one hand and an apple in the other, rotating the apple while peeling.

The Outcome:

• Without their system, the robot succeeded only about 15% of the time on average.
• With the IMCopilot and MoDE-VLA, success jumped to 34% (which is huge in robotics!).
• Most impressively, they achieved the first autonomous robot to peel an apple. The robot could hold the fruit, rotate it, and peel a full ring of skin off without dropping it.

The Big Picture Analogy

Imagine you are learning to juggle.

• Old Robots: You try to juggle by just looking at the balls. You drop them constantly because you can't feel when they are slipping.
• IMCopilot: A friend holds the balls for you while you learn the arm motions, then lets go when you are ready.
• MoDE-VLA: You put on special gloves that vibrate when a ball is about to slip, and your brain instantly adjusts your hand position without you having to think about it.

By combining smart training tools (IMCopilot) with a brain that understands both sight and touch (MoDE-VLA), the authors have taken a giant step toward robots that can do the delicate, messy, human-like tasks we take for granted every day.

Technical Summary

Here is a detailed technical summary of the paper "Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA".

1. Problem Statement

The paper addresses the significant gap between current Vision-Language-Action (VLA) models and human-like dexterous manipulation. While VLAs have succeeded in simple pick-and-place tasks using low-degree-of-freedom (DoF) grippers, they struggle with contact-rich, bimanual in-hand manipulation (e.g., peeling an apple). The authors identify three critical bottlenecks:

• Data Acquisition: Teleoperating high-DoF (63 DoF) bimanual systems for precise in-hand manipulation is cognitively overwhelming for humans, leading to low-quality or inconsistent demonstrations.
• Multi-Skill Learning: Complex tasks require switching between distinct phases (gross vision-guided motion, force-guided cutting, and tactile-guided rotation), which is difficult for a single monolithic policy to master in a high-dimensional action space.
• Modality Heterogeneity: Naively concatenating force and tactile data into a VLA backbone often degrades performance because these modalities have different physical semantics and temporal dynamics compared to vision and language.

2. Methodology

The authors propose an integrated framework consisting of two synergistic components: IMCopilot and MoDE-VLA.

A. IMCopilot (In-hand Manipulation Copilot)

IMCopilot is a suite of Reinforcement Learning (RL) trained atomic skills designed to serve a dual role:

1. Shared-Autonomy Assistant (Data Collection): During teleoperation, human operators control gross arm movements via an exoskeleton but trigger pre-trained in-hand skills (e.g., stable grasp maintenance, object rotation) via foot pedals. This bypasses the human inability to perform precise multi-finger coordination, yielding high-fidelity demonstration data.
2. Callable Low-Level Primitive (Autonomous Execution): During inference, the VLA can invoke IMCopilot as a low-level skill. The VLA outputs a trigger signal ($c$); if $c > 0.5$, IMCopilot takes over hand control for in-hand manipulation while the VLA continues to control the arms.

Training: Skills are trained using Proximal Policy Optimization (PPO) in simulation (IsaacLab) with an asymmetric actor-critic architecture and teacher-student distillation to ensure sim-to-real transfer.

B. MoDE-VLA (Mixture-of-Dexterous-Experts VLA)

MoDE-VLA is a novel architecture that fuses force and tactile modalities into a pretrained VLA backbone (based on OpenPI-0/π0) without degrading its pretrained knowledge.

• Architecture: It introduces a dedicated pathway for force (arm joint torques) and tactile (fingertip 6-DoF wrench) signals.
• Token Construction: Force and tactile inputs are projected into the embedding space and replicated across the action horizon with positional encodings to allow timestep-specific processing.
• Sparse Mixture-of-Experts (MoE): A token-level MoE layer routes force/tactile tokens through specialized experts. This allows the network to dynamically select different experts for different manipulation regimes (e.g., free-space reaching vs. contact onset vs. stable grasping).
• Residual Injection: The output of the MoE layer is injected as a residual correction to the backbone's action prediction. This ensures that when contact information is irrelevant (free space), the correction vanishes, preserving the robust pretrained behavior. The injection is modality-specific: force tokens refine arm actions, while tactile tokens refine hand actions.

3. Key Contributions

1. IMCopilot Framework: A unified system that solves the data acquisition bottleneck by augmenting teleoperation with RL skills and solves the multi-skill challenge by acting as a hierarchical, callable primitive for the VLA.
2. MoDE-VLA Architecture: A novel method for integrating heterogeneous force and tactile modalities into VLAs using dedicated self-attention, sparse MoE routing, and residual injection, enabling contact-aware refinement without catastrophic forgetting.
3. First Autonomous Dual-Dexterous Apple Peeling: The authors demonstrate the first successful autonomous dual-hand apple peeling, a task requiring the full synergy of vision, force, tactile, and in-hand rotation.

4. Experimental Results

The framework was evaluated on four tasks of increasing complexity: Gear Assembling, Charger Plugging, Tube Rearranging, and Apple Peeling.

• Data Acquisition (Q1):

  • IMCopilot vs. Teleoperation: IMCopilot achieved an 89% success rate in in-hand manipulation tasks (e.g., rotating objects), compared to only 34% for pure teleoperation. For small objects like ping-pong balls, teleoperation succeeded only 10% of the time, while IMCopilot reached 83%.
  • Feedback: Adding force/tactile visual feedback to the teleoperator reduced task completion time and increased demonstration success rates.

• Policy Performance (Q2):

  • MoDE-VLA vs. Baseline (π0): MoDE-VLA achieved an average success rate (SR) of 34%, outperforming the baseline by 19%.
  • Task Specifics:
    • Gear Assembling: +20% improvement (critical for contact onset detection).
    • Charger Plugging: +10% improvement.
    • Apple Peeling: Achieved 30% Success Rate and 73% Peel Completion Ratio (PCR). The baseline failed to complete full rings due to slippage.

• Ablation Studies (Q3):

  • Force Sensing: Removing force input caused the largest drop (11% average SR decrease), confirming its necessity for insertion tasks.
  • Tactile Sensing: Removing tactile input reduced SR by 8%, primarily affecting grasp stability and slip detection.
  • IMCopilot: Removing IMCopilot and forcing the VLA to learn rotation directly caused the Apple Peeling PCR to plummet from 73% to 25%, proving the necessity of dedicated low-level skills for in-hand rotation.

5. Significance

This work represents a major step toward human-like robotic dexterity. By combining shared autonomy (to solve the data bottleneck) with hierarchical control (VLA for planning, RL for execution) and specialized sensory fusion (MoDE), the authors demonstrate that robots can perform complex, contact-rich tasks that were previously infeasible for end-to-end learning. The successful execution of apple peeling by a dual-arm robot with dexterous hands serves as a benchmark for future research in contact-rich manipulation, proving that integrating physical feedback and modular skill learning is essential for advancing beyond simple pick-and-place robotics.