💻 computer science

Tri-Manual Visuomotor Imitation Learning of Robot Policies

This paper introduces TriManPolicy, an imitation learning system featuring Dependency-Aware Tri-Arm Scheduling (DATS) that enables a single human operator to effectively demonstrate and train synchronous policies for tri-manual robots by offline retiming independent arm motions while preserving task constraints, thereby overcoming the limitations of traditional bimanual teleoperation.

Original authors: James Zhao, Mingyuan Ba, Weiming Zhi

Published 2026-08-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: James Zhao, Mingyuan Ba, Weiming Zhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots are learning to do chores not by trial and error, but by watching humans do them first. This field is called "imitation learning," and it's like teaching a robot to cook by letting it watch you chop vegetables and stir the pot. Usually, this works best when the robot and the human have the same number of hands. If a human uses two hands to fold a shirt, a two-armed robot can copy the move perfectly. But what happens when the robot is super-powered and has three arms, yet the human teacher only has two? This is the puzzle scientists are trying to solve. They want to know if a robot can learn to use a third "helper" arm just by watching a human who can only control two arms at a time. If the robot tries to copy the human's timing exactly, it might get stuck waiting for the human to switch hands, even though the robot could be doing three things at once. The big question is: Can we teach the robot to ignore the human's "switching delays" and figure out how to use all three arms simultaneously?

Enter TriManPolicy, a clever new system designed to teach three-armed robots using only a two-handed human teacher. The researchers found that when a human controls a three-arm robot, they have to take turns: they control two arms, then switch to a different pair, leaving the third arm frozen in place. If a robot simply copies this video, it learns to freeze too, waiting for the human to switch modes. To fix this, the team created a digital "time-travel" tool called DATS (Dependency-Aware Tri-Arm Scheduling). Think of DATS as a movie editor that watches the human's recording and re-edits the timeline. It keeps all the actual movements—the human holding a bag open with one hand while the other two fold a towel—but it removes the awkward pauses where the human had to stop and switch controls. DATS figures out which actions must happen in a specific order (like putting a lid on a box before placing it down) and which actions can happen at the same time (like holding a box steady while another arm wipes it). By re-arranging the video to show these actions happening together, it trains the robot to be a true three-armed superhero.

The results are impressive. The team tested this on six different real-world tasks, like folding towels, taping bags, and placing erasers in boxes. They ran 25 trials for each task. The robots trained with the "time-traveled" DATS method were significantly faster. In fact, on every single task, the DATS-trained robots finished their jobs in less time than the robots trained on the raw, unedited videos. For example, in a task involving placing erasers, the DATS robots were nearly 50% faster. They also succeeded just as often, if not more often, than the other robots. The data shows that by teaching the robot to ignore the human's switching delays and focus on the logical flow of the task, the robot learns to coordinate all three arms smoothly, turning a clunky, stop-and-go performance into a fluid, synchronized dance.

Technical Summary: Tri-Manual Visuomotor Imitation Learning of Robot Policies

Problem Statement

Standard bimanual teleoperation assumes a one-to-one mapping between an operator's two hands and a robot's two end-effectors. This assumption fails for tri-manual systems, where a robot possesses three arms but a single human operator can only continuously control two channels simultaneously. To demonstrate tasks for three arms, operators must use pairwise mode switching, serializing independent motions that could otherwise occur concurrently.

When training imitation learning policies (e.g., via behavior cloning) on such data, the policy often learns the interface-imposed delays rather than the task's intrinsic requirements. The robot may reproduce unnecessary waiting periods where one arm idles while the operator switches control to another pair, even though the task geometry would allow simultaneous execution. The core challenge is to separate the demonstrated local motions (which are valid) from the collection timing (which is suboptimal due to interface constraints) to enable synchronous three-arm control.

Methodology: TriManPolicy and DATS

The authors propose TriManPolicy, a system that enables a single operator to demonstrate tri-manual behaviors by decoupling motion content from global timing. The system pipeline consists of three main stages:

1. Mode-Switched Demonstration Collection

Operators control two of the three arms (Left, Right, or Other) at any given time, switching between pairs ({LR, LO, RO}) as needed. All robot states, observations, and commands are logged under a unified clock, recording the active teleoperation pair at each timestep.

2. Subtask Annotation and Graph Construction

For each recorded episode, a Vision-Language Model (VLM) proposes segment boundaries, required arms, and predecessor relations. Human annotators review and refine these into a task graph $G_n = (V_n, E_n)$ , where:

Nodes ( $V_n$ ): Represent temporally localized subtasks with fixed durations, required arm sets, and optional semantic labels.
Edges ( $E_n$ ): Encode finish-to-start dependencies (task prerequisites) and resource constraints (e.g., an arm cannot be used by two segments simultaneously).
Key Distinction: The graph distinguishes between task-required orderings (which must be preserved) and interface-induced serializations (which can be relaxed).

3. Dependency-Aware Tri-Arm Scheduling (DATS)

The core innovation, DATS, treats demonstration retiming as a constrained scheduling problem. It takes the fixed-duration sensorimotor segments and repositions them on a new global timeline to minimize the episode makespan while satisfying:

Precedence Constraints: If segment $A$ must precede $B$ , $A$ must finish before $B$ starts.
Resource Exclusivity: A single arm cannot execute two segments simultaneously.
Goal: Maximize concurrency of disjoint-arm segments that have no dependency path, effectively "parallelizing" the demonstration.

Retiming Process:
DATS computes an optimal schedule using CP-SAT (Google OR-Tools). It applies a forward time map to shift the timestamps of all messages within a segment by a constant offset. This preserves the local sensorimotor timing within each segment while altering the global alignment between arms. The resulting retimed data trains a single synchronous policy that controls all three arms jointly.

4. Policy Learning

The system trains an action-chunked transformer policy. Given an observation $o_t$ , the policy predicts a horizon- $H$ sequence of synchronized actions for all three arms. Crucially, the training objective minimizes the same loss function for both raw and DATS data, but the joint supervision windows differ: DATS places compatible cross-arm targets within the same training window, whereas raw data often pairs progress in one arm with idle holds in others.

Key Contributions

Problem Formulation: Extends teleoperated imitation learning to settings where the robot's manipulation channels exceed the operator's simultaneous control capacity, specifically formulating the problem for three cooperating arms.
DATS Algorithm: Introduces a scheduling-based approach to retiming demonstrations. It preserves task dependencies and arm-resource constraints while relaxing interface-imposed serializations, allowing independent arm motions to overlap.
System Evaluation: Trains and evaluates synchronous three-arm policies across six challenging real-world tasks, demonstrating that retimed data leads to faster execution without sacrificing success rates.

Experimental Results

The system was evaluated on six real-world tasks (e.g., TowelHang, BinTowel, BagTape) using 237 processed demonstrations and 300 real-robot trials (25 trials per task per condition).

Success Rates: DATS-trained policies achieved equal or higher success counts compared to the baseline (Raw) policies across all six tasks.
Completion Time: DATS policies significantly reduced the mean completion time for successful trials on every task, with reductions ranging from 31.3% to 49.9%. The aggregate macro-mean time dropped from 83.1s (Baseline) to 47.8s (DATS).
Offline Analysis:
- Duration: In some tasks (e.g., TowelHang, BinTowel), DATS shortened the episode timeline by removing idle intervals.
- Co-Window Coverage: In all tasks, DATS significantly increased the co-window coverage (the fraction of eligible cross-arm segment pairs appearing in the same training window). For instance, in TrayWipe, coverage jumped from ~9.5% (Raw) to ~71.2% (DATS) with negligible change in total duration. This indicates that DATS successfully changes what the policy learns to do simultaneously, not just how fast it does it.

Significance and Claims

The paper claims that supervision timing is a critical factor in tri-manual imitation learning. By treating mode-switch staging as a property of the training timeline rather than a constraint the deployed policy must inherit, TriManPolicy allows a single operator to demonstrate complex, concurrent three-arm behaviors.

The authors emphasize that:

The local sensorimotor content of demonstrations remains valid and useful even when their global timing is suboptimal.
DATS effectively reconstructs the global timing to align with task dependencies and available resources, preventing the policy from learning unnecessary delays.
The approach requires no dependency graph or scheduler at deployment; the learned policy acts directly on live, synchronous observations, having internalized the concurrent execution patterns during training.

The results demonstrate that separating demonstration content from collection timing enables efficient, synchronous tri-manual control using data collected via standard pairwise teleoperation.