💻 computer science

TriPilot-FF: Coordinated Whole-Body Teleoperation with Force Feedback

TriPilot-FF is an open-source whole-body teleoperation system for bimanual mobile manipulators that utilizes a foot-operated pedal with lidar-driven haptics for collision-averse base control, alongside arm-side force feedback and manipulability guidance to enhance operator coordination and task performance.

Original authors: Zihao Li, Yanan Zhou, Ranpeng Qiu, Hangyu Wu, Guoqiang Ren, Weiming Zhi

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihao Li, Yanan Zhou, Ranpeng Qiu, Hangyu Wu, Guoqiang Ren, Weiming Zhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to drive a massive, two-armed construction robot through a crowded, narrow hallway while simultaneously trying to pick up a delicate egg with both hands.

If you use a standard joystick, your thumbs are doing all the work. You’re trying to steer the heavy base, avoid hitting walls, and move your fingers with precision—all at once. It’s like trying to play a high-stakes video game where your hands are exhausted, and you’re constantly bumping into things because you can't "feel" the environment.

The researchers behind TriPilot-FF decided to fix this by giving the robot a "whole-body" interface that works more like a human body.

The Core Idea: "The Three-Point Control"

Instead of cramming all the controls into your hands, they split the work across your limbs, much like how you drive a car or ride a bike:

The Feet (The Steering Wheel): They invented a special foot pedal to control the robot's wheels. Instead of using your hands to steer, you use your feet. But it’s not just a regular pedal; it has "haptic feedback."
- The Analogy: Imagine you are driving a car, and as you get closer to a curb, the steering wheel suddenly becomes very heavy and hard to turn in that direction. You don't hit the curb because the car "pushed back" against you. That’s exactly what the pedal does using lasers (Lidar) to sense obstacles. It physically resists your foot if you try to drive into a wall.
The Hands (The Precision Tools): Your hands use "leader-follower" arms. When you move your hands, the robot's arms mimic you perfectly.
- The Analogy: It’s like wearing a pair of high-tech "force-feedback" gloves. If the robot’s hand touches a table, you actually feel that resistance in your own hands. This prevents the robot from accidentally crushing things or pushing too hard.
The "Co-Pilot" (The Intuitive Guide): The system acts like a smart navigator. If the robot's arms are reaching too far (like you stretching your arms to grab something on a high shelf), the foot pedal will give you a little "nudge" or a vibration, telling you, "Hey, move the base closer so you don't strain yourself!"

Why does this matter for AI?

The researchers didn't just want to make it easier for humans; they wanted to make it better for Robot Learning.

Currently, we teach robots by having humans "show" them what to do (Imitation Learning). If the human is clumsy or bumps into things, the robot learns to be clumsy. By using this "TriPilot" system, the human provides much "cleaner" demonstrations.

They even taught the robot's "brain" (an AI called ACT) to pay attention to the force and pressure felt during the task.

The Analogy: It’s the difference between teaching a child to pick up a glass by showing them a video (visual only) versus actually holding their hand and letting them feel the weight and the texture of the glass (visual + touch). The second way makes them much better at it.

The Result

In their tests, the robot was able to:

Carry bulky objects through narrow doors without crashing.
Perform complex "hand-overs" (passing an object from one hand to the other).
Navigate a whole kitchen, opening fridges and moving items around.

In short: TriPilot-FF turns robot teleoperation from a stressful "one-handed" struggle into a coordinated, full-body dance.

Technical Summary: TriPilot-FF: Coordinated Whole-Body Teleoperation with Force Feedback

1. Problem Statement

Mobile manipulators (robots with arms mounted on mobile bases) offer large operational envelopes but present a significant control challenge: whole-body coordination. Operators must simultaneously manage base navigation, collision avoidance, and complex bimanual manipulation.

Existing teleoperation interfaces are primarily "hand-centric" (using VR controllers or joysticks), which overburdens the operator's hands and leaves the feet—a robust, low-latency control channel—underutilized. Furthermore, relying solely on visual feedback (cameras) forces operators to mentally simulate physical constraints like reachability, contact forces, and obstacle proximity, leading to high cognitive load and frequent task failures.

2. Methodology

The authors propose TriPilot-FF, an open-source whole-body teleoperation system designed to "co-pilot" the human operator through multi-modal haptic and visual guidance. The system architecture is divided into three main components:

Hardware Design:
- Foot-Operated Pedal: A low-cost, 3-DoF (x, y, yaw) pedal that controls the mobile base. It uses a turntable-linkage mechanism to allow bidirectional interaction (sensing movement and rendering resistance).
- Bimanual Leader-Follower Arms: Isomorphic arm setups where the operator's movements are mapped to the robot's arms.
Software & Haptic Guidance (The "Co-Pilot"):
- Lidar-Driven Pedal Haptics: The system uses a base-mounted lidar to detect obstacles. It calculates a repulsive potential field based on proximity; if an operator commands a direction toward an obstacle, the pedal renders a resistive force, physically biasing the user toward collision-averse behavior.
- Arm-Side Force Reflection: To improve contact awareness, the system estimates interaction torques at the robot's joints and reflects them back to the leader arms (using gravity compensation), allowing the operator to "feel" contact.
- Manipulability Guidance: To prevent the arms from reaching kinematic limits (singularities), the system computes a "manipulability field." It provides haptic cues via the pedal to suggest base repositioning and overlays visual dexterity indicators on the camera feed.
Policy Learning (Imitation Learning):
- The authors extend the Action Chunking with Transformers (ACT) policy. Unlike standard ACT, which uses only visual and joint state data, TriPilot-FF incorporates joint torque signals into the observation space. This allows the learned policy to capture the nuances of contact and load-sharing demonstrated during teleoperation.

3. Key Contributions

TriPilot-FF System: A complete, open-source whole-body teleoperation framework coupling foot-based base control with bimanual arm control.
Multi-Modal Feedback: The integration of lidar-driven pedal resistance, arm-side force reflection, and manipulability-based guidance.
Torque-Augmented Learning: A novel approach to training imitation learning policies by injecting structured haptic/torque information into the transformer-based architecture.
Open-Source Release: Provision of the pedal design, software stack, and datasets.

4. Results

The system was evaluated through several rigorous benchmarks:

Tight Space Coordination: In "BlindCarry" (carrying an object that occludes the camera), the addition of pedal feedback increased success rates from 55% to 100% and eliminated collisions.
Reachability: In "GuidedReach" tasks, manipulability guidance reduced completion time and the amount of time spent in "kinematically fragile" (low-dexterity) configurations.
Contact Stability: In "MobileSwipe" (maintaining contact with a surface while moving), force reflection significantly reduced torque oscillations ( $\sigma_{tor}$ ) and energy consumption, leading to smoother contact regulation.
Imitation Learning Performance:
- In Simulation (MuJoCo): Adding torque to the ACT policy increased success rates in "CubeTransfer" from 22% to 50%.
- In Real-World: For the "BasketPack" task, the "ACT + Torque + Co-training" approach achieved a 60% success rate, significantly outperforming standard ACT (12%). The policy also demonstrated robustness in Out-of-Distribution (OOD) scenarios.

5. Significance

TriPilot-FF shifts the paradigm of teleoperation from a simple "input-to-action" mapping to an interactive, embodied partnership. By reclaiming the feet for base control and using haptics to communicate environmental constraints, the system reduces the cognitive burden on humans. Most importantly, it demonstrates that better teleoperation interfaces lead to better robot learning; by providing high-quality, force-aware demonstrations, the resulting autonomous policies are more robust, efficient, and capable of handling complex, contact-rich tasks.