M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation

Robots that can move around a house and pick things up have long been a dream of science fiction, but turning that dream into reality requires solving a difficult physical puzzle. To be truly useful, a robot must do two things at once: navigate through a cluttered room and gently manipulate objects without breaking them. This requires a delicate balance between moving its body and feeling the world it touches. If a robot pushes too hard, it might knock over a vase; if it moves too stiffly, it cannot open a drawer or wipe a surface clean. For a robot to learn these skills from a human, the human must be able to guide it with a level of sensitivity that feels natural, while the robot must be able to listen to its own sensors and adjust its grip and pressure in real time. Without this ability to feel and adapt, the data collected for teaching the robot is often clumsy and inconsistent, making it hard for the machine to learn complex tasks later.

Researchers at Shenzhen Technology University have developed a new system called M3-Tele to solve this problem. They built a framework that allows a human to control a mobile robot arm using a smartphone, but with a crucial twist: the system is designed to feel the contact between the robot and the world. The robot is equipped with special sensors on its gripper that can see how much the soft pads on its fingers are squishing when they touch an object, as well as sensors that measure the force pushing against its wrist. When a human operator guides the robot to wipe a whiteboard or open a drawer, the system does not just follow the hand's path blindly. Instead, it constantly checks the pressure and the squish of the gripper, automatically adjusting the robot's movement to keep the contact gentle and steady. This creates a smooth, responsive experience where the robot behaves like a compliant partner rather than a rigid machine.

The team tested this system on four different tasks that ranged from simple to difficult. They asked human volunteers to use the system to pick up a block, rotate a bar, pull open a drawer, and wipe a whiteboard. The results showed that the new system was significantly better at keeping the robot in contact with objects without losing its grip or applying too much force. When the researchers compared their system to older methods that did not use these force-sensing adjustments, the difference was stark. The new controller reduced the error in tracking the desired force from over 4 newtons down to less than 0.35 newtons. In practical terms, this means the robot could hold a steady pressure against a surface with much greater precision. Furthermore, the system prevented the robot from accidentally losing contact with an object, reducing the number of times the grip slipped from nearly 2.5 times per attempt to almost zero.

The study also looked at how easy the system was for humans to use. The researchers found that giving the operator feedback about the robot's touch made the task feel much more intuitive. Users rated the ease of using the new feedback-enabled interface significantly higher than standard control methods, scoring it around 8.4 out of 10 compared to 5.8 for a basic version. This suggests that when a human can "feel" what the robot is feeling through the controller, they can guide it more effectively. The system worked well across all the different tasks, successfully completing the challenging wiping task 80% of the time and simpler tasks up to 94% of the time. This reliability is essential because it means the robot can collect high-quality demonstrations that are consistent enough to be used for training.

Finally, the researchers used the data collected by this system to teach a robot how to wipe a board on its own, using a learning method known as a diffusion policy. They tested whether the robot learned better when it had access to all the sensory information—the camera view, the force measurements, and the tactile squish data—or if it could get by with just the camera. The experiments showed that the robot learned the most effective wiping behavior when it had all three types of information combined. Even when the researchers gave the learning algorithm only a small amount of data, the combination of touch and force sensors helped the robot understand the task better than vision alone. This confirms that for robots to master tasks involving physical contact, they need to be taught with data that captures not just what the robot sees, but exactly how it feels the world. The work demonstrates that by integrating these senses into the control system, we can build robots that are not only capable of moving through our homes but are also gentle and precise enough to interact with the objects inside them.