ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction
ForceTwin is a system that identifies physics-informed digital twins of articulated objects by analyzing instrumented human interactions with force-sensing grippers, enabling significantly more accurate manipulation and control compared to existing kinematics-only or visual-language prior-based approaches.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots have long been masters of the factory floor, where every object is fixed in place and every movement is pre-programmed. But the moment a robot steps into a human home, it faces a chaotic reality: a drawer might be empty or packed with heavy dishes, a door might have a simple hinge or a hidden spring that fights against being opened. To move these objects, a robot needs more than just a picture of what they look like; it needs to understand how they feel. It must know the weight of the object, the friction of its sliding tracks, and the hidden forces inside its mechanisms. Without this physical intuition, a robot pushing a door might simply stall, unable to tell the difference between a stuck object and one that is naturally resistant.
This is the challenge that researchers at ETH Zurich and their collaborators have tackled with a new system called ForceTwin. Instead of relying on cameras to guess how heavy or stiff an object is, they let a human do the work. A person holds a special tool equipped with a force sensor and simply pushes, pulls, and wiggles a door, drawer, or cabinet. As the human interacts with the object, the tool records exactly how the object moves and, crucially, how much force is required to move it. From these simple, natural interactions, the system builds a "digital twin"—a virtual copy of the object that includes not just its shape, but its specific physical personality. This twin knows exactly how much effort is needed to open a specific door, accounting for invisible springs, heavy contents, and sticky hinges that a camera would never see.
The core of this discovery lies in the realization that appearance is a poor guide for physics. Two doors that look identical might require vastly different amounts of strength to open. One might be light and loose, while the other is heavy with a powerful closer mechanism that slams it shut. Previous attempts to create digital copies of objects relied on visual data or language descriptions to guess these properties, often resulting in simulations that looked right but felt wrong. When robots tried to use these flawed models, they frequently failed, especially with objects that had strong internal mechanisms. The researchers found that by measuring the actual forces during human interaction, they could identify the true physical parameters of the object. They separated the constant properties, like the object's mass and basic friction, from the complex, changing forces of mechanisms like door closers, which behave differently depending on how fast or far the object is moving.
To test if this approach actually worked, the team put their digital twins to the test in the real world. They used the data gathered from human probing to program two very different robots: a four-legged robot named Spot and a stationary arm called Franka. The robots were tasked with opening a variety of objects, including drawers, sliding doors, and heavy wooden and metal doors. When the robots used the new, physics-informed twins, they succeeded in opening the objects 87 percent of the time. In contrast, when they relied on older methods that only guessed the physics based on what the objects looked like, or methods that only knew the shape but not the forces, their success rates dropped to 60 percent and 57 percent, respectively. The difference was most dramatic with the difficult objects. On a heavy metal door with a strong spring mechanism, the older methods caused the robots to stall completely, unable to generate enough force to move the door. The ForceTwin system, however, allowed the robots to push through the resistance and complete the task.
The researchers also demonstrated that these digital twins could be used to teach robots new skills through simulation. By feeding the accurate physical data into a computer simulation, they trained a robot to walk through a doorway. When they sent this trained robot into the real world to try the same task on a real door, it succeeded. This proved that the physical model captured by the human interaction was accurate enough to guide complex, real-world behavior. The system works by taking the raw data of the human's push and pull, calculating the object's inertia and friction, and then using a smart algorithm to model the remaining tricky forces. This creates a complete picture of the object's behavior that can be used immediately by a robot controller or exported to a simulation environment.
While the system is powerful, it does have limits. It requires a human to physically interact with each object to gather the data, meaning it cannot instantly guess the properties of an object it has never touched. It also assumes the object behaves in a predictable way while being moved, so it cannot account for things like a drawer that suddenly gets stuck or a latch that changes state. However, for the objects it does probe, the results are strikingly accurate. The system reduced the error in estimating an object's weight and resistance by nearly half compared to previous guessing methods. It successfully identified that a heavy drawer was indeed heavy, and that a specific door required a specific amount of extra force to overcome its internal spring.
This work represents a shift in how robots learn about the physical world. Instead of trying to infer everything from a static image, it embraces the idea that to truly understand an object, one must touch it. By combining the dexterity of human interaction with the precision of force sensors, the researchers have created a way to give robots a sense of touch that extends beyond their own grippers. They have shown that a simple, human-led exploration can generate a detailed physical model that allows a machine to navigate the messy, resistant reality of a human home. The result is a robot that does not just see a door, but understands the effort required to open it, turning a potential failure into a successful interaction.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.