💻 computer science

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos

BoxTwin is an interactive digital twin framework that learns the full elastoplastic dynamics of articulated objects from videos, enabling robots to accurately track joint trajectories and predict long-term plastic deformation and damage during physical interactions.

Original authors: Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song, Shivansh Patel, Sonny Hu, Yunzhu Li, Changxi Zheng, Peter Yichen Chen

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song, Shivansh Patel, Sonny Hu, Yunzhu Li, Changxi Zheng, Peter Yichen Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fold a piece of cardboard or close a flimsy metal door. If you ask a standard robot to do this, it might treat the object like a solid brick or a perfect spring. It would try to bend the cardboard, but because real-world materials are messy, the robot would get confused. The cardboard might bend back, stay bent, or even crack, and the robot wouldn't know why. This is the world of "digital twins"—a fancy term for creating a perfect, virtual copy of a real object inside a computer so a robot can practice before touching the real thing. But until now, these virtual copies have been terrible at handling things that are part rigid, part squishy, and part broken. They struggle with objects that bend like rubber, stay bent like wet clay, and get weaker every time you fold them. This paper tackles that specific messiness, aiming to give robots a brain that understands how real-world, bendy, breakable things actually behave.

The researchers behind this work, Heng Zhang and their team, have built a new system called BoxTwin. Think of BoxTwin as a super-smart video detective that watches a robot play with a bendy object and learns exactly how that object behaves. Instead of guessing the rules, BoxTwin watches the video, figures out the hidden physics, and builds a digital twin that can predict the future.

Here is the magic trick: most digital twins assume that if you bend something, it snaps back to its original shape (like a rubber band). BoxTwin knows that's not always true. It understands three tricky things that happen when you mess with objects like cardboard boxes or sheet-metal assemblies:

Non-linear Elasticity: Sometimes, the harder you push, the more it resists in a weird, non-straight way.
Plastic Deformation: Sometimes, if you push too hard, the object stays bent forever. It doesn't snap back.
Damage: Every time you bend it, the material gets a little bit weaker, like a paperclip that eventually snaps after too many folds.

BoxTwin doesn't just guess these rules; it learns them. The system takes a video of a real-world object being folded or manipulated. It then reconstructs the scene and figures out the specific "constitutive model" (a fancy way of saying the rulebook for how that specific material moves) for that object. It tracks how the "rest angle" (where the object wants to sit) changes as it gets bent, and it even counts how much "damage" has accumulated to make the object weaker over time.

To test if this actually works, the team ran two experiments. First, they manually folded a simple object made of two panels and a joint, keeping one side stuck to a table. They used colored markers to track the angles and compared the real-world movement to what BoxTwin predicted in the computer. The result? BoxTwin was spot on. It didn't just get the movement right for a second; it kept getting it right over a long sequence, accurately tracking how the object slowly changed its shape and got "tired" from being folded again and again.

In the second, more impressive test, they used a dual-armed robot (called Trossen Aloha) to grab, fold, and move these tricky objects. They recorded every move the robot made and then replayed those exact moves in the BoxTwin simulation. The digital twin predicted the robot's actions and the object's reaction with high accuracy, even for complex objects with multiple joints. The simulation matched the real world so well that the team believes robots can now use these digital twins to plan their moves safely without needing to crash into things or fail repeatedly in the real world.

In short, BoxTwin is a leap forward because it stops pretending that bendy, breakable objects are simple. By combining video learning with a deep understanding of how materials bend, stay bent, and break, it gives robots a way to anticipate the messy, unpredictable nature of the real world. This means robots might soon be able to handle delicate tasks—like folding laundry or assembling flimsy parts—with the same confidence they currently have when moving heavy, solid boxes.

Technical Summary: BoxTwin – Learning Elastoplastic Articulated Object Dynamics from Videos

Problem Statement

Robotic manipulation in unstructured environments requires digital twins capable of anticipating physical interactions. While digital twins exist for rigid bodies and linearly elastic materials, they fail to model Elastoplastic Articulated Objects (EAOs). EAOs, such as folding boxes, sheet-metal assemblies, and hinged doors, exhibit a complex triad of physical phenomena that current models cannot capture simultaneously:

Non-linear Elasticity: Stress-strain relationships that deviate from Hookean behavior, causing energy dissipation and inter-degree-of-freedom coupling.
Plastic Deformation: Irreversible shape alterations that occur once critical loads are exceeded, establishing new rest configurations.
Damage Evolution: The gradual degradation of material parameters (e.g., stiffness loss) over repeated loading cycles due to micro-fractures and delamination.

Existing approaches typically address either rigid dynamics or static elastic deformation, leaving the intertwined nature of plasticity, non-linear elasticity, and progressive damage in articulated objects unaddressed. This limitation causes robots to fail in tasks involving grasping, folding, or forecasting post-interaction states.

Methodology: The BoxTwin Framework

BoxTwin is an interactive digital-twin framework designed to learn the full dynamics of EAOs directly from video data. The system integrates a video-driven reconstruction pipeline with a physics-aware constitutive model.

1. Articulated Body Representation

The EAO is modeled as a collection of rigid links connected by ideal hinge joints, representing flexible creases. The topology is defined by a rooted kinematic tree $G = (V, E)$ . The configuration is described by a vector of joint angles $q$ . The system utilizes standard Lagrange formulation for joint-space dynamics:
$M(q)\ddot{q} + C(q, \dot{q})\dot{q} + g(q) = \tau$
where $\tau$ represents generalized forces from external contacts.

2. Physics-Aware Constitutive Modeling

Instead of static parameters, BoxTwin employs a calibrated model that evolves over time to capture three specific behaviors:

Non-linear Elastic Response: The elastic torque $\tau^{el}_e$ for a hinge is modeled as a smooth, differentiable mapping $f_{\theta_e}(\Delta_e)$ of the deformation gap $\Delta_e = q_e - \bar{q}_e$ . This captures non-linear hinge behavior while remaining amenable to analytical differentiation.
Plastic Deformation: Permanent set is modeled by allowing the rest angle $\bar{q}_e$ to evolve via an internal scalar variable $\alpha_e$ (accumulated plastic rotation). Plastic flow activates when the elastic gap exceeds a yield threshold $\Delta^y_e$ . The evolution of $\alpha_e$ follows a smooth flow rule $g_{\psi_e}(\Delta_e)$ .
Damage Evolution: To account for stiffness loss and reduced capacity for permanent set, a scalar damage variable $d_e \in [0, 1]$ is introduced. This variable multiplicatively attenuates both the elastic torque and the plastic flow rate. The damage rate $\dot{d}_e$ is driven by a smooth, monotone function of the cumulative plastic slip $P_e(t)$ , which records the total absolute plastic rotation over time.

3. Learning Pipeline

The framework reconstructs the scene from input videos to extract EAO parameters. It then uses these parameters to predict future states. The system continuously updates material properties to reflect time-varying damage accumulation, enabling robust simulation and control.

Experimental Evaluation

The authors evaluated BoxTwin using the MuJoCo simulator in two distinct scenarios:

A. Folding Test (Manual Deformation)

Setup: A human manually deformed an EAO consisting of two panels and a single joint, with one panel anchored. Color markers on the panels allowed for the derivation of joint angles.
Result: The same actuation applied to the physical object was transferred to the simulation. The model accurately tracked joint-angle trajectories over long sequences, faithfully reproducing instantaneous kinematic responses and the gradual evolution of material properties (hysteresis and wear).

B. Prediction Test (Dual-Arm Manipulation)

Setup: Trossen Aloha dual-arm robots manipulated various EAOs (single-joint and multi-joint assemblies). The system operated in a leader-follower configuration, where leader arms received high-level commands and follower arms executed physical actions.
Result: Full trajectories (positions, velocities, torques, end-effector poses) were logged and replayed in a physics-based simulator with identical kinematic/dynamic parameters and experimentally identified EAO parameters. The simulation showed close correspondence with physical hardware measurements across all test cases, validating the environment for algorithm development and transfer learning without repetitive physical trials.

Key Contributions and Significance

The paper claims to introduce the first interactive digital-twin framework capable of capturing the full dynamics of EAOs during real-world robotic manipulation. Its primary contributions include:

Integration of Complex Dynamics: Successfully combining video-driven reconstruction with a physics-aware model that simultaneously handles non-linear elasticity, plastic deformation, and damage evolution.
Predictive Capability: The ability to predict post-contact plastic behavior and long-horizon dynamics, a capability absent in prior EAO models which were limited to static simulations or rigid-body assumptions.
Adaptive Control: Enabling adaptive, predictive control of deformable objects in unstructured environments where traditional rigid-body assumptions fail.

The authors position BoxTwin as a foundational step toward safer, more adaptable human-robot collaboration, allowing robots to foresee interactions and refine manipulation tactics for deformable items that undergo irreversible physical changes.