🔢 mathematics

Dynamic robotic cloth folding with efficient Koopman operator-based model predictive control

This paper presents a novel model predictive control framework that combines high-fidelity physics-based simulation with efficient kernel-based Koopman operator regression to generate fast, accurate, and generalizable dynamic robotic cloth folding trajectories, effectively overcoming the challenges of nonlinear system identification and simulation-to-reality transfer.

Original authors: Edoardo Caldarelli, Franco Coltraro, Adrià Colomé, Lorenzo Rosasco, Carme Torras

Published 2026-05-19

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Edoardo Caldarelli, Franco Coltraro, Adrià Colomé, Lorenzo Rosasco, Carme Torras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fold a large, floppy piece of fabric on a table. If you move slowly, the fabric just drags along. But if you try to fold it quickly—like a professional in a busy factory—the fabric flies, flutters, and behaves like a wild animal. It has its own momentum, air resistance, and inertia. Getting a robot to do this fast and accurately is incredibly hard because the math describing that "wild" behavior is messy and complex.

This paper presents a clever solution to teach a robot how to fold clothes quickly and precisely, using a mix of high-tech simulation and a machine learning trick called the Koopman Operator.

Here is how they did it, broken down into simple concepts:

1. The Problem: The "Wild" Fabric

When a human folds a shirt quickly, they use the fabric's own momentum to help them. A robot, however, usually struggles. If it moves too slowly, it's inefficient. If it moves too fast, the robot's computer gets confused because the fabric's physics are too complicated to calculate in real-time. It's like trying to predict exactly where a kite will go in a sudden gust of wind just by looking at it; the math is too heavy for the computer to handle quickly enough.

2. The Solution: Two Models Working Together

The authors built a system that uses two different "brains" to solve this:

The "Super-Realistic" Simulator (The Physics Brain):
First, they created a highly detailed computer simulation of the cloth. Think of this as a video game engine that knows exactly how fabric bends, hits the table, and fights the air. This simulator is accurate but very slow to run. It's like a master chef who can cook a perfect meal but takes hours to do it.
The "Fast Learner" (The Koopman Brain):
To make the robot fast, they didn't use the slow simulator directly. Instead, they fed data from the simulator into a machine learning technique called Koopman Operator regression.
- The Analogy: Imagine the fabric's movement is a chaotic, swirling dance. The Koopman method is like a translator that takes that chaotic dance and turns it into a simple, straight-line walk. It doesn't change the dance; it just finds a way to describe it using simple, linear math that a computer can solve instantly.
This creates a "surrogate model"—a simplified, fast version of the fabric that behaves almost exactly like the real thing but is easy for the robot to calculate.

3. The Controller: The "Crystal Ball" Planner

Once they have this fast, simplified model, they use a Model Predictive Controller (MPC).

How it works: Imagine you are driving a car and you want to park perfectly. You don't just look at the spot right in front of you; you look ahead, imagine different paths, and pick the one that gets you there smoothly without hitting anything.
The Robot's Job: The robot uses its "Crystal Ball" (the Koopman model) to look 1.5 seconds into the future. It calculates the perfect path to grab the corner of the cloth and swing it into place. It checks constraints like "don't hit the table" and "keep the motion smooth" before making a move.

4. The Result: Fast and Accurate Folding

The team tested this on a real robot arm (a UR5) with two types of cloth: a wool suit and denim jeans.

The Speed: The robot folded the clothes in less than 1.5 seconds. That is fast enough to use the fabric's own momentum to help the fold, just like a human expert.
The Accuracy: Even though the robot learned everything from a computer simulation (never seeing the real cloth before), it worked perfectly on the real world. This is called a "zero-shot sim-to-real transfer." It's like a pilot practicing in a flight simulator and then landing a real plane perfectly on their first try.
The Grip: Interestingly, the robot only needed to hold one corner of the cloth with a fixed grip (like holding a flagpole) to fold the whole thing. It didn't need two hands or complex grasping.

Summary

The paper shows that by using a high-fidelity simulator to teach a machine learning model how to "linearize" (simplify) the chaotic physics of flying fabric, robots can finally fold clothes quickly and accurately. They turned a complex, messy problem into a simple, solvable math puzzle, allowing a robot to swing a piece of cloth into a perfect fold in under a second and a half.

Technical Summary: Dynamic Robotic Cloth Folding with Efficient Koopman Operator-Based Model Predictive Control

Problem Statement
Robotic cloth manipulation remains a significant challenge, particularly for dynamic folding tasks that rely on fast motions to leverage cloth inertia. Unlike quasi-static manipulation, dynamic folding requires navigating complex, nonlinear cloth dynamics, which hinders system identification and trajectory planning. Traditional physical models often suffer from a large "sim-to-real" gap when applied to high-speed motions, while existing robotic approaches are typically either too slow or fast but imprecise, often requiring multiple attempts to achieve a fold. Furthermore, human experts perform these tasks dynamically and efficiently, yet robotic systems struggle to replicate this dexterity, especially when constrained to single-arm manipulation of a single corner.

Methodology
The proposed approach addresses these challenges through a novel Model Predictive Control (MPC) framework that integrates a high-fidelity physics-based simulator with an efficient, data-driven surrogate model. The core methodology consists of three main components:

Physics-Based Simulation (Simulation-Oriented Model - SOM):
The system utilizes a high-fidelity, inextensible cloth model that accounts for aerodynamics, gravity, stiffness, and damping. This model is discretized into a quadrilateral mesh and solved using an iterative process involving quadratic programming to handle equality constraints (inextensibility) and inequality constraints (collisions with the table and self-collision). This SOM serves as the ground truth for generating training data and provides accurate feedback in simulation.
Data-Driven Surrogate Model (Control-Oriented Model - COM):
To overcome the computational cost of nonlinear MPC, the authors employ Koopman operator regression. This technique lifts the nonlinear dynamics of the cloth into a higher-dimensional (potentially infinite-dimensional) space where the dynamics become linear.
- Training: The model is trained on synthetic data generated by the SOM. The control input is defined as the displacement of two adjacent nodes on the cloth mesh (simulating a vertical grasp with fixed orientation), rather than a single point, to better reflect real-world grasping constraints.
- Approximation: The infinite-dimensional operator is approximated using the Nyström method, which subsamples training data to create a finite-dimensional linear system. This results in a linear state-space model that can be learned efficiently.
- Reconstruction: A reconstruction matrix maps the surrogate state back to the physical cloth state.
Linear Model Predictive Control (LMPC):
The linear surrogate model is embedded within an LMPC loop. The controller solves a finite-horizon optimal control problem (OCP) to generate trajectories that minimize a cost function (balancing state error and control effort) while satisfying specific constraints:
- Safety Constraints: Preventing the grasped corner from colliding with the table or moving outside valid regions.
- Smoothness Constraints: Enforcing limits on velocity and acceleration changes to ensure the trajectory is executable by a real robot.
- Orientation Constraints: Ensuring the two controlled mesh points move with identical displacement to maintain the fixed end-effector orientation.
- Boundary Constraints: Limiting the magnitude of displacement to avoid unstable behaviors.

The control loop operates by solving the OCP using the linear COM for prediction, while the SOM is used to update the state in a closed-loop fashion (in simulation) or to provide the initial state for the next step.

Key Contributions

Hybrid Modeling Architecture: The paper combines a physics-based, collision-aware SOM with an efficient Koopman operator regression algorithm to create a linear, data-driven COM for cloth folding.
Constrained Linear MPC: The authors utilize this COM within a linear MPC strategy to generate constrained robot trajectories in closed-loop with the SOM, enabling the folding of cloth to unseen target poses.
Zero-Shot Sim-to-Real Transfer: The methodology is executed and evaluated on a real robotic platform (UR5), demonstrating successful transfer of high-speed folding trajectories (under 1.5 seconds) without requiring retraining on real-world data.

Experimental Results
The proposed pipeline was evaluated in both simulation and on a real robot using two different materials: wool (suit) and denim (jeans).

Simulation: Experiments showed that increasing the size of the surrogate state (number of Nyström landmarks) improved folding accuracy. Furthermore, a longer prediction horizon was found to be critical for generating trajectories that initially move toward the center of the cloth, a behavior necessary for successful folding.
Real-Robot Performance: The system achieved a zero-shot sim-to-real transfer, successfully folding both wool and denim garments to target poses in less than 1.5 seconds.
- Accuracy: The relative folding error ( $E_{fold}$ ) ranged from approximately 0.77% to 7.23% across different target folds and materials.
- Observations: While denim folds showed lower numerical error ratios, visual analysis indicated that wool folds achieved a flatter surface and more accurate corner placement, attributed to the different physical properties of the materials.
- Execution: Trajectories were executed in open-loop due to the latency of current high-frequency visual feedback systems relative to the speed of the motion.

Significance and Claims
The paper claims that its primary significance lies in demonstrating that accurate, physics-based modeling can be effectively used to train a linear surrogate model via Koopman operator regression. This linearization allows for the efficient computation of folding trajectories without derivatives of the nonlinear simulator, while reducing the dimensionality of the state space.

The authors emphasize that their approach successfully bridges the sim-to-real gap for high-speed dynamic folding, a task where inertial effects typically cause significant failure. By achieving successful folds in under 1.5 seconds with a single manipulator and a single motion, the work challenges the notion that robotic cloth folding must be slow or quasi-static. The paper concludes that while the current system relies on open-loop execution, the framework provides a robust foundation for future research incorporating real-time feedback and more complex grasping primitives.