💻 computer science

Learning to unfold cloth: Scaling up world models to deformable object manipulation

This paper presents a modified DreamerV2 reinforcement learning architecture that incorporates surface normals and enhanced data augmentation to enable a robot to generalize in-air cloth unfolding across various shapes, sizes, and patterns through both simulation and zero-shot physical deployment.

Original authors: Jack Rome, Stephen James, Subramanian Ramamoorthy

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Rome, Stephen James, Subramanian Ramamoorthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to shake out a crumpled towel after a shower. To a human, this is easy: you grab a corner, give it a flick, and watch it snap open. But to a robot, this task is a nightmare. Cloth is floppy, unpredictable, and hides its own corners. If a robot tries to grab it, it might just grab a bunched-up knot instead of a flat edge, and the whole thing stays crumpled.

This paper is about teaching a robot how to master this "flick and unfold" trick, not by following a rigid set of instructions, but by learning to dream about how cloth behaves.

Here is the story of how they did it, broken down into simple concepts:

1. The Problem: The "Shape-Shifting" Enemy

Robots are great at moving solid boxes (like a toaster or a book) because the shape never changes. But cloth is a shape-shifter. It twists, folds, and wrinkles in millions of ways.

The Old Way: Previous robots tried to use cameras to take a 3D picture of the cloth, calculate exactly where every fold is, and then move their arm. This was like trying to solve a math equation while the equation was changing its numbers every second. It was too slow and too complicated.
The New Way: Instead of calculating the math, the researchers taught the robot to learn by experience, similar to how a baby learns to walk by falling down and getting back up.

2. The Brain: The "Dreamer"

The robot uses a special AI brain called DreamerV2. Think of this brain as a video game simulator inside the robot's head.

How it works: Instead of just reacting to what it sees right now, the robot "dreams" about what will happen next. It asks itself, "If I pull this corner to the left, will the towel open up, or will it just bunch up more?"
The Benefit: Because it can simulate thousands of "what-if" scenarios in its head, it learns the best way to move without needing to physically crash into things a million times.

3. The Superpower: Seeing with "Surface Normals"

This is the paper's biggest trick. Usually, robots look at cloth using standard cameras (RGB), which see color and patterns.

The Problem: If you train a robot on a red towel, it might get confused when you give it a blue towel. It's too focused on the color rather than the shape.
The Solution: The researchers replaced the color camera with a "Surface Normal" view. Imagine looking at the towel not as a colorful object, but as a topographic map or a 3D relief sculpture.
- In this view, a red shirt and a blue shirt look exactly the same.
- What matters is the geometry: where the wrinkles are, where the fabric is flat, and where the corners are sticking out.
- It's like teaching a blind person to read Braille by feeling the bumps, rather than trying to read the ink on the page. The robot learns to feel the shape of the cloth, ignoring the distracting colors.

4. The Training: Cheating a Little (The Replay Buffer)

Training a robot to do something this hard usually takes forever because it has to fail a lot before it succeeds.

The Fix: The researchers didn't just let the robot wander around randomly. They first showed it a few videos of a human (or a perfect script) successfully unfolding a towel. They loaded these "expert demonstrations" into the robot's memory bank (the Replay Buffer).
The Analogy: It's like a student taking a test. Instead of guessing every answer from scratch, the teacher gives them a few solved examples first. The student then studies those examples and learns the pattern of how to solve the problem, rather than just guessing.
Augmentation: They also taught the robot to ignore minor distractions (like slight changes in lighting or camera angles) by randomly "shaking up" the training images. This makes the robot robust, so it doesn't panic if the real world isn't perfect.

5. The Result: From Simulation to Real Life

They trained the robot in a computer simulation (a virtual world) and then sent it to a real robot arm in a lab.

The Test: They threw all sorts of clothes at the robot: t-shirts, long-sleeved shirts, dishcloths, and even shorts.
The Outcome: The robot successfully unfolded about 74% of the items on the very first try, without needing to be retrained for each specific shirt.
The Catch: It struggled a bit with shorts. Why? Because shorts are small. In the robot's "vision," they took up very few pixels, making it hard to see the corners. It's like trying to thread a needle while wearing thick winter gloves.

Summary: Why This Matters

This paper shows that if you give a robot the right "eyes" (surface normals) and the right "dreaming" ability (world models), it can learn to handle messy, floppy objects like a human.

It moves us one step closer to robots that can actually help us at home—folding laundry, making beds, or helping elderly people get dressed—without needing a human to hold their hand and guide them through every single fold. They aren't just following a script; they are learning the physics of the fabric.

1. Problem Statement

The paper addresses the challenge of in-air cloth manipulation, specifically the task of unfolding deformable objects (like towels, shirts, and pants) using a robotic arm.

Complexity: Unlike rigid objects, cloth undergoes high-dimensional, non-linear deformations. The state of the cloth changes rapidly, and self-occlusion often hides target grasp points (e.g., corners).
Limitations of Existing Methods:
- Traditional 3D Mapping: Methods relying on 3D reconstruction often require slow, multi-step processes (rotating the cloth, multiple sensors) and fail if the cloth is disturbed during the process.
- Standard Reinforcement Learning (RL): Applying standard RL to cloth is difficult due to the vast visual variability (colors, textures) and the difficulty of generalizing from simulation to reality (Sim-to-Real). Standard RGB inputs often lead to overfitting on specific visual appearances rather than learning the underlying geometry.
- Real-time Constraints: Many existing methods are too slow for dynamic, real-time interaction.

2. Methodology

The authors propose a modified DreamerV2 reinforcement learning architecture designed to learn a "world model" (a latent representation of the environment) to plan actions without constant real-world interaction.

A. Input Modality: Surface Normals vs. RGB

Instead of using standard RGB images, the system converts depth images into surface normal images.

Rationale: RGB inputs are sensitive to color and texture variations, which are irrelevant to the physics of unfolding. Surface normals encode the geometric orientation of the cloth (wrinkles, creases, folds) while filtering out appearance noise.
Implementation: Depth images are processed using Sobel filters (kernel size 9) to generate 64x64x3 normal maps. This allows the policy to focus on the physical state of the object.

B. Architecture Modifications

The authors modified the standard DreamerV2 pipeline in two critical ways:

Replay Buffer Enhancement (Expert Bootstrapping):
- Standard DreamerV2 relies on random exploration to build the initial world model, which is inefficient for complex tasks like cloth manipulation where reaching the goal state (unfolded) via random actions is rare.
- Solution: The authors load a set of demonstration episodes (expert data) into the replay buffer before training begins. This ensures the agent immediately learns the latent representation of the goal state.
Data Augmentation:
- To prevent overfitting to specific camera angles or lighting conditions, the system applies random batch augmentations (brightness/contrast shifts, hue/saturation changes, slight rotations, translations, and zooms) to images sampled from the replay buffer.
- Note: Flipping and heavy rotation are avoided to maintain consistency with the control axes.

C. Environment and Reward

Simulation: Built using Unity with the ObiCloth physics engine and RFUniverse for communication.
State Space: The agent receives two camera views (a stand-off camera for global context and a wrist-mounted "ego" camera for grasping), surface normal images, and proprioceptive data (gripper position, width, tension flag).
Action Space: End-effector displacement (XYZ) and gripper open/close.
Reward Function: Based on the normalized surface area of the cloth. The agent is rewarded for maximizing the unfolded area ( $A_{norm}$ ) while penalized for time steps ( $t$ ) to encourage speed.

3. Key Contributions

Surface Normal Input: Demonstrated that replacing RGB with surface normals significantly improves generalization across different cloth types by prioritizing geometry over appearance.
Replay Buffer Strategy: Introduced a method of pre-loading expert demonstrations to bootstrap the world model, solving the "cold start" problem in complex deformable object tasks where random exploration fails.
Zero-Shot Sim-to-Real: Achieved successful deployment of policies trained in simulation directly onto a physical Franka Emika robot without fine-tuning, handling real-world sensor noise and domain gaps.
Comprehensive Evaluation: Tested on five distinct garment categories (plane cloth, short/long-sleeve shirts, shorts, unbuttoned shirts) and compared against multiple baselines (Pick-and-Place scripts, vanilla DreamerV2, SAC).

4. Results

Simulation Results

Performance: The combination of Surface Normals + Modified Replay Buffer yielded the best results.
- Achieved a 15% increase in success rate compared to traditional Pick-and-Place algorithms.
- Reduced failure rates by 35%.
- Outperformed vanilla DreamerV2 and SAC models significantly.
Garment Types: The method worked well for towels, shirts, and unbuttoned shirts.
Outlier: Performance dropped significantly on shorts. The authors attribute this to the small physical size of shorts, which occupy fewer pixels in the 64x64 observation space, making grasp point identification difficult.

Real-World Deployment (Zero-Shot)

Setup: A Franka robot with stereo cameras (one fixed, one wrist-mounted) manipulating various real-world cloths.
Success Rate: Achieved an average 74% success rate across 25 trials per garment type (towel, dishcloth, facecloth, pillowcase).
Comparison: Comparable to state-of-the-art literature (which reports 82–86%), but with the distinct advantage of real-time operation and no need for pre-processing beyond loading the agent.
Robustness: The augmented training allowed the robot to handle slight misalignments between the simulation and real-world camera positions.

5. Significance and Future Work

Significance: This work bridges the gap between complex deformable object manipulation and scalable RL. It proves that world models, when fed geometrically rich data (normals) and bootstrapped with expert data, can learn robust policies for tasks previously considered too difficult for autonomous robots.
Limitations Identified:
- Sensor Noise: Real-world depth sensors struggle when the gripper gets too close (below minimum sensing distance), causing data loss.
- Single-Arm Constraint: The current single-arm setup limits the ability to re-orient large garments (e.g., trousers, dresses) in mid-air.
- Generalization: Currently, a separate agent is trained for each garment type.
Future Directions:
- Implementing dual-arm robots to enable mid-air repositioning of larger garments.
- Exploring tactile sensing to complement visual inputs when depth data is occluded.
- Developing policy transfer methods to train a single agent capable of handling diverse garment types.

In conclusion, the paper presents a scalable, high-performance framework for robotic cloth manipulation that leverages geometric inputs and strategic data augmentation to achieve robust, real-time, zero-shot Sim-to-Real transfer.