💻 computer science

Image2Garment: Simulation-ready Garment Generation from a Single Image

This paper proposes a feed-forward framework that generates simulation-ready garments from a single image by fine-tuning a vision-language model to infer material attributes and mapping them to physical parameters via a lightweight predictor, thereby overcoming the lack of image-to-physics datasets and eliminating the need for iterative optimization.

Original authors: Selim Emir Can, Jan Ackermann, Kiyohiro Nakayama, Ruofan Liu, Tong Wu, Yang Zheng, Hugo Bertiche, Menglei Chai, Thabo Beeler, Gordon Wetzstein

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Selim Emir Can, Jan Ackermann, Kiyohiro Nakayama, Ruofan Liu, Tong Wu, Yang Zheng, Hugo Bertiche, Menglei Chai, Thabo Beeler, Gordon Wetzstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single photograph of a person wearing a beautiful, flowing dress. You want to turn that photo into a 3D character for a video game or a movie.

The hard part isn't just making the shape of the dress (which many AI tools can already do). The hard part is figuring out what the dress is made of. Is it heavy velvet that drags on the floor? Is it light silk that flutters in the wind? Or is it stiff denim that holds its shape?

If you get the material wrong, the dress will look fake. A silk dress made of "denim physics" will look like a cardboard box; a denim jacket made of "silk physics" will look like a wet rag.

"Image2Garment" is a new AI system that solves this problem. It looks at a single photo and instantly figures out not just the shape of the clothes, but exactly how the fabric should behave in a physics simulation.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Black Box" of Physics

Previously, if you wanted to simulate cloth, you had two bad options:

Option A (The Detective): You had to film the person from every angle with a studio of cameras and spend hours manually tweaking the physics settings until it looked right. This is slow and expensive.
Option B (The Guess): You could take a photo and guess the material. But without a guide, the AI would just pick random numbers, resulting in clothes that look like they are made of jelly or steel.

There was no "dictionary" that connected what a fabric looks like in a photo to the specific math numbers a computer needs to simulate it.

2. The Solution: A Two-Step Detective Story

The researchers realized they couldn't teach the AI to go straight from "Photo" to "Physics Math" because there isn't enough data to teach that direct link. Instead, they built a two-step translator.

Step 1: The "Fashion Expert" (The Vision-Language Model)

First, the AI acts like a super-smart fashion stylist. It looks at the photo and asks:

"What is this made of?" (e.g., 80% Cotton, 20% Polyester).
"What kind of weave is it?" (e.g., Knit, Woven, Lace).
"How heavy is it?" (e.g., Thick and heavy, or thin and light).

To do this, they taught a massive AI model (based on a technology called Qwen) using a huge new dataset they created called FTAG. Think of this dataset as a giant library of clothing tags from online stores. The AI learned to read the visual cues in a photo and match them to the text descriptions found on real clothing tags.

Step 2: The "Physics Translator" (The Material-to-Physics Map)

Once the "Fashion Expert" says, "This is a heavy, woven cotton," the system passes that description to a second, simpler tool.

This second tool uses a new dataset called T2P (Tag-to-Physics). This is a small, precise database that links fabric descriptions to the actual math numbers needed for simulations.

Input: "Heavy Woven Cotton."
Output: "Stiffness: 400, Bend Resistance: 200, Damping: 0.5."

It's like a translator who speaks "Fashion" and "Physics." The Fashion Expert speaks the language of the photo, and the Physics Translator converts that into the language of the video game engine.

3. The Result: A "Simulation-Ready" Garment

The final output is a 3D dress that is ready to be dropped into a video game engine (like Marvelous Designer or Blender).

If the character jumps, the dress bounces realistically.
If the character spins, the dress flares out correctly.
If the character stops, the fabric settles naturally.

All of this happens in a single pass (instantly), without needing to film the person from multiple angles or spend hours tweaking settings.

Why This Matters

Think of it like this:

Old Way: You hire a sculptor to carve a statue of a dress, then hire a physicist to figure out how that statue would move if it were made of real cloth. It takes days.
Image2Garment: You take a photo, and an AI instantly hands you a digital dress that knows exactly how to move, flutter, and drape, just by "reading" the fabric in the picture.

The "Secret Sauce"

The paper's biggest breakthrough is realizing that you don't need to teach the AI the complex math of physics directly from photos. Instead, teach it to identify the ingredients (the fabric type) first, and then let a simple rulebook convert those ingredients into physics.

It's the difference between trying to memorize every possible way a cake can fall (hard/impossible) versus learning the recipe (flour, eggs, sugar) and knowing that if you change the flour, the texture changes (easy/achievable).

In short: Image2Garment turns a flat photo into a living, breathing piece of clothing that knows its own physics, making virtual worlds feel much more real.

1. Problem Statement

The core challenge addressed by this work is the generation of simulation-ready garments from a single RGB image. While recent advances in computer vision can reconstruct 3D garment geometry (meshes) from single images, they typically fail to predict the physical material properties (e.g., stiffness, damping, density, friction) required for realistic physics-based simulation.

The Gap: Existing methods either require expensive multi-view capture setups, rely on slow iterative optimization (inverse rendering/physics), or predict geometry without material properties, resulting in static or physically inaccurate animations.
The Bottleneck: Directly mapping a single image to physical simulation parameters is an ill-posed problem due to the lack of large-scale datasets pairing real-world garment images with ground-truth physical measurements. Furthermore, visual cues for material composition are often ambiguous (e.g., different blends can look identical).

2. Methodology

The authors propose a feed-forward, two-stage factorization framework that decomposes the ill-posed image-to-physics problem into two learnable, data-efficient sub-problems. The pipeline consists of three main stages:

A. Stage 1: Garment Geometry Reconstruction

The system first estimates the 3D garment shape (sewing pattern) using an existing state-of-the-art model, ChatGarment.
The predicted sewing pattern is draped onto a standard human body (SMPL) to generate the initial 3D mesh geometry ( $\hat{S}$ ).

B. Stage 2: Fabric Attribute Prediction (Image $\to$ Semantics)

Instead of predicting raw physics numbers directly, the model predicts interpretable fabric attributes ( $M, Z$ ) from the single image.
Model: A Vision-Language Model (VLM), specifically Qwen-2.5-VL, is fine-tuned using LoRA (Low-Rank Adaptation).
Task: The VLM is prompted to output a structured JSON string containing:
- Material Composition: Fiber-level mixture (e.g., 70% Cotton, 30% Polyester).
- Fabric Family: Microstructure and appearance (e.g., Denim, Chiffon, Jersey).
- Structure Type: Construction method (Knit, Woven, Others).
- Auxiliary Attributes: Areal density (gsm) and thickness (mm).
Strategy: The authors leverage the abundance of web-scale image-to-text data (garment tags) to train this stage, as "image-to-material" supervision is far more available than "image-to-physics."

C. Stage 3: Physics Parameter Estimation (Semantics $\to$ Physics)

A lightweight Random Forest Regressor (RFR) maps the predicted semantic attributes to the specific physical parameters required by industry-standard simulators (e.g., Marvelous Designer/CLO3D).
Inputs: The predicted composition, family, structure, density, and thickness.
Outputs: Simulator-compatible parameters including bending stiffness, shear stiffness, stretch stiffness, buckling stiffness, buckling ratio, friction, and damping.
Density/Thickness Estimation: Since these are hard to predict visually, the system uses a hierarchical retrieval strategy on a curated dataset to sample density and thickness based on the predicted composition and structure.

3. Key Contributions

A. New Datasets

The paper introduces two critical datasets to bridge the data gap:

FTAG (Fabric Attributes from Garment Tags): A dataset of 16,026 images of people wearing garments, annotated with vendor-provided material composition, fabric family, and structure type. This enables the training of the VLM for semantic attribute prediction.
T2P (Tag-to-Physics): A dataset of 1,254 digitized fabrics linking intrinsic attributes (composition, family, structure, density, thickness) to measured CLO3D physics parameters. This enables the training of the material-to-physics mapper.

B. Feed-Forward Framework

Unlike prior optimization-based methods (e.g., Dress 1-to-3) that require hours of iterative fitting per garment, Image2Garment is a single-pass, optimization-free pipeline. It jointly predicts geometry and physics, enabling real-time or near-real-time generation.

C. Latent Decomposition Strategy

The authors reformulate the inverse problem by introducing a latent semantic space ( $M, Z$ ). This regularizes the learning task, transforming the ambiguous "Image $\to$ Physics" mapping into a well-posed "Image $\to$ Semantics" task followed by a supervised "Semantics $\to$ Physics" mapping.

4. Experimental Results

Quantitative Performance

Geometry & Dynamics: Evaluated on a synthetic benchmark (4D-Dress) and the authors' own dataset, Image2Garment outperforms state-of-the-art baselines (ChatGarment, AIpparel, GarmentRecovery) in Chamfer Distance (CD) and Intersection-over-Union (IoU).
Physics Accuracy: The method achieves significantly lower Normalized Mean Absolute Error (NMAE) in predicting physical parameters compared to baselines that use random or fixed material parameters.
Attribute Prediction: The fine-tuned VLM significantly outperforms zero-shot and few-shot baselines (like GPT-5) in predicting material composition, fabric family, and structure type (e.g., +30% accuracy gain in fabric family prediction).

Qualitative Performance

Simulation Fidelity: Animations generated with Image2Garment's predicted parameters exhibit realistic draping, wrinkle formation, and dynamic response to motion (e.g., jumping, spinning).
Comparison: Baselines using random materials often result in garments that are too stiff (no movement) or too loose (excessive collapse), whereas Image2Garment matches the ground-truth dynamics closely.

5. Significance and Impact

Accessibility: The method democratizes high-fidelity garment simulation by removing the need for multi-view capture setups, manual material measurement, or expensive optimization loops.
Industry Applicability: By outputting parameters directly compatible with industry-standard tools (Marvelous Designer, CLO3D, Browzwear), the generated assets are immediately usable in virtual reality, gaming, and digital fashion workflows.
Scalability: The feed-forward nature allows for rapid processing of "in-the-wild" images, making it feasible to generate simulation-ready assets from social media photos or e-commerce catalogs at scale.

6. Limitations and Future Work

Single-Layer Constraint: The current model assumes single-layer garments; complex multi-layered outfits (e.g., jackets over dresses) are filtered out during dataset curation and remain a challenge.
Visual Ambiguity: Materials with nearly identical visual appearances but different physical properties (e.g., certain cotton vs. viscose blends) can still lead to minor prediction errors.
Accessories: Heavy accessories (buttons, zippers) are not explicitly modeled, which can cause local discrepancies in simulation.

In summary, Image2Garment represents a significant leap forward in digital fashion by successfully decoupling geometry reconstruction from material estimation, leveraging large language models for semantic understanding, and providing a practical, data-driven path to simulation-ready 3D assets from a single image.