Controllable Clothing: Precise Labels and Generation for Virtual Try-On with Latent Diffusion Models

Imagine you are trying to teach a robot how to dress up. You show it a photo of a person and a photo of a cool new jacket, and you ask, "Can you put the jacket on the person?" This is the world of Virtual Try-On (VITON), a branch of computer vision where machines try to solve the puzzle of how clothes look on different bodies. To do this, the robot uses a special kind of "brain" called a diffusion model. Think of a diffusion model like a magical artist who starts with a canvas covered in static (like TV snow) and slowly, step-by-step, clears away the noise to reveal a picture. If you give this artist a text description like "a red shirt," it can paint one. But if you want it to paint exactly that specific shirt on that specific person, with the sleeves ending at the elbow and the collar sitting just right, the artist often gets confused. It might paint the shirt too long, too short, or in the wrong color. This is a big problem for online stores, because if a customer buys a shirt based on a fake picture that doesn't match reality, they will be unhappy.

This report by Max Rehman Linder tackles that confusion. The author asks: "What if we could give the robot a ruler and a set of instructions, not just a vague description?" The paper proposes a new method called ControlableClothing. Instead of just showing the robot the pictures, the author's team measures the clothes and the person's body with mathematical precision—calculating exactly how much of the shoulder is covered, how deep the neckline is, and how long the sleeves are. They turn these measurements into a secret code (labels) and feed them to the robot alongside the images. The result is a system that doesn't just guess where the clothes go; it follows a map. The paper suggests that by using these precise measurements and a special "adapter" (a tool that helps the robot understand the code), the generated images become much more controllable. However, the author is careful to note that while the robot can now follow these rules, it still needs more practice to get every tiny detail perfect, and some of the new tools they built are still being tested to see if they work as well as hoped.

The Magic of the "Noise-Clearing" Artist

To understand how this works, let's look at the robot's brain. The paper uses a type of AI called a latent diffusion model. Imagine a bucket of muddy water. If you keep stirring it, it gets messier and messier until it's just brown sludge. A diffusion model learns to do the opposite: it learns how to take that brown sludge and slowly, carefully, separate the dirt from the water until you have a clear picture again. In the computer world, the "sludge" is random noise, and the "clear picture" is a photo of a person wearing clothes.

For a long time, these robots were great at making new pictures from scratch but terrible at following strict rules. If you asked for a shirt with sleeves ending exactly at the elbow, the robot might just guess. It's like asking a child to draw a house with a door exactly 3 inches wide; they might draw a door, but it's rarely the exact size you wanted. Previous attempts to fix this involved using GANs (another type of AI), but they were often stiff and couldn't create realistic folds or lighting. Then came Diffusion Models, which are much better at making things look real, but they still struggled with the "exact placement" problem.

The New Solution: Giving the Robot a Ruler

The author's main idea is simple: stop guessing and start measuring.

In the past, if you wanted to try on a shirt, you'd just show the robot the shirt and the person. The robot would try to "warp" the shirt to fit. But the author realized that to get it right, the robot needs to know how the shirt fits. Is it tight? Is it loose? Does the sleeve stop at the wrist or the elbow?

So, the team built a pre-processing pipeline. Before the robot even sees the picture, they use other smart tools to:

1. Cut out the clothes: They use a "mask" to hide the part of the person where the new shirt will go.
2. Measure everything: They use computer vision to measure the distance from the person's hips to the bottom of the shirt, the width of the neck, and how much of the shoulder is covered.
3. Create a "Scorecard": They turn these measurements into numbers and labels. For example, instead of just saying "long sleeves," they might say "sleeves end at 0.8 of the arm length."

The "Adapter" Tool: The Translator

The robot's brain (the diffusion model) is already very good at painting, but it doesn't speak the language of "measurements." To fix this, the author built a special tool called an adapter. Think of this adapter as a translator.

The translator takes the "Scorecard" (the measurements and labels) and the picture of the shirt, and it converts them into a language the robot understands. It then whispers these instructions into the robot's ear while it is painting.

• The IP-Adapter: This part helps the robot understand the look of the shirt (the pattern, the color).
• The T2I-Adapter: This part helps the robot understand the details and where to put them.

The author created a new, custom version of this translator specifically for the "up-sampling" part of the painting process. This is the stage where the robot adds the fine details, like the texture of the fabric or the way light hits a fold. By injecting the measurement data right at this stage, the robot can adjust the details based on the ruler's instructions.

What the Experiments Showed

The author trained this new system on thousands of images of people and clothes. Here is what they found:

• It works, but it's a work in progress: When they turned the "dials" (the measurement labels), the robot actually changed the clothes. For example, if they told the robot to make the neckline deeper (by changing a number from 1 to -2), the robot drew a deeper neckline. If they told it to cover only one shoulder, it did that too.
• Numbers are better than words: The author found that giving the robot a specific number (like a float value) worked better than giving it a category (like "long sleeve"). It's like telling a chef "add 5 grams of salt" instead of "add a little salt." The robot could follow the number more precisely.
• The "Mask" trick: The team also taught the robot to ignore parts of the image that were hidden behind a mask. They found that if they told the robot "the mask ends here, but you can paint above it," the robot would generate new fabric above the mask line. This is huge because it means you could take a photo of someone in a long shirt and tell the robot to turn it into a crop top, and the robot would know to paint the missing stomach area.
• The Limits: The paper admits that the robot isn't perfect yet. When the clothes had very complex patterns or details, the robot sometimes lost them. The author suspects this is because the new "up-sampling" tool they built hasn't been trained long enough. The robot is still learning how to use its new ruler.

What the Author Says "No" To

The author was very clear about what doesn't work or isn't the focus:

• Just using text prompts isn't enough: They tried asking the robot to "make the sleeves longer" using just words, but the robot often ignored them or got it wrong. The measurements were necessary.
• One-hot encoding isn't ideal: They tried using simple "on/off" switches for things like sleeve length (e.g., "is it long? Yes/No"), but found that continuous numbers (like a slider from 0 to 1) gave much better control.
• It's not a magic fix for everything: The author notes that if the robot is trained on a dataset where it has already memorized the answers, adding new instructions might not help. They had to start fresh to make the new system work.

The Verdict

This paper suggests that by combining precise measurements with smart "adapter" tools, we can give virtual try-on systems much more control. It's not a solved problem—the robot still needs more training to get the tiny details perfect, and the new tools need more testing—but the direction is promising. The author shows that if you give the AI a ruler and a set of instructions, it can stop guessing and start following orders, creating images that are much closer to what a real customer would see. It's a step toward a future where you can try on clothes online, and the picture you see is exactly what you'll get when the package arrives.