Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation

This paper introduces LOTS, a framework that enhances fashion image generation by combining global sketch guidance with multi-level localized text-sketch conditioning, validated on the newly proposed Sketchy dataset featuring both professional and "in-the-wild" sketches.

The Gist

Imagine you are a fashion designer sitting at a drafting table. You have a rough sketch of an outfit: a jacket, a pair of pants, and a shirt. But a sketch alone is just lines on paper. To make it real, you need to tell a tailor exactly what to do: "The jacket should be velvet and red, the pants should be denim and blue, and the shirt needs to be silk and white."

For a long time, computers trying to turn these sketches into realistic photos were like confused interns. If you gave them a sketch of a whole outfit and said, "Make the jacket red and the pants blue," the computer often got mixed up. It might paint the entire outfit red, or put the blue color on the jacket instead of the pants. This is called "attribute confusion."

This paper introduces a new system called LOTS (LOcalized Text and Sketch) that acts like a super-organized project manager to fix this mess.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-All" Mistake

Previous AI tools treated the whole outfit as one big blob. They looked at the sketch and the text as a single, giant instruction.

• The Analogy: Imagine you are giving a recipe to a chef, but you shout, "Make the soup spicy and the cake sweet!" while pointing at a picture of a table with both items. A confused chef might put hot peppers in the cake or sugar in the soup.
• The Result: The AI would generate an image where the pants are the color of the jacket, or the texture of the shirt ends up on the shoes.

2. The Solution: LOTS (The "Team of Specialists")

LOTS changes the game by treating every piece of clothing as a separate project with its own manager.

• Step 1: The Local Teams (The Specialists)
  Instead of one big instruction, LOTS breaks the outfit down. It creates a tiny "team" for the jacket, a team for the pants, and a team for the shirt.

  • How it works: Each team gets their own sketch (just the outline of that specific item) and their own text description (e.g., "Red velvet").
  • The Magic: These teams work independently so they don't accidentally steal each other's colors. The "Red Velvet" team knows only about the jacket, so it doesn't accidentally paint the pants red.

• Step 2: The Global Architect (The Big Picture)
  While the specialists work on their specific items, LOTS also has a "Global Architect." This part looks at the entire sketch of the whole outfit.

  • Why? If the specialists work alone, they might draw a jacket that is too big for the pants, or a shirt that doesn't fit the body. The Architect ensures that all the pieces fit together perfectly in terms of shape, pose, and style.
  • The Analogy: Think of it like a construction site. You have electricians wiring specific rooms (Local Teams), but you also have a site foreman (Global Architect) making sure the house doesn't collapse and all the rooms connect logically.

• Step 3: The Slow Cook (The Diffusion Process)
  Most AI tools try to mix all these instructions together instantly. LOTS is different; it "slow cooks" the image.

  • The Analogy: Instead of dumping all the ingredients into a pot at once, LOTS adds them gradually, step-by-step, while the image is being formed. This ensures the "Red Velvet" instruction stays stuck to the jacket and doesn't bleed onto the pants.

3. The New Playground: "Sketchy"

To teach this new system, the researchers built a massive new dataset called Sketchy.

• The Professional Sketches: They took thousands of real fashion designs and turned them into clean, professional-looking sketches with detailed descriptions for every single piece of clothing.
• The "Wild" Sketches: They also asked regular people (non-artists) to draw outfits using mice or styluses. These drawings are messy and imperfect.
• The Goal: They wanted to see if LOTS could handle not just perfect professional drawings, but also the messy, "I-drew-this-in-5-minutes" sketches that real designers might use.

4. The Results: Why It Matters

When they tested LOTS against other top AI tools:

• No More Mix-ups: LOTS was the only one that consistently got the colors and patterns on the right clothes. If you asked for a striped shirt and polka-dot pants, LOTS delivered exactly that.
• Sticking to the Sketch: Even when the input sketch was messy (drawn by a non-expert), LOTS still managed to create a realistic outfit that looked like the sketch.
• Human Approval: When real humans looked at the results, they preferred LOTS because it felt like the AI actually understood the assignment, rather than just guessing.

In a Nutshell

LOTS is like a fashion design AI that finally learned to listen. Instead of shouting instructions at the whole picture, it whispers specific instructions to specific parts of the outfit, while keeping an eye on the big picture to make sure everything fits. It turns a chaotic mix of lines and words into a perfectly coordinated, realistic fashion look.

Technical Summary

1. Problem Formulation

The paper addresses the challenge of Multi-Localized Conditional Image Generation in the fashion domain.

• The Gap: Existing sketch-to-image and text-to-image methods typically rely on global conditioning (a single sketch and a single text prompt). When generating complex outfits with multiple garments, this leads to attribute confusion, where properties of one item (e.g., color, pattern) incorrectly "leak" onto another (e.g., a brown blazer pattern appearing on black pants).
• The Goal: To synthesize realistic fashion images from a set of localized sketch-text pairs (e.g., a sketch of a shirt + text "red silk shirt" AND a sketch of pants + text "blue denim pants") while maintaining the global structural coherence of the entire outfit.
• The Challenge: The model must associate specific text descriptions with their corresponding sketch regions without merging them prematurely, while simultaneously preserving the overall silhouette and pose defined by the global sketch.

2. Methodology: LOTS Framework

The authors propose LOTS (LOcalized Text and Sketch with multi-level guidance), a two-stage framework built upon Stable Diffusion.

A. Multi-Level Conditioning Stage

This stage constructs a conditioning representation that separates local details from global structure.

1. Modularized Pair-Centric Representation (Local Level):
   • Each localized sketch-text pair $(S_i, T_i)$ is processed independently.
   • Modality-specific encoders (frozen) extract features for the sketch and text.
   • A Pair-Former module fuses these features using learnable tokens and self-attention. This creates a spatially grounded embedding for each garment pair without allowing cross-talk between different pairs at this stage.
2. Global Conditioning (Global Level):
   • A Global Sketch ($S_g$, the union of all local sketches) is encoded to capture the overall outfit structure and pose.
   • A Cross-Attention mechanism fuses the global sketch representation with the local pair tokens. This ensures that while local details are preserved, the overall composition remains coherent.

B. Diffusion Pair Guidance Stage

Instead of merging all conditioning signals into a single vector before generation (which causes attribute leakage), LOTS injects the multi-level representations into the diffusion process iteratively.

• Progressive Integration: The model uses learnable cross-attention adapters inserted into the pre-trained diffusion backbone.
• Mechanism: At every denoising step, the adapters condition the generation on the multi-level embeddings ($P_{m-l}$). This allows the model to gradually assimilate local and global cues, preventing early fusion errors and maintaining structural integrity throughout the generation process.
• Flexibility: The attention-based design supports a variable number of sketch-text pairs without architectural changes.

3. Key Contributions

1. Novel Framework (LOTS): The first approach explicitly designed for multi-localized sketch-text conditioning. It effectively mitigates attribute confusion by decoupling local pair processing from global structural guidance until the iterative denoising phase.
2. The Sketchy Dataset:
   • A new large-scale dataset (47k outfits, 104k localized pairs) built upon Fashionpedia.
   • Features: Includes hierarchical garment annotations, automated professional-style sketches, and fine-grained textual descriptions (including color, extracted via Vision-Language Models).
   • "In the Wild" Split: A subset containing 141 outfits with non-expert, casual sketches (drawn by laypeople using mice or styluses) to test robustness against noise and variability.
3. New Evaluation Metrics: Introduction of Localized-VQAScore (L-VQAScore), which uses Visual Question Answering to verify if attributes are not only present but correctly assigned to specific garment regions.
4. State-of-the-Art Performance: Demonstrated superior results in balancing global structural adherence with fine-grained semantic control.

4. Experimental Results

The authors evaluated LOTS against state-of-the-art baselines (ControlNet, IP-Adapter, T2I-Adapter, GLIGEN, etc.) on the Sketchy dataset.

• Quantitative Performance:

  • Global Quality: LOTS achieved the best FID (0.74) and GlobalCLIP scores, indicating high realism and semantic alignment.
  • Local Alignment: LOTS outperformed all baselines in LocalCLIP (0.826) and L-VQAScore (0.700), proving it successfully assigns attributes to the correct garments.
  • Structural Adherence: LOTS achieved the highest SSIM (0.691), showing strong adherence to the input sketches.
  • Note: While some baselines (like ControlNet) had high VQAScore, they failed L-VQAScore, confirming they suffered from attribute confusion (e.g., putting a pattern on the wrong garment).

• Generalization (Sketchy in the Wild):

  • LOTS maintained top performance on casual, non-expert sketches, demonstrating robustness to domain shifts and imperfect inputs.

• Human Evaluation:

  • Attribute Localization: LOTS achieved the highest F1 score (0.729) and Precision (0.870) in user studies, significantly reducing attribute leakage compared to competitors.
  • Sketch Adherence: Users preferred LOTS and ControlNet for structural alignment, but LOTS was superior in maintaining semantic correctness (no attribute confusion).

• Ablation Studies:

  • Removing global conditioning (LOTS*) degraded structural alignment.
  • Naive feature concatenation (CONCAT) hurt structural metrics.
  • Deferring integration to the diffusion process (vs. early pooling) was critical for preventing attribute confusion.

5. Significance

This work represents a paradigm shift in fashion image generation, moving from global control to fine-grained, localized control.

• Practical Impact: It enables designers to automate the creation of complex, multi-garment outfits with precise control over the material, color, and style of individual items, closely mimicking the human design workflow.
• Technical Advancement: It solves the "attribute confusion" problem in multi-condition diffusion models through a novel multi-level conditioning strategy and iterative guidance, setting a new standard for controllable image synthesis.
• Resource Availability: The release of the Sketchy dataset and the LOTS codebase provides a crucial benchmark for future research in multimodal fashion synthesis and localized image generation.