FabricGen: Microstructure-Aware Woven Fabric Generation

FabricGen is an end-to-end framework that generates realistic woven fabric materials from text by decoupling macro-scale texture synthesis via fine-tuned diffusion models from micro-scale weaving pattern generation using a specialized LLM-driven procedural geometric model.

The Gist

Imagine you want to design a beautiful piece of fabric for a video game character or a movie costume. In the old days, you'd need to be a master weaver and a digital artist simultaneously. You'd have to understand how threads twist, how they slide against each other, and how light hits every tiny fiber. It's like trying to build a house by hand, brick by brick, while also painting the walls.

FabricGen is a new tool that changes the game. It's like having a super-smart, two-person design team that works together to create realistic fabric from just a simple sentence you type, like "a cozy beige sweater with a log cabin pattern."

Here is how this team works, broken down into simple parts:

1. The Problem: The "Blurry Photo" Issue

Previous AI tools were like a photographer trying to take a picture of a woven blanket. If they zoomed out, the blanket looked okay. But if they zoomed in close to the camera, the AI got confused. The threads would melt into each other, the patterns would look like a blurry mess, or the fabric would look like it was made of plastic instead of cotton. The AI didn't understand the rules of weaving.

2. The Solution: Splitting the Job

The creators of FabricGen realized that making fabric is actually two different jobs. They split the work between two specialists:

Specialist A: The "Macro" Artist (The Diffusion Model)

• What they do: This AI is the painter. It focuses on the big picture: the colors, the overall pattern (like a plaid or a floral print), and the general "vibe" of the fabric.
• The Trick: The team taught this AI to ignore the tiny threads. They gave it a special dataset of "smooth" fabric images so it learns to paint the color pattern without accidentally trying to draw individual threads. This ensures the colors are crisp and the pattern is perfect, without any weird 3D glitches.

Specialist B: The "Micro" Engineer (The WeavingLLM)

• What they do: This is the structural engineer. It doesn't paint; it builds. It uses a specialized Large Language Model (called WeavingLLM) that acts like a master weaver who has read thousands of weaving manuals.
• How it works: When you say "twill pattern," this AI doesn't guess. It writes a precise "recipe" (called a weaving draft) that tells a computer exactly how to interlock the threads.
• The Magic Touch: Real fabric isn't perfect. Threads slide around, and tiny fuzzies (flyaway fibers) stick out. This AI adds those imperfections naturally. It simulates threads slipping out of place and little hairs sticking up, which makes the fabric look alive and realistic when light hits it.

3. Putting It Together: The "Sandwich"

Once both specialists are done, FabricGen combines their work:

1. The Bottom Layer: The "Micro" Engineer builds a 3D map of the threads (the geometry).
2. The Top Layer: The "Macro" Artist paints the colors on top of those threads.

When you render (show) the final result, you get a fabric that looks perfect from far away (because of the painter) and looks incredibly detailed and realistic when you zoom in (because of the engineer).

Why is this a big deal?

Think of it like this:

• Old AI: Like a child drawing a picture of a sweater. From a distance, it looks like a sweater. Up close, it's just a scribble of lines.
• FabricGen: Like a high-end tailor who can instantly weave a sweater based on your description, complete with the right twist in the yarn and the right amount of fuzz, ready to be worn in a movie.

The Result

With FabricGen, you don't need to know what a "warp" or a "weft" is. You just type what you want, and the system handles the complex physics and math. It creates fabrics that are so real, you can almost feel the texture just by looking at the screen. It's a bridge between your imagination and a photorealistic digital world.

Technical Summary

Here is a detailed technical summary of the paper "FabricGen: Microstructure-Aware Woven Fabric Generation."

1. Problem Statement

Woven fabric materials are essential for photorealistic rendering in digital human modeling, interior design, and film. However, creating realistic fabric materials traditionally requires a multi-stage, expert-driven process involving weaving pattern design, texture authoring, and shading tuning.

• Limitations of Current AI Methods: Recent diffusion-based approaches (e.g., DressCode, MatFuse) have lowered the barrier to entry but suffer from critical flaws:
  • They often generate physically implausible patterns or artificial stripes.
  • They fail to capture yarn-level microstructures (micro-details), leading to blurred or unrealistic close-up renderings.
  • They struggle to enforce domain-specific structural rules (e.g., valid weaving drafts).
• The Core Challenge: There is a need for an end-to-end framework that can generate both macro-scale textures (color/albedo) and micro-scale weaving patterns (yarn geometry) from natural language prompts while adhering to the physical rules of weaving.

2. Methodology: FabricGen Framework

FabricGen proposes a two-scale decomposition strategy, separating the generation of macro-scale textures from micro-scale geometric patterns. The framework consists of three primary components:

A. Macro-Scale Texture Generation (Diffusion Model)

• Goal: Generate seamless, tileable albedo (color) maps that are free of microstructures (no yarn details, shadows, or wrinkles).
• Approach:
  • The authors collected a dataset of 600 microstructure-free fabric textures.
  • They fine-tuned the pre-trained FLUX.1-dev diffusion model using LyCORIS (a parameter-efficient fine-tuning method).
  • Noise Rolling: A mechanism is incorporated to ensure the output is perfectly tileable.
  • Multimodal Conditioning: The model supports text prompts and optional image inputs (flat or wrinkled) to guide the style while suppressing geometric wrinkles.

B. Micro-Scale Pattern Generation (WeavingLLM)

• Goal: Automatically generate valid weaving drafts (binary matrices representing warp/weft interlacing) and fabric parameters (e.g., ply count, roughness) from text.
• Approach:
  • WeavingLLM: A domain-specific Large Language Model based on Qwen2.5-14B-it.
  • Training: Fine-tuned via QLoRA on a dataset of 1,142 annotated weaving drafts (sourced from Handweaving.net).
  • Prompt Tuning: The model is further tuned with domain-specific fabric expertise to learn the priors of weaving rules.
  • Output: Given a text prompt, WeavingLLM outputs a formatted binary weaving draft and a set of geometric parameters.

C. Procedural Geometric Model

• Goal: Synthesize natural, yarn-level 3D geometry (normals, tangents, height) based on the draft and parameters.
• Key Innovations:
  • Curved Helix Model: Unlike previous models that treat yarns as simple cylinders, this model represents plies as curved helices. This allows for multi-ply configurations and analytical calculation of normals and orientations directly from UV coordinates.
  • Global Irregularities: The model introduces two critical physical effects often missed by procedural methods:
    1. Yarn Sliding: Simulates the random positional shifting of yarns using Perlin noise, creating natural gaps and revealing lower layers.
    2. Flyaway Fibers: Adds a stochastic fiber layer escaping the surface to create realistic specular highlights and fuzziness.
  • On-Demand Querying: The model does not pre-render high-res maps; it calculates geometric data on-the-fly, enabling ultra-fine detail without storage overhead.

D. Rendering

• The generated macro-albedo and micro-geometry (normal, tangent, height) are combined using SpongeCake, a layered shading model, to produce the final photorealistic render.

3. Key Contributions

1. FabricGen Framework: A novel end-to-end system that decouples macro-texture and micro-structure generation, enabling the creation of highly detailed woven fabrics from text.
2. WeavingLLM: A specialized LLM that learns to design valid weaving drafts and fabric parameters from natural language, bridging the gap between text and procedural geometry.
3. Enhanced Procedural Model: A new geometric model featuring curved helix plies, yarn sliding, and flyaway fibers, which significantly improves the realism of microstructures compared to existing surface-based methods.
4. Microstructure-Free Diffusion: A fine-tuned diffusion model specifically trained to generate clean albedo maps, avoiding the "baked-in" artifacts common in general material generators.

4. Results and Evaluation

The authors evaluated FabricGen against state-of-the-art baselines (DressCode and MatFuse) using text and image conditioning.

• Visual Quality:
  • Baselines produce plausible results at a distance but suffer from blurring and distortion in close-ups.
  • FabricGen preserves fine yarn details and structural integrity in both macro and micro views.
• Quantitative Metrics:
  • CLIP Score: FabricGen achieved 0.317 (vs. 0.307 for MatFuse), indicating better semantic alignment with prompts.
  • CLIP-I Score (Image Conditioning): FabricGen scored 0.827 (vs. 0.722 for MatFuse), showing superior consistency with input images.
  • User Study: FabricGen was preferred by 82.55% of participants over baselines (MatFuse: 16.02%, DressCode: 1.43%).
• Ablation Studies:
  • WeavingLLM: Fine-tuning and prompt-tuning significantly improved pattern accuracy (Cosine Similarity of Fourier spectra) and user preference compared to base LLMs (Qwen2.5, GPT-5).
  • Irregular Effects: Removing yarn sliding or flyaway fibers resulted in higher LPIPS (perceptual difference) from real photos, proving their necessity for realism.
  • Diffusion Fine-tuning: The fine-tuned model successfully eliminated unintended 3D artifacts and implicit stripes found in the base model.

5. Significance

FabricGen represents a significant leap forward in procedural material generation. By explicitly separating the color pattern (macro) from the weaving structure (micro) and using an LLM to enforce physical weaving rules, it solves the "unrealistic microstructure" problem that plagues current diffusion models.

• Accessibility: It allows non-experts to generate complex, physically accurate fabrics without knowledge of weaving drafts or 3D modeling.
• Application: The framework is highly applicable to digital human clothing, virtual try-ons, interior design, and cinematic production where close-up realism is critical.
• Future Work: The authors note that extending this approach to knitted fabrics, embroidery, and jacquard patterns is a promising direction for future research.