Technical Summary: Fashion-3DLR

Problem Statement

While AI-generated content (AIGC) has matured in 2D fashion design, 3D garment generation remains in its nascent stages. The primary challenge lies in the intricate coupling relationships between diverse design elements (e.g., structural sketches and appearance textures) within 3D representations. Current approaches struggle with three fundamental issues:

1. Cross-modal Semantic Disambiguation: Bridging the semantic gap between sketch-based structural representations and appearance-based auxiliary features (textures, colors) while preserving the unique characteristics of both in a unified 3D space.
2. High-Quality Generation: Existing 3D generation architectures often suffer from significant geometric detail loss during feature extraction and decoding, failing to produce models with precise texture details and realistic structures.
3. Intuitive Edit-ability: Traditional 3D modeling requires specialized expertise and manual parameter tuning. There is a lack of user-friendly workflows that allow for interactive editing as intuitive as natural language interaction.

Furthermore, existing systems face limitations in integrating heterogeneous design elements, maintaining fine-grained details under sparse training data, and achieving physically plausible simulations.

Methodology

The authors propose Fashion-3DLR, a unified framework for controllable 3D garment generation that synthesizes high-quality assets from paired fashion elements (e.g., sketches and textures). The pipeline consists of three core components:

1. Garment Feature Fusion Diffusion Transformer (GFF-DiT)

To address cross-modal semantic disambiguation, the framework introduces the GFF-DiT module. This module integrates heterogeneous inputs (sketches and textures) into a unified latent space through a bidirectional modulation process:

• Feature Extraction: Sketch ($b$) and texture ($o$) inputs are down-sampled via convolutional layers.
• Bidirectional Modulation: The texture features generate learnable factors ($\Lambda_i, \Theta_i$) that modulate the normalized sketch features. Conversely, the modulated sketch features generate factors to modulate the texture features. This ensures interactive communication between the structural and appearance modalities.
• Fusion: The modulated features are passed through a zero convolution layer to produce fusion features ($x_i$), which are then processed by a DiT (Diffusion Transformer) block to generate fused conditional features ($M$).

2. Rectified Flow Transformer (RFT) for Latent Generation

The fused conditional features ($M$) are injected into a Rectified Flow Transformer to generate garment latents ($S$).

• Architecture: The transformer utilizes self-attention for noise latents and cross-attention to incorporate the conditional features ($M$) as keys and values.
• Voxelization: 3D garment assets are divided into voxel grids. Features are projected from multi-view rendered images onto these voxels to capture local structural and appearance details.
• Decoding: The resulting garment latents are decoded via convolutional up-sampling blocks into multiple output formats, specifically 3D Gaussian Splatting (3DGS) and meshes.

3. Physics-Aware 3D Gaussian Simulation

To enable realistic cloth dynamics, the framework models 3D Gaussians as deformable particles using the Material Point Method (MPM).

• Parameter Identification: The system identifies fabric-specific physical parameters (bending stiffness $k_b$, stretching stiffness $k_s$, shear stiffness $k_{sh}$, and area density $\rho$) by minimizing the discrepancy between simulated Gaussian dynamics and reference cloth videos.
• Simulation: This allows for the simulation of diverse materials (cotton, silk, wool, nylon) directly within the 3D Gaussian domain, preserving high-frequency visual details without relying on intermediate mesh representations.

Key Contributions

The paper outlines four primary contributions:

1. Unified Framework: Fashion-3DLR synthesizes high-quality 3D garments from paired design elements by integrating sketch-driven structural modeling with texture-guided appearance generation, supporting both 3DGS and mesh outputs.
2. GFF-DiT Module: A novel diffusion-based cross-modal fusion module that jointly encodes sketch geometry and texture semantics via bidirectional modulation, enabling diverse and high-quality synthesis under limited 3D supervision.
3. Physics-Integrated Simulation: The framework supports 3D Gaussian-based physical simulation where Gaussians act as deformable particles, enabling realistic, fabric-specific dynamics while maintaining visual fidelity.
4. Downstream Applications: The system supports mesh decoding for virtual try-on, achieving superior realism and structural fidelity compared to existing sewing pattern-based methods.

Experimental Results

The authors evaluated Fashion-3DLR against state-of-the-art (SOTA) methods (e.g., Hunyuan3D, Tripo, Trellis, DressCode) using both quantitative metrics and qualitative analysis.

• Quantitative Performance: On a paired instance-element test set derived from the SewFactory dataset, Fashion-3DLR achieved superior scores across all metrics:
  • FIDCLIP: 0.2743 (Lower is better; indicates better color/detail alignment).
  • LPIPS: 0.5946 (Lower is better; indicates better contour similarity).
  • CLIP-Iscore: 0.9271 (Higher is better; indicates stronger semantic correspondence).
  • These results significantly outperform baselines like Hunyuan3D (FIDCLIP 0.4648) and Tripo (FIDCLIP 0.4524).
• Qualitative Analysis:
  • Structure: Unlike baselines that often produce "watertight" shells or collapsed geometries, Fashion-3DLR generates non-watertight, wearable topologies with accurate openings (necklines, cuffs).
  • Texture & Style: The model successfully fuses artistic oil-brush inputs with sketches, preserving handcrafted aesthetics and preventing the "white-wash" artifacts seen in other models.
  • Virtual Try-On: In comparisons with ChatGarment and DressCode, Fashion-3DLR produced garments with more natural yarn-like texture distributions and correct structural alignment (e.g., flat hems, correct sleeve lengths).
• User Study: In a study with 24 participants evaluating 200 generated sets, Fashion-3DLR received the highest preference rate (83.82%) for virtual try-on quality, surpassing DressCode (64.41%) and ChatGarment (73.25%).
• Ablation: Removing the GFF-DiT module resulted in a marked decline in performance (FIDCLIP increased to 0.6874), confirming its critical role in feature consistency.

Significance and Claims

The paper claims that Fashion-3DLR establishes the first end-to-end pipeline capable of generating high-fidelity 3D garments with dynamic simulation properties while maintaining precise artistic control. By bridging the gap between 2D design elements and 3D physical reality, the framework significantly reduces the manual refinement typically required in professional design workflows.

The authors position the work as a versatile tool for intelligent fashion design that enables:

• Dynamic Simulation: Realistic cloth dynamics for various materials directly in the 3D Gaussian domain.
• Controllability: Intuitive editing via paired sketches and textures.
• Versatility: Support for both 3DGS (for simulation and rendering) and mesh (for virtual try-on) formats.

The paper concludes with a modest acknowledgment of current limitations, noting that 3D Gaussian garments cannot yet interact with mesh-based human bodies and that mesh garments lack sewing-pattern structures. The authors identify cross-representation simulation and pattern-level modeling as key directions for future work.