Technical Summary: Stitched Embeddings (StEm)

Problem Statement

While 3D human digitization has advanced significantly through parametric body models (e.g., SMPL), the reconstruction and manipulation of garments remain challenging due to their vast topological variability and non-rigid deformation. Traditional garment modeling relies on 2D sewing patterns—piece-wise CAD formalizations consisting of discrete panels defined by parametric curves (e.g., Bézier curves) and seam constraints—which serve as the blueprint for physical manufacturing and realistic animation.

Currently, bridging the gap between 2D patterns and 3D geometry typically requires physics-based simulations (e.g., spring-mass systems) to "sew" panels together. This reliance on simulators presents three critical limitations:

1. Computational Cost: Simulators are slow, hindering real-time applications.
2. Non-Differentiability: Most simulators are not differentiable, preventing end-to-end learning and optimization (e.g., refining predictions via backpropagation).
3. Lack of Flexibility: Existing methods struggle to support test-time adaptation or editing of the number and type of sewing patterns without re-running complex simulations.

Furthermore, while recent works have explored implicit representations like Unsigned Distance Fields (UDFs) for 3D garments, these lack the semantic structure required for downstream tailoring applications.

Methodology: Stitched Embeddings (StEm)

The authors propose Stitched Embeddings (StEm), a unified latent space that jointly represents 3D garments and their corresponding 2D sewing patterns. The core framework, StEm-Net, is the first end-to-end, fully differentiable, and simulation-free method to connect these representations.

1. Core Representation: BoxMesh

To align 2D patterns with 3D geometry without a simulator, the authors introduce the BoxMesh.

• Concept: A BoxMesh is a 3D representation where 2D sewing patterns are arranged in 3D space around a body model (similar to a "pre-simulation" state).
• Function: It serves as a spatial proxy that aligns 2D panels into a canonical 3D configuration based on their semantic function (e.g., sleeves, front panel).
• Topology: The BoxMesh maintains the exact topology of the sewing patterns but occupies a 3D volume. When a cloth simulation is applied to a BoxMesh, it yields the final draped garment mesh.
• Alignment: By using BoxMeshes as an input modality alongside simulated garment meshes, the network receives 3D spatial priors that are topologically aligned with the final output.

2. Network Architecture (StEm-Net)

The architecture is built upon a modified Hunyuan3D 2.1 VAE framework, adapted to use Unsigned Distance Fields (UDF) to represent non-watertight, open manifolds (essential for necklines and open hems).

• Encoders:

  • Garment Encoder ($E_g$): Processes a 3D garment point cloud ($P$) with normals ($N$).
  • BoxMesh Encoder ($E_{bm}$): Processes the pre-simulated BoxMesh geometry.
  • Alignment Strategy: To ensure the latent space is invariant to the input modality (BoxMesh vs. Simulated Mesh), the model exploits the known 1:1 correspondence between the two. During training, corresponding points are sampled from both geometries to map the same physical locations to the same latent regions.
  • Output: Both encoders map to a common latent set $z \in \mathbb{R}^{256 \times 64}$.

• Decoders:

  • 3D Decoder ($D_{3D}$): Reconstructs the UDF field of the 3D garment.
  • Sewing Pattern Decoder ($D_{SP}$): Regresses 122 design parameters (from the GarmentCode space) that define the sewing patterns. These parameters include continuous values (dimensions), boolean flags (e.g., sleeve presence), and categorical values (garment type). A decoding transformer with learnable per-panel queries maps the latent set to these parameters.

3. Training and Loss Functions

The model is trained end-to-end using a composite loss function:
$$\mathcal{L} = \mathcal{L}_{UDF} + \lambda_1 \mathcal{L}_{SP} + \lambda_2 \mathcal{L}_{KL}$$

• 3D Reconstruction Loss ($\mathcal{L}_{UDF}$): Minimizes Mean Squared Error (MSE) between predicted and ground truth distance fields for both garment and BoxMesh inputs. It uses stratified sampling to capture both high-frequency details and global structure.
• Sewing Pattern Loss ($\mathcal{L}_{SP}$): A multi-task loss handling the heterogeneous nature of the 122 parameters:
  • MSE for continuous dimensions.
  • Binary Cross-Entropy (BCE) for boolean flags.
  • Cross-Entropy (CE) for categorical values.
• Latent Alignment Loss ($\mathcal{L}_{KL}$): Enforces consistency between the two input modalities by regularizing the latent space against a standard normal distribution and minimizing the KL-divergence between the BoxMesh encoding ($z_B$) and the Garment encoding ($z_G$).

4. Inference and Test-Time Adaptation

• 3D to 2D: Given a 3D point cloud, the model encodes it to the latent space and decodes sewing pattern parameters.
• Test-Time Optimization: To refine predictions for unseen geometries, the latent vector $z$ is iteratively updated via backpropagation to minimize the reconstruction error between the predicted UDF and the input surface. Because the latent space is shared, this optimization simultaneously refines the 3D geometry and the discrete sewing pattern parameters.
• 2D to 3D (Editing): Users can modify 2D pattern parameters, and the network (acting as a "neural simulator") instantly predicts the updated 3D UDF field without physical simulation.

Key Contributions

1. StEm-Net: The first end-to-end, fully differentiable, and simulation-free method that unifies 3D garment realizations and 2D sewing patterns in a common latent space.
2. State-of-the-Art Performance: The method achieves new state-of-the-art results in sewing pattern prediction from 3D point clouds, reducing error by up to 15% compared to the closest competitor (NeuralTailor).
3. Test-Time Adaptation & Editing: The framework enables direct garment editing from 2D patterns and refinement of predictions via backpropagation, all without requiring physical simulations.

Experimental Results

The authors evaluated StEm-Net on two datasets: NeuralTailor (simpler topology) and GarmentCodeData (complex topology with up to 75 panels).

• Comparison with NeuralTailor: On the NeuralTailor test set, StEm-Net outperformed NeuralTailor across almost all metrics, including Panel L2 error, edge detection, and stitch precision. Notably, NeuralTailor failed to converge on the more complex GarmentCodeData, whereas StEm-Net succeeded.
• Comparison with Image-Based Methods: When compared to ChatGarment (a state-of-the-art image-based method) on the GarmentCodeData test set, StEm-Net (operating on 3D point clouds) achieved superior accuracy in parameter regression, demonstrating that 3D geometric cues provide more definitive information for pattern inference than 2D visual textures alone.
• Generalization: The model generalized effectively to real-world scans from the 4DDress and CloSe datasets, significantly outperforming NeuralTailor in Chamfer Distance metrics despite being trained on synthetic data.
• Editing: The framework successfully demonstrated seamless 3D editing based on 2D pattern modifications (e.g., changing garment length, removing sleeves, or swapping trousers for skirts) without physical simulation.

Significance and Claims

The paper positions Stitched Embeddings as a significant milestone in bridging the gap between neural 3D vision and the physical garment manufacturing pipeline.

• Simulation-Free Efficiency: By eliminating the need for differentiable physics simulators, the method offers a computationally efficient alternative for garment reconstruction and editing.
• Unified Representation: It solves the fundamental challenge of representing both continuous 3D geometry and discrete 2D manufacturing blueprints within a single, learnable latent space.
• Practical Applicability: The ability to perform test-time adaptation and direct editing makes the tool practically flexible for the Vision and Graphics communities, potentially streamlining digital fabrication pipelines.

The authors acknowledge that while their results are not yet fully aligned with high-fidelity physical simulations, the work establishes a scalable foundation for future research in multi-layered attire, varying materials, and fine-grained garment nuances.