NI-Tex: Non-isometric Image-based Garment Texture Generation

NI-Tex addresses the challenge of generating diverse, production-ready PBR textures for existing 3D garment meshes from non-isometric images by introducing a physically simulated dataset, leveraging Nano Banana for cross-topology editing, and employing an iterative uncertainty-guided baking method to fuse multi-view predictions.

The Gist

Imagine you have a digital mannequin (a 3D model of a shirt, dress, or pants) and a photo of a real, beautiful piece of clothing you want to put on it. Your goal is to "dress" the mannequin so it looks exactly like the photo.

This is the job of NI-Tex. But here's the catch: usually, this is like trying to fit a square peg into a round hole. If the photo shows a person standing straight, but your 3D mannequin is twisting, or if the photo shows a short skirt but your mannequin is wearing long pants, most computer programs get confused. They either stretch the pattern weirdly, make it blurry, or fail to put the design on the back of the shirt.

The authors of this paper built a new system called NI-Tex to solve this "mismatch" problem. Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Square Peg" Issue

Think of existing 3D texture generators like a strict tailor who only works if the customer stands perfectly still and wears the exact same outfit as the photo. If the customer moves or wears a different style, the tailor gets lost.

• The Old Way: If you show a photo of a T-shirt and ask for it on a pair of jeans, the computer tries to force the T-shirt pattern onto the jeans, resulting in a messy, distorted mess.
• The NI-Tex Way: It's like a master tailor who can look at a photo of a T-shirt and say, "Ah, I see the pattern and the fabric style. I can apply that style to these jeans, even though they are shaped differently."

2. The Secret Sauce: "3D Garment Videos" (The Dance Class)

To teach the computer how to handle these mismatches, the researchers didn't just show it static photos. They created a dataset called 3D Garment Videos.

• The Analogy: Imagine taking a single piece of clothing and filming it on a dancer who spins, jumps, and stretches. The computer watches the fabric move and stretch in 3D space.
• The Result: The AI learns that a pattern on a shirt stays the same pattern even when the shirt wrinkles or twists. It learns the "soul" of the fabric, not just the flat picture.

3. The Magic Trick: "Nano Banana" (The Photoshop Wizard)

Sometimes, the photo you have and the 3D model you want to dress are too different (e.g., a photo of a dress vs. a model of a jacket). The computer needs help bridging that gap.

• The Analogy: They used a tool called Nano Banana (an AI image editor) to act like a magical photo-shopper.
• How it works: If you show a photo of a skirt, Nano Banana can digitally "morph" the skirt in the photo to look more like pants, while keeping the original fabric pattern intact.
• The Benefit: This teaches the AI: "Hey, even if the shape changes from a skirt to pants, the floral pattern stays the same." This allows the AI to transfer textures between completely different types of clothing.

4. The Final Polish: "The Uncertainty Detective" (Baking)

Once the AI generates the texture, it has to "bake" it onto the 3D model. Think of baking like wrapping a gift. If you wrap a gift from just one angle, you might miss a corner or leave a gap.

• The Problem: If the AI guesses the texture from the front, the back might be blurry or have holes.
• The Solution: NI-Tex uses an Uncertainty Quantification (UQ) system. Imagine a detective with a magnifying glass walking around the 3D model.
  • The detective asks: "Where is the picture blurry? Where did we miss a spot?"
  • If the detective finds a blurry spot on the back, the system says, "Okay, let's take a new picture from a different angle specifically to fix that spot."
  • It repeats this process, taking new photos only where needed, until the whole model is perfectly wrapped with a crisp, high-quality texture.

Why Does This Matter?

Before NI-Tex, making 3D clothes for video games, movies, or virtual reality was slow and required human artists to manually paint every inch.

• Now: You can take a photo from your phone (even if the person is posing weirdly) and instantly dress a 3D character with professional-quality, realistic fabric.
• The Impact: It makes creating virtual worlds faster, cheaper, and more realistic, allowing for better avatars, games, and fashion design without needing a team of 3D artists for every single item.

In short: NI-Tex is a smart, flexible digital tailor that learns from dancing clothes, uses magic photo-editing to bridge gaps, and has a detective that ensures no part of the outfit is ever left unfinished.

Technical Summary

1. Problem Statement

The core challenge addressed by NI-Tex is non-isometric image-based garment texture generation.

• Context: While 3D garment datasets cover diverse geometries, they lack texture diversity. Generative methods are used to transfer textures from 2D images to 3D meshes.
• Limitation of Existing Methods: Current state-of-the-art approaches (e.g., Hunyuan3D, Meshy) rely heavily on topological and geometric consistency between the input image prompt and the target 3D mesh.
  • They often fail when there are significant discrepancies in pose, topology (e.g., transferring a skirt texture to a pants mesh), or geometry.
  • They require strict alignment or accurate mesh deformation, leading to distorted textures, holes, or artifacts when these conditions are not met.
• Goal: To create a framework that can robustly generate high-quality Physically-Based Rendering (PBR) textures (Albedo, Roughness, Metallic) from an image prompt to a target garment mesh, even when they exhibit non-isometric differences (topological or geometric mismatches).

2. Methodology

NI-Tex employs a three-pronged approach: a novel dataset construction strategy, a dual-branch feedforward architecture, and an iterative baking algorithm.

A. Dataset Construction: 3D Garment Videos & Cross-Topology Augmentation

To train a model robust to non-isometric variations, the authors constructed a specialized dataset:

• 3D Garment Videos: Built upon the BEDLAM dataset, they created physically simulated garment videos where a single garment texture is applied to hundreds of frames of human motion. This provides consistent geometry and material supervision across diverse poses and deformations.
  • Cross-pose Augmentation: Randomly sampling two frames from a video (one as condition, one as supervision) allows the model to learn texture consistency across different human poses.
• Cross-Topology Augmentation (Nano Banana): To handle topological mismatches (e.g., image of a skirt $\to$ mesh of pants), they use Nano Banana, a high-quality image editing tool.
  • They render views from the 3D videos and use Nano Banana to edit the garment topology while preserving texture identity.
  • Constraints: To prevent semantic drift, they enforce category-wise consistency (tops/bottoms don't swap), inner-outer consistency (layers remain distinct), and allow auxiliary human parts to help the model ignore irrelevant background.

B. Network Architecture: Feedforward Dual-Branch

The model is based on a feedforward dual-branch architecture (inspired by Hunyuan3D) designed for speed and alignment:

• Guidance Branch: Extracts hierarchical features from the input image prompt.
• Generation Branch: Takes multi-view normal and position maps of the target mesh as geometric constraints.
• Multi-Channel Aligned Attention (MCAA):
  • Injects reference features from the guidance branch into the Albedo channel.
  • Aligns the Metallic/Roughness (MR) latent representation with the Albedo attention to ensure spatial and geometric consistency between material properties.
• Switchable Multi-channel U-Net: A specialized training mechanism where the MR channel can be deactivated if the input image (edited by Nano Banana) has inconsistent reflection properties, optimizing training efficiency.

C. Iterative Baking via Uncertainty Quantification (UQ)

Generating multi-view textures often results in artifacts (holes, blurriness) when merged onto a mesh. NI-Tex introduces an iterative baking process:

1. Uncertainty Quantification (UQ) Model: A residual network trained to predict per-pixel uncertainty scores on generated texture maps. It learns to identify "bad" areas (holes, inconsistencies) by simulating errors between generated and ground-truth maps.
2. View Selection: The algorithm selects new candidate views that maximize the average per-pixel uncertainty of the current bake.
3. Iterative Refinement: The model re-generates textures for the selected views, and a reweighting strategy merges them. This loop continues until uncertainty drops below a threshold or a max view count is reached, ensuring full mesh coverage and seamless integration.

3. Key Contributions

1. First Feedforward Non-Isometric Framework: NI-Tex is the first to effectively tackle non-isometric texture generation using a feedforward architecture, eliminating the need for slow optimization or strict pose alignment.
2. Novel Data Strategy: Introduction of 3D Garment Videos and a Nano Banana-based augmentation pipeline to create training pairs with diverse topologies and poses, bridging the sim-to-real gap.
3. Uncertainty-Aware Iterative Baking: A novel algorithm that uses a learned UQ model to dynamically select views and reweight predictions, significantly reducing baking artifacts like holes and blurriness.
4. Public Release: The authors release the code and the new dataset to facilitate future research in garment texture generation.

4. Results

• Qualitative Performance: NI-Tex demonstrates superior robustness compared to baselines (Paint3D, Hunyuan3D, Meshy 6 Preview, Hyper3D). It successfully transfers textures from wild images (DeepFashion2) to industrial meshes and generated meshes, even with large pose and topology differences. It produces realistic PBR materials (Albedo, Roughness, Metallic) without the distortion seen in other methods.
• Quantitative Performance: Evaluated using Kernel Inception Distance (KID) and Fréchet Inception Distance (FID) on 10 textured objects with 42 random seeds.
  • NI-Tex achieved the lowest KID (0.0364) and FID (237.59), outperforming all commercial and academic baselines.
• Baking Efficiency: The UQ-based view selection method achieved full mesh coverage, whereas standard coverage-based methods left significant gaps due to self-occlusion.

5. Significance

• Industry Applicability: The method generates production-ready PBR textures suitable for virtual reality, digital avatars, and physical simulation, addressing a critical bottleneck in 3D garment asset creation.
• Flexibility: By decoupling the requirement for topological consistency, NI-Tex allows designers to use any 2D image (even from the wild) to texture any 3D garment mesh, drastically expanding the creative possibilities for digital fashion.
• Efficiency: As a feedforward approach, it avoids the time-consuming optimization steps (like SDS) common in other PBR generation methods, enabling scalable and fast inference.

Limitations: The current model is specialized for garments and lacks generalization to complex rigid deformations of non-garment objects due to a lack of physically simulated data for general objects. Future work aims to improve 3D self-awareness for broader object deformation.