CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customization

Imagine you have a digital 3D model of a room—like a video game level or a virtual reality house. Right now, this room is like a ghost house: it has the walls, the sofa, and the cabinets, but they are all painted in a boring, flat gray. They have no personality, no wood grain, no fabric weave, and no style.

Your goal is to "decorate" this ghost house to look real. This is what the paper calls texturing.

The Problem: The "Bad Painter" and the "Vague Client"

Previously, trying to decorate these 3D rooms was like hiring a painter who only speaks in vague riddles or tries to copy a messy collage.

The Text Problem: If you tell a computer, "Make the sofa look like a vintage leather chair," the computer gets confused. It might make the whole room look like leather, or the sofa might look like a weird, blurry blob. It lacks precision.
The Reference Problem: If you show the computer a photo of a nice sofa and say, "Copy this," older methods would try to paste that photo over the whole room. The result? The sofa looks okay, but the chair next to it suddenly looks like a sofa too, or the whole room gets weird shadows baked into the paint (like the sun is permanently stuck in one spot).

The Solution: CustomTex (The "Master Decorator")

The authors created CustomTex, a new tool that acts like a master interior designer with a very specific set of skills.

Here is how it works, using a simple analogy:

1. The "Reference Board" (Instance-Level Control)

Instead of giving the designer one big, confusing instruction, you give them a reference board with specific photos for specific items.

You point to the sofa in the 3D room and say, "Here is a photo of the exact leather sofa I want."
You point to the cabinet and say, "Here is a photo of the dark wood I want."
You point to the wall and say, "Here is the wallpaper pattern."

CustomTex understands that the sofa photo only applies to the sofa, and the wood photo only applies to the cabinet. It doesn't mix them up.

2. The "Two-Step Painting Process" (Dual-Distillation)

This is the secret sauce. The paper says CustomTex uses a "dual-distillation" approach. Let's break that down into two painters working together:

Painter A (The Architect): This painter looks at your reference photos and says, "Okay, I understand the idea." They ensure the sofa looks like a sofa, the wood looks like wood, and the patterns match. They handle the meaning and the shape.
- Analogy: This is like sketching the outline of the furniture to make sure it fits the room and matches the style you asked for.
Painter B (The Detail Artist): This painter is obsessed with crispness. They take the sketch from Painter A and zoom in to 400%. They add the tiny scratches on the wood, the weave of the fabric, and the subtle shine. They make sure the image isn't blurry or "mushy."
- Analogy: This is like taking a low-resolution photo and using a magic lens to make it look like a high-definition 4K photograph.

3. The "No-Baked-Shadows" Rule

Old methods often painted the shadows into the texture. If you moved the virtual sun, the shadows on the wall would look wrong because they were "baked in" (frozen) like a tattoo.

CustomTex is smart enough to paint only the material (the color and pattern), leaving the shadows for the lighting engine to handle later. This means you can change the lighting in your virtual room, and the textures will still look perfect.

Why is this a big deal?

Think of it like the difference between stamping a sticker on a wall versus hand-painting a mural.

Old methods were like stamping a sticker: It looked okay from far away, but up close, it was blurry, the edges were messy, and it didn't fit the specific shape of the object.
CustomTex is like hand-painting a mural: It fits the sofa perfectly, the wood grain is sharp, the fabric feels real, and you can change the lighting without ruining the look.

In Summary

CustomTex is a new way to decorate 3D rooms. Instead of guessing what you want with words, you show it pictures of exactly what you want for each piece of furniture. It uses a smart two-step process to ensure the room looks semantically correct (the right objects have the right look) and visually stunning (crisp, high-definition details without weird shadows).

It turns a gray, boring 3D model into a photorealistic, customizable home in a matter of minutes, making it much easier for designers, game developers, and architects to create beautiful virtual worlds.

1. Problem Statement

The creation of high-fidelity, customizable 3D indoor scene textures remains a significant challenge in computer vision and graphics. Existing approaches face three primary limitations:

Lack of Instance-Level Control: Text-driven methods (e.g., using prompts like "wooden chair") are often ambiguous and struggle to convey precise visual characteristics (e.g., specific wood grain or fabric weave) for individual objects within a complex scene.
Insufficient Visual Fidelity: Current diffusion-based methods often produce textures that are soft, blurry, or unnaturally uniform, lacking the sharpness and richness of artist-created or physically scanned materials.
"Baked-in" Shading: Diffusion models tend to replicate lighting and depth cues from their training data, resulting in textures with baked-in highlights and shadows. This makes the textures unsuitable for re-lighting or rendering in different environments.
Global vs. Local Control: Reference-image-based methods often provide only global, coarse-level control, failing to allow users to specify fine-grained attributes for specific object instances (e.g., sofa vs. cabinet) simultaneously.

2. Methodology: CustomTex Framework

CustomTex is a novel framework designed for instance-level, high-fidelity texturing driven by multiple reference images. It takes an untextured 3D scene mesh and a set of reference images (one for each object instance) and generates a unified, high-resolution texture map.

The core innovation is a Dual-Distillation Approach unified within a Variational Score Distillation (VSD) optimization framework. This separates semantic control from pixel-level enhancement.

A. Semantic-Level Distillation (Instance-Guided VSD)

Goal: Ensure semantic plausibility and precise alignment between the generated texture and the specific reference image for each object instance.
Mechanism:
- Uses a pre-trained depth-to-image diffusion model (e.g., Stable Diffusion) as a prior.
- Employs an Instance Cross-Attention mechanism. Reference image features are extracted via an IP-Adapter encoder.
- Masking: Instance masks ( $m_i$ ) derived from the 3D scene are used to align specific reference features ( $f_i^{ref}$ ) with the corresponding object parts in the rendered RGB image. This prevents "bleeding" of styles between objects (e.g., ensuring the sofa gets the sofa texture, not the wall texture).
- Optimization: Uses an alternating two-step strategy to optimize both the texture field parameters ( $\theta$ ) and a lightweight LoRA adapter ( $\phi_{LoRA}$ ) that learns the specific style prior from the reference images.

B. Pixel-Level Distillation (Super-Resolution)

Goal: Enhance visual fidelity, sharpness, and high-frequency details while suppressing artifacts.
Mechanism:
- Leverages a pre-trained Super-Resolution (SR) diffusion model.
- Unlike standard post-processing, this distillation occurs during the optimization loop. The SR model provides a gradient that guides the texture field toward high-quality, sharp details.
- This process is distinct from semantic generation, ensuring that the enhancement does not disrupt the semantic alignment achieved in the first stage.

C. Unified Optimization

The final texture is optimized by combining the gradients from both distillation processes:
$\nabla_\theta \mathcal{L} = \nabla_\theta \mathcal{L}_{VSD} + \lambda_{SR} \nabla_\theta \mathcal{L}_{SR}$
Where $\mathcal{L}_{VSD}$ handles semantic consistency and $\mathcal{L}_{SR}$ handles pixel-level quality. The texture is represented as an implicit multi-resolution hash grid (from Instant-NGP), allowing for arbitrary resolution output.

3. Key Contributions

Instance-Level Customization: A framework that allows users to specify the appearance of each object instance in a 3D scene using separate reference images, achieving fine-grained control impossible with text-only prompts.
Dual-Distillation Strategy: A novel training approach that decouples semantic generation (ensuring "what" the object is) from pixel enhancement (ensuring "how good" it looks), effectively solving the trade-off between semantic alignment and visual fidelity.
Reduction of Artifacts and Baked-in Shading: By using instance masks to decompose global image generation into local object generation, the method prevents the diffusion model from applying global lighting/shading cues, resulting in clean albedo maps suitable for re-lighting.
High-Fidelity Output: The method produces 4K+ resolution textures with superior sharpness and reduced blurriness compared to state-of-the-art (SOTA) methods.

4. Experimental Results

The authors evaluated CustomTex against SOTA baselines (e.g., SceneTex, Paint3D, HY3D-2.1, TEXture) on the 3D-FRONT dataset.

Quantitative Performance:
- Image-to-Texture: CustomTex achieved the highest scores in CLIP-I (similarity to reference), CLIP-FID, and user-centric metrics (Q-Align IQA and IAA).
- Text-to-Texture: Even when using GPT-4v to convert text prompts into reference images, CustomTex outperformed text-driven baselines in semantic fidelity (CLIP-T) and image quality.
Qualitative Results:
- Consistency: CustomTex demonstrated precise alignment with reference images for individual objects (sofas, cabinets, walls), whereas baselines often produced repetitive patterns or mixed styles.
- Quality: Generated textures showed significantly higher sharpness and fewer artifacts (noise, blurriness) compared to baselines.
- Shading: The method successfully eliminated "baked-in" shadows and highlights, producing clean material maps.
User Study: In a study with 60 participants, CustomTex received the highest ratings for both visual quality and prompt consistency (4.008/5 and 4.125/5, respectively).
Ablation Studies:
- Removing pixel-level distillation ( $\mathcal{L}_{SR}$ ) resulted in blurry textures.
- Applying super-resolution as a post-processing step (rather than distillation) failed to improve UV texture quality, confirming the necessity of in-loop distillation.
- Removing instance masks caused lighting instability and loss of instance separation.

5. Significance and Impact

Direct Path to High-Quality Editing: CustomTex establishes a more direct and user-friendly workflow for 3D scene editing, moving away from ambiguous text prompts to precise image-based control.
Practical Applicability: The ability to generate clean, high-resolution textures without baked-in lighting makes the output immediately usable in professional pipelines (e.g., game engines, architectural visualization) where re-lighting is required.
Scalability: The method supports arbitrary texture resolutions (up to 12K) with efficient inference times (seconds for 4K), making it viable for large-scale indoor scene generation.
Future Directions: While currently focused on diffuse albedo, the framework lays the groundwork for generating other material maps (normal, roughness) and addressing the computational intensity of training.

In summary, CustomTex represents a significant leap forward in 3D scene texturing by effectively combining instance-level semantic control with high-fidelity pixel enhancement, solving long-standing issues of quality, consistency, and lighting artifacts in generative 3D workflows.