💻 computer science

Dual-Diffusion Grid Texture Generation Based on Multimodal Geometric Perception

This paper proposes a dual-diffusion mesh texture generation method that integrates multimodal geometric perception to overcome UV mapping distortions and resolution limits, significantly enhancing the visual realism, geometric consistency, and diversity of generated 3D textures compared to existing approaches.

Original authors: Shijie Zhao, HongJuan Gao

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shijie Zhao, HongJuan Gao

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Creating a realistic 3D object for a video game or a movie often feels like painting a flat sheet of paper and then trying to wrap it perfectly around a complex, irregular shape without tearing or stretching the image. This is the core challenge of texturing 3D models. For decades, artists have manually painted these surfaces, a slow and skill-intensive process. More recently, computers have learned to generate these textures using artificial intelligence, but the results often suffer from visible seams, distorted patterns, or a lack of detail where the computer struggles to understand the object's true shape. The goal of modern research is to teach machines to create textures that are not only visually stunning but also geometrically perfect, meaning the pattern fits the curve of the object exactly as it would in the real world.

In a study published in August 2026, researchers Shijie Zhao and HongJuan Gao from institutions in Ningxia, China, introduced a new method called FM-TeD to solve these persistent problems. Their approach moves away from the traditional way of simply projecting a 2D image onto a 3D surface, which often leads to the distortion mentioned above. Instead, they built a system that acts like a highly observant artist who looks at a 3D object, reads a written description of what it should look like, and studies a reference photo all at the same time. By combining these three different types of information—the shape of the object, a text description, and a visual example—the system learns to generate a texture map that respects the physical geometry of the object while filling in details that are missing from the single reference photo.

The researchers found that previous methods often struggled when an object had deep curves or complex curves, leading to "seams" where different parts of the image didn't match up, or "Janus" problems where the same face appeared twice on an object. Their new system addresses this by using a dual-diffusion process. Imagine a process where the computer starts with a blank, noisy canvas and slowly cleans it up, step by step, using the 3D shape as a guide to ensure the texture follows the curves correctly. At the same time, it uses the text and the reference image to decide what colors and patterns to add. This allows the computer to "hallucinate" or invent the parts of the texture that are hidden from the reference photo, ensuring the final result looks complete and consistent from every angle.

When the team tested their system against existing methods using a large collection of 3D chair models and other objects, the results were clear. The new method produced textures that were significantly more realistic and diverse. In technical measurements used to judge image quality, their system improved the visual fidelity by a substantial margin compared to the best previous tools. Specifically, the generated images were much closer to real-world photos, with fewer errors in how the patterns aligned with the 3D shape. The system also proved capable of creating a wide variety of unique textures for the same object, showing that it could generate many different designs rather than just repeating the same pattern.

Despite these successes, the researchers noted that their system is not without its costs. Generating these high-quality textures requires a significant amount of computing power and time. The model needs to run through thousands of steps to refine the details, which means it takes longer to train and run than simpler methods. The authors suggest that future work will focus on making the system more efficient, perhaps by finding smarter ways to organize the data it processes, so that it can achieve the same high quality in less time. For now, however, this work represents a meaningful step forward in teaching computers to understand the relationship between a 3D shape and the surface that covers it, paving the way for more realistic and automatically generated 3D worlds.

Technical Summary: Dual-Diffusion Grid Texture Generation Based on Multimodal Geometric Perception

1. Problem Statement

Existing 3D texture generation methods, which typically map 2D image textures onto 3D mesh surfaces, face significant limitations in practical applications. These include:

UV Mapping Distortion: Issues arising from projecting rendered images onto surfaces with high depth variation or curvature.
Geometric Inconsistency: Difficulty in maintaining alignment between the generated texture and the underlying 3D geometric structure.
Seams and Artifacts: Visible stitching artifacts occur when merging images from diverse perspectives, particularly when neighboring pixels originate from different viewpoints.
Resolution and Detail Limitations: Current approaches often struggle to produce high-resolution textures with fine-grained details or fail to capture intricate topological features accurately.
Inefficiency: Many methods rely on time-consuming per-instance optimization or rudimentary heuristic approaches (e.g., simple RGB averaging) that hinder computational efficiency and global consistency.

2. Methodology: FM-TeD

The paper proposes FM-TeD (Fusion of Multimodal Textured Diffusion), a novel framework designed for high-resolution, high-fidelity 3D surface UV texture generation. The method integrates 3D geometric information, 2D images, and textual descriptions to produce texture maps that are geometrically consistent and visually realistic.

Core Architecture

The framework operates through the following stages:

Multimodal Data Representation:
- 3D Geometry: A 3D mesh $M = \{V, F\}$ is defined by vertices and triangular patches.
- Conditional Inputs: A single image ( $I$ ) and a text prompt ( $T$ ) serve as multimodal conditions.
- Feature Extraction: The BLIP-2 model is utilized for visual-to-language representation learning. Image features are extracted using a trainable query matrix (Q-Former), while text features are processed by a language model. These are aligned via a cross-modal attention mechanism.
- Fusion: A Multi-Layer Perceptron (MLP) and conditional embedding module fuse the 3D geometric features with the 2D visual-linguistic features.
Attention-Enhanced Dual-Diffusion Blocks:
- Dual-Diffusion Block: Inspired by DiffusionNet, this block combines a thermal diffusion module, a time-step-aware MLP, and a farthest-point sampling attention layer.
- Thermal Diffusion: This acts as a geometric feature extractor on the manifold. It uses a mixed-mesh Laplace operator ( $L_{mix}^R$ ) and its eigenvalue decomposition to compute a heat diffusion operator. By controlling the diffusion distance $h$ (learned from the time step), the model captures both local geometric details (short-range) and global geometric features (long-range) while maintaining geometric consistency.
- Feature Fusion Module: This hybrid 2D-3D block processes inputs through:
  - A UV Head (2D convolution) to extract local features in UV space efficiently.
  - A 3D Point Cloud Module that maps UV features back to the 3D mesh via rasterization.
  - A Time-Step-Aware Encoder (MLP).
- The final fused features ( $f_{out} = f_{uv}^{out} + f_{point}^{in}$ ) guide the denoising process, ensuring the 2D texture patterns align with 3D geometry to avoid stretching and seams.
Texture Generation Pipeline:
- The process begins by initializing a texture map with pure Gaussian noise in UV space.
- Using conditional inputs (image and text), the model iteratively denoises the map.
- During training, pixel information from the single-view image is projected into UV space to form a partially initialized map, guiding the model to fill in invisible regions.
- The final 2D texture map is mapped back onto the 3D mesh surface.

3. Key Contributions

The paper outlines three primary contributions:

Attention-Enhanced Dual-Diffusion Block: A specialized 3D geometric feature extractor designed to capture fine-grained local geometric dependencies on mesh surfaces, streamlining the texture generation process.
Multimodal Feature Fusion: A denoising block that optimizes texture features in UV space by integrating 2D image processing capabilities with 3D geometric information from point clouds, facilitating collaborative denoising of multi-modal data.
End-to-End High-Resolution Framework: A system enabling direct mapping from point cloud data to high-resolution texture maps, eliminating complex intermediate processing steps found in traditional methods and improving workflow efficiency.

4. Experimental Results

The authors evaluated FM-TeD on the ShapeNet Chair dataset and the Amazon Berkeley Objects (ABO) dataset, comparing it against four baselines: PointUVDiff (UV-based), PointUVDiff (PCL-texture-based), DiffusionNet with DDPM, and UV3-TeD.

Quantitative Performance (ShapeNet Chair)

FM-TeD demonstrated significant improvements over existing methods:

FID (Fréchet Inception Distance): Achieved 46.54 (lower is better), outperforming UV3-TeD (54.20) and DiffusionNet DDPM (116.58).
KID (Kernel Inception Distance): Achieved 30.14 (lower is better), significantly better than UV3-TeD (42.17) and DiffusionNet DDPM (468.09).
LPIPS (Learned Perceptual Image Patch Similarity): Achieved 0.23 (higher is better for diversity), comparable to UV3-TeD (0.21) and superior to PointUVDiff variants (0.08–0.09).

Qualitative Observations

Visualizations indicate that FM-TeD generates textures with greater diversity and richer semantic information. The method effectively captures local semantic details, producing refined and realistic textures without requiring additional segmentation information.

Ablation Studies

Experiments on the ABO dataset confirmed the necessity of the proposed components:

Removing the optimized 2D-3D hybrid denoising module resulted in higher FID (52.16 vs. 46.54) and KID (40.91 vs. 30.14).
Removing the attention-enhanced dual-diffusion block led to a substantial drop in quality (FID 98.40 vs. 76.08), validating the importance of the dual-diffusion design.

5. Significance and Limitations

Significance:
The paper claims that FM-TeD effectively mitigates geometric distortion caused by UV mapping and produces detailed textures highly consistent with the mesh's geometric structure. By integrating multimodal inputs, the method enables the generation of 3D models with more photorealistic visual effects and finer details compared to mainstream methods. The reported improvements include a 14.13% improvement in FID and a 28.53% improvement in KID over existing approaches.

Limitations:
The authors modestly acknowledge a notable limitation in learning efficiency regarding high-frequency texture details. The model requires substantial computational resources and training time, typically needing thousands of iterations to effectively capture and generate fine-grained features. This results in significant training overhead and prolonged training cycles.

Future Work:
The authors suggest that future research should focus on developing more efficient network architectures, such as advanced feature aggregation or downsampling strategies (e.g., optimized pooling), to process high-resolution texture information more effectively and reduce training time while preserving fidelity.