💻 computer science

Semantic Segmentation of Textured Non-manifold 3D Meshes using Transformers

This paper introduces a texture-aware transformer model that integrates raw pixel information from mesh faces with geometric descriptors via a hierarchical learning scheme and Two-Stage Transformer Blocks to achieve state-of-the-art semantic segmentation performance on textured non-manifold 3D meshes.

Original authors: Mohammadreza Heidarianbaei, Max Mehltretter, Franz Rottensteiner

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammadreza Heidarianbaei, Max Mehltretter, Franz Rottensteiner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, intricate 3D puzzle made of thousands of tiny triangular tiles. Some tiles are smooth and blue (like water), some are rough and green (like trees), and some are cracked and brown (like old roof tiles). Your job is to look at this puzzle and tell a computer exactly what every single tile is.

This is the challenge of Semantic Segmentation for 3D meshes. But here's the catch: these puzzles are messy. They aren't perfect grids like a chessboard; they are irregular, jagged, and often have "non-manifold" structures (think of a wall where three different rooms meet at a single point, which confuses standard computer programs).

The paper you shared introduces a new, super-smart computer brain (a Transformer) designed specifically to solve this messy puzzle by looking at two things at once: the shape of the tiles and the picture painted on them.

Here is a simple breakdown of how they did it, using some everyday analogies:

1. The Problem: The "Blurry Photo" Mistake

Previous computer programs tried to understand these 3D puzzles by taking a "snapshot" of the texture (the picture on the tile) and turning it into a single, average color.

The Analogy: Imagine trying to identify a specific breed of dog just by looking at a blurry, pixelated photo of its fur color. You might guess "brown," but you can't tell if it's a Golden Retriever or a Chihuahua.
The Issue: By averaging the colors, the computer lost all the fine details (patterns, cracks, specific textures) that are crucial for telling things apart. Also, many old methods required the puzzle to be perfectly structured, which real-world 3D scans rarely are.

2. The Solution: The "Two-Brain" System

The authors built a new AI that acts like a detective with two specialized senses working together.

Sense A: The "Shape Detective" (Geometry Branch)

This part of the AI looks at the physical shape of the triangle. Is it flat? Is it steep? How big is it? It uses hand-crafted rules to understand the 3D structure.

Sense B: The "Texture Detective" (The New Superpower)

This is the big innovation. Instead of averaging the colors, this branch looks at every single pixel inside the triangle's texture map.

The Analogy: Instead of just saying "it's brown," this detective zooms in and sees, "Ah, I see a specific pattern of moss growing in a crack, and the paint is peeling in a specific way."
How it works: They use a mini-Transformer (a type of AI famous for understanding language) to read the pixels of each triangle like a sentence, summarizing the visual details into a smart "token" (a digital summary).

3. The "Team Huddle" Strategy (Two-Stage Transformer)

Now, the AI has a list of thousands of triangles, each with a "Shape Summary" and a "Texture Summary." But how does it know if a group of triangles forms a whole building or just a random patch of grass?

They use a clever Local vs. Global strategy:

Stage 1: The Neighborhood Chat (Local Block)
The AI groups nearby triangles into small "neighborhoods" (clusters). Inside each neighborhood, the triangles talk to each other.
- Analogy: Imagine neighbors in a cul-de-sac chatting. "Hey, I'm a roof tile, and my neighbor is also a roof tile. We probably belong to the same house." This preserves fine details.
Stage 2: The Town Hall Meeting (Cross-Cluster Block)
Here is where they fixed a major flaw in older methods. Old methods would take the "average opinion" of the whole town and force every neighbor to agree with it, which made the results look "blurry" (over-smoothed).
- The Fix: This new method has a special "Town Crier" (a Cluster Token) for each neighborhood. These Town Criers meet in a global meeting to share big-picture context ("The whole area is a historic church").
- The Magic: The Town Criers share their wisdom with the neighborhoods, but they don't force the neighbors to change their minds completely. The local details (like a specific crack in a tile) are preserved, while the big picture (it's a church) is added.

4. The Results: Sharper and Smarter

The team tested this on two very different puzzles:

The City (SUM Dataset): A huge 3D map of Helsinki.
- Result: The AI got it right 94% of the time, beating all previous methods. It could clearly distinguish between cars, boats, and buildings without blurring them together.
The Historic Roof (Cultural Heritage Dataset): A 3D scan of an old Spanish roof to find damage (moss, salt, cracks).
- Result: This was much harder because the damage is tiny and rare. The AI still outperformed everyone else, correctly identifying specific types of rot and cracks that other methods missed.

Why This Matters

Think of this technology as upgrading from a black-and-white sketch to a high-definition, 4K video for 3D analysis.

For Cities: It helps planners understand urban environments better.
For History: It allows conservators to automatically find tiny cracks or biological growth on ancient monuments without needing to climb up there with a magnifying glass.

In a nutshell: The authors built a computer brain that doesn't just guess what a 3D object is based on its average color. Instead, it looks at the fine-grained details of the picture and the shape, and it uses a smart "local chat, global meeting" system to understand the whole picture without losing the small details.

1. Problem Statement

The paper addresses the challenge of semantic segmentation for textured 3D meshes, specifically those that are non-manifold (a common occurrence in real-world photogrammetric data where topological constraints like manifoldness are violated).

Limitations of Existing Methods:
- Geometric Constraints: Many deep learning methods for meshes rely on manifold assumptions (well-defined neighborhoods), which restricts their applicability to real-world data.
- Texture Representation: Existing approaches that handle non-manifold meshes often treat texture simplistically (e.g., using single RGB vectors per vertex, mean color histograms, or low-dimensional statistics). These methods fail to capture high-frequency spatial patterns and fine-grained details available in texture maps.
- Over-smoothing: Recent transformer-based approaches (e.g., NoMeFormer) use cross-attention mechanisms where face tokens are updated by weighted averages of cluster tokens. The authors argue this leads to over-smoothing, blurring fine-grained local features and reducing classification accuracy for small or distinct objects.
- Computational Complexity: Standard self-attention scales quadratically with the number of faces, making it infeasible for large meshes without reducing spatial resolution (patchification), which loses detail.

2. Methodology

The authors propose a Texture-Aware Transformer architecture that jointly processes geometric and raw textural information using a hierarchical learning scheme.

A. Input Representation

The input is a textured 3D triangulated mesh $M = (V, F, T, I)$ , where $F$ are faces, and $I$ is the texture image. Each face is linked to a specific region in $I$ via texture coordinates.

B. Feature Extraction Branch

The network extracts a unified feature vector for each face by combining two independent sub-branches:

Geometry Branch: Uses 16 hand-crafted geometric descriptors (normalized coordinates, normals, area, angles) projected via a linear layer.
Texture Branch (Novelty): Instead of statistical descriptors, this branch processes raw pixel data associated with each face.
- Pixels are extracted from the texture map based on face coordinates.
- A learnable "texture token" is appended to the pixel sequence.
- A Transformer Block (with multi-head self-attention) processes this sequence.
- Only the texture token is retained as the final texture feature vector ( $F_T$ ), effectively summarizing the face's appearance into a learnable embedding.
- The geometric ( $F_G$ ) and texture ( $F_T$ ) vectors are concatenated and passed through an MLP to form the final face feature vector.

C. Two-Stage Transformer Blocks (TSTB)

To handle the irregular structure and computational cost, the authors employ a Local-Global strategy inspired by clustering, but with a modified attention mechanism to prevent over-smoothing.

Clustering: Faces are clustered into $K$ groups using K-means. A cluster token is prepended to the sequence of face tokens within each cluster.
Stage 1: Local Block (Intra-cluster):
- Self-attention is applied only within each cluster.
- Face tokens interact with their neighbors and the local cluster token.
- This preserves fine-grained local details and reduces computational complexity.
Stage 2: Cross-Cluster Block (Inter-cluster):
- Key Innovation: Instead of using cross-attention to update face tokens directly (which causes over-smoothing), this block applies self-attention only among the cluster tokens.
- Cluster tokens exchange global context.
- The updated cluster tokens are then propagated back to the local blocks of subsequent layers.
- Benefit: This allows global context to flow back to faces indirectly via the cluster token, preserving the unique identity of individual face features while still capturing long-range dependencies.

D. Classification

The final output of the TSTB stack is passed to a classification head that discards cluster tokens and predicts class scores for each face using a softmax layer.

3. Key Contributions

Unified Texture-Aware Architecture: A novel transformer-based architecture that directly learns from raw pixel data associated with mesh faces, rather than relying on coarse statistical descriptors. This captures high-frequency textural patterns crucial for distinguishing similar geometries.
Hierarchical Attention Mechanism (TSTB): A modified attention strategy that separates local and global processing. By restricting global interaction to cluster tokens (via self-attention) rather than direct cross-attention on face tokens, the method mitigates the over-smoothing problem common in previous mesh transformers.
Non-Manifold Capability: The method is explicitly designed to handle non-manifold meshes without requiring topological constraints, making it suitable for real-world photogrammetric data.
New Dataset: Introduction of a curated Cultural Heritage (CH) dataset featuring textured roof tiles with triangle-level annotations for damage types (e.g., biological colonization, salt efflorescence), addressing a gap in existing benchmarks.

4. Experimental Results

The method was evaluated on two datasets: SUM (Semantic Urban Meshes) and the new CH (Cultural Heritage) dataset.

Performance Metrics

SUM Dataset:
- mF1: 81.9%
- OA (Overall Accuracy): 94.3%
- Comparison: Outperformed the previous state-of-the-art (NoMeFormer) by +15.3% mF1 and +11.7% OA. It also surpassed point-cloud based methods (KPConv) and other baselines (DiffusionNet, RF-MRF).
CH Dataset:
- mF1: 49.7%
- OA: 72.8%
- Comparison: Outperformed NoMeFormer by +15.6% mF1 and +4.9% OA. The improvement was particularly significant for minority classes, demonstrating the model's ability to handle class imbalance better than previous methods.

Ablation Studies

Texture vs. Geometry: Using only geometry yielded 66.9% mF1; using only texture yielded 70.2%. Combining both achieved 81.9%, proving the complementary nature of the modalities.
Attention Mechanism: Replacing the proposed Self-Attention (S-A) on cluster tokens with the previous Cross-Attention (C-A) approach resulted in a drop of 0.7% mF1 on SUM and 3.4% mF1 on CH. This confirms that the S-A mechanism better preserves fine-grained details, especially for small damage clusters.
Feature Dimension: Increasing the embedding dimension ( $N_F$ ) from 32 to 256 significantly improved performance.

5. Significance and Conclusion

This work represents a significant advancement in 3D mesh semantic segmentation by:

Bridging the Gap: Successfully integrating high-resolution texture information directly into mesh processing, moving beyond the limitations of point clouds or low-dimensional descriptors.
Solving Over-smoothing: Providing a robust architectural solution (cluster-token self-attention) that balances global context integration with the preservation of local geometric and textural details.
Real-World Applicability: Demonstrating superior performance on non-manifold data in diverse domains, from urban planning to Cultural Heritage preservation, where detecting subtle damage on complex surfaces is critical.

The authors conclude that their approach sets a new state-of-the-art for textured mesh segmentation, though future work will address memory constraints and explore explicit positional encoding in the texture branch.