💻 computer science

Learning Human Visual Attention on 3D Surfaces through Geometry-Queried Semantic Priors

This paper introduces SemGeo-AttentionNet, a dual-stream architecture that integrates geometry-conditioned diffusion-based semantic priors with point cloud transformers to model human visual attention on 3D surfaces by explicitly formalizing the interplay between bottom-up geometric processing and top-down semantic recognition.

Original authors: Soham Pahari, Sandeep C. Kumain

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Soham Pahari, Sandeep C. Kumain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a 3D statue in a museum. Why do your eyes stop on the face? Why do they linger on a shiny button on a coat, even if the button is flat and the coat has lots of wrinkles?

For a long time, computer scientists tried to answer this by looking only at the shape of the object. They thought, "If it sticks out, has a sharp edge, or is bumpy, that's what humans will look at." But this didn't work well. Humans often ignore bumpy, weird shapes and instead stare at smooth, flat things that mean something to us (like a face or a logo).

This paper introduces a new computer program called SemGeo-AttentionNet that finally solves this puzzle. Here is how it works, explained simply:

1. The Two Brains Working Together

The authors realized that human attention is a team effort between two different "brains":

The "Bottom-Up" Brain (Geometry): This is your reflex. It screams, "Look at that sharp edge! Look at that weird bump!" It reacts to the physical shape.
The "Top-Down" Brain (Semantics): This is your knowledge. It whispers, "Wait, that's a face. That's a tool. That's important." It reacts to what the object is, even if it's perfectly flat.

Previous computer models only had the "Bottom-Up" brain. They were like a security camera that only notices movement but doesn't know what a person looks like.

2. The New Solution: A Smart Detective

The new model, SemGeo-AttentionNet, acts like a detective who uses both brains at once. It has two main parts:

The Shape Scanner (Geometry Stream): It uses a powerful tool (Point Transformer V3) to map out every bump, curve, and edge of the 3D object, just like a sculptor feeling the clay.
The Knowledge Library (Semantic Stream): This is the magic part. Since the computer doesn't have a human brain, the authors gave it a "crystal ball" made of Diffusion Models (the same AI technology that creates art from text).
- They take the 3D object, take 100 different photos of it from all angles, and ask the AI, "What is this?"
- The AI says, "That's a face," or "That's a cup."
- This gives the model a "semantic prior"—a pre-loaded understanding of what the object is, without needing to be taught with labeled 3D data.

3. The "Query" Mechanism: Who is in Charge?

Here is the clever trick. The model doesn't just mix these two brains together; it makes them talk in a specific order.

Think of the Shape as a Search Query and the Knowledge as a Database.

The Shape says: "I see a flat area here. Let me check the database: Is this a face? Is this a screen?"
The Knowledge replies: "Yes, that flat area is a face. Pay attention to it!"

This ensures that even if a face is flat and boring geometrically, the model knows to look at it because the "Knowledge" part confirms its importance. However, the Shape still has a "veto power." If the Knowledge says "That's a face," but the Shape says "That's actually just a flat wall with no features," the model won't get distracted. It needs both to agree.

4. The "Eye-Tracking" Game (Reinforcement Learning)

So far, the model just predicts where you look. But humans look at things in a sequence (a scanpath). First the eyes go to the nose, then the mouth, then the hat.

The authors taught the model to play a game to simulate this movement:

The Game: The model is an agent walking on the surface of the 3D object.
The Rules:
1. Go to interesting spots: Move toward high-saliency areas (the face, the tool handle).
2. Don't get bored: If you've looked at a spot too many times, you get a penalty (this is called "Inhibition of Return"). You must move on.
3. Explore: Try to visit new areas you haven't seen yet.
The Result: The model learns to "walk" its eyes across the object in a way that looks exactly like how a human would explore it, respecting the fact that you can't jump through the air; you have to follow the surface of the object.

5. The Results

The team tested this on three different datasets (collections of 3D objects with human eye-tracking data).

The Score: Their new model significantly outperformed all previous methods. It was much better at predicting exactly where humans would look.
The Proof: When they looked at the results, the model correctly identified that people stare at the eyes of a gargoyle (even though the face is smooth) and the handle of a worn tool, whereas old models only looked at the bumpy, noisy parts of the statue.

Summary

In short, this paper built a computer vision system that understands 3D objects not just by their shape, but by their meaning. By combining a geometric scanner with a "knowledge base" derived from AI art generators, and then teaching it to move its "eyes" like a human, they created the most accurate model yet for predicting where we look in a 3D world.

Technical Summary: Learning Human Visual Attention on 3D Surfaces Through Geometry-Queried Semantic Priors

Problem Statement
Human visual attention on three-dimensional objects arises from the interplay between bottom-up geometric processing (reacting to edges, protrusions, and surface discontinuities) and top-down semantic recognition (identifying faces, functional parts, or contextual meaning). Existing 3D saliency prediction methods suffer from a critical dichotomy: hand-crafted geometric approaches (e.g., Gaussian curvature, mesh saliency) fail to explain why humans fixate on semantically meaningful but geometrically unremarkable regions (e.g., a character's eyes or text on a flat surface). Conversely, recent deep learning approaches either rely on geometric encoders lacking semantic awareness or utilize 2D semantic models that ignore 3D surface topology. Furthermore, existing scanpath generation models are limited to 2D images and do not respect 3D mesh connectivity or inhibition-of-return dynamics.

Methodology: SemGeo-AttentionNet
The authors propose SemGeo-AttentionNet, a dual-stream architecture designed to explicitly formalize the bottom-up/top-down dichotomy through asymmetric cross-modal fusion.

Semantic Priors via Diffusion Distillation:
- To inject semantic awareness without labeled 3D data, the method leverages pretrained text-to-image diffusion models (Stable Diffusion) and vision transformers (DINOv2).
- The input mesh is rendered from 100 viewpoints. For each view, geometry-conditioned maps (depth and surface normals) guide a conditional diffusion inpainting process.
- Intermediate U-Net decoder activations are aggregated from the final 75% of denoising steps to capture fine-grained semantic details. These are fused with DINOv2 features to create 2048-dimensional per-pixel descriptors.
- A unprojection operator maps these 2D features back to 3D mesh vertices using ball-query aggregation, creating per-vertex semantic descriptors ( $S$ ). This pipeline runs offline, treating diffusion as a frozen perceptual prior.
Geometric Encoding:
- The mesh is uniformly sampled (2,048 points) to extract geometric descriptors (position and normals).
- These are processed by Point Transformer V3, which captures multi-scale spatial structure through voxelization and space-filling curves, outputting 32-dimensional geometric features ( $H_{geo}$ ).
Asymmetric Geometry-to-Semantics Fusion:
- The core innovation is an asymmetric cross-attention mechanism where geometric features act as the query and semantic features act as the key and value.
- This formalizes the cognitive principle that bottom-up geometric distinctiveness guides the retrieval of top-down semantic knowledge.
- A residual connection ensures geometric features are never discarded, preventing the model from collapsing into pure semantic recognition. A final MLP head predicts per-vertex saliency scores.
Temporal Scanpath Generation via Reinforcement Learning:
- The framework is extended to generate temporal scanpaths (sequences of fixations) using a Partially Observable Markov Decision Process (POMDP).
- State: Current vertex and a visit count vector.
- Action: Moving to one of the $K=6$ nearest neighbors on the mesh surface, respecting 3D topology rather than Euclidean grids.
- Reward: A composite function balancing saliency exploitation, inhibition-of-return (penalizing revisits), exploration bonuses for new regions, and diversity.
- Training: The policy is optimized using Proximal Policy Optimization (PPO).

Key Contributions

Diffusion-Distilled Semantic Priors: The first application of geometry-conditioned diffusion features to enable zero-shot semantic grounding in 3D attention modeling.
Asymmetric Fusion Architecture: A novel mechanism where geometry queries semantics, explicitly modeling the bottom-up/top-down cognitive dichotomy.
Topology-Aware Scanpath RL: A reinforcement learning framework for 3D scanpath generation that respects mesh connectivity and incorporates inhibition-of-return dynamics, addressing a gap in existing 2D-only models.

Experimental Results
The method was evaluated on three datasets: SAL3D, NUS3D, and 3DVA.

SAL3D: The model achieved a Correlation Coefficient (CC) of 0.8492, a 28% improvement over the previous best (0.6616). KL Divergence decreased significantly to 0.1638.
NUS3D: Achieved an LCC of 0.609 and AUC-Judd of 0.935, outperforming MIMO-GAN by 129% in LCC.
3DVA: Achieved a mean LCC of 0.762 with low variance (0.093), outperforming MIMO-GAN-CRF by 49%.
Scanpath Generation: The RL agent achieved an NSS of 2.05 and a MultiMatch score of 0.51, indicating that the generated temporal sequences closely match human viewing behavior.
Qualitative Analysis: Visualizations confirm the model correctly identifies semantically salient regions (e.g., gargoyle faces, octopus tentacles) that geometric-only methods miss, while still attending to geometrically distinct features.

Significance and Claims
The paper claims to establish a new paradigm for modeling human visual attention on 3D surfaces by unifying geometric and semantic processing within a cognitively motivated architecture. The authors assert that their approach effectively bridges the gap between low-level geometric distinctiveness and high-level semantic recognition, validating that such architectures can explain human fixation patterns on 3D objects more accurately than prior geometry-only or 2D-based methods. The work also pioneers the formulation of scanpath generation directly on 3D mesh topologies.