💻 computer science

Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition

This paper proposes a dynamic graph neural network with an adaptive node selection mechanism to effectively fuse RGB and depth local features across multiple spatial levels for superior indoor scene recognition.

Original authors: Qiong Liu, Ruofei Xiong, Xingzhen Chen, Muyao Peng, You Yang

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qiong Liu, Ruofei Xiong, Xingzhen Chen, Muyao Peng, You Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify a room just by looking at a photo. If you only see the colors and textures (the RGB image), you might see a red chair and a wooden table. But if you also have a "3D depth map" (the Depth image), you can see how far away those objects are and how they fit together in space.

The problem is that a room is messy. There are too many objects, some are hidden, and the lighting is weird. If you try to look at every single pixel in the photo, your brain (or a computer) gets overwhelmed and confused. You need to focus on the important parts to figure out what the room is.

This paper presents a smart new way for computers to do exactly that: pick the most important clues and figure out how they relate to each other.

Here is the breakdown of their method using simple analogies:

1. The Problem: The "Too Much Information" Overload

Previous methods were like trying to read a whole encyclopedia to find out what a room is. They looked at the whole picture globally, or they tried to look at every single object.

Global features: Like looking at a room from far away and guessing, "It looks like a living room." (Good, but not precise).
Local features: Like zooming in on every single object. (Too much data, hard to connect the dots).

The authors realized that not all objects are equally important. A "lamp" might be a strong clue for a "bedroom," but a "random sock" on the floor isn't. The computer needs to adaptively select the best clues.

2. The Solution: The "Dynamic Graph" Detective Team

The authors built a system called a Dynamic Graph Neural Network. Let's break down what that means:

A. The "Adaptive Node Selection" (The Scout)

Imagine you are a detective entering a crime scene. You don't look at the dust on the floor; you look for the key evidence.

The computer uses a special "Attention Mechanism" (like a spotlight) to scan the room.
It asks: "Which objects are most important for identifying this room?"
It picks the top 16 most important objects (nodes) from both the color image and the depth image.
Analogy: Instead of interviewing 1,000 people in a crowd, the detective only interviews the 16 most suspicious or relevant witnesses.

B. The "Three-Level Hierarchy" (The Organized Team)

Once the detective picks the 16 witnesses, they aren't all treated the same. The system organizes them into three levels:

The Main Captain: The single most important object (e.g., the bed in a bedroom).
The Lieutenants: The next few most important objects (e.g., the nightstand, the lamp).
The Foot Soldiers: The remaining important objects that support the main clues.

This hierarchy helps the computer understand that the "bed" is the boss, and the "nightstand" is related to it, but they are both different from a "sock" on the floor.

C. The "Dynamic Graph" (The Conversation)

Now, these 16 objects need to talk to each other to solve the puzzle.

Static Graph (Old Way): Imagine a meeting where everyone is forced to sit in fixed seats and can only talk to the person next to them, no matter what.
Dynamic Graph (New Way): This is a flexible meeting. The connections between objects change based on the situation.
- If the "bed" and "nightstand" are close, they talk closely.
- If the "lamp" is far away, the connection is weaker.
- Crucially, the system updates itself. As the computer learns more, it changes who talks to whom and how much weight they give to the "Color" clues vs. the "Depth" clues.

D. The "Fusion" (The Final Verdict)

Finally, the system combines the "Global View" (the whole room) with the "Local View" (the 16 important objects and their relationships).

It takes the best of both worlds: the general vibe of the room + the specific details of the key objects.
It then makes a final guess: "This is definitely a Bedroom."

3. Why is this better?

The authors tested this on two famous datasets (SUN RGB-D and NYU Depth v2).

The Result: Their method got the highest accuracy scores ever recorded on these tests.
Why it won:
1. It ignores the noise: It doesn't waste time on irrelevant objects.
2. It understands relationships: It knows how objects relate to each other in 3D space, not just 2D pictures.
3. It adapts: It knows that sometimes the color is more important, and sometimes the depth (distance) is more important, and it switches focus accordingly.

Summary Metaphor

Think of previous methods as trying to solve a jigsaw puzzle by looking at the box cover (global) or trying to force every single piece together at once (local).

This new method is like a smart puzzle solver who:

Scans the box and picks out the 16 most unique, colorful edge pieces (Adaptive Selection).
Arranges them on the table based on how they fit together (Graph Structure).
Constantly moves pieces around and swaps them if they don't fit the picture (Dynamic Updates).
Finally, looks at the completed section to guess what the whole picture is (Scene Recognition).

By focusing on the right pieces and how they connect, the computer becomes much better at recognizing indoor scenes.

1. Problem Statement

Indoor scene recognition using RGB-D (color and depth) data faces significant challenges due to:

Complexity of Indoor Scenes: Cluttered objects, diverse spatial layouts, and mutual occlusions make global feature extraction insufficient.
Feature Selection: While local features are crucial, existing methods struggle to adaptively select the most discriminative local features from both RGB and depth modalities.
Modality Integration: Previous works often treat RGB and depth features with fixed weights or simple concatenation, failing to account for the varying contributions of each modality across different scenes.
Static Relationships: Existing Graph Neural Network (GNN) approaches often use fixed graph connections, which limits their ability to model dynamic spatial relationships between objects in different scenes.

2. Methodology

The authors propose a Dynamic Graph Neural Network (DGNN) framework equipped with an Adaptive Node Selection (ANS) mechanism. The framework consists of four main modules:

A. Feature Extraction

RGB and Depth images are processed through two separate ResNet-101 backbones to extract global feature maps.
Depth maps are encoded using HHA (Horizontal disparity, Height above ground, Angle with gravity) representation to better utilize geometric information.

B. Adaptive Node Selection (ANS)

Mechanism: Based on the CBAM (Convolutional Block Attention Module) attention mechanism.
Process:
1. Global average and max pooling are applied to the feature maps to generate attention maps ( $SA_{rgb}$ and $SA_{d}$ ).
2. These maps highlight the importance of local regions.
3. The top- $k$ positions with the highest attention scores are selected as nodes for the graph.
4. Node Hierarchy: Selected nodes are categorized into three levels based on their attention scores:
  - Main-central nodes (Top 1 most important).
  - Sub-central nodes (Next $n$ most important).
  - Leaf nodes (Remaining selected nodes).
Goal: To filter out interference noise and focus only on discriminative local features.

C. Dynamic Graph Construction & Update

Graph Structure: A dynamic graph $G=(V, E)$ is constructed where $V$ contains RGB appearance nodes and Depth geometry nodes.
Edge Connections:
- Intra-modality: Sub-central nodes connect to the main-central node; leaf nodes connect to the nearest sub-central node (Euclidean distance).
- Inter-modality: Corresponding nodes between RGB and Depth modalities are connected (Main-to-Main, Sub-to-Sub).
Dynamic Update: Unlike static graphs, the graph is updated iteratively using a GNN aggregation mechanism.
- Dual Attention: A second attention mechanism is used during the update phase to automatically learn the varying contributions of RGB and depth modalities for each specific scene.
- Update Rule: Node features are updated by aggregating information from neighbors (both intra- and inter-modality) weighted by learned attention coefficients ( $\alpha_{ij}$ ).

D. Feature Fusion

The optimized local features from the dynamic graph are concatenated with the global features extracted from the initial ResNet layers.
A linear classifier (Fully Connected layer) followed by a cross-entropy loss function performs the final scene classification.
Loss Function: The total loss combines losses from the global RGB branch, global Depth branch, and the final fused local/global branch.

3. Key Contributions

Adaptive Node Selection: Introduced a mechanism to dynamically select $k$ discriminative local features from both modalities based on attention weights, rather than using fixed patches or object proposals.
Dynamic Graph Modeling: Proposed a graph structure with hierarchical node levels (Main, Sub, Leaf) and sparse connections that adaptively update based on scene-specific attention, allowing the model to capture complex object relations.
Modality-Aware Fusion: Unlike previous methods that treat modalities equally, the proposed model learns the varying contributions of RGB and depth features dynamically during the graph update process.
End-to-End Learning: The entire framework (selection, graph construction, and fusion) is trained end-to-end, avoiding the high computational complexity of two-stage object detection approaches.

4. Experimental Results

The method was evaluated on two standard public datasets: SUN RGB-D and NYU Depth v2.

SUN RGB-D Dataset:
- Achieved a Mean-Class Accuracy of 57.7%.
- Outperformed the previous state-of-the-art (Xiong et al., 2021) which scored 57.3%.
- Showed significant improvement over single-modality baselines and simple fusion methods.
NYU Depth v2 Dataset:
- Achieved a Mean-Class Accuracy of 70.4%.
- Surpassed the previous best (Xiong et al., 2021) at 69.3%.
- Demonstrated near-perfect accuracy (approaching 100%) on specific categories like "classroom," "bookstore," and "bathroom."
Ablation Studies:
- Single vs. Multi-modality: Multi-modality fusion significantly outperformed single-modality (RGB or Depth alone).
- Graph Validity: Removing the GNN structure (simple cascading of features) caused a ~3.5% drop in accuracy, proving the value of modeling feature relations.
- Attention Mechanism: Removing the attention module during graph updates reduced accuracy by ~1.8% (57.7% $\to$ 55.9%), confirming the importance of dynamic weight adjustment.
- Node Count ( $k$ ): Optimal performance was found at $k=16$ . Too few nodes ( $k=4, 9$ ) lacked scene representation, while too many ( $k=25, 36$ ) introduced noise.

5. Significance

This paper addresses a critical gap in RGB-D scene recognition by moving beyond static feature extraction and fixed graph structures. By integrating adaptive selection with dynamic graph updates, the model effectively handles the clutter and complexity of indoor environments. The ability to learn modality-specific contributions dynamically allows the system to be more robust across diverse scenes. The state-of-the-art results on benchmark datasets demonstrate that leveraging local feature relationships through a dynamic graph is a superior strategy for indoor scene understanding compared to global feature-only or static graph approaches.