💻 computer science

DEGround: An Effective Baseline for Ego-centric 3D Visual Grounding with a Homogeneous Framework

DEGround introduces a homogeneous, single-stage framework for ego-centric 3D visual grounding that unifies detection and grounding through shared object queries and transformer heads, augmented by specialized plug-in modules to achieve state-of-the-art performance on multiple benchmarks.

Original authors: Yani Zhang, Dongming Wu, Hao Shi, Yingfei Liu, Tiancai Wang, Xingping Dong

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yani Zhang, Dongming Wu, Hao Shi, Yingfei Liu, Tiancai Wang, Xingping Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot walking through a cluttered room, holding a camera and a depth sensor (like a 3D scanner). Someone gives you a spoken instruction: "Find the red pillow that is sitting in front of the door." Your job is to look at the 3D world you see and point a digital box around that specific pillow. This task is called Ego-centric 3D Visual Grounding.

The paper introduces a new system called DEGround to solve this problem. Here is how it works, explained simply:

The Problem: The "Two-Step" Mess

Before this paper, most robots used a clumsy, two-step process to find objects:

Step 1 (The Detective): First, the robot had to act like a detective, scanning the whole room to find every object it could see (chairs, tables, pillows) and draw boxes around them.
Step 2 (The Translator): Then, it had to take that list of boxes and try to figure out which one matched the sentence.

The Analogy: Imagine you are trying to find a specific book in a library. The old way was to first write down the title, author, and location of every single book in the library on a giant list. Then, you would take that list and read through it again to find the one you wanted. It was slow, repetitive, and the robot often forgot what it learned in step one when it started step two.

The Solution: DEGround (The "All-in-One" Brain)

The authors created DEGround, which acts like a single, efficient brain that does both jobs at once.

1. The "Shared Query" (The Common Language)
Instead of having two different systems that don't talk to each other, DEGround uses a single set of "queries" (think of these as digital sticky notes) to represent objects.

How it works: The robot places these sticky notes on objects it sees. These notes are used simultaneously to say, "This is a pillow" (Detection) AND "This is the pillow the human asked for" (Grounding).
The Benefit: Because the robot uses the same notes for both tasks, it doesn't have to re-learn how to find objects. It transfers its "know-how" instantly, making it much faster and more accurate.

2. The "Regional Activation" Module (The Highlighter)
Sometimes, the room has two identical pillows. One is near the door (what you want), and one is near the window (a distractor).

The Analogy: Imagine the robot has a magical highlighter. When it hears "in front of the door," the highlighter automatically glows bright red over the area near the door and dims the rest of the room.
The Result: This helps the robot ignore the wrong pillow and focus only on the region that matches the description.

3. The "Query-wise Modulation" Module (The Context Switch)
This module helps the robot understand the intent of the sentence right from the start.

The Analogy: Before the robot even looks at the room, it "primes" its sticky notes based on the sentence. If the sentence is about a "pillow," the notes become extra sensitive to soft, squishy shapes. If the sentence is about a "lamp," they become sensitive to light and metal.
The Result: The robot starts the search already knowing what to look for, rather than guessing.

The Results: Why It Matters

The authors tested this system on a massive benchmark called EmbodiedScan, which is like a giant, difficult test of 3D rooms with thousands of objects.

Speed and Accuracy: DEGround didn't just win; it crushed the competition. It improved the robot's accuracy by 7.52% compared to the previous best method.
Efficiency: In a small test, the old method took 12 rounds of training to get a decent score. DEGround reached a much higher score in just 3 rounds. It's like a student who learns a subject in a week that others take a month to master.
Robustness: In pictures showing the robot's view, DEGround correctly identified the right object even when there were many confusing, identical-looking objects nearby, whereas older methods got confused.

Summary

DEGround is a new, streamlined way for robots to understand 3D spaces. Instead of using a slow, two-step process, it uses a single, shared system to find objects and understand language at the same time. It uses "highlighting" and "contextual priming" to ignore distractions and find exactly what the human is asking for, making it the new state-of-the-art for robots that need to navigate and interact with the real world.

1. Problem Statement

Ego-centric 3D Visual Grounding (3D VG) is a core task in embodied intelligence, requiring an agent to locate a specific object in a 3D scene based on a natural language instruction, using only sparse, first-person multi-view RGB-D observations.

Current Limitations:
Existing methods typically employ a two-stage, heterogeneous pipeline:

Stage 1: Train a 3D object detector (often convolution-based or anchor-based).
Stage 2: Transfer the backbone weights to a separate grounding model (often Transformer-based/DETR-style) to perform the grounding task.

Key Issues with Heterogeneous Pipelines:

Incomplete Knowledge Transfer: While backbone weights are shared, the decoder and bounding box heads are architecturally different. Consequently, object-level priors (learned object representations and geometric reasoning) are not transferred, forcing the grounding model to re-learn these features.
Redundant Optimization: The grounding model must re-optimize object decoding and box regression on the same data, leading to slow convergence, high computational cost, and suboptimal accuracy.
Lack of Fine-Grained Alignment: Existing methods struggle with "same-class distractors" (objects of the same category that do not match the description) and fail to effectively integrate linguistic intent into the initial query representations.

2. Methodology: DEGround

The authors propose DEGround, a homogeneous framework that unifies 3D detection and grounding into a single architecture.

A. Core Architecture: Homogeneous Framework

Shared Object Representation: Instead of using distinct decoders, DEGround uses a single set of object queries as the common representation for both detection and grounding.
Shared Decoder & Heads: Both tasks utilize the same Transformer-based decoder and a shared bounding box head.
- Mechanism: A lightweight scoring head selects the top- $K$ voxel centers based on visual features to form the initial query embeddings ( $Q$ ).
- Benefit: This alignment allows the grounding task to directly inherit robust localization and categorization priors from the detection task, eliminating redundant learning and accelerating convergence.

B. Feature Extraction & Fusion

Input: Multi-view RGB-D images.
3D Representation: Depth maps are projected to world coordinates to create a pseudo-point cloud, which is voxelized and encoded ( $F_{3d}$ ).
2D Representation: Multi-scale features are extracted from RGB images ( $F_{2d}$ ).
Fusion: 2D features are sampled onto 3D points via bilinear interpolation and concatenated with 3D geometric features to create a unified visual representation ( $F_{visual}$ ).

C. Task-Specific Plug-in Modules

To enhance fine-grained instruction grounding, two modules are introduced:

Regional Activation Grounding (RAG):
- Goal: Address "same-class distractors" by highlighting regions consistent with the text.
- Mechanism: Uses text-driven attention to fuse $F_{visual}$ with text embeddings ( $F_{text}$ ), generating a text-enhanced feature map ( $F'_{region}$ ).
- Supervision: Introduces a Spatial Relevance Loss ( $L_{spatial}$ ) that supervises point-level predictions, forcing the model to assign high relevance scores to points inside the ground-truth bounding box and low scores to others.
Query-wise Modulation (QIM):
- Goal: Inject global sentence-level semantics directly into the initial object queries to make them "instruction-aware."
- Mechanism: Applies a feature-wise affine transformation to the visual queries ( $Q_{vis}$ ) using modulation vectors ( $\beta, \gamma$ ) generated from the sentence embedding.
- Formula: $Q_{mod} = \beta \odot Q_{vis} + \gamma \odot \mathbf{1}S$ . This adapts the query space to better integrate visual evidence with linguistic intent from the very start of decoding.

3. Key Contributions

Homogeneous Framework: Proposed DEGround, the first framework to unify detection and grounding via sharing queries, enabling true object-level knowledge transfer and removing redundant optimization.
Regional Activation Grounding (RAG): A novel module that uses point-wise supervision and text-guided attention to resolve same-class distractor confusion.
Query-wise Modulation (QIM): A mechanism that dynamically modulates decoder queries based on global sentence semantics, improving query discriminability.
State-of-the-Art Performance: Established a new baseline on the EmbodiedScan benchmark, significantly outperforming existing methods.

4. Experimental Results

The model was evaluated on the EmbodiedScan benchmark (aggregating ScanNet, 3RScan, and Matterport3D), which features realistic, sparse, ego-centric observations.

3D Visual Grounding (Mini Validation Set):
- DEGround achieved 61.28% Overall Precision (AP@25).
- This is a 15.49% improvement over the previous SOTA (BIP3D, 45.79%).
- It surpassed the ResNet-50 baseline (DenseGrounding) by 17.77%.
- Convergence: DEGround reached 59.90% mAP in just 3 epochs, whereas the baseline required 12 epochs to reach 35.84%.
3D Visual Grounding (Full Validation Set):
- Achieved 62.18% Overall Precision.
- Outperformed BIP3D (Swin-T backbone) by 7.52%, despite using a lighter ResNet-50 backbone.
Official Test Set:
- Achieved 42.04% AP@50, setting a new SOTA and outperforming the nearest competitor (BIP3D) by 2.35%.
3D Object Detection:
- Achieved 24.68% AP@25 on the detection benchmark, outperforming BIP3D by 3.77%.

5. Significance

Paradigm Shift: DEGround challenges the prevailing two-stage heterogeneous paradigm in 3D VG, demonstrating that a unified, homogeneous architecture is superior for transferring object-level priors.
Efficiency: By sharing the decoder and box head, the method significantly reduces training time and data requirements while improving accuracy.
Robustness: The introduction of RAG and QIM specifically addresses the challenges of ego-centric perception (sparse views, distractors), making the model more suitable for real-world embodied agents that cannot rely on pre-reconstructed scene priors.
Benchmarking: The results on EmbodiedScan establish a new standard for 3D visual grounding, proving that simple, elegant architectural unification can outperform complex, multi-stage pipelines.