💻 computer science

REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting

The paper introduces REALM, an MLLM-agent framework that enables open-world 3D reasoning segmentation and editing on Gaussian Splatting by employing a novel Global-to-Local Spatial Grounding strategy to bridge the gap between complex human instructions and precise 3D object manipulation without extensive 3D-specific post-training.

Original authors: Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou Yu

Published 2026-03-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical, invisible 3D world inside your computer, built out of millions of tiny, glowing sparkles (this is called 3D Gaussian Splatting). Now, imagine you want to tell a robot to "find the blue toy elephant that the child likes" or "remove the chair that's blocking the apple."

The problem is, most robots are like people who only have a single, fixed camera eye. If you ask them to find something based on a complex clue, they get confused if the object is hidden or if the clue requires thinking (like "the chair near the apple"). They might look at the wrong angle and miss it entirely.

Enter REALM, a new "brain" for robots that solves this problem. Here is how it works, explained simply:

1. The Problem: The "One-Eye" Blind Spot

Imagine you are trying to find a specific key in a messy room. If you only look at the room from one angle (say, standing in the doorway), you might miss the key because it's hidden behind a chair. If you try to look at every angle at once, your brain gets overwhelmed and you can't focus.

Existing AI models are like that. They either look at one picture and miss the context, or they try to look at too many pictures and get confused. They are great at saying "That's a cup," but terrible at saying "That's the cup next to the book."

2. The Solution: The "Sherlock Holmes" Agent

REALM is an AI agent that acts like a detective with a superpower: it can talk to a giant, super-smart brain (a Multimodal Large Language Model, or MLLM) and ask it to reason through the scene.

Here is the step-by-step process REALM uses, using a Search Party analogy:

Step A: The "Wide-Angle Scan" (Global Grounding)

Instead of staring at one spot, REALM sends out a team of scouts to look at the room from 8 different angles at the same time.

The Analogy: Imagine you are looking for a lost dog. You don't just stand in one spot; you send friends to the front, back, left, and right of the house.
What happens: Each "scout" (a camera view) asks the super-smart brain: "I see a blue elephant here. Is this the one?" The brain says, "Yes, that looks like the blue elephant!"
The Magic: REALM takes all these answers and votes. If 7 out of 8 scouts say, "That's the elephant," the robot knows exactly where the elephant is in 3D space, even if one scout couldn't see it.

Step B: The "Zoom-In" (Local Refinement)

Once the team knows roughly where the elephant is, they don't stop there. They zoom in.

The Analogy: Now that you know the dog is in the backyard, you walk right up to the fence and look closely to see exactly where its paws are.
What happens: The AI generates close-up, high-definition pictures of just that area. It asks the brain again: "Okay, now that we are close, exactly which pixels belong to the elephant?" This creates a perfect, detailed outline (a mask) of the object.

3. The Superpower: "Reasoning"

This is the coolest part. REALM doesn't just look for keywords; it thinks.

The Clue: "Find the toy that would make a child who likes blue happy."
The Robot's Thought Process:
1. Scan the room.
2. Identify objects: There's a red ball, a green car, and a blue elephant.
3. Apply logic: The child likes blue. The blue elephant is the best match.
4. Action: "Target acquired: Blue Elephant."

Old robots would get stuck on the word "toy" and might grab the red ball. REALM understands the context.

4. The "Magic Wand" (Editing)

Once REALM has found the object and drawn a perfect 3D outline around it, it can perform "magic tricks" on the scene without breaking anything else.

Remove: "Take away the chair near the apple." (The chair vanishes, but the apple stays).
Replace: "Swap the blue elephant for a teddy bear." (The elephant is gone, a bear appears in the exact same spot).
Style Change: "Turn the green chair into gold." (The chair changes material, but its shape and position remain perfect).

Why is this a big deal?

Think of it like upgrading from a 2D map to a 3D hologram.

Before: You could tell a computer "Find the apple," and it would find the apple.
Now (with REALM): You can tell a computer, "Find the apple that is behind the book but in front of the lamp, and paint it gold." The computer understands the spatial relationships and the complex instructions, just like a human would.

In short: REALM is a smart robot assistant that uses a team of cameras and a super-brain to understand complex, "riddle-like" instructions about 3D spaces, finds the right objects, and lets you edit the world around them with a simple sentence.

1. Problem Statement

The paper addresses a critical gap in the intersection of computer vision, robotics, and natural language processing: bridging complex, reasoning-based human instructions with precise 3D object grounding.

The Challenge: Existing 3D segmentation methods (e.g., those based on NeRF or 3DGS) excel at explicit queries (e.g., "segment the cup") but fail at implicit, reasoning-based instructions (e.g., "remove the chair closer to the apple" or "find the toy for a child who likes blue").
The Limitation of Current MLLMs: While Multimodal Large Language Models (MLLMs) possess strong 2D reasoning capabilities, they lack intrinsic 3D spatial understanding. Directly feeding a single rendered view of a 3D scene to an MLLM leads to high sensitivity to viewpoint selection, while feeding multiple views simultaneously overwhelms the model, causing it to lose consistency.
Goal: To create a framework that enables open-world 3D reasoning segmentation without extensive 3D-specific post-training, allowing agents to understand spatial relationships, context, and semantics to perform 3D editing tasks.

2. Methodology: The REALM Framework

REALM leverages 3D Gaussian Splatting (3DGS) as a high-fidelity proxy for the 3D world due to its ability to render photorealistic novel views suitable for MLLM comprehension. The framework consists of three core components:

A. 3D Feature Field Construction

To enable consistent 3D identification, REALM constructs a 3D Feature Field:

Initialization: It uses the Segment Anything Model (SAM) to extract instance masks from input images.
Propagation: A temporal propagation model associates instances across different views to assign a consistent identity ( $id_i$ ) to each 3D Gaussian primitive.
Optimization: Each Gaussian is assigned an instance feature vector. A classifier is trained to predict pixel-wise identity maps from rendered feature maps, allowing the system to group 3D Gaussians into specific instances based on learned features.

B. MLLM-Based Visual Segmenter (LMSeg)

This is the core reasoning agent that operates on 2D rendered views:

Input: An image from a specific viewpoint and a natural language query.
Process: The LMSeg agent prompts an MLLM (e.g., Qwen-2.5-VL) to output a bounding box, object category, and an explanatory rationale.
Grounding: The predicted bounding box is fed into SAM to generate a 2D binary mask. This mask is intersected with the predicted 2D instance map (derived from the 3D Feature Field) to identify the specific 3D Instance ID corresponding to the query.

C. Global-to-Local Spatial Grounding (GLSpaG)

To overcome the viewpoint sensitivity of MLLMs, REALM employs a hierarchical two-stage strategy:

Global Stage (Coarse Localization):
- View Selection: Instead of random views, the system clusters training camera poses using K-means and selects representative views. It further filters these to select the top- $N$ views that observe the highest number of distinct instances (ensuring diverse context).
- Aggregation: The LMSeg agent processes these global views in parallel. The resulting instance ID predictions are aggregated via a voting scheme to determine the most probable target instance ID.
- Output: A coarse 3D segmentation mask.
Local Stage (Fine-Grained Refinement):
- View Selection: The system selects a new set of "local" cameras specifically centered on the coarse target object identified in the global stage.
- Refinement: LMSeg generates fine-grained 2D masks for these close-up views.
- Optimization: A differentiable rasterizer renders the 3D mask to the image plane, and a loss function aligns it with the 2D masks. This process refines the 3D Gaussian grouping to produce accurate, consistent 3D boundaries.

3. Key Contributions

REALM Framework: A novel MLLM-agent framework that lifts 2D reasoning capabilities into the 3D domain using 3DGS, enabling reasoning-based segmentation without heavy 3D-specific training.
GLSpaG Strategy: A robust "Global-to-Local" spatial grounding mechanism that mitigates viewpoint sensitivity by aggregating multi-view reasoning and refining results with targeted close-up views.
LMSeg Agent: An image-level reasoning segmenter that combines MLLM priors with SAM and a 3D feature field to infer 3D instance identities from implicit instructions.
REALM3D Benchmark: The introduction of a new large-scale dataset comprising 100+ scenes and 1,000+ prompt-mask pairs, specifically annotated with implicit, reasoning-based instructions to evaluate 3D reasoning capabilities (re-annotating LERF and 3D-OVS as well).
3D Editing Capabilities: The framework extends beyond segmentation to support downstream tasks like object removal, replacement, and style transfer directly on the 3DGS representation.

4. Experimental Results

The authors evaluated REALM on LERF, 3D-OVS, and the new REALM3D benchmarks.

Quantitative Performance:
- REALM significantly outperforms state-of-the-art baselines (Gaga, GAGS, GS-Group) on implicit queries.
- On LERF, REALM achieved 92.88% mIoU (vs. 44.82% for Gaga).
- On 3D-OVS, REALM achieved 93.68% mIoU (vs. 58.46% for GAGS).
- On REALM3D, REALM achieved 82.30% mIoU.
- The performance gap is attributed to the inability of baselines to handle spatial relationships and contextual reasoning.
Qualitative Performance:
- Spatial Relationships: Successfully identified objects based on relative position (e.g., "drink held by the teddy bear").
- Ambiguous Descriptions: Correctly inferred objects based on function or attributes (e.g., "refreshing beverage made from fruit" $\rightarrow$ orange juice).
- Contextual Understanding: Selected objects based on scenario context (e.g., "earphone" when everyone else is asleep).
- Editing: Demonstrated precise object removal and style transfer (e.g., changing material to gold) without affecting occluded or adjacent objects.
Efficiency:
- The total inference time for a prompt is under 10 seconds (approx. 8.68s), with MLLM calls executed in parallel.
- The method maintains high rendering speeds (354 FPS) as it only renders single-channel masks during optimization.

5. Significance

REALM represents a significant step forward in embodied AI and human-robot interaction. By enabling agents to understand and act upon complex, implicit natural language instructions in 3D space, it moves beyond simple object detection to true scene understanding and manipulation. The framework's ability to operate on 3DGS makes it computationally efficient and suitable for real-time applications, while the introduction of the REALM3D benchmark provides a crucial standard for future research in 3D reasoning. This work effectively bridges the gap between the reasoning power of large language models and the spatial requirements of the physical 3D world.