💻 computer science

OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention

OpenGaFF is a novel open-vocabulary 3D scene understanding framework built on 3D Gaussian Splatting that enhances spatial semantic coherence and object-level consistency by coupling a Gaussian Feature Field with a structured codebook and a codebook-guided attention mechanism.

Original authors: Kunyi Li, Michael Niemeyer, Sen Wang, Stefano Gasperini, Nassir Navab, Federico Tombari

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kunyi Li, Michael Niemeyer, Sen Wang, Stefano Gasperini, Nassir Navab, Federico Tombari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to "see" a 3D room, not just as a collection of floating dots, but as a place filled with meaningful objects like a "red apple," a "glass of water," or a "sofa."

The paper introduces a new system called OpenGaFF. Its goal is to make computers understand 3D scenes so well that you can ask them, "Where is the toy elephant?" or "Show me the glass of water," and they will point to the exact right spot, even if they've never seen that specific object before.

Here is how it works, explained through simple analogies:

The Problem: The "Fuzzy" 3D Map

Previous methods tried to teach computers about 3D objects by taking 2D pictures and projecting them into 3D space. Think of this like trying to build a 3D sculpture by gluing together flat stickers.

The Issue: Because the computer looks at the object from different angles (like walking around a table), the "stickers" often don't match up perfectly. One side might think the object is a "chair," while the other side thinks it's a "table." This leads to a messy, fragmented 3D map where the computer gets confused about what things are.
The "Transparent" Problem: Some objects, like a glass of water, are see-through. Old methods often ignored them because they didn't block light well, making the computer "blind" to the glass even if you asked for it.

The Solution: OpenGaFF

OpenGaFF fixes this by using two main tricks: a Smart Blueprint and a Shared Dictionary.

1. The Smart Blueprint (The Gaussian Feature Field)

Instead of letting every single tiny 3D dot (called a "Gaussian") learn its own identity independently, OpenGaFF treats the whole scene like a continuous landscape.

The Analogy: Imagine a weather map. You don't teach every single pixel of the map what the temperature is. Instead, you have a rule: "If you are near the mountain, it's cold; if you are near the beach, it's warm."
How it helps: OpenGaFF uses a "Feature Field" that says, "If a 3D dot is in the shape of a chair and looks like wood, it must be part of the chair." This forces the computer to understand that objects have a consistent shape and identity, no matter which angle you look at them from. It ties the shape (geometry) tightly to the meaning (semantics).

2. The Shared Dictionary (The Structured Codebook)

Old methods tried to compress complex language ideas into tiny, simple numbers, which often lost important details. OpenGaFF uses a "Codebook," which is like a shared dictionary of semantic building blocks.

The Analogy: Imagine a group of artists trying to paint a picture of a "red apple."
- Old Way: Each artist tries to invent their own definition of "red" and "apple" from scratch. The result is a messy mix of colors.
- OpenGaFF Way: They all agree to use a specific, pre-defined "Red Apple" card from a shared deck. If they see something that looks like a red apple, they all grab that same card.
How it helps: This ensures that every part of the "red apple" in the 3D scene uses the same definition. It stops the computer from getting confused and calling the apple a "ball" or a "tomato." It also uses a special attention mechanism to make sure the computer picks the exact right card from the dictionary, reducing noise.

3. The "Ghost" Opacity

For tricky objects like a glass of water, the system gives them a special "semantic opacity."

The Analogy: In normal 3D rendering, if something is see-through, the computer ignores it. OpenGaFF says, "Even though I can see through the glass, I can still see the concept of the glass." It creates a separate layer just for "meaning" so that transparent objects don't disappear from the computer's understanding.

The Results

The paper tested this system against other top methods on real-world datasets (like rooms with furniture and toys).

Better Accuracy: When asked to find objects, OpenGaFF found them more accurately and completely than previous systems.
Less Noise: It didn't accidentally highlight random background items as the target object.
Faster & Lighter: It runs faster and uses less computer memory than some competitors because it doesn't need to store a unique, complex definition for every single 3D dot; it just follows the "Smart Blueprint" rules.

In Summary

OpenGaFF is like giving a computer a 3D map where the shape of the object dictates its name, and a shared dictionary that ensures everyone agrees on what that name means. This allows the computer to understand a room not just as a pile of pixels, but as a collection of consistent, understandable objects, even if they are transparent or viewed from weird angles.

Technical Summary: OpenGaFF

Problem Statement

Understanding open-vocabulary 3D scenes using Gaussian-based representations faces significant challenges regarding spatial coherence and semantic consistency. While 3D Gaussian Splatting (3DGS) offers efficient and explicit scene representations, existing methods for incorporating semantics often suffer from a weak coupling between geometry and semantics. This leads to fragmented predictions, spatially inconsistent object representations, and unreliable semantic attribution, particularly for transparent objects or complex multi-view scenarios. Current approaches generally fall into three categories with distinct limitations:

Compress-then-decode: These methods project high-dimensional language features into low-dimensional embeddings, discarding fine-grained semantic information and limiting open-vocabulary understanding.
Feature aggregation: These fuse multi-view features using appearance-based opacity weights, which often fail to align with semantic relevance (e.g., assigning near-zero opacity to transparent glass).
Codebook-based: While these discretize semantics into shared embeddings, they often assign features in a geometry-agnostic manner, resulting in spatially inconsistent predictions and fragmented object representations.

Methodology

OpenGaFF (Open-Vocabulary Gaussian Feature Field with Codebook Attention) is a framework built upon 3DGS designed to enforce geometric consistency and object-level semantic coherence. The method operates in two primary stages:

1. Preprocessing and Codebook Initialization

The pipeline begins by processing RGB images using foundation models (SAM for segmentation, CLIP for vision-language features) to generate 2D language feature maps. Unlike prior works that use randomly initialized codebooks of fixed size, OpenGaFF constructs a Structured Language (SL) Codebook in a data-driven manner. All CLIP features are aggregated and clustered via k-means, where the number of clusters ( $N_c$ ) is automatically determined by the scene's structure. The cluster centroids form the codebook, serving as a set of shared semantic primitives. Additionally, Principal Component Analysis (PCA) is applied to project features into a lower-dimensional (LD) space to provide efficient supervision for the feature field.

2. Stage 1: Gaussian Feature Field

Instead of assigning independent learnable semantic features to each Gaussian (which decouples geometry and semantics), OpenGaFF introduces a Gaussian Feature Field. This field models semantics as a continuous function of the Gaussian's geometry and appearance. Specifically, the low-dimensional semantic feature $m_i$ for a Gaussian is predicted by an MLP-based decoder conditioned on the Gaussian's position (via Fourier positional encoding) and color:
$m_i = F(\phi(\mu_i), c_i)$
This formulation explicitly couples geometry and semantics, encouraging spatially consistent feature representations. To handle objects where appearance opacity does not align with semantic importance (e.g., transparent glass), a separate semantic opacity ( $\alpha^s_i$ ) is introduced. The LD features are rasterized using this semantic opacity and supervised against the ground-truth LD feature maps using a combination of cosine similarity and L1 loss.

3. Stage 2: Attention with Codebook

To recover high-dimensional language expressiveness lost during compression, OpenGaFF employs a codebook-guided attention mechanism. The rendered LD feature map acts as the query, while the structured codebook serves as the key and value. Through an attention mechanism, the model retrieves corresponding high-dimensional language features. This process is treated as commutative with rendering, allowing the model to learn the codebook using only 2D supervision while naturally extending to 3D Gaussians for inference.
To ensure that each query focuses on a single, distinct semantic unit, an entropy regularization term is applied to the attention weights, promoting independence across codebook entries. The final language features are supervised against ground-truth high-dimensional language maps.

Key Contributions

Gaussian Feature Field: A novel mechanism that conditions semantic predictions on Gaussian geometry and appearance, aligning 3D semantic representation with spatial coherence and reducing feature fragmentation.
Structured Language Codebook: A data-driven initialization of a semantic dictionary that enforces object-level consistency by constraining language features to be selected from shared semantic primitives, thereby reducing intra-object variation.
Codebook-Guided Attention: A retrieval mechanism that enables robust open-vocabulary reasoning by matching query embeddings to learned codebook entries, ensuring consistent and expressive semantic representation in 3D scenes.

Experimental Results

OpenGaFF was evaluated on standard 2D and 3D open-vocabulary benchmarks, specifically LERF-OVS and ScanNet-v2.

Quantitative Performance: The method consistently outperforms state-of-the-art approaches (including LangSplat, LangSplatV2, GALA, and OpenGaussian) in both 2D and 3D open-vocabulary segmentation. On LERF-OVS, OpenGaFF achieved a mean mIoU of 64.98 in 2D and 54.36 in 3D, surpassing the previous best (LangSplatV2) by significant margins. On ScanNet-v2, it achieved the best performance across all metrics (e.g., 36.55 mIoU for 19 classes), demonstrating superior spatial consistency.
Qualitative Improvements: Visualizations show that OpenGaFF produces more complete and spatially coherent segmentations. Unlike baselines that often yield fragmented results or miss parts of transparent objects, OpenGaFF successfully segments complex objects like "Red Apple" and "Glass of Water" with high fidelity.
Codebook Interpretability: Analysis of the learned codebook reveals that OpenGaFF learns disentangled semantic units, where individual entries correspond to unique, coherent semantic regions. In contrast, baseline methods like LangSplatV2 exhibit entangled feature bases where semantics are distributed across multiple entries or activate on background noise.
Efficiency: OpenGaFF demonstrates high efficiency, requiring only 12 GB of memory and 15 minutes of training time on a single RTX 4090, while maintaining an inference speed of 35 fps.

Significance

The paper posits that the primary limitation in current 3D scene understanding is the weak coupling between geometry and semantics, leading to fragmented and inconsistent representations. OpenGaFF addresses this by treating semantics as a continuous function of geometry and enforcing object-level consistency through a structured codebook. The significance of this work lies in its ability to produce geometrically consistent and semantically interpretable 3D representations without the computational overhead of dense volumetric methods or the information loss of aggressive feature compression. By demonstrating that a structured, attention-based approach can effectively bridge 2D foundation models and 3D Gaussian representations, OpenGaFF advances the capability of machines to perceive complex 3D environments in a manner that is both spatially coherent and semantically meaningful.