💻 computer science

Learning Representations from 3D Gaussian Splats

This paper conducts a comparative evaluation of various geometric deep learning architectures to assess their effectiveness in learning latent representations from 3D Gaussian Splatting for scene classification, highlighting the impact of architectural choices and Gaussian-specific attributes on representation quality.

Original authors: Julia Farganus, Krzysztof Żurawicki, Arkadiusz Gaweł, Weronika Jakubowska, Halina Kwaśnicka

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Julia Farganus, Krzysztof Żurawicki, Arkadiusz Gaweł, Weronika Jakubowska, Halina Kwaśnicka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize different objects, like airplanes, chairs, or bags. Usually, we teach computers by showing them a "point cloud"—a digital cloud made of thousands of tiny, invisible dots that mark the surface of an object. It's like trying to guess the shape of a statue by feeling only a few scattered grains of sand on its surface.

But recently, a new technology called 3D Gaussian Splatting (3DGS) came along. Instead of just dots, this method uses "splats." Think of these splats not as tiny points, but as fuzzy, glowing, 3D paint blobs. Each blob has a specific size, a specific tilt (rotation), a specific transparency (how see-through it is), and a specific color.

The authors of this paper asked a simple question: "Can we teach a computer to understand the shape of an object just by looking at these fuzzy, glowing blobs, even though they were originally designed just to make pretty pictures?"

Here is how they explored this, using some fun analogies:

The Experiment: Three Ways to Look at the Blobs

The researchers tested three different "brain architectures" (computer models) to see which one could best learn from these blobs.

The "Global Connector" (MLP): Imagine looking at a bag of mixed Lego bricks and trying to guess what you can build by just glancing at the whole pile at once, without looking at how any specific brick touches another. This method treats every piece of data as if it's equally important to every other piece, ignoring the actual 3D layout.
The "Independent Observer" (PointNet family): Imagine looking at each Lego brick individually, describing it, and then taking a "vote" to decide what the object is. This method looks at each blob on its own, then combines the opinions. It's very good at not getting confused if the bricks are shuffled around.
The "Local Neighbor" (Graph Neural Networks): Imagine looking at a brick and asking, "Who are my neighbors?" This method builds a map of which blobs are touching or close to each other, trying to understand the object by studying the relationships between neighbors.

The Results: What Worked and What Didn't

1. The "Pretty Picture" Features Helped (Sometimes)
The researchers found that using the extra information from the blobs (like their size, tilt, and transparency) was like giving the computer a superpower.

The Analogy: If you are trying to identify a wire chair versus a solid wooden chair, looking at just the "dots" (the points) is hard because they might look similar. But if you look at the "blobs," the wire chair is made of long, thin, semi-transparent blobs, while the wooden chair is made of short, thick, solid blobs.
The Finding: For some models, adding these extra "blob features" made them much smarter. For others, it actually made things more confusing, like trying to solve a puzzle with too many pieces.

2. The "Independent Observer" Won the Race
The models that looked at each blob individually and then voted (the PointNet family) were the most consistent winners. They were like a team of detectives who each gathered their own facts and then agreed on a conclusion. They worked well whether the data was perfect or messy.

3. The "Local Neighbor" Struggled
The models that tried to map out neighbors (Graph Neural Networks) had a harder time.

The Analogy: Imagine trying to navigate a city where the streets keep changing every time you look at them. Because the "blobs" are fuzzy and irregular, the computer kept getting confused about who was actually a neighbor to whom. The "neighbor" of one blob might be a completely different blob the next time the computer looked, causing the model to lose its way.
The Exception: One specific "neighbor" model (SplineCNN) did okay at the main task (identifying the object), but when the researchers asked it to prove it really understood the shape (by grouping similar objects together without being told the answers), it failed. It was like a student who could pass a multiple-choice test but couldn't explain the concept to a friend.

The Big Takeaway

The paper concludes that while 3D Gaussian Splatting creates beautiful, detailed images, using it for understanding shapes is tricky.

The "fuzzy blob" data is very rich, but standard computer brains designed for simple dots get confused by all the extra information (size, tilt, transparency).
The best approach right now is to use models that treat each blob as an independent piece of evidence rather than trying to map out complex neighborhoods between them.
The authors suggest that in the future, we might need to invent a new way to "sample" these blobs—instead of just picking the ones closest together, we should pick the ones that are most important for describing the object's shape.

In short: 3D Gaussian Splatting is a fantastic artist, but it's still learning how to be a good teacher for computers trying to understand 3D shapes.

Technical Summary: Learning Representations from 3D Gaussian Splats

Problem Statement
While 3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art technique for high-fidelity scene rendering and visualization, its potential for downstream scene understanding tasks, such as classification, remains underexplored. Unlike traditional 3D point clouds, which consist of unordered, sparse centroids, 3DGS representations comprise a set of Gaussian primitives characterized by heterogeneous attributes: spatial position, anisotropic scaling, orientation (quaternion), opacity, and view-dependent color (spherical harmonics). The core challenge addressed in this work is determining whether neural networks can successfully extract meaningful geometric representations from these features, which were originally optimized strictly for visual rasterization rather than geometric fidelity. The authors investigate whether the inclusion of auxiliary visual features (opacity, scale, rotation) enhances or hinders the learning of latent representations compared to standard centroid-based point clouds.

Methodology
The study conducts a comprehensive comparative evaluation of various geometric deep learning architectures adapted for 3DGS data. The research framework involves the following components:

Data Sources: The authors utilize two standard point cloud datasets (ModelNet40 and ShapeNet-29) and two dedicated Gaussian Splatting datasets (ShapeSplat and MACGS). ShapeSplat features objects reconstructed from 360-degree views, while MACGS relies on single-view supervision, resulting in partial reconstructions.
Input Representations: The models are tested with three distinct input configurations:
1. Centroids only ( $p$ ): Position coordinates $(x, y, z)$ .
2. Full Splat ($psqo$): Position, scale, quaternion, and opacity.
3. Auxiliary features only ($sqo$): Scale, quaternion, and opacity (excluding position) to isolate the discriminative power of non-geometric attributes.
Architectural Taxonomy: The authors benchmark three connectivity paradigms:
1. Global Connectivity: Multi-Layer Perceptrons (MLP) treating points as a flattened set.
2. Self-Connectivity: The PointNet family (PointNet, PointNet++, PointNeXt), which processes points independently and aggregates via symmetric pooling (e.g., max-pooling).
3. Local Connectivity: Graph Neural Networks (GNNs) including SplineCNN, Dynamic Graph Convolutional Neural Network (DGCNN), and Graph Attention Network (GAT). These models construct graphs where nodes are Gaussian primitives and edges represent local spatial or feature-space relationships.
Evaluation Metrics: The quality of learned embeddings is assessed through:
- End-to-End Classification: Standard accuracy on the training objective.
- Linear Probing: Training a linear classifier on frozen embeddings to measure the intrinsic discriminative power of the representation.
- Clustering Analysis: Unsupervised evaluation using K-Means (Adjusted Mutual Information - AMI) and Gaussian Mixture Models (Mean Log Ratio) to assess intra-class compactness and inter-class separability.

Key Results
The experimental results reveal distinct performance patterns across architectures and datasets:

Classification Performance:
- PointNet Family: Consistently achieves robust performance across all datasets. On ShapeSplat and MACGS, PointNet and PointNet++ reach accuracy levels of 84–86%.
- GNNs: Performance is highly variable. SplineCNN is the only GNN that competes closely with the PointNet family in end-to-end classification (≈85% on MACGS). In contrast, DGCNN and GAT generally struggle, often achieving <70% accuracy, particularly on the MACGS dataset.
- Feature Impact: Incorporating full splat attributes ($psqo$) generally improves performance for PointNeXt, DGCNN, and GAT on the MACGS dataset (e.g., PointNeXt gains +33.8 percentage points). However, for SplineCNN and PointNet++, coordinate-only inputs often perform comparably or better in specific clustering metrics.
Linear Probing & Clustering:
- A significant discrepancy exists between end-to-end accuracy and embedding quality. While PointNet family models maintain high linear probing scores (≈80%), GNNs like DGCNN and GAT suffer severe drops (often to ~20%), indicating that their high end-to-end accuracy may rely heavily on the classification head rather than robust latent representations.
- In clustering tasks (AMI and Log Ratio), PointNet and PointNet++ dominate, achieving high AMI scores (≈0.7 on ShapeSplat). SplineCNN shows a substantial drop in clustering performance relative to its classification scores, suggesting its learned embeddings lack the compactness required for unsupervised tasks.
- Feature Sensitivity: PointNet architectures often perform best with coordinate-only inputs or standalone Gaussian features ($sqo$), whereas GNNs generally require the full feature set to surpass MLP baselines, though they struggle to effectively fuse heterogeneous feature types.

Significance and Claims
The paper claims to provide the first systematic evaluation of latent representations learned by geometric deep learning architectures specifically for 3D Gaussian Splatting. The authors highlight several critical insights:

Architecture Suitability: While GNNs are theoretically well-suited for modeling local relationships, standard message-passing mechanisms (like those in DGCNN and GAT) struggle with the non-uniform distribution and heterogeneous attributes of Gaussian Splats. SplineCNN emerges as a notable exception, likely due to its local processing mechanisms resembling CNNs and its reliance on relative distances.
Feature Utility: The study demonstrates that auxiliary visual features (opacity, scale, rotation) can significantly enhance class separation, particularly for distinguishing objects with similar geometry but different visual textures (e.g., wire-like vs. solid structures). However, the efficacy of these features is architecture-dependent.
Representation Quality: The PointNet family proves to be the most robust and generalizable approach for 3DGS classification, maintaining high performance in both supervised and unsupervised settings. In contrast, many GNNs exhibit a gap between end-to-end accuracy and the quality of their learned embeddings.
Future Directions: The authors suggest that future work should focus on developing dedicated message-passing mechanisms that account for the distinct semantic meanings of Gaussian attributes. They also propose that standard preprocessing techniques like Farthest Point Sampling (FPS) may be suboptimal for 3DGS, as they ignore auxiliary features, and recommend exploring feature-aware sampling routines based on the probabilistic interpretation of Gaussians.

In conclusion, the work establishes that while 3DGS representations offer rich geometric and visual data, current standard geometric deep learning architectures require adaptation to fully leverage these properties, with the PointNet family currently offering the most reliable baseline for scene understanding tasks in this domain.

The Experiment: Three Ways to Look at the Blobs

The Results: What Worked and What Didn't

The Big Takeaway

Technical Summary: Learning Representations from 3D Gaussian Splats

More like this