💻 computer science

C3G: Learning Compact 3D Representations with 2K Gaussians

C3G is a novel feed-forward framework that reconstructs and understands 3D scenes from sparse, unposed views by generating a compact set of essential 3D Gaussians guided by learnable tokens and self-attention, thereby significantly reducing memory overhead while improving novel view synthesis and scene understanding compared to existing methods that rely on redundant per-pixel Gaussians.

Original authors: Honggyu An, Jaewoo Jung, Mungyeom Kim, Chaehyun Kim, Minkyeong Jeon, Jisang Han, Kazumi Fukuda, Takuya Narihira, Hyuna Ko, Junsu Kim, Sunghwan Hong, Yuki Mitsufuji, Seungryong Kim

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Honggyu An, Jaewoo Jung, Mungyeom Kim, Chaehyun Kim, Minkyeong Jeon, Jisang Han, Kazumi Fukuda, Takuya Narihira, Hyuna Ko, Junsu Kim, Sunghwan Hong, Yuki Mitsufuji, Seungryong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room using only a few photos taken from different angles, but you don't have a ruler or a map to tell you exactly where the camera was. This is a tricky puzzle for computers.

Most current computer programs try to solve this by placing a tiny "3D dot" (called a Gaussian) for every single pixel in your photos. If you have a high-resolution photo, that means millions of dots. It's like trying to describe a whole city by listing the exact location of every single brick in every building. It's incredibly heavy, slow, and often results in a messy, blurry model because the computer gets confused by all the extra, unnecessary dots.

C3G (Compact 3D Gaussians) is a new method that changes the game. Instead of placing a dot for every pixel, it acts like a smart, efficient architect who only places a few key markers where they actually matter.

Here is how it works, broken down into simple concepts:

1. The "Smart Search Team" vs. The "Brick-by-Brick" Approach

The Old Way (Pixel-Aligned): Imagine a construction crew that sends a worker to every single square inch of a wall to place a brick. They end up with millions of bricks, many of which are in the wrong place or overlapping. It takes forever to build and requires a massive warehouse to store them.
The C3G Way (Learnable Queries): Imagine a team of 2,000 smart scouts (called "learnable tokens"). Instead of checking every inch, these scouts are trained to look at the photos and ask, "Where are the important parts of this room?"
- One scout might say, "I'll cover the chair."
- Another says, "I'll cover the floor."
- Another says, "I'll cover the wall."
- They ignore the empty air and the redundant details.
- The Result: They build the entire room using only 2,000 dots instead of millions. This is about 65 times fewer dots than the old methods, yet the room looks just as good, if not better.

2. The "Group Chat" for Understanding

How do these 2,000 scouts know what to do? They use a "group chat" (a Transformer architecture).

They look at all the photos together.
They talk to each other to agree on what the room looks like.
If Scout A sees a chair in Photo 1 and Scout B sees the same chair in Photo 2, they realize, "Hey, we are both looking at the same thing!"
Because they agree on the location, they can place their dot exactly where the chair is, creating a clean, sharp 3D model without the confusion of the old "millions of dots" method.

3. The "Universal Translator" for Features

One of the hardest things for computers is taking a 2D picture of an object and understanding what it is (e.g., "that's a chair") from different angles. Usually, the computer gets confused because the chair looks different from the left side than the right side.

C3G has a special trick called C3G-F.

Because the scouts (the 2,000 dots) already agreed on where the chair is, C3G-F can take any "description" of the chair from any photo and attach it to that specific dot.
It's like having a universal translator that ensures the word "chair" means the same thing whether you are looking at it from the front, back, or side.
This allows the computer to not just see the shape, but to understand the scene (e.g., "This is a bedroom with a bed and a table") with incredible accuracy, even though it's using very few dots.

4. Why This Matters (The "Lightweight" Advantage)

The paper claims that by using this "smart scout" method instead of the "brick-by-brick" method:

Memory: It uses 15 times less memory. You could fit this model on a device that the old models couldn't even run.
Speed: It renders (draws) new views of the room much faster.
Quality: Despite using fewer dots, the images look sharper and the 3D understanding is more accurate.

Summary Analogy

Think of the old method as trying to draw a portrait by placing a drop of paint on every single square millimeter of the canvas. It's messy, expensive, and slow.

C3G is like a master painter who looks at the subject, identifies the key features (eyes, nose, mouth, hair), and uses just a few precise brushstrokes to capture the entire essence of the face. It's faster, cheaper, and surprisingly, the result is often clearer because there's no "noise" from unnecessary paint.

The paper demonstrates that you don't need millions of tiny pieces to understand a 3D world; you just need the right pieces in the right places.

1. Problem Definition

The paper addresses the challenge of reconstructing and understanding 3D scenes from unposed, sparse multi-view images in a feed-forward manner.

Limitations of Current Methods: Recent state-of-the-art approaches (e.g., NoPoSplat, LSM) rely on per-pixel 3D Gaussian Splatting. These methods predict one or multiple Gaussians for every pixel in the input images.
- Redundancy: This leads to millions of Gaussians (e.g., 131K+), many of which are redundant or misaligned in 3D space, causing artifacts.
- Computational Overhead: The sheer volume of primitives creates massive memory consumption and computational bottlenecks, especially when lifting 2D semantic features to 3D.
- Feature Compression: To manage the overhead, existing methods often compress semantic features into low-dimensional embeddings, resulting in information loss and sub-optimal scene understanding.
Core Question: Do we truly need pixel-aligned Gaussians to reconstruct and understand 3D scenes? The authors argue that, similar to human cognition, a compact, semantically meaningful abstraction of key objects and spatial relationships is sufficient and more efficient.

2. Methodology

The authors propose C3G (Compact 3D Gaussians), a novel feed-forward framework consisting of two main components: C3G-G (Gaussian Decoder) and C3G-F (Feature Decoder).

A. C3G-G: Compact 3D Gaussian Decoder

Instead of predicting Gaussians per pixel, C3G-G uses a learnable query-based approach to discover essential 3D locations.

Architecture:
- Visual Encoder: Extracts multi-view features using a pre-trained encoder with geometric priors (default: VGGT).
- Learnable Query Tokens: A compact set of $N$ learnable tokens (e.g., $N=2048$ ) is introduced. These tokens act as abstraction units.
- Transformer Processing: The query tokens and image features are concatenated and processed through Transformer layers with self-attention.
  - Tokens learn to aggregate relevant visual information from specific regions across all input views.
  - Tokens exchange information to avoid redundancy and ensure comprehensive scene coverage.
- Gaussian Head: The refined query tokens are decoded into 3D Gaussian parameters (position $\mu$ , covariance $\Sigma$ , opacity $\sigma$ , and color $c$ ) via lightweight MLPs.
Training:
- Trained solely on photometric reconstruction loss (MSE + LPIPS) for novel view synthesis. No ground-truth depth or segmentation labels are required.
- Progressive Low-Pass Filtering: To stabilize training and prevent Gaussian collapse, the authors adopt a strategy (from RAIN-GS) where the projected 2D Gaussian size is annealed from large to small during training. This allows the model to learn coarse positions first before refining details.

B. Emergent Properties

A key finding is that without explicit supervision on where to place Gaussians, the model emergently learns to allocate tokens to geometrically coherent regions.

Attention Visualization: The attention maps show that each query token focuses sharply on spatially consistent regions across multiple views, effectively discovering multi-view correspondences automatically.

C. C3G-F: View-Invariant Feature Decoder

Leveraging the emergent attention patterns from C3G-G, the authors introduce a method to lift arbitrary 2D features to 3D without compression.

Mechanism:
- C3G-F reuses the attention weights (Query and Key projections) learned by C3G-G but initializes new learnable feature tokens and value projections.
- It aggregates features from any visual encoder (e.g., LSeg, MaskCLIP, DINOv3) by attending to the same spatial regions identified by the Gaussian tokens.
Advantages:
- No Backward Mapping: Eliminates the need for expensive backward mapping to find 3D correspondences.
- No Heuristic Aggregation: Uses learned attention weights as interpolation weights to resolve multi-view feature inconsistencies naturally.
- No Compression: Since the number of Gaussians is small (2K), features can be stored directly without dimensionality reduction, preserving rich semantic information.

3. Key Contributions

Compact Representation: Demonstrates that a scene can be accurately represented by only ~2,000 Gaussians (approx. 65× fewer than per-pixel methods), drastically reducing memory usage (4.1MB vs. 61.5MB) and computational cost.
Query-Based Decoding: Introduces a learnable token mechanism that replaces rigid per-pixel prediction, allowing the model to focus computational capacity on geometrically significant regions.
Emergent Correspondence: Shows that self-attention in a feed-forward transformer naturally learns multi-view correspondences and spatial coherence without explicit supervision.
Efficient Feature Lifting: Proposes C3G-F, which leverages learned attention to lift high-dimensional, view-invariant features to 3D without compression, enabling superior 3D scene understanding.
State-of-the-Art Performance: Achieves competitive or superior performance in Novel View Synthesis (NVS) and 3D Open-Vocabulary Segmentation while being significantly faster and more memory-efficient.

4. Experimental Results

Novel View Synthesis (RealEstate10K)

Efficiency: Uses only 2K Gaussians compared to 131K+ in per-pixel methods.
Quality: Achieves 23.89 PSNR (without test-time optimization), comparable to methods using 65× more Gaussians.
Speed: Significantly faster inference and rendering (e.g., 2742 FPS vs. ~400 FPS for baselines).
Optimization: With short Test-Time Optimization (TTO), C3G reaches 30.25 PSNR, outperforming all baselines.

3D Scene Understanding (ScanNet & Replica)

Segmentation: Outperforms previous feed-forward methods (LSM, CF3) and matches or exceeds per-scene optimization methods in mIoU (e.g., 0.513 on ScanNet vs. 0.503 for LSM).
Feature Fidelity: The ability to lift features without compression leads to more accurate open-vocabulary segmentation maps compared to methods that compress features.

Multi-View Feature Correspondence

Performance: When used as a feature upsampler/aggregator, C3G significantly improves PCK@10px scores for correspondence tasks across various encoders (VGGT, DINOv2, DINOv3).
Consistency: The aggregated features show superior multi-view consistency compared to raw encoder outputs or features processed by other upsamplers (e.g., AnyUp).

5. Significance

Paradigm Shift: C3G challenges the prevailing "per-pixel" paradigm in feed-forward 3D reconstruction, proving that sparse, learned primitives are more effective than dense, redundant ones.
Scalability: The drastic reduction in memory and compute makes high-fidelity 3D reconstruction and understanding feasible for real-time applications, robotics, and edge devices.
Generalizability: The framework is model-agnostic regarding the visual encoder and feature type, allowing it to integrate with various foundation models (e.g., SAM, LLMs) for future 3D scene understanding tasks.
Efficiency: By eliminating the need for autoencoders to compress features, the method avoids information loss and accelerates training convergence (7.1× faster convergence reported in ablation studies).

In summary, C3G provides a highly efficient, high-quality, and scalable solution for 3D scene reconstruction and understanding by learning to "ask the right questions" (via query tokens) rather than blindly processing every pixel.