Fused-Planes: Why Train a Thousand Tri-Planes When You Can Share?

The paper introduces Fused-Planes, a novel 3D representation that significantly improves the training speed and memory efficiency of Tri-Plane NeRFs for large object collections by sharing a set of global base planes across objects while retaining high rendering quality.

The Gist

Imagine you are an architect tasked with building a massive city of 3D models. You need to create thousands of unique houses, cars, and faces.

In the past, the standard way to do this was like hiring a separate, dedicated construction crew for every single building. Even if you were building 1,000 identical-looking red brick houses, you would send 1,000 different crews to the site, each carrying their own full set of blueprints, tools, and materials. They would all work in isolation, ignoring the fact that they were all building the same type of house. This is incredibly wasteful: it takes forever, costs a fortune in materials, and fills up your warehouse with redundant blueprints.

This is exactly what the old method, called Tri-Planes, did in the world of 3D computer vision. It treated every 3D object as a unique project, ignoring the fact that all "cars" share a chassis, all "faces" share a nose, and all "chairs" share legs.

The paper introduces a new method called Fused-Planes. Think of it as a Master Blueprint System.

The Core Idea: The "Shared Library" vs. The "Personal Notebook"

Instead of building 1,000 separate crews, Fused-Planes sets up a central library of shared parts (the "Macro" component) and gives each object a tiny personal notebook (the "Micro" component).

1. The Shared Library (Macro Planes):
   Imagine a library filled with pre-made, high-quality 3D "ingredients." There are generic car wheels, standard face structures, and common chair legs. These are the Base Planes. They capture the structural similarities that all objects in a category share.

   • Analogy: Think of this like a "Lego base set." You don't need to invent the concept of a wheel; you just grab a wheel from the box.

2. The Personal Notebook (Micro Planes):
   For each specific object (e.g., your specific red Ferrari), you only need a tiny notebook that says: "Take 3 wheels from the library, paint them red, and add a spoiler."

   • Analogy: This is just a short list of instructions on how to customize the shared parts. It's incredibly small because it doesn't store the whole car, just the differences.

How It Works (The "Fused" Part)

When the computer needs to render a specific object, it doesn't start from scratch. It goes to the Shared Library, grabs the relevant base parts, and mixes them together according to the instructions in the Personal Notebook.

• Old Way: "Here is the entire 3D model of a car, stored in a massive file." (Heavy, slow to load).
• Fused-Planes Way: "Here is a tiny note: 'Mix 50% Library Part A, 30% Library Part B, and add a tiny red sticker.'" (Tiny file, instant to load).

Why This is a Game-Changer

The paper shows that this approach is a massive win for efficiency:

• Speed: Because the computer doesn't have to re-learn the basics of what a "car" is for every single car, it learns 7.2 times faster. It's like learning to bake 100 cakes by mastering the dough recipe once, rather than learning to bake dough 100 separate times.
• Memory: The storage savings are staggering. The "Ultra-Lightweight" version of this method uses 1,875 times less memory per object than the old way.
  • Analogy: If the old method required a warehouse the size of a football stadium to store the blueprints for 2,000 cars, Fused-Planes could fit all those blueprints in a single shoebox.
• Quality: Despite being so small and fast, the final images look just as good (or even better) than the old, bloated methods.

The Secret Sauce: The "Latent Space"

The paper also mentions using a "3D-aware latent space." Think of this as a translator.
Normally, computers see 3D objects as a chaotic mess of pixels (RGB space). It's hard to find patterns in chaos. The "Latent Space" is like a translator that converts that chaotic mess into a clean, organized spreadsheet where similar things (like "nose" or "headlight") are grouped together neatly. This makes it much easier for the computer to realize, "Hey, these 1,000 faces all have the same underlying structure," and build the Shared Library efficiently.

Summary

Fused-Planes is a smart way to build 3D worlds. Instead of reinventing the wheel for every single object, it builds a shared library of common parts and uses tiny, custom instructions to assemble unique objects.

• Result: You get the same beautiful 3D images, but you build them 7 times faster and store them in a space 1,000 times smaller.

It's the difference between carrying a full encyclopedia for every book you read, versus just carrying a small index card that tells you which pages of the encyclopedia to open.

Technical Summary

1. Problem Statement

Tri-Plane NeRFs have become the dominant representation for modeling large collections of 3D objects because they represent 3D scenes using three orthogonal 2D feature planes. This structure allows them to be seamlessly integrated with standard 2D vision models (e.g., CNNs, Diffusion models) for downstream tasks like generation, editing, and classification.

However, training Tri-Planes for large-scale datasets (thousands of objects) is computationally inefficient and resource-intensive. Current approaches train a separate, independent Tri-Plane for every single object in the dataset. This strategy:

• Ignores structural similarities shared across objects of the same class (e.g., all cars share wheels, chassis, and windows).
• Leads to massive redundancy in computation and memory usage.
• Creates a bottleneck where the cost of reconstructing the dataset often outweighs the benefits of the downstream tasks.

The paper addresses the need for a method that can model large object classes efficiently by leveraging shared structural information without sacrificing the planar properties that make Tri-Planes useful.

2. Methodology: Fused-Planes

The authors propose Fused-Planes, a novel object representation that decomposes the scene modeling task into shared global components and object-specific local components, trained within a 3D-aware latent space.

A. Micro-Macro Decomposition

Instead of learning a full Tri-Plane for each object $T_i$, Fused-Planes splits the representation into two parts:

1. Micro Component ($T^{mic}_i$): An object-specific plane that captures unique details and variations for the specific object $i$. This is learned independently for each object.
2. Macro Component ($T^{mac}_i$): A representation of class-level structural similarities. Instead of learning this from scratch for every object, it is computed as a weighted sum of a small set of globally shared base planes ($B = \{B_1, ..., B_M\}$).
   • Formula: $T^{mac}_i = \sum_{k=1}^{M} w^k_i B_k$
   • Here, $w^k_i$ are learned coefficients specific to object $i$, and $B_k$ are the shared base planes trained across the entire dataset.
   • The final representation is the concatenation: $T_i = T^{mic}_i \oplus T^{mac}_i$.

This approach drastically reduces the number of trainable parameters per object, as the heavy lifting of capturing common structures is offloaded to the shared base planes.

B. 3D-Aware Latent Space Training

Traditional Tri-Planes operate in RGB space. Fused-Planes trains in the latent space of an image autoencoder (Encoder $E_\phi$, Decoder $D_\psi$).

• Why? High-dimensional RGB space lacks the structure needed to effectively disentangle shared features from object-specific details. A 3D-aware latent space provides a continuous, structured encoding that facilitates learning shared patterns.
• Joint Training: Unlike previous latent NeRF works that pre-train a generic latent space, Fused-Planes jointly trains the autoencoder and the Fused-Planes representations. This ensures the latent space is specifically optimized for the micro-macro decomposition, preserving rendering quality while enabling speed.

C. Ultra-Lightweight Variant (Fused-Planes-ULW)

The authors introduce a variant where the Micro component is removed entirely ($F^{mic} = 0$). Objects are represented solely by the weighted sum of base planes. This sacrifices minor rendering quality for massive gains in memory efficiency.

3. Key Contributions

1. Novel Representation: Introduced Fused-Planes, the first method to explicitly integrate shared representations with planar structures, allowing for the sharing of structural priors across thousands of objects.
2. Efficiency Breakthrough: Achieved a 7.2× faster training speed and 3.2× lower memory footprint compared to standard Tri-Planes while maintaining comparable rendering quality.
3. Ultra-Lightweight Scaling: The ULW variant reduces per-object memory usage by 1875× compared to Tri-Planes, making it feasible to store thousands of 3D objects in minimal space.
4. Joint Latent Optimization: Demonstrated that jointly training the latent space with the scene representation avoids the quality degradation typically seen in latent NeRFs, achieving speed without compromise.
5. Multi-Class Scalability: Showed that the method scales effectively to multi-class datasets (e.g., mixing cars, sofas, and speakers) without significant performance drops.

4. Experimental Results

The method was evaluated on large-scale datasets including ShapeNet (Cars, Furniture, Speakers, Sofas) and Basel Faces (2,000 objects per experiment).

• Performance vs. Tri-Planes:
  • Training Time: 8.96 min vs. 64.32 min (Tri-Planes).
  • Memory Footprint: 0.48 MB per object vs. 1.5 MB (Tri-Planes).
  • Quality: Maintained similar PSNR/SSIM/LPIPS scores (e.g., 30.47 PSNR vs. 28.15 for Tri-Planes on ShapeNet Cars).
• Performance vs. K-Planes:
  • Fused-Planes is 8.4× faster and 854× more memory efficient than K-Planes, while closing the quality gap.
• Performance vs. Shared Representations (CodeNeRF):
  • Fused-Planes outperforms CodeNeRF in both rendering quality and training speed, while retaining the crucial planar structure for 2D model compatibility.
• Scaling: As the number of objects ($N$) increases, the total memory and training time for Fused-Planes scale linearly with a much smaller slope than baselines, proving its suitability for massive datasets.

5. Significance and Impact

• Democratizing 3D Research: By drastically reducing the computational cost of training large 3D datasets, Fused-Planes makes it feasible for researchers to train and deploy large-scale 3D models without requiring massive GPU clusters.
• Enabling 2D-to-3D Pipelines: Since Fused-Planes retains the planar structure, they can be directly reshaped into 2D tensors. This unlocks the use of powerful pre-trained 2D foundation models (like Stable Diffusion or CLIP) for 3D tasks (generation, editing, classification) on massive datasets, a task previously hindered by the inefficiency of training the 3D representations themselves.
• New State-of-the-Art: The paper establishes a new benchmark for resource efficiency in planar scene representations, proving that "sharing" structural knowledge is a viable and superior strategy to independent object modeling.

In conclusion, Fused-Planes solves the scalability bottleneck of Tri-Plane NeRFs by introducing a shared base-plane architecture within a jointly optimized latent space, offering a highly efficient, high-quality solution for large-scale 3D reconstruction and generation.