Technical Summary: Fast and Lightweight Novel View Synthesis with Differentiable Multiplane Image

Problem Statement

Novel View Synthesis (NVS) has advanced significantly with methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, existing approaches face critical trade-offs between rendering speed, model size, and training efficiency.

• NeRF and 3DGS: While capable of high-quality synthesis, they often require dense multi-view observations for optimization. Their training is time-consuming, and the resulting models can be computationally expensive.
• Feed-forward 3DGS: Recent feed-forward variants reduce optimization time but typically employ a "pixel-aligned" formulation where each pixel maps to a Gaussian. This creates severe redundancy, generating millions of Gaussians from sparse inputs, which hinders deployment on resource-constrained devices like mobile platforms.
• Traditional Multiplane Images (MPI): MPI offers a compact, discrete representation using planar layers, which is more efficient than continuous volumetric fields. However, existing MPI methods often rely on feed-forward predictions from single images, limiting them to the camera frustum. They struggle with disocclusions, depth discretization artifacts, and repetitive texture patterns, especially under sparse-view conditions or wide camera motions.

Methodology

The authors propose Multi-view MPI, a framework that reimagines MPI as a learnable, differentiable representation optimized via multi-view supervision, augmented by a generative neural enhancer.

1. Geometric Initialization via Foundation Models

To overcome the lack of reliable geometric priors in raw sparse images, the method utilizes large visual foundation models (specifically PI3 and VGGT).

• Process: Given a set of sparse input images, PI3 estimates camera poses, dense depth maps, and per-pixel confidence scores.
• Initialization: These outputs are used to back-project pixels into 3D point clouds. The point clouds are aligned to a world coordinate system and segmented based on depth, then projected onto multiple planes distributed along the depth axis to initialize the MPI structure.

2. Expanded MPI and Differentiable Optimization

Unlike traditional MPI predictors that hallucinate occluded regions, this approach treats MPI as a set of learnable parameters.

• Expanded Frustum: To handle viewpoints beyond the initial reference frustum, the method employs an "expanded" view angle. Larger planes are defined to cover a wider spatial range, but rendering is restricted to the target camera's field of view via homography transformation.
• Differentiable Rendering: The MPI layers (density and color) are optimized directly via volume rendering and gradient descent. This bypasses the need for Multi-Layer Perceptrons (MLPs) to decode features, allowing for efficient parallelization and faster convergence.
• Optimization Objective: The system minimizes Mean Squared Error (MSE) between rendered views and ground truth, as well as between rendered pseudo-views and their refined versions.

3. Neural Enhancer with Single-Step Diffusion

To address artifacts (e.g., noise, flickering, and missing geometry) inherent in sparse-view synthesis, the pipeline integrates DIFIX, a fast one-step diffusion model, in two capacities:

• During Optimization (Distillation): The system renders intermediate novel views, passes them through DIFIX to remove artifacts and generate "clean" pseudo-views, and uses these refined images as additional supervision to update the MPI parameters. This distills strong generative priors into the 3D representation.
• During Inference (Post-processing): DIFIX acts as a lightweight post-processor to further enhance the final rendered output, reducing artifacts caused by insufficient supervision.

Key Contributions

1. Multi-view MPI Framework: A novel NVS framework that formulates MPI as a learnable planar representation optimized via multi-view supervision. This enables broader viewpoint exploration and 3D-consistent rendering while avoiding the redundancy of pixel-aligned Gaussian methods.
2. Dual-Role Neural Enhancer: The integration of a one-step diffusion model (DIFIX) serves both as a training prior to refine the underlying 3D geometry and as a real-time post-processor to enhance visual fidelity.
3. Efficiency and Compactness: The method achieves a balance between high rendering quality and low resource consumption, making it suitable for mobile deployment.

Experimental Results

The method was evaluated on the LLFF (real-world forward-facing) and NeRF Synthetic datasets, comparing against NeRF, PixelNeRF, DeepView, and AnySplat.

• Rendering Speed: The proposed method achieves 135.62 FPS, significantly outperforming NeRF (0.212 FPS) and AnySplat (103.72 FPS).
• Model Size: The model size is 22.83 MB, which is approximately 14.8% of the size of AnySplat (153.85 MB) and smaller than PixelNeRF (112.82 MB).
• Quality:
  • On the LLFF dataset, the method achieved the highest PSNR (25.469) among the compared methods.
  • On the NeRF Synthetic dataset, it achieved the highest PSNR (29.161).
  • Qualitative analysis shows sharper structures and fewer artifacts (e.g., depth discretization, texture repetition) compared to DeepView and PixelNeRF.
• Ablation Studies: Removing PI3 initialization, differentiable optimization, or the diffusion enhancer resulted in significant drops in PSNR (up to 10.4 dB), confirming the necessity of each component.

Significance and Claims

The paper claims that by revisiting the MPI representation and combining it with modern foundation models and generative priors, it is possible to overcome the limitations of both continuous volumetric fields (NeRF) and redundant explicit representations (pixel-aligned 3DGS).

The authors assert that their approach:

• Bridges the gap between high-quality synthesis and practical deployment on mobile devices by drastically reducing model size and increasing rendering speed.
• Mitigates the "hallucination" problem of traditional MPI methods by using differentiable optimization with multi-view supervision rather than relying solely on feed-forward prediction.
• Provides a robust pipeline that handles sparse-view inputs effectively, producing 3D-consistent results even under wide camera motions, a scenario where traditional MPI methods typically fail.

The work positions itself as a solution for scenarios requiring fast, lightweight, and high-fidelity 3D reconstruction, such as robotics and 3D photography, without the heavy computational burden of current state-of-the-art methods.