⚡ electrical engineering

Fast Wave-optics Rendering of Multiplane Images for 3D Holographic Displays

This paper proposes a wave-optics rendering pipeline that converts multiplane images (MPIs) into holograms, achieving unprecedented speedups of up to 250,000x over state-of-the-art methods while maintaining high image quality and superior performance in 3D focal stack and 4D light field reconstruction.

Original authors: Brian Chao, Dario Seyb, Nathan Matsuda, Oliver Cossairt, Yang Zhou, Douglas Lanman, Gordon Wetzstein, Grace Kuo, Changwon Jang

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brian Chao, Dario Seyb, Nathan Matsuda, Oliver Cossairt, Yang Zhou, Douglas Lanman, Gordon Wetzstein, Grace Kuo, Changwon Jang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to build a movie screen that doesn't just show a flat picture, but actually tricks your eyes into seeing a real, three-dimensional object floating in mid-air. This is the dream of holographic displays. To make this happen, scientists use a special screen called a Spatial Light Modulator (SLM) that acts like a super-fast, microscopic puppet master. Instead of just lighting up pixels, it bends light waves to create interference patterns, essentially sculpting light itself so that when it hits your eye, it looks like a solid object is there.

However, there's a catch: calculating how to bend all those light waves is incredibly hard. It's like trying to predict how every single drop of water in a tsunami will splash if you throw a pebble in it, but you have to do it a million times a second. The current methods are either too slow to be useful in real-time (like trying to solve a giant puzzle by hand) or they produce images that look "glitchy" with weird halos and light leaking through objects that should be solid. This paper tackles that exact problem: how to make these 3D holograms fast enough to watch, without losing the magic of them looking real.

The researchers, a team from Stanford and Meta, propose a clever new way to do this called MPI-based Computer-Generated Holography. Think of a 3D scene not as a solid block of clay or a million tiny floating dots, but as a stack of transparent glass sheets, like a deck of cards. Each card has a picture painted on it, and they are spaced out at different distances. This is what they call a "Multiplane Image" (MPI).

In the past, trying to turn these stacks of cards into a hologram was slow if you treated every tiny dot on the cards individually, or it looked bad if you treated the whole card as a single flat layer. The authors found a "sweet spot." They developed a pipeline that takes these stacks of transparent cards and mathematically blends them together using wave optics (the physics of how light waves ripple and overlap).

Here is the magic trick they discovered: By treating the scene as a manageable stack of layers (usually fewer than 100) and adding a little bit of "randomness" to the light waves on each card, they can blend them into a single hologram incredibly fast. It's like taking a messy pile of transparent overlays and snapping them together instantly, rather than trying to calculate the path of every single grain of sand in the pile.

The results are staggering. When they tested their method, they found it was up to 250,000 times faster than the previous best method that used millions of tiny 3D dots (Gaussians). But speed isn't the only win. While other fast methods made images that looked blurry or had "light leaks" (where you could see through solid objects), their MPI method created holograms that looked just as sharp and realistic as the slow, super-complex ones. They showed this in computer simulations and with real-world experiments using a laser and a camera, proving that the holograms could reconstruct accurate 3D focus stacks (where you can focus on the front or back of an object) and 4D light fields (where the view changes naturally as you move your head).

Essentially, this paper suggests that by organizing 3D scenes into a smart stack of transparent layers and using a specific wave-optics recipe, we can finally make holographic displays fast enough to be practical for things like Virtual Reality (VR) and Augmented Reality (AR), without sacrificing the realistic look that makes them cool. They didn't just suggest it; they measured it, showing that this approach bridges the gap between "too slow to use" and "too blurry to believe."

Technical Summary: Fast Wave-optics Rendering of Multiplane Images for 3D Holographic Displays

Problem Statement
Holographic near-eye displays offer the potential for physically accurate, immersive 3D visual experiences by reconstructing wavefronts. However, their practical deployment is severely constrained by the extreme computational cost of Computer-Generated Holography (CGH). Existing CGH algorithms face a fundamental trade-off between reconstruction fidelity and computational efficiency:

Primitive-based methods (e.g., point clouds, meshes, Gaussian splats) offer high visual fidelity but require propagating millions of individual wavefronts, resulting in prohibitive runtimes unsuitable for real-time applications.
Layer-based methods (e.g., RGBD images) are computationally efficient but suffer from significant artifacts, such as light leakage, halo effects, and incorrect occlusion behavior at depth transitions, due to the lack of continuous volumetric information.
Optimization-based methods can achieve high fidelity but require iterative updates, making them too slow for interactive rates.

There is a critical need for a 3D scene representation and rendering pipeline that bridges the gap between the efficiency of layered inputs and the visual quality of densely populated primitives.

Methodology
The authors propose a novel wave-optics rendering pipeline that utilizes Multiplane Images (MPIs) as an intermediate 3D representation for efficient hologram synthesis. MPIs represent a scene as a set of $L$ RGBA images defined at discrete depth planes, allowing for novel view synthesis via planar homographies and alpha compositing.

The proposed pipeline consists of four key steps (illustrated in Fig. 2):

Representation: The input is a set of camera frustum-aligned MPIs, where each layer $i$ contains color ( $c_i$ ) and opacity ( $\alpha_i$ ) channels at depth $d_i$ .
Wavefront Transformation: Each RGBA image slice is transformed into a complex wavefront aligned with the Spatial Light Modulator (SLM) plane. This is achieved by applying a unique random phase mask ( $\phi_i$ ) to each slice.
Sequential Wave Propagation and Compositing: The complex wavefronts are propagated and composited sequentially from the back-most layer to the front-most layer. The authors utilize a recursive formulation (Eq. 2) based on the angular spectrum method and a recently proposed wave-optics alpha blending technique for random-phase wavefronts. This process accounts for occlusion and interference between layers.
Time Multiplexing: To further expand the eyebox and improve space-bandwidth product utilization, multiple hologram frames ( $T$ ) with different random phase patterns are generated and time-multiplexed. The final reconstructed intensity is the time-averaged accumulation of these frames.

This approach allows the system to convert MPIs—traditionally limited to ray-optics rendering—into complex-valued holograms capable of reconstructing accurate 3D focal stacks and 4D light fields under a full wave-optics model.

Key Contributions

A Novel CGH Algorithm: The paper introduces a direct CGH pipeline that composites MPIs into complex holograms with optional time multiplexing.
Performance Analysis: A detailed evaluation comparing MPI-based holograms against existing baselines (RGBD-based CGH and the state-of-the-art primitive-based Random-phase Wave Splatting, or RPWS). The study analyzes the impact of the number of depth layers on both image quality and runtime.
Experimental Validation: Extensive validation using both simulation and experimentally captured results on diverse 3D scene datasets (NeRF Blender and Mip-NeRF 360). The results demonstrate accurate 3D focal stack reconstruction and 4D light field parallax.

Results

Efficiency: The MPI-based method achieves speedups of up to 250,000× compared to the state-of-the-art primitive-based RPWS algorithm. The runtime scales approximately linearly with the number of depth layers, similar to RGBD methods, but with significantly higher quality.
Quality: MPI-based holograms achieve image quality (measured by SSIM and PSNR) comparable to the high-fidelity RPWS method, significantly outperforming conventional RGBD-based CGH.
Artifact Reduction: Unlike RGBD holograms, which exhibit light leakage, halos, and "depth-tearing" at occlusion boundaries, MPI holograms reconstruct natural defocus blur and accurate occlusion effects even with a small number of layers (e.g., 4 layers).
Scalability: Perceptual quality saturates beyond approximately 30 layers, suggesting that MPI-based CGH offers an optimal balance for interactive applications without requiring excessive computational resources.

Significance and Claims
The paper claims that MPI-based CGH successfully resolves the long-standing trade-off between rendering efficiency and visual fidelity in holographic displays. By leveraging the structural sparsity of MPIs (typically fewer than 100 layers) rather than millions of primitives, the method enables the synthesis of high-quality holograms that accurately reconstruct 3D focal stacks and 4D light fields.

The authors state that this significant advance in rendering efficiency "paves the way for practical and real-time computer-generated holography for interactive AR and VR experiences." While the current implementation is not yet real-time, the massive speedup over primitive-based methods suggests that combining this pipeline with real-time synthesis networks could achieve the necessary performance for future holographic near-eye displays. The work emphasizes that MPIs provide the "best of both worlds," matching the visual fidelity of Gaussian-based approaches while maintaining the computational efficiency of RGBD-based approaches.

Technical Summary: Fast Wave-optics Rendering of Multiplane Images for 3D Holographic Displays

More like this