3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

The paper presents 3DTV, a feedforward interpolation network that achieves real-time, high-quality sparse-view synthesis for interactive applications by combining Delaunay-based triplet selection with a pose-aware depth module, eliminating the need for scene-specific optimization or explicit proxies.

The Gist

Imagine you are at a concert, but you can only see the stage from three specific seats in the audience. Now, imagine you want to "teleport" your view to a spot right in the middle of the crowd, or even behind the drummer, without actually moving your body.

That is the problem 3DTV solves. It's a new computer program that lets you create a brand-new, realistic view of a scene using just three photos taken from different angles, and it does it instantly (in real-time).

Here is how it works, broken down with some everyday analogies:

1. The Problem: Too Much Data, Too Slow

Usually, to make a 3D movie where you can look around freely, you need hundreds of cameras or a supercomputer that takes hours to "learn" the scene. It's like trying to build a house by hand-picking every single brick one by one. It's accurate, but it's way too slow for things like video calls, VR games, or live sports broadcasts.

2. The Solution: The "Smart Trio" (Delaunay Triangulation)

Most computer programs try to guess which photos to use by looking for the ones closest to where you want to look. This often leads to messy results, like trying to build a table using three legs that are all on the same side.

3DTV uses a clever trick called Delaunay Triangulation.

• The Analogy: Imagine you have three friends standing in a circle. If you want to stand in the middle of them, you need them to surround you evenly. 3DTV mathematically picks the perfect "triangle" of three cameras that surround your desired new viewpoint.
• The Result: It ensures the three photos it uses are the best possible "team" to create a new angle, avoiding gaps and weird distortions.

3. The Engine: The "Ghost" Chef

Once it picks the three photos, it needs to blend them together. Old methods are like heavy, slow trucks trying to carry a massive load of data.

3DTV uses a lightweight network based on something called GhostNet.

• The Analogy: Imagine a master chef (the main convolution) who cooks a delicious soup. Instead of hiring 100 new chefs to make more soup, the master chef uses a "ghost" technique: they take the soup they already made and use a simple, cheap trick (like adding a specific spice or stirring it differently) to create new flavors without doing all the heavy lifting again.
• The Result: The computer does the heavy thinking once, then uses "cheap" tricks to generate the rest of the image. This makes it fast enough to run on a standard gaming laptop or phone.

4. The Secret Sauce: Depth and "Coarse-to-Fine"

The hardest part of making a new view is knowing what is in front of what (depth). If you get it wrong, people's faces might float in mid-air or look like they are melting.

3DTV builds the image in layers, like peeling an onion or sketching a drawing.

• The Analogy: First, it draws a rough, blurry sketch of the whole scene (Coarse). It asks, "Is the person generally here?" Then, it zooms in and adds details (Fine), asking, "Is that a nose or a mole?"
• The Magic: It uses a "Depth Module" to act like a 3D ruler. It doesn't just guess; it calculates how far away every pixel is. This allows it to "warp" the three photos so they fit together perfectly, hiding the parts that should be blocked (occlusions) and revealing the parts that should be visible.

5. Why This Matters: The "Magic Window"

The biggest breakthrough is that 3DTV doesn't need to learn the scene first.

• Old Way: To make a 3D model of your living room, you had to take a video, wait 10 minutes for the computer to "train" on your room, and then you could look around. If you moved a chair, you had to start over.
• 3DTV Way: It's a "feedforward" system. It's like a magic window that works on any scene instantly. You point three cameras at a room, and it instantly lets you walk around virtually. No training, no waiting.

Summary

Think of 3DTV as a real-time 3D teleportation machine.

1. It picks the best three photos to form a triangle around your new view.
2. It uses a lightweight "Ghost" brain to process the data quickly.
3. It builds the image layer by layer, using a 3D ruler to make sure everything lines up perfectly.
4. It does all this in 40 frames per second, meaning you can look around a virtual room as smoothly as you would in real life, without needing a supercomputer or a long wait time.

This technology could revolutionize VR/AR, video calls (where you can look around the person you are talking to), and live sports, letting you watch the game from any angle you want, instantly.

Technical Summary

1. Problem Statement

Real-time free-viewpoint rendering faces a fundamental trade-off between fidelity and efficiency.

• The Challenge: Traditional Novel View Synthesis (NVS) methods, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting, achieve high photorealism but typically require heavy per-scene optimization (minutes to hours of training) and dense camera arrays. This makes them unsuitable for interactive applications like AR/VR, telepresence, and live streaming where low latency and immediate rendering are critical.
• The Gap: Existing real-time or feedforward methods often struggle with sparse inputs (few cameras), leading to geometric artifacts (ghosting, floating structures), depth ambiguity under wide baselines, and temporal instability. Furthermore, many rely on heuristic nearest-neighbor view selection, which can result in poor angular coverage.

2. Methodology

The authors propose 3DTV, a feedforward network designed to synthesize novel views from only three sparse input cameras in real-time (40 FPS at 1k resolution) without per-scene retraining. The pipeline consists of four key stages:

A. Geometric View Selection (Delaunay Triangulation)

Instead of using simple k-Nearest Neighbor (k-NN) selection, which can yield poorly conditioned camera configurations, 3DTV employs a projected Delaunay triangulation strategy:

1. Projection: Camera centers are projected onto a fitted cylinder and then mapped to a 2D plane to remove depth bias.
2. Triangulation: A Delaunay triangulation is computed on this 2D plane.
3. Selection: For any target query view, the algorithm identifies the enclosing triangle in the triangulation. This ensures the selected triplet of source cameras provides balanced angular coverage and a geometrically consistent basis for interpolation.

B. Efficient Feature Extraction (GhostNet Backbone)

To meet real-time constraints, the network uses a lightweight hierarchical backbone based on GhostNet:

• Ghost Modules: Intrinsic feature maps are generated via standard convolution, while remaining channels are produced via inexpensive depthwise operations, reducing redundancy.
• Feature Pyramid: A 7-level feature pyramid is extracted.
• Context Aggregation: A lightweight Atrous Spatial Pyramid Pooling (L-ASPP) module is added at the deepest level to capture multi-scale context without significant computational overhead.

C. Coarse-to-Fine Depth Estimation & Refinement

The core of the synthesis is a coarse-to-fine depth pyramid that estimates depth and alpha maps for the target view:

• Plane-Sweep Stereo: At the coarsest level, 32 depth hypotheses are sampled.
• Recursive Refinement: For finer levels, the search space is refined using a local window around the upsampled depth prediction from the previous level.
• Grouped Correlation: Features from the three source views are warped to the target view using homographies induced by depth hypotheses. A grouped correlation volume is constructed to match features across views.
• Residual Learning: The network predicts depth residuals ($\Delta l$) and opacity maps rather than absolute values, stabilizing training and enabling sub-pixel accuracy.

D. Hierarchical Feature Fusion & Synthesis

• Confidence Weighting: A confidence prediction network generates per-view weights to handle occlusions and view-dependent effects.
• Fusion: Warped features are fused using the predicted weights and depth maps.
• Decoder: A strictly hierarchical decoder aggregates fused features, depth, and opacity to synthesize the final RGB image, ensuring global structure from coarse levels regularizes high-frequency details in fine levels.

3. Key Contributions

1. Geometric View Selection: A novel Delaunay-based triplet selection strategy that ensures geometric consistency and angular coverage, outperforming heuristic k-NN selection for sparse inputs.
2. Feedforward Architecture: A lightweight, depth-guided network that performs real-time interpolation from only three cameras without any per-scene optimization or retraining.
3. Coarse-to-Fine Depth Pyramid: A robust mechanism for handling wide baselines and occlusions by progressively refining depth hypotheses, replacing motion-centric flow reasoning with geometry-aware depth estimation.
4. Real-Time Performance: The method achieves 40 FPS at 1024×1024 resolution on an NVIDIA RTX 4090 (optimized with TensorRT), with a memory footprint of only 2.2 GB.

4. Experimental Results

The authors evaluated 3DTV on six diverse benchmarks (DNA Rendering, LLFF, MVHumanNet, RIFTCast, THuman2.1, ZJUMoCap) covering human captures and general scenes.

• Quality vs. Efficiency: 3DTV consistently outperforms recent real-time sparse-view baselines (e.g., GPS-Gaussian+, ENeRF) in PSNR, SSIM, and LPIPS metrics.
  • Example: On the DNA Rendering dataset, 3DTV achieved 25.7 PSNR (vs. 22.0 for GPS-Gaussian+) and 0.941 SSIM.
• Robustness: Unlike 2-view methods that suffer from floating artifacts and depth ambiguity, the 3-view geometric conditioning produces stable geometry and preserves fine details (faces, extremities).
• Generalization: Despite being trained only on synthetic data, the model generalizes robustly to real-world captures and diverse scene configurations. It also scales to higher resolutions (2048×2048) without fine-tuning.
• Comparison to Offline Methods: While offline optimization methods (e.g., Splatfacto-big) achieve slightly higher metrics, they require minutes of training per scene. 3DTV offers a near-optimal quality-efficiency trade-off for interactive applications.

5. Significance

3DTV represents a significant step forward in scalable real-time free-viewpoint video synthesis. By combining principled geometric reasoning (Delaunay triangulation) with efficient deep learning (GhostNet, coarse-to-fine depth), it bridges the gap between classical image-based rendering and modern neural rendering.

• Practical Impact: It enables low-latency multi-view streaming and interactive rendering for AR/VR and telepresence, removing the bottleneck of per-scene training.
• Artifact Reduction: The method effectively mitigates common sparse-view artifacts like ghosting and geometric instability, making it a practical solution for dynamic, real-world scenarios.
• Future Direction: The paper suggests that combining lightweight geometric constraints with feedforward neural synthesis is a promising path for future real-time 3D vision systems.

Limitations: The current method is constrained to three-view inputs and bounded interpolation (it struggles with large extrapolations beyond the camera hull). It also relies on foreground masks, which can be imperfect in real-world captures.