💻 computer science

One View Is Enough! Monocular Training for In-the-Wild Novel View Generation

The paper introduces OVIE, a novel monocular training framework that synthesizes novel views from unpaired internet images by leveraging a depth-based geometric scaffold with masked losses during training, resulting in a geometry-free inference model that achieves state-of-the-art zero-shot performance with significantly faster speed.

Original authors: Adrien Ramanana Rahary, Nicolas Dufour, Patrick Perez, David Picard

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adrien Ramanana Rahary, Nicolas Dufour, Patrick Perez, David Picard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a single photograph of a cathedral. You can see the front, but you can't see the back. A human brain is amazing; it can instantly imagine what the cathedral looks like from the side, the back, or even from a bird's-eye view, filling in the missing parts based on what it knows about how buildings work.

For a long time, computers have struggled to do this. To teach a computer to "see" around corners, researchers usually needed multi-view datasets—basically, thousands of photos of the same object taken from every possible angle. It's like trying to teach someone to drive a car only by showing them a video of a specific car on a specific track. If you put them in a different car on a different road, they get lost. This requirement for perfect, paired data limited computers to very specific, boring scenarios.

Enter OVIE (One View Is Enough! Monocular Training for In-the-Wild Novel View Generation). This new method is like a wizard that can learn to drive any car on any road just by looking at a single photo.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Pair" Bottleneck

Traditionally, to teach a computer to generate a new view, you had to show it a "Source" photo and a "Target" photo (the same scene from a different angle) side-by-side. It's like a teacher holding up two flashcards and saying, "See how the tree moved? Now guess where it goes."

The Issue: Finding these perfect pairs on the internet is nearly impossible. Most photos are just single snapshots. This meant AI could only learn from tiny, curated collections of data.

2. The Solution: The "Magic Scaffold"

The authors realized they didn't need real "Target" photos. They could fake them using a clever trick.

The Trick: They use a pre-trained "Depth Estimator" (a tool that guesses how far away things are in a flat photo).
The Analogy: Imagine taking a 2D drawing of a house and using a magic ruler to turn it into a 3D clay model. Once it's a 3D model, you can walk around it in your mind and take a picture from the back.
The Catch: Since the original photo was flat, the "clay model" is a bit messy. When you walk around it, you might see "holes" where the back of the house should be (because the AI didn't know what was there).
The Fix: The AI is trained to ignore those holes. It only learns from the parts of the "fake photo" that make sense. It's like a student studying a map that has some missing pieces; they learn the roads that are there and get really good at guessing the rest.

3. The Training: Learning from the "Wild"

Instead of using a tiny, perfect dataset, OVIE was fed 30 million random photos from the internet.

The Analogy: Imagine trying to learn a language. Most students use a textbook with perfect grammar (the old methods). OVIE, however, spent 30 million hours eavesdropping on conversations in a busy, chaotic city (the "wild" internet).
Because it saw so many different things—indoor rooms, outdoor streets, paintings, and objects—it learned the general rules of how 3D space works, rather than just memorizing specific rooms.

4. The Result: Super Speed and Super Generalization

Once trained, OVIE doesn't need the "magic ruler" (depth estimator) anymore. It just takes a photo and a desired camera angle and instantly spits out the new view.

Speed: It is 600 times faster than the next best method.
- Analogy: If the old methods were like a snail trying to paint a masterpiece, OVIE is a jet fighter painting a masterpiece while flying at Mach 1. It can run in real-time, letting you "fly" through a room just by moving your mouse.
Versatility: Because it learned from 30 million random images, it works on things it was never explicitly taught. It can generate new views of a painting, a video game screenshot, or a real-world street, all with the same ease.

Why This Matters

Before this, if you wanted to explore a 3D version of a historical building or a piece of art, you needed a team of photographers to take hundreds of pictures. Now, with OVIE, you can take one single photo and explore the scene from any angle, instantly.

It turns the entire internet of 2D photos into a 3D playground, proving that you don't need perfect data to learn perfect geometry—you just need a lot of data and a smart way to learn from it.

1. Problem Statement

Monocular Novel View Synthesis (NVS) aims to generate a plausible view of a scene from a new camera pose given only a single source image.

Current Limitation: State-of-the-art methods rely heavily on multi-view supervision (datasets with posed, static multi-view captures like RealEstate10K or DL3DV). These datasets are scarce, cover only narrow visual domains, and often fail to generalize to "in-the-wild" images (e.g., paintings, historical archives, or dynamic scenes).
The Bottleneck: The dependency on multi-view data restricts the scale and diversity of training, preventing models from achieving the generalization capabilities of human vision, which can imagine 3D structures from a single 2D observation.
Goal: To train a geometry-free, feed-forward NVS model entirely on unpaired, single-image collections (internet-scale data) without requiring multi-view ground truth, depth annotations, or controlled recording conditions.

2. Methodology: OVIE

The core insight of OVIE is that modern monocular depth estimators can serve as a geometric scaffold to generate "pseudo-ground-truth" training pairs from single images. The method operates in two phases:

A. Training Phase: Pseudo-Pair Generation

Instead of using real multi-view pairs, OVIE constructs training pairs on-the-fly:

3D Lifting: A frozen, pre-trained metric monocular depth estimator (specifically MoGe-2) processes the source image ( $I_0$ ) to predict a depth map and surface normals. This lifts the 2D image into a 3D point cloud ( $P$ ) with metric scale.
Pose Sampling & Reprojection: A relative camera transformation ( $T_{0\to1} \in SE(3)$ ) is sampled from a distribution of plausible viewpoint changes. The point cloud is rigidly transformed and reprojected onto the target image plane.
Pseudo-Target Creation: This reprojection yields a sparse "pseudo-target" image ( $I^*_1$ $I_{1}^{*}$ ) and a binary validity mask ( $M$ $M$ ).
- $M=1$ for valid, visible pixels.
- $M=0$ for disoccluded regions, occlusions, backfaces, or out-of-frame content.
Metric Awareness: Because the depth estimator is metric, the sampled camera translations are in real-world units (meters), providing the model with metric scale awareness without manual calibration.

B. Model Architecture

OVIE is a geometry-free, feed-forward image-to-image network:

Input: Source image ( $I_0$ ) and a 7D camera transformation vector (3D translation + unit quaternion for rotation).
Backbone: A convolutional encoder downsamples the image, followed by a ViT-B (Vision Transformer) bottleneck where the camera pose is injected via Adaptive Layer Normalization (AdaLN). A symmetric convolutional decoder upsamples the features to the original resolution.
Output: A synthesized novel view ( $\hat{I}_1$ ) in pixel space. No intermediate 3D representation is stored or rendered during inference.

C. Training Objective

The loss function is designed to handle the sparsity of the pseudo-targets:

Masked Reconstruction Loss ( $L_{recon}$ ): An $L_2$ loss applied only to valid regions ( $M \odot \hat{I}_1$ vs. $M \odot I^*_1$ ).
Perceptual Loss ( $L_{perc}$ ): Combines LPIPS and P-DINO (feature loss from a pretrained DINO model). Both the prediction and pseudo-target are masked before feature extraction to avoid penalizing unobserved regions.
Adversarial Loss ( $L_{adv}$ ): A PatchGAN discriminator distinguishes between the source image ( $I_0$ ) and the generated view ( $\hat{I}_1$ ). Since the pseudo-target is incomplete, the source image serves as the "real" distribution proxy to enforce realistic texture synthesis in unobserved regions.
Total Loss: $L_{total} = L_{recon} + L_{perc} + \lambda_{adv} \cdot w_{adap} \cdot L_{adv}$ .

3. Key Contributions

Data-Scalable, Domain-Agnostic Training:
- Demonstrates that multi-view data is not strictly necessary.
- Trained on 30 million uncurated, single images from ImageNet-21K, Places, Open Images, and OpenStreetView.
- Achieves strong generalization across diverse domains (indoor, outdoor, objects, paintings) without in-domain fine-tuning.
Metric Scale Awareness:
- By leveraging metric depth estimators, the model learns real-world scale relationships (parallax) directly from the data, unlike methods trained on SLAM-derived poses which are scale-ambiguous.
Efficient Geometry-Free Design:
- The model performs a single forward pass with no 3D reconstruction, point clouds, or iterative sampling at inference.
- Speed: Achieves 116 FPS (8.6 ms), making it 600× faster than the next best baseline (VIVID) while maintaining or exceeding quality.
Zero-Shot Generalization:
- The model outperforms state-of-the-art methods trained in-domain (on RealEstate10K) when tested on out-of-domain datasets (DL3DV), proving that large-scale data diversity compensates for the lack of specific geometric supervision.

4. Experimental Results

Benchmarks: Evaluated on RealEstate10K and DL3DV.
Performance:
- On RealEstate10K (in-domain for baselines, out-of-domain for OVIE): OVIE rivals or outperforms in-domain baselines (GeoGPT, PhotoNVS) and is competitive with the strongest baseline (VIVID).
- On DL3DV (out-of-domain for all): OVIE surpasses all baselines on all metrics (PSNR, SSIM, LPIPS, FID), confirming superior robustness to distribution shifts.
Ablation Studies:
- Loss Terms: Removing perceptual or adversarial losses significantly degrades FID (realism), showing their necessity for high-frequency detail.
- Data Scale: Performance improves monotonically with dataset size (3K to 30M images). Scale is the primary driver; diversity provides marginal but consistent gains.
- Reconstruction Loss: Surprisingly, $L_2$ (MSE) outperforms robust losses like $L_1$ or Charbonnier, likely because the pseudo-targets are sparse enough that MSE preserves high-frequency details better.
Qualitative Results: OVIE successfully synthesizes views for non-photographic inputs (e.g., paintings) and handles complex occlusions and parallax correctly, whereas other methods often fail to enforce the target pose or produce geometric inconsistencies.

5. Significance and Impact

Democratization of 3D Vision: OVIE removes the barrier of needing expensive, controlled multi-view capture setups. It enables 3D-aware applications for historical archives, artwork, and single-shot photography where multi-view data is impossible to obtain.
Real-Time Interaction: The extreme inference speed (100+ FPS) enables real-time interactive navigation from a single image, allowing users to explore a scene via keyboard/mouse controls as if it were a 3D environment.
Paradigm Shift: The paper challenges the prevailing belief that multi-view supervision is essential for NVS. It suggests that internet-scale 2D data, combined with modern geometric priors (depth estimators), is sufficient to learn universal 3D geometric understanding.

In summary, OVIE proves that "one view is enough" by leveraging the geometric priors of modern depth estimators to create a scalable, fast, and highly generalizable framework for novel view synthesis.