💻 computer science

The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images with Minimal 3D Knowledge

This paper proposes a scalable, data-centric feed-forward Novel View Synthesis framework that achieves state-of-the-art performance without relying on explicit 3D structure or camera poses, validating the principle that methods requiring less 3D knowledge ultimately learn more effectively as training data increases.

Original authors: Haoru Wang, Kai Ye, Minghan Qin, Yangyan Li, Wenzheng Chen, Baoquan Chen

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoru Wang, Kai Ye, Minghan Qin, Yangyan Li, Wenzheng Chen, Baoquan Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Using Training Wheels

Imagine you want to learn how to ride a bicycle.

The Old Way (Bias-Driven Methods):
Most previous AI systems for creating 3D scenes from 2D photos are like a child learning to ride with training wheels and a strict instructor.

The Training Wheels: The AI is given a pre-built 3D skeleton (like 3D Gaussian Splatting or NeRF) that forces it to see the world in a specific geometric way.
The Instructor: The AI is given a map of exactly where the camera was for every photo (called "camera poses").
The Problem: These training wheels are great when you only have a few photos. But if you try to ride on a massive, complex highway (millions of photos), the training wheels become a cage. They stop the AI from learning the true nature of the road because it's too busy following the rigid rules.

The New Way (The "Less You Depend, The More You Learn" Approach):
This paper introduces a new method called UP-LVSM. It's like taking the training wheels off and letting the child learn by feeling the road.

No Training Wheels: The AI isn't forced to use a pre-made 3D skeleton. It builds its own understanding of 3D space from scratch.
No Instructor: The AI isn't told where the camera was. It has to figure out the camera's position just by looking at the pictures, like a detective solving a mystery.

The Core Discovery: The More Data, The Better They Get

The authors ran a massive experiment to see what happens when you feed these two types of learners more and more data.

The "Training Wheel" Learner: When you give them a little data, they do great. But as you give them more data, they hit a ceiling. They can't get much better because their rigid rules prevent them from adapting to the complexity of the new data.
The "Feeling" Learner (UP-LVSM): When you give them a little data, they stumble. They are confused without the rules. But, as you feed them more and more data, they start to understand the patterns intuitively. Their performance skyrockets.

The Analogy:
Think of it like learning a language.

Method A gives you a grammar book and a dictionary (explicit rules). You speak perfectly for simple sentences, but you can't understand slang or complex poetry because you're stuck following the book.
Method B throws you into a country where everyone speaks the language (massive data). At first, you are silent and confused. But after living there for a year, you understand the soul of the language, the jokes, and the nuances better than the person who just memorized the grammar book.

The Paper's Slogan: "The Less You Depend (on rules), The More You Learn (from data)."

How Does the New AI Actually Work?

Since the AI doesn't have a map (camera poses) or a skeleton (3D structure), how does it know what to do?

The "Latent Plücker Learner" (The Internal Compass)
Imagine you are in a dark room with a friend. You can't see each other, but you can hear voices.

The AI looks at two photos. It doesn't know the camera moved "5 feet left."
Instead, it learns a secret code (a "latent space") that represents the relationship between the photos. It figures out, "Ah, if I shift my internal view this way, the objects in the picture line up perfectly."
It creates its own internal compass. It doesn't need to know "North" (absolute coordinates); it just needs to know how to get from Point A to Point B relative to the images.

Why Is This a Big Deal?

It Scales Forever: Because it doesn't rely on rigid rules, it can keep getting smarter as we feed it more and more internet photos. The old methods will eventually stop improving; this one keeps climbing.
It Works with "Messy" Photos: You don't need perfect, labeled photos. You can take a bunch of random snapshots from your phone, and this AI can figure out the 3D scene and generate new views from angles you never took.
It's Surprisingly Good: Even though it doesn't use the "cheat codes" (camera poses) that other methods use, it actually produces higher quality 3D views than the methods that do use them, once it has enough data.

The Catch (and the Fix)

The Catch: Because the AI learns its own "secret code" for camera movement, it's hard for a human to say, "Hey, move the camera 10 feet to the right." The AI speaks a language of "latent codes" that humans don't understand.

The Fix: The authors showed that you can teach the AI a simple translator. You can show it a few examples of "Real World Movement" vs. "Secret Code," and it learns to translate. This means you can eventually control the camera again, but now the AI is smart enough to handle the messy, real-world data that breaks older systems.

Summary

This paper argues that in the age of Big Data, rigid rules are holding us back. By letting AI learn 3D geometry purely from 2D images without being told the rules or the camera positions, we unlock a system that learns faster, scales better, and creates more realistic 3D worlds than anything we've built before. It's the difference between memorizing a map and learning to navigate by the stars.

1. Problem Statement

Novel View Synthesis (NVS) aims to render high-fidelity, unseen views of a 3D scene from a collection of 2D images. While traditional methods rely on per-scene optimization (e.g., NeRF, 3D Gaussian Splatting) requiring dense inputs and precise camera poses, recent feed-forward approaches have emerged. However, these existing feed-forward methods generally fall into two categories with significant limitations:

Bias-Driven Methods: Rely on explicit 3D knowledge, such as handcrafted 3D representations (NeRF, 3DGS) and camera poses annotated by Structure-from-Motion (SfM) algorithms (e.g., COLMAP). These methods often struggle to scale effectively with increasing data volumes because their architectural biases limit their ability to learn complex patterns directly from data.
Data-Centric Methods: Attempt to learn 3D structure implicitly from large-scale 2D data. While promising, many still rely on SfM-annotated poses (which contain noise and geometric biases) or predefined 3D structures, creating a performance bottleneck at scale.

Core Question: Which paradigm is more scalable in an era of ever-increasing data availability? The authors hypothesize that reducing dependence on explicit 3D knowledge leads to superior scalability and performance.

2. Key Findings & Hypothesis

The authors conducted a systematic analysis of existing methods across varying data regimes (from 1K to 66K scenes). They uncovered a critical trend they term "The Less You Depend, The More You Learn":

Explicit 3D Bias: Methods relying on explicit scene structures (e.g., MVSplat, NoPoSplat) perform well in low-data regimes but plateau as data increases.
Pose Annotations: Even data-centric methods relying on SfM-generated poses (e.g., LVSM) suffer from scalability limits due to noise and inconsistencies in these annotations.
The Trend: Methods that minimize dependence on explicit 3D structures and pose annotations (purely data-driven) show a steeper performance curve as training data scales, eventually outperforming bias-driven counterparts.

3. Methodology: UP-LVSM

To validate this hypothesis, the authors propose UP-LVSM (Unposed Large View Synthesis Model), a novel feed-forward NVS framework designed to operate in the unposed setting (no input or target camera poses provided).

Architecture

Transformer-Based Encoder-Decoder: The model uses a pure Transformer architecture (similar to LVSM) to process image patches.
Latent Scene Representation: Instead of explicit 3D representations (like 3DGS or NeRF), the scene is modeled implicitly as latent tokens.
DINOv2 Backbone: Input images are encoded into patchified tokens using a pre-trained DINOv2 encoder to leverage strong visual priors.

Core Innovation: Latent Plücker Learner

The primary challenge in the unposed setting is the lack of target pose ( $P_T$ ) for supervision. The authors introduce the Latent Plücker Learner to solve this:

Self-Supervised Pose Inference: Instead of using ground-truth poses, the model learns to infer a latent pose directly from the target image in a self-supervised manner.
Bottlenecked Representation: To prevent the latent pose from simply memorizing the target image (information leakage), the learner compresses the pose into a compact 7D latent token (3D translation + 4D quaternion).
Plücker Ray Embedding: This compact token is analytically upsampled into fine-grained, pixel-level Plücker ray embeddings. This provides the decoder with rich, per-ray geometric conditioning without requiring explicit 3D supervision.
Training: The model is trained end-to-end using only 2D image reconstruction loss, learning the mapping from images to latent poses and scene representations simultaneously.

Camera Control Extension

Although UP-LVSM learns implicit poses, the authors demonstrate that the learned latent space forms a meaningful manifold. They introduce a simple linear mapper to translate explicit SE(3) camera poses into the latent space, enabling explicit camera control with negligible performance degradation.

4. Experimental Results

The authors evaluated UP-LVSM on multiple datasets (RealEstate10K, DL3DV, ACID, Objaverse) across different data scales.

Scalability: UP-LVSM demonstrated the highest scalability. On RealEstate10K, as the training set grew from 1K to 66K scenes, UP-LVSM's performance (PSNR) increased by 2.63 dB, significantly outperforming bias-driven methods like MVSplat (0.39 dB gain) and pose-dependent data-centric methods like LVSM (0.64 dB gain).
State-of-the-Art Performance: Despite having no pose supervision during training, UP-LVSM achieved SOTA results, outperforming methods that rely on explicit 3D structures and annotated poses (e.g., MVSplat, NoPoSplat, LVSM).
- RealEstate10K Full Eval: UP-LVSM achieved 28.82 PSNR vs. LVSM's 27.60 and MVSplat's 26.45.
Zero-Shot Generalization: When trained on RealEstate10K and tested on the unseen ACID dataset, UP-LVSM outperformed models trained directly on ACID, proving the benefits of large-scale data scaling.
3D Awareness Probing: Analysis of attention weights and latent pose mapping confirmed that the model successfully learns implicit 3D spatial relationships and camera geometry, despite never seeing explicit 3D labels.

5. Key Contributions

Paradigm Shift Analysis: The paper provides a systematic analysis revealing that reducing dependence on explicit 3D knowledge (structure and poses) is the key to unlocking scalability in NVS.
UP-LVSM Framework: A novel, data-centric NVS model that eliminates reliance on explicit scene structures and pose annotations, learning 3D awareness purely from 2D images.
Latent Plücker Learner: A self-supervised mechanism to infer camera geometry directly from images, bypassing the need for noisy SfM annotations.
Empirical Validation: Extensive experiments validating that "less dependence" leads to "more learning," with the proposed method achieving SOTA performance even without pose supervision.

6. Significance

This work fundamentally challenges the prevailing design philosophy in Novel View Synthesis. It demonstrates that inductive biases (like handcrafted 3D representations and SfM poses), while helpful for small datasets, become bottlenecks in the era of big data. By removing these dependencies, UP-LVSM unlocks the full potential of data-driven learning, offering a more scalable, generalizable, and robust path forward for 3D vision tasks. The findings suggest that future NVS systems should prioritize learning implicit 3D priors from massive 2D datasets rather than relying on explicit geometric scaffolding.