CuriGS: Curriculum-Guided Gaussian Splatting for Sparse View Synthesis

CuriGS is a curriculum-guided framework that enhances sparse-view 3D Gaussian Splatting reconstruction by progressively training with pseudo-views of increasing perturbation levels, which are selectively promoted to the training set based on multi-signal quality metrics to overcome supervision scarcity and overfitting.

The Gist

Imagine you are an artist trying to paint a realistic 3D sculpture of a cat, but you are only allowed to look at the cat through a tiny keyhole from three different angles. You have to guess what the rest of the cat looks like.

If you just try to guess, you might end up with a weird, lopsided cat where the tail is on the wrong side or the ears are floating in mid-air. This is the problem computer scientists face with 3D Gaussian Splatting (3DGS), a popular technology for creating 3D worlds from photos. When there are very few photos (sparse views), the computer gets confused, "hallucinates" fake details, and creates a messy 3D model.

The paper introduces CuriGS, a clever new way to teach the computer how to build these 3D models even when it has very few photos. Here is how it works, using some simple analogies:

1. The Problem: The "Keyhole" Limitation

Think of the computer as a student trying to learn a subject. Usually, it has a whole library of textbooks (hundreds of photos) to study. But in "sparse view" scenarios, the student only has three pages of a book. If the teacher just says, "Draw the rest of the story," the student will likely make things up that don't make sense.

2. The Solution: The "Curriculum" Approach

Instead of throwing the student into the deep end, CuriGS uses a Curriculum. In school, a curriculum means you learn easy things first, then gradually move to harder things. CuriGS does this with camera angles.

• The Teacher (Ground Truth): The few real photos you actually have are the "Teachers." They are the only facts we know for sure.
• The Students (Pseudo-Views): The computer creates fake, "imaginary" photos around the real ones. These are the "Students."

3. How the Training Works: The "Baby Steps" Strategy

The computer doesn't just ask the student to imagine a completely new angle immediately. That would be too hard and lead to mistakes. Instead, it follows a step-by-step plan:

• Step 1: The Baby Steps. The computer starts by asking the student to imagine a view that is just a tiny bit different from the real photo (like moving your head an inch to the left). Because the change is small, it's easy to guess correctly.
• Step 2: The Gradual Stretch. Once the student gets good at those tiny moves, the computer says, "Okay, now try moving two inches." Then three inches.
• Step 3: The Filter. Not every guess is good. The computer acts like a strict art critic. It checks every "imaginary" photo the student made.
  • If the imaginary photo looks weird or blurry, it gets thrown in the trash.
  • If it looks sharp and realistic, it gets promoted. It becomes a new "Teacher" for the next round of training.

4. The Safety Net: "Anchors" and "Double-Checking"

To make sure the computer doesn't get too creative and start inventing a cat with six legs, CuriGS uses two safety nets:

• The Anchor: The computer constantly looks back at the original, real photos (the Teachers) to make sure it hasn't drifted away from reality. It's like a sailor checking the compass to ensure they haven't sailed off course.
• The Double-Check: The computer runs two different "brains" (models) at the same time. If both brains agree on what the imaginary photo should look like, it's probably real. If they disagree, it's likely a hallucination, and the computer fixes it.

5. The Result: A Better 3D World

By using this "Curriculum" method, the computer learns to fill in the gaps of the 3D world without making up nonsense.

• Without CuriGS: The 3D model is shaky, blurry, and has floating artifacts (like ghostly blobs).
• With CuriGS: The model is sharp, the geometry is solid, and you can walk around the object in the 3D space without seeing weird glitches, even though the computer only started with three photos.

Summary

CuriGS is like a smart tutor that teaches a computer how to imagine 3D worlds. Instead of asking the computer to guess the whole story at once, it gives it small, manageable clues, checks the work constantly, and only lets the best guesses become part of the final lesson. This allows the computer to create high-quality 3D scenes from very few photos, solving the problem of "not enough data."

Technical Summary

1. Problem Statement

3D Gaussian Splatting (3DGS) has revolutionized real-time scene reconstruction and rendering due to its efficiency and high fidelity. However, applying 3DGS to sparse-view synthesis (reconstructing scenes from very few input images) remains a significant challenge.

• Core Issues: Limited viewpoint coverage leads to supervision scarcity, causing severe overfitting to the observed views and geometric inconsistency (e.g., floaters, texture drift, and collapsed geometry) in unobserved regions.
• Limitations of Existing Methods: Current approaches often rely on external priors (e.g., depth maps) or ensemble regularization. While helpful, these do not fundamentally resolve the lack of supervisory signals inherent in extreme data sparsity. They often struggle to generalize to unseen viewpoints or maintain geometric stability without external constraints.

2. Methodology: CuriGS Framework

The authors propose CuriGS, a curriculum-guided framework that dynamically expands the training distribution by generating and selecting high-quality "pseudo-views" (student views) around the original "ground-truth" cameras (teacher views).

A. Student View Generation

• Teacher-Student Paradigm: For each original camera pose (Teacher), the system generates multiple groups of Student Views (pseudo-cameras).
• Perturbation Strategy: Student poses are created by applying controlled perturbations to the Teacher's rotation and translation. These perturbations include:
  • Angular shifts (yaw/pitch) sampled from a Gaussian distribution.
  • Radial shifts along the viewing direction.
• Hierarchical Levels: Students are grouped by perturbation magnitude ($\sigma$), ranging from small (local consistency) to large (global diversity).

B. Curriculum Scheduling

Instead of introducing all pseudo-views at once, CuriGS employs a curriculum learning strategy:

1. Progressive Unlocking: Training begins with students having small perturbations ($\sigma_{min}$) to ensure local geometric stability.
2. Dynamic Expansion: As training iterations proceed, the system gradually unlocks higher perturbation levels ($\sigma_{max}$), exposing the model to increasingly diverse viewpoints.
3. Active Sampling: At each iteration, a subset of students from the currently active perturbation level is randomly sampled to assist in training.

C. Evaluation and Promotion Mechanism

To prevent the model from learning from low-quality or hallucinated pseudo-views, a strict selection process is used:

• Multi-Signal Metric: Each candidate student view is evaluated using a composite score combining:
  • SSIM: Structural similarity to the Teacher view.
  • LPIPS: Perceptual similarity.
  • No-Reference Image Quality: A metric to assess global rendering plausibility without ground truth.
• Promotion: The best-performing student for a specific Teacher and perturbation level is retained. If its quality score exceeds a predefined threshold, it is promoted to the official training set, effectively augmenting the sparse supervision.

D. Optimization Objective

The total loss function ($L_{total}$) consists of three components:

1. Dynamic Training Loss ($L_{train}$): Computed over the current training set (original + promoted students), using $L_1$ and D-SSIM to drive primary optimization.
2. Anchor Loss ($L_{anchor}$): A strict constraint applied to the original ground-truth views to prevent semantic drift and ensure the model remains anchored to reliable observations.
3. Student Regularization ($L_{reg}$): Since student views lack ground-truth pixels, they are regularized via:
   • Depth-Correlation: Aligning the rendered depth with a proxy depth map from a pretrained monocular estimator using Pearson correlation.
   • Co-Regularization: Enforcing photometric consistency between two independently initialized models ($M_A$ and $M_B$) rendering the same student view to suppress stochastic artifacts.

3. Key Contributions

1. Curriculum-Guided Sparse View Expansion: The first framework to dynamically generate and promote student views in 3DGS, directly addressing supervision scarcity by expanding the viewpoint distribution rather than just adding regularization.
2. Unified Pseudo-View Learning: A principled mechanism for generating, evaluating, and integrating pseudo-views, opening new directions for virtual-view learning in reconstruction tasks.
3. Superior Performance: Demonstrated state-of-the-art results in rendering fidelity, perceptual quality, and geometric consistency across synthetic and real-world sparse-view benchmarks.

4. Experimental Results

The authors evaluated CuriGS on three standard benchmarks: LLFF (forward-facing real scenes), MipNeRF-360 (unbounded scenes), and DTU (object-centric scenes).

• Quantitative Performance:
  • LLFF (3 views): Achieved the highest PSNR (21.10 dB) and SSIM (0.732) among 3DGS variants, outperforming baselines like FSGS, CoR-GS, and DropGaussian.
  • MipNeRF-360 (24 views): Achieved best-in-class PSNR (24.21 dB) and SSIM (0.761), showing robust generalization in large-scale scenes.
  • DTU (3 views): Showed significant gains in geometric sensitivity, achieving the highest SSIM (0.873) and preserving fine details better than competitors.
• Qualitative Improvements: Visual comparisons show CuriGS recovers sharper high-frequency details, reduces texture drift, and maintains structural integrity (e.g., thin structures and edges) where other methods fail or produce artifacts.
• Ablation Studies:
  • Removing the curriculum guidance resulted in significant performance drops (e.g., ~2-4 dB PSNR loss), confirming the necessity of progressive perturbation.
  • Removing the Anchor Loss led to geometric drift, while removing Student Regularization caused severe overfitting to sparse inputs.

5. Significance

CuriGS represents a paradigm shift in sparse-view 3D reconstruction. Instead of relying solely on external priors or static regularization, it treats the expansion of the training distribution as a learnable, curriculum-driven process. By intelligently curating pseudo-views that balance local consistency with global diversity, CuriGS effectively mitigates the overfitting inherent in sparse data. This approach not only improves the visual quality of novel view synthesis but also ensures geometric stability, making 3DGS more viable for applications with limited data capture, such as cultural heritage preservation, rapid prototyping, and mobile AR/VR.