💻 computer science

ReSplat: Learning Recurrent Gaussian Splatting

ReSplat introduces a feed-forward recurrent Gaussian splatting model that iteratively refines 3D Gaussians using rendering errors as a feedback signal, achieving state-of-the-art performance with significantly fewer Gaussians and faster rendering speeds across diverse datasets, resolutions, and view counts.

Original authors: Haofei Xu, Daniel Barath, Andreas Geiger, Marc Pollefeys

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haofei Xu, Daniel Barath, Andreas Geiger, Marc Pollefeys

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room, but you only have a few blurry photos taken from different corners. This is the challenge of Sparse View Synthesis: creating a full 3D world from very limited 2D pictures.

For a long time, there were two ways to do this:

The Slow, Perfectionist Way: Take your photos, then spend hours (or thousands of computer steps) tweaking the model until it looks perfect. It's accurate, but it's painfully slow.
The Fast, "One-Shot" Way: Use a smart AI to guess the 3D model instantly. It's super fast, but because it only gets one guess, it often makes mistakes, especially if the photos are tricky or low-quality.

ReSplat is a new method that combines the best of both worlds. It's like having a Fast Artist who also knows how to Self-Correct.

Here is how it works, broken down into simple analogies:

1. The "Subsampled" Sketch (The Smart Start)

Most fast AI models try to draw a 3D dot (called a "Gaussian") for every single pixel in your photo. If you have a high-res photo, that's millions of dots. It's like trying to paint a masterpiece by placing a single grain of sand for every pixel. It's heavy and slow to process.

ReSplat is smarter. It starts by drawing a rough sketch in a "subsampled" space.

The Analogy: Imagine you are drawing a map. Instead of drawing every single tree and house, you first draw the major roads and landmarks on a smaller, simplified map (1/16th the size).
The Result: This means ReSplat starts with 16 times fewer 3D dots than its competitors. It's lighter, faster, and less cluttered right from the start.

2. The "Feedback Loop" (The Magic of Iteration)

This is the paper's biggest breakthrough. Traditional fast AI models make one guess and stop. If the guess is wrong, they stay wrong.

ReSplat uses a Recurrent process. Think of it like a Sculptor refining a statue:

The Guess: The AI makes an initial 3D model (the rough sketch).
The Test: It takes that model and tries to "re-take" the photos you gave it.
The Error Check: It compares its "re-taken" photos with your original photos. Where do they look different? That difference is the Rendering Error.
The Correction: Instead of just saying "oops," the AI uses that error as a feedback signal. It asks, "Okay, the sky looks too blue in my model compared to the photo. Let me tweak the blue dots."
Repeat: It does this over and over (usually 4 times), getting slightly better with every pass.

Why is this special?
Usually, to fix a mistake, you need to do complex math (gradients) which is slow. ReSplat learns to fix mistakes without doing that heavy math. It's like a student who learns from a teacher's red pen marks instantly, rather than re-deriving the whole math formula from scratch.

3. Why It's a Game Changer

Speed: Because it starts with fewer dots and fixes them quickly, it is 100 times faster than the slow, perfectionist method.
Quality: Because it corrects its own mistakes, it is sharper and more accurate than the fast, one-shot methods.
Flexibility: It works great even if you give it different numbers of photos (2, 8, or 16) or different resolutions. It adapts on the fly because it's constantly checking its own work.

The Bottom Line

ReSplat is like a smart, iterative editor.

Old fast AI: "Here is my draft. Goodbye." (Often messy).
Old slow AI: "I will rewrite this draft 4,000 times until it's perfect." (Takes forever).
ReSplat: "Here is a rough draft. Let me read it, find the typos, fix them, read it again, and polish it up. Done in 4 seconds."

It allows us to create high-quality 3D worlds from just a few photos, instantly, making it a huge step forward for VR, AR, and digital mapping.

1. Problem Statement

Existing feed-forward Gaussian Splatting (3DGS) models offer computational efficiency and generalization capabilities for sparse view synthesis (2–32 views). However, they face two critical limitations:

Scalability: Most current methods predict one Gaussian per pixel. This results in millions of Gaussians for high-resolution images or many input views, creating a massive computational burden for rendering and memory.
Inflexibility: These models rely on a single-step forward pass. Once the network predicts the Gaussians, the process stops. This limits their ability to correct errors, generalize to unseen datasets/resolutions, or handle complex scenes where a single-step mapping is insufficient.

In contrast, traditional optimization-based 3DGS achieves high quality through thousands of iterative gradient updates but is computationally expensive (taking minutes to hours) and does not generalize well to new scenes without re-optimization.

Goal: Develop a method that combines the efficiency of feed-forward inference with the adaptability and quality of iterative optimization, while significantly reducing the number of Gaussians required.

2. Methodology: ReSplat

ReSplat introduces a feed-forward recurrent Gaussian splatting framework. Instead of a single prediction, it iteratively refines 3D Gaussians without explicitly computing gradients.

A. Compact Initialization (Subsampled Space)

To address the scalability issue, ReSplat does not predict Gaussians per pixel.

Subsampling: The model operates in a 16× subsampled 3D space. It first predicts depth maps at 1/4 resolution, unprojects them into a point cloud, and generates Gaussians from these points.
Result: This produces 16× fewer Gaussians than per-pixel baselines (e.g., MVSplat, DepthSplat), drastically reducing memory and rendering overhead.
Context Aggregation: To compensate for the loss of resolution, the model uses k-Nearest Neighbors (kNN) attention and Global Attention to aggregate local and global 3D context features before decoding Gaussian parameters (position, opacity, covariance, spherical harmonics).

B. Recurrent Refinement via Rendering Error

The core innovation is a weight-sharing recurrent network that iteratively updates the initial Gaussians.

Feedback Signal: Instead of using ground-truth 3D labels (which are unavailable at test time), the model uses the rendering error as a feedback signal.
1. Render the current Gaussian prediction to the input views.
2. Compute the difference (error) between the rendered image and the ground-truth input image.
3. This error is computed in both pixel space (L1) and feature space (using a pre-trained ResNet-18).
Error Propagation: The rendering errors are propagated back to the 3D Gaussians using Global Attention. This allows each Gaussian to "see" errors from all rendered pixels, not just the one it directly projects to, enabling global correction.
Update Mechanism: The recurrent network takes the current Gaussian parameters, a hidden state, and the propagated error to predict incremental updates ( $\Delta g, \Delta z$ ). This process is gradient-free and feed-forward.
Iterations: The process typically converges after 4 iterations, offering a flexible speed-accuracy trade-off.

C. Training Strategy

The model is trained in two stages:

Initialization Stage: Train the compact reconstruction model to predict initial Gaussians using rendering loss and depth smoothness loss.
Recurrent Stage: Freeze the initialization model and train the recurrent update module end-to-end. The loss is applied to all intermediate iterations with exponentially increasing weights to encourage convergence.

3. Key Contributions

Recurrent Feed-Forward Architecture: First to introduce a recurrent, iterative refinement mechanism for feed-forward 3DGS that operates without explicit gradient computation, bridging the gap between single-pass speed and iterative quality.
16× Subsampled Representation: A novel compact initialization strategy that reduces the Gaussian count by 16× compared to per-pixel models, enabling efficient processing of high-resolution and multi-view inputs.
Rendering Error as Feedback: Demonstrates that rendering error serves as a powerful, self-supervised signal that allows the model to adapt to unseen data distributions (datasets, view counts, resolutions) at test time.
State-of-the-Art Performance: Achieves superior results across diverse benchmarks (DL3DV, RealEstate10K, ACID) with significantly fewer parameters and faster rendering speeds than existing methods.

4. Experimental Results

Experiments were conducted on DL3DV, RealEstate10K, and ACID datasets with varying input views (2, 8, 16, 32) and resolutions.

Performance vs. Optimization (3DGS):
- On DL3DV (8 views, 512×960), ReSplat achieves 27.70 PSNR after 4 iterations.
- This outperforms standard optimization-based 3DGS (which requires 4,000 iterations) by 4.2 dB PSNR.
- Speed: ReSplat is 100× faster than 3DGS optimization (0.8s vs. ~80s) and benefits from additional iterations, whereas 3DGS requires re-running the entire optimization loop.
Performance vs. Feed-Forward Baselines:
- Outperforms DepthSplat by 3.5 dB PSNR.
- Uses 16× fewer Gaussians (246K vs. 3.9M) and achieves 4× faster rendering speed.
- On 16-view DL3DV, ReSplat outperforms Long-LRM by 0.8 dB PSNR while using 4× fewer Gaussians.
Generalization:
- Cross-Dataset: Trained only on DL3DV, ReSplat generalizes effectively to RealEstate10K and ACID, outperforming single-step models by large margins (e.g., +1.1 dB on ACID).
- Cross-Resolution: Shows robustness when testing on resolutions different from training (e.g., 320×640), improving by 5 dB over baselines.
- Parameter Efficiency: A small recurrent model (77M params) outperforms a massive single-step model (1.2B params, WorldMirror) by 5.53 dB PSNR.

5. Significance and Impact

Efficiency: ReSplat solves the scalability bottleneck of Gaussian Splatting, making high-quality 3D reconstruction feasible for high-resolution videos and large scenes on consumer hardware.
Robustness: By leveraging rendering error as a feedback loop, the model overcomes the "domain gap" problem common in feed-forward learning, allowing it to adapt to unseen scenes without retraining.
Paradigm Shift: It challenges the notion that high-quality 3D reconstruction requires either slow optimization or low-quality single-pass inference. ReSplat proves that iterative, learned refinement can be both fast and accurate.
Future Directions: The paper suggests that adaptive pruning/densification (currently fixed count) and exploring more informative feedback mechanisms are promising next steps.

In summary, ReSplat represents a significant leap in 3D Gaussian Splatting, offering a practical, high-performance solution for sparse-view synthesis that is both computationally efficient and highly generalizable.