💻 computer science

FluSplat: Sparse-View 3D Editing without Test-Time Optimization

FluSplat is a feed-forward framework that enables sparse-view 3D scene editing with cross-view consistency and high fidelity by introducing a cross-view regularization scheme during training, thereby eliminating the need for computationally expensive, scene-specific test-time optimization.

Original authors: Haitao Huang, Shin-Fang Chng, Huangying Zhan, Qingan Yan, Yi Xu

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haitao Huang, Shin-Fang Chng, Huangying Zhan, Qingan Yan, Yi Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 3D model of a room, like a digital twin of your living room. You want to use a text command like "Replace the dog with a cat" to change it.

The Old Way (The "Slow & Messy" Method):
Think of the old methods like trying to paint a 3D sculpture by looking at it from only two angles. You paint the front, then you paint the side. But because you aren't looking at the whole object at once, you might paint the dog's tail on the front but forget to paint it on the side. When you walk around the sculpture, the dog looks broken or glitchy.

To fix this, the old computers had to do a massive amount of math every single time you wanted to make a change. They would paint, check for errors, repaint, check again, and repeat this hundreds of times until the dog looked right from every angle. This took minutes or even hours, and it was like trying to solve a puzzle while blindfolded.

The New Way (FluSplat: The "Magic Instant" Method):
The authors of this paper, FluSplat, came up with a smarter way. They realized that if you can make the pictures look consistent before you even build the 3D object, the 3D object will build itself perfectly.

Here is how they did it, using a few simple analogies:

1. The "Twin Photographers" Analogy

Imagine you have two photographers standing in different spots, taking pictures of the same scene.

The Problem: If you ask them to "Change the red car to a blue car," they might do it differently. One might make the car bright blue, the other dark blue. If you try to stitch these photos together into a 3D movie, the car will look weird and flicker.
The FluSplat Solution: Before the photographers take the picture, they put on special "Magic Glasses" (this is the Cross-View Regularization). These glasses force the photographers to agree on exactly what blue to use and where the car is, even before they snap the photo. They are trained to look at each other's work and say, "No, we need to match that exactly."

2. The "Two-Step Magic Trick"

FluSplat breaks the process into two super-fast steps that happen in a single blink of an eye (a "forward pass"):

Step 1: The Consistent Edit (The 2D Part)
Instead of editing the 3D object directly, they edit the two flat photos first. But they don't just edit randomly. They use a special training technique (called LoRA and Feature Loss) that acts like a strict teacher.
- Global Loss: This ensures the whole scene looks consistent (e.g., the sky is the same blue in both photos).
- Local Loss: This ensures the specific thing you changed (the car) looks consistent in both photos.
- Result: You get two perfect, matching photos where the dog is definitely a cat, and the cat looks the same in both pictures.
Step 2: The Instant 3D Lift (The 3D Part)
Once the photos are perfect, they feed them into a "3D Printer" (a neural network called NoPoSplat). Because the photos already agree with each other, the printer doesn't need to guess or fix anything. It just instantly "lifts" the 2D images into a 3D world.
- Result: In about 20 seconds, you have a perfect 3D scene with a cat, ready to walk around and view from any angle.

Why is this a Big Deal?

Speed: The old way took minutes (like waiting for a slow oven to bake a cake). FluSplat takes seconds (like microwaving a meal). It's 10 to 20 times faster.
No "Reheating": The old methods had to "re-bake" the scene every time you wanted a new edit. FluSplat just does it once and is done.
No Glitches: Because the photos were forced to agree before becoming 3D, the final 3D object doesn't have those weird "glitchy" moments where the object looks different from different angles.

The Bottom Line

FluSplat is like having a magic editor that learns to "think in 3D" while it's still working on 2D photos. Instead of struggling to fix a broken 3D model after the fact, it ensures the ingredients (the photos) are perfect before they are mixed, resulting in a delicious, consistent 3D meal instantly.

1. Problem Statement

Existing text-guided 3D scene editing methods face three critical limitations:

Computational Inefficiency: Current pipelines (e.g., InstructNeRF, InstructGS2GS) rely on iterative "edit-and-fit" loops at test time, alternating between 2D diffusion editing and 3D reconstruction optimization. This process is computationally expensive and slow (taking minutes to hours per scene).
Cross-View Inconsistency: When 2D edits are applied independently to different views and then lifted to 3D, they often suffer from geometric and semantic misalignment. These inconsistencies amplify under novel-view rendering, breaking spatial realism.
Scene-Specific Tuning: Many methods require per-scene optimization or manual hyperparameter tuning, limiting their applicability in real-time or sparse-view scenarios.

The core challenge is to achieve high-fidelity, cross-view consistent 3D editing from sparse inputs (e.g., two views) in a single forward pass, eliminating the need for test-time optimization.

2. Methodology: FluSplat

FluSplat is a fully feed-forward framework that decouples the editing process from the 3D reconstruction process. It operates in two sequential stages:

A. Cross-View Consistent Image Editing

Instead of editing views independently, FluSplat fine-tunes a rectified-flow diffusion model (specifically FLUX.1 Kontext) to enforce consistency across multiple views during the generation process.

Architecture: The model uses a LoRA (Low-Rank Adaptation) strategy, inserting rank- $r=4$ modules into the deeper layers of the transformer backbone. This allows for parameter-efficient fine-tuning while preserving the pretrained generative prior.
Inference: It utilizes single-step rectified-flow sampling. The model predicts a conditional velocity field to transport noise to data latents in a single Euler update ( $\Delta t = 1$ ), enabling rapid inference.
Self-Supervised Regularization: The model is fine-tuned without paired edited data. Instead, it uses two novel loss functions to enforce consistency between two input views:
1. Global Diffusion Feature Loss (GDFL): Aligns intermediate features from the diffusion U-Net across concatenated views. This ensures high-level structural and semantic consistency (e.g., lighting, global geometry) across the scene.
2. Local Editing Feature Loss (LEFL): Uses CLIP to identify the specific region of interest based on the text prompt, then aligns dense features (via DINOv3) of those specific regions across views. This ensures localized edits (e.g., changing an object's color) remain geometrically consistent without distorting the background.

B. Feedforward 3D Gaussian Reconstruction

Once the 2D views are edited and aligned, they are lifted into 3D without iterative optimization.

Mechanism: The system employs a NoPoSplat-style feedforward network. It takes the edited RGB images and camera intrinsics as input.
Process: A shared Vision Transformer (ViT) encoder processes the views, and a cross-view decoder fuses information using multi-head cross-attention.
Output: Dedicated regression heads directly predict the parameters for 3D Gaussian Splatting (3DGS) primitives (center, opacity, rotation, scale, and spherical harmonics) in a single forward pass. Notably, this module is pose-free, meaning it does not require known camera extrinsics.

3. Key Contributions

First Fully Feed-Forward Sparse-View 3D Editor: FluSplat eliminates the need for per-scene optimization or iterative score distillation, reducing inference time by orders of magnitude.
Self-Supervised Cross-View Regularization: The introduction of GDFL and LEFL allows the model to learn view-consistent editing priors directly in the image domain, solving the inconsistency problem before 3D lifting.
Decoupled Editing and Reconstruction: By ensuring geometric coherence in 2D first, the method enables a direct, stable, and generalizable 3D lift, achieving superior semantic fidelity.
Robust Generalization: Despite being trained on a single dataset (RealEstate10K), the method generalizes effectively to object-centric datasets (DTU) and diverse scene types without dataset-specific adaptation.

4. Experimental Results

The authors evaluated FluSplat on DTU (object-centric), IN2N (2D editing benchmark), and RealEstate10K (large-scale scenes).

Efficiency: FluSplat completes the entire editing and reconstruction pipeline in approximately 20 seconds per scene. In contrast, optimization-based baselines (DGE, EditSplat, ViP3DE) take 4 to 9 minutes.
Semantic Consistency: FluSplat achieves the highest CLIP similarity and directional similarity scores on both 2D and 3D (novel view) evaluations.
- RE10K CLIP3D sim: Ours (0.218) vs. DGE (0.195).
- DTU CLIP3D sim: Ours (0.241) vs. ViP3DE (0.230).
Cross-View Consistency: Quantitative metrics (LEFL distance) show FluSplat significantly reduces structural drift and cross-view artifacts compared to baselines.
Qualitative Performance: Visual comparisons demonstrate that FluSplat produces consistent color transformations, better structural preservation, and fewer artifacts (e.g., color bleeding or incomplete object removal) than prior methods.

5. Significance

FluSplat represents a paradigm shift in 3D content creation. By moving the burden of consistency enforcement from test-time optimization to training-time regularization, it makes high-quality 3D editing feasible for real-time applications, robotics simulation, and immersive media. The ability to generate coherent 3D scenes from just two sparse views in a single forward pass removes a major bottleneck in the adoption of 3D generative AI.