💻 computer science

SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Training

SplatPainter is a state-aware feedforward model that enables interactive, high-fidelity editing of 3D Gaussian assets through 2D user inputs by leveraging test-time training for rapid, precise, and identity-preserving refinements.

Original authors: Yang Zheng, Hao Tan, Kai Zhang, Peng Wang, Leonidas Guibas, Gordon Wetzstein, Wang Yifan

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Zheng, Hao Tan, Kai Zhang, Peng Wang, Leonidas Guibas, Gordon Wetzstein, Wang Yifan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, photorealistic 3D model of a character or object, created using a technology called 3D Gaussian Splatting. Think of this model not as a solid statue, but as a cloud of millions of tiny, glowing, colored dots (Gaussians) that, when viewed from any angle, look like a perfect photograph.

The problem is that once this "cloud" is created, it's hard to fix. If you want to add a specific detail—like a tear drop under an eye, a graffiti tag on a wall, or change the lighting to look like sunset—current tools are either too slow (taking minutes or hours) or they break the model, making it look blurry or changing parts you didn't want to touch.

SplatPainter is a new tool designed to fix this. Here is how it works, explained through simple analogies:

1. The "Smart Compressor" (Stage I)

Before you can edit, the system takes your massive cloud of millions of dots and organizes it.

The Analogy: Imagine your 3D model is a giant, messy library with millions of books scattered everywhere. To find what you need quickly, SplatPainter first groups these books into neat, labeled boxes (called voxels).
What it does: It doesn't just stack them; it reads the most important books in each box and creates a "summary card" (a compact latent representation) that captures the essence of that area. This makes the model small and fast to work with, but it keeps all the high-quality details safe inside those summary cards.

2. The "Instant Learner" (Stage II: Test-Time Training)

This is the magic part. Usually, AI models are trained once and then frozen. If you want to teach them something new, you have to retrain them from scratch, which takes forever.

The Analogy: Imagine a master chef who knows how to cook a perfect steak. If you ask them to add a specific spice you brought from home, a normal chef might need to go back to culinary school for a week to learn it.
SplatPainter's Chef: This chef has a special "instant learning" ability. When you hand them your spice (your 2D edit, like a drawing or a photo), they instantly taste it, adjust their internal recipe for just this moment, and apply that change to the steak immediately. They don't forget how to cook the steak; they just temporarily adapt to your new ingredient.
In Tech Terms: This is called Test-Time Training (TTT). The system looks at your 2D edit (like a zoomed-in photo of a face), instantly updates a tiny part of its "brain" (the fast weights) to understand your specific request, and then applies that change to the 3D model.

3. What Can You Do With It?

The paper highlights three main ways you can use this "instant learning" tool:

Super-Resolution (The "Zoom-In" Fix):
- Scenario: You have a 3D model of a face, but the eyes look a bit blurry. You take a high-quality photo of just the eye and show it to SplatPainter.
- Result: The tool instantly updates the 3D dots in that specific area to match your sharp photo. It's like taking a blurry painting and instantly repainting just the eyes with perfect detail, while the rest of the face stays exactly the same.
Local Painting (The "Graffiti" Tool):
- Scenario: You want to draw a blue tear under the character's eye or add a sticker to a helmet.
- Result: You draw it on a 2D screen, and SplatPainter "lifts" that drawing into 3D space. Crucially, it knows exactly where to put it. It won't accidentally paint the tear on the other side of the head or change the color of the whole helmet. It preserves the original identity of the object perfectly.
Global Relighting (The "Sunset" Switch):
- Scenario: Your 3D scene looks like it's in a bright office, but you want it to look like it's in a warm, sunset room.
- Result: You show the tool one or two photos of the scene with the new lighting. The tool instantly adjusts the lighting and shadows across the entire 3D model so it looks consistent from every angle, without needing to wait for hours of calculation.

Why Is This a Big Deal?

Speed: It works in seconds (interactive speed). Other methods take hundreds of seconds or minutes.
Safety: It doesn't destroy the original model. It's like editing a photo in Photoshop where you can undo or change your mind instantly, rather than having to repaint the whole canvas.
Precision: It understands exactly what you want. If you ask for a tear under the left eye, it puts it there. It doesn't get confused and put it on the nose or the forehead.

In short, SplatPainter turns the difficult, slow process of fixing 3D models into a fast, intuitive experience where you can paint, zoom, and light up your 3D worlds just like you would with a 2D drawing tablet.

Technical Summary: SplatPainter

Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized photorealistic 3D asset creation, a critical gap remains in interactive appearance refinement and editing. Existing workflows face three primary challenges:

Identity Preservation: Current methods often fail to maintain the original asset's identity and intricate structural details when applying edits, leading to "identity drift" or unintended global changes.
Precision and Control: Approaches relying on intermediate multiview generation or diffusion models often suffer from "spatial edit leakage," where local edits bleed into unintended regions, or fail to match specific user intent due to blurry reconstructions.
Latency: State-of-the-art methods based on iterative diffusion sampling or per-scene optimization are prohibitively slow (often requiring hundreds of seconds), breaking the fluid creative process required for real-time interaction.

There is a need for a framework that allows users to make precise, fine-grained adjustments using familiar 2D tools while strictly preserving the original 3D structure and operating at interactive speeds.

Methodology

SplatPainter is a state-aware feedforward framework designed for the continuous, high-fidelity editing of 3DGS assets. It bypasses the need for destructive multiview regeneration or per-scene optimization by directly predicting updates to Gaussian attributes from 2D image-space edits. The pipeline consists of two main stages:

1. Compact and Feature-Rich 3DGS Representation (Stage I)

To enable efficient editing, the method first transforms the input 3DGS asset into a compact, structured latent representation:

Gaussian LRM Initialization: The input 3DGS is rendered from canonical viewpoints and processed by a Gaussian Large Reconstruction Model (LRM) to generate a dense, feature-rich cloud of Gaussians.
Voxelization: The dense Gaussian cloud is reorganized into a regular latent grid by grouping nearby Gaussians into local voxel cells.
Local Voxel Transformer: A transformer-based module compresses the information within each voxel. It aggregates intra-voxel relations via self-attention and then distills the information by selecting the top- $K$ most significant Gaussians (based on opacity) to serve as queries in a cross-attention mechanism. This results in a compact, structured set of voxel latents that retain global context while reducing memory overhead.

2. Refinement via Test-Time Training (TTT) (Stage II)

The core innovation for interactive editing is the use of Test-Time Training (TTT) to make the model state-aware:

Fast vs. Slow Weights: The refinement network consists of transformer blocks containing "slow weights" (frozen backbone) and "fast weights" (adaptable mapping networks).
Adaptation Mechanism: When a user provides a new 2D edit (e.g., a zoomed-in view or a painted stroke), the model performs a single step of gradient descent on a self-supervised loss using the new input tokens to update the fast weights.
Iterative Application: Once adapted, the updated fast weights are applied in parallel to both the image tokens and the 3D voxel latents. This allows the network to integrate new appearance information on the fly without overwriting the knowledge of the original asset.
Local vs. Global Updates:
- Local Edits: Ray tracing identifies visible voxels corresponding to the edited view. Updates are applied only to these voxels, and the token set is expanded to introduce fine-grained structures.
- Global Edits: The updated fast weights are applied to all Gaussian tokens, enabling coherent global changes (e.g., relighting) while preserving geometry.

Key Contributions

Novel Feedforward Editing Framework: SplatPainter introduces a state-aware, feedforward model for interactive 3DGS editing. By leveraging TTT, it iteratively incorporates 2D edits to directly update Gaussian attributes without destructive regeneration.
Versatile Applications: The unified architecture supports diverse tasks, including:
- High-Fidelity Local Detail Refinement: Recovering missing details from sparse, zoomed-in views.
- Interactive Creative Editing: Integrating user paint-overs, graffiti, or generative edits (e.g., from Nano Banana) while strictly preserving the asset's identity.
- Consistent Global Relighting: Propagating lighting changes across the entire scene based on few-shot relit views.
State-of-the-Art Performance at Interactive Speeds: Extensive experiments demonstrate that SplatPainter significantly outperforms existing methods in both visual quality and speed, operating at interactive rates (approx. 0.3s per local edit) compared to hundreds of seconds for optimization-based baselines.

Experimental Results

The authors evaluated SplatPainter on three primary tasks using datasets from Objaverse and TexVerse:

Local Refinement: The method achieved superior PSNR, SSIM, and LPIPS scores compared to direct reconstruction (GS-LRM), feedforward densification (GenDen), and optimization-based approaches (3DGS, SRGS). Crucially, it achieved this quality in ~0.3 seconds per edit, whereas optimization methods required 300–500 seconds.
Creative Editing: In comparisons with VoxHammer and VineDresser3D, SplatPainter demonstrated significantly better identity preservation. Baseline methods exhibited "spatial leakage" and global identity drift (e.g., distorting facial structures when editing a specific region), whereas SplatPainter kept unedited regions bit-exact to the original. Quantitative metrics on non-edited regions showed SplatPainter achieved a PSNR of 33.02 compared to 13.86 for VoxHammer.
Global Relighting: The method outperformed ReLitLRM and 3DGS optimization in reconstruction accuracy (PSNR 25.03 with 4 views vs. 21.68 for ReLitLRM) and produced more consistent shading without color tone mismatches.
Ablation Studies: The two-stage pipeline was validated, showing that Stage I achieves aggressive compression (reducing Gaussian count by ~70%) with minimal quality loss, while Stage II effectively refines details. The TTT update mechanism was shown to be superior to standard cross-attention baselines in maintaining coherence.

Significance and Claims

The paper claims that SplatPainter addresses the fundamental limitations of current 3D editing workflows by providing a non-destructive, fast, and precise alternative. Its significance lies in:

Bridging the Gap: It enables a fluid, real-time creative workflow where artists can use intuitive 2D tools to refine 3D assets, a capability previously missing in the 3DGS ecosystem.
Preserving Asset Identity: By operating directly on the Gaussian representation with state-aware adaptation, it solves the "identity drift" problem common in generative and optimization-based methods.
Scalability: The feedforward nature of the approach allows for immediate feedback, making it a practical solution for unlocking real-time 3D content authoring in industries ranging from gaming to e-commerce.

The authors note that the current design focuses on appearance updates and does not introduce new geometry, leaving geometric editing as a direction for future work.