Improved 3D Scene Stylization via Text-Guided Generative Image Editing with Region-Based Control

This paper presents an improved 3D scene stylization framework that leverages text-guided generative image editing with a reference-based attention mechanism and multi-depth view generation to ensure high-quality, view-consistent results, while introducing a novel region-controlled loss function for applying distinct styles to specific semantic areas within a scene.

The Gist

Imagine you have a beautiful, high-resolution 3D photograph of a real-world scene—maybe a cozy living room or a bustling city street. Now, imagine you want to turn that photo into a Van Gogh painting, a cyberpunk city, or a watercolor sketch, but you want to keep the 3D structure intact so you can walk around it and look at it from different angles.

That's exactly what this paper is about. The authors, Haruo Fujiwara and his team, have built a "magic wand" that turns real 3D scenes into stylized art using text prompts, while solving three major headaches that previous methods had: flickering views, messy colors, and inability to paint specific parts.

Here is how their method works, broken down into simple concepts:

1. The Problem: The "Flickering TV" and the "Spilled Paint"

Previous attempts to do this were like trying to paint a 3D object by painting 2D photos of it from different angles.

• The Flicker: If you painted the front view, then the side view, the colors might not match perfectly. When you looked at the 3D object, it would look like a broken TV screen flickering between different styles.
• The Spilled Paint: If you wanted to paint a red car blue, the old methods often accidentally painted the sky blue too, or the grass turned red. They couldn't tell the difference between the "car" and the "background."

2. The Solution: A Two-Step Dance

The authors use a technique called Gaussian Splatting (think of it as a cloud of millions of tiny, colored, 3D confetti pieces that form the scene). They don't try to paint the confetti directly. Instead, they do a two-step dance:

Step 1: The "Group Photo" Trick (Multi-View Generation)

Instead of asking an AI to paint one picture at a time, they ask it to paint a grid of four pictures at once.

• The Analogy: Imagine you are directing a photoshoot. Instead of telling the photographer, "Take a photo of the left side," then "Take a photo of the right side," you tell them, "Take a photo of all four corners of the room simultaneously."
• The Secret Sauce: They use a "depth map" (a blueprint of how far away things are) as a guide. They tile four of these blueprints together and feed them to the AI. This forces the AI to understand that the "left" wall in the first photo is the same wall as the "right" wall in the second photo.
• The Result: The AI generates four images that are perfectly consistent. No more flickering. It's like the AI is looking at the whole room at once, not just one slice of it.

Step 2: The "Smart Paintbrush" (Region-Based Control)

Once the AI has generated these perfect 2D images, they use them to "re-train" the 3D scene. But here is where they get clever with the Multi-Region Loss.

• The Analogy: Imagine you are painting a model of a city. You want to paint the buildings gold but keep the grass green.
  • Old Method: You throw gold paint at the whole model. The grass gets gold, the sky gets gold. You have to try to scrape it off later (which ruins the 3D shape).
  • New Method: You put a stencil (a mask) over the grass. Now, when you throw the gold paint, it only hits the buildings.
• How it works: The system uses a segmentation tool (like a smart highlighter) to identify "objects" (e.g., the bear, the sky, the floor). It then calculates the "style" (the colors and textures) separately for each object. It ensures the bear gets the "Van Gogh" style, but the sky stays "realistic," or maybe gets a "watercolor" style.

3. The "Efficiency" Hack

Calculating all these styles is usually very slow, like trying to count every single grain of sand on a beach.

• The Analogy: Instead of counting every grain of sand, the authors realized that if you look at the "most interesting" grains (the ones that tell you the most about the beach), you can guess the total count very quickly.
• The Tech: They use a mathematical trick called Importance-Weighted Sliced Wasserstein Distance. In plain English: They ignore the boring, repetitive parts of the image and focus only on the parts that matter most for the style. This makes the process 5 times faster without losing quality.

Why This Matters

This paper is a big deal because it makes 3D art creation:

1. Stable: No more flickering or glitchy 3D models.
2. Precise: You can paint the car without painting the sky.
3. Fast: It doesn't take days to render; it takes minutes.

In summary: The authors built a system that takes a real 3D scene, asks an AI to "paint" it from all angles at once using a blueprint to keep things consistent, and then uses smart stencils to make sure the paint only goes where you want it to. The result is a high-quality, 3D world that looks exactly like the art style you described in text.

Technical Summary

1. Problem Statement

Recent advances in 3D scene editing and stylization leverage powerful 2D generative models (like Stable Diffusion) to avoid the scarcity of 3D data. However, existing methods face three critical challenges:

1. View Consistency: Ensuring that stylized images remain geometrically and semantically consistent across different viewpoints is difficult. Inconsistencies lead to flickering or artifacts when rendering the 3D scene.
2. Semantic Control: Applying styles consistently to specific regions or objects (e.g., changing only the sky or a specific statue) while maintaining semantic correspondence across views is challenging. Most methods apply style uniformly, leading to "color bleeding" where styles spill into unintended areas.
3. Efficiency: Many current approaches rely on iterative optimization (e.g., SDS) or complex training loops that are computationally expensive and slow to converge.

2. Methodology

The authors propose a two-stage, training-free pipeline based on 2D Gaussian Splatting (2DGS). The process involves generating consistent stylized 2D multi-view images first, followed by fine-tuning the 3D scene.

A. Multi-View Editing with Tiled Depth Reference

To generate consistent stylized images from text prompts, the authors enhance the standard diffusion pipeline:

• Tiled Depth Conditioning: Instead of using a single depth map, the method constructs a tiled grid of depth maps (e.g., 2x2) representing multiple views. This unified reference input is fed into a depth-conditioned ControlNet (based on SDXL).
• Reference-Based Attention Sharing: The authors replace the fully shared attention mechanism used in prior work with a single reference-based attention-sharing mechanism. The generation process is anchored on the tiled depth maps, forcing the model to align structural features across different camera angles. This significantly improves geometric and appearance consistency compared to methods using fully shared attention.
• Workflow: The system predicts depth maps for source views, tiles them, and uses them as a conditioning reference to generate stylized target views via an image-to-image diffusion process.

B. 3D Scene Refinement with Novel Loss Functions

Once stylized 2D images are generated, the source 2D Gaussian Splatting scene is fine-tuned using three specific loss components:

1. Multi-Region Importance-Weighted Sliced Wasserstein Distance (MR-IW-SWD) Loss:

   • Region-Based Control: Unlike standard style transfer which treats the whole image uniformly, this loss partitions features based on segmentation masks (e.g., from SAM2). It calculates the Sliced Wasserstein Distance (SWD) separately for each semantic region (e.g., foreground vs. background). This prevents style bleeding and allows for selective stylization (e.g., stylizing only a statue while keeping the background original).
   • Importance Weighting: To improve efficiency, the authors introduce an importance-weighted sampling strategy. Instead of computing SWD over all random projection directions, they use a Softmax-based weighting to prioritize directions that yield higher 1D Wasserstein distances (more informative projections). This reduces the number of required projections by 95% (using only 5%) while maintaining convergence quality.

2. Content Loss:

   • A standard Mean Squared Error (MSE) loss on VGG19 activations is used to preserve the original geometric structure and content of the scene, preventing the stylization from distorting the 3D geometry.

3. Key Contributions

1. Improved Multi-View Generation Pipeline: A training-free diffusion pipeline that uses tiled depth maps and reference-based attention sharing to achieve superior view consistency and structural alignment compared to state-of-the-art methods.
2. MR-IW-SWD Loss: A novel loss function that combines multi-region segmentation with importance-weighted sampling. This enables:
   • Semantically consistent style transfer (preventing color bleeding).
   • Selective stylization of specific objects/regions.
   • Significant computational efficiency (faster convergence with fewer projection samples).
3. High-Quality 3D Stylization: Demonstrating that fine-tuning 2D Gaussian Splatting scenes with these generated views yields competitive, high-fidelity 3D stylization results.

4. Experimental Results

The method was evaluated on datasets like Instruct-NeRF2NeRF and Mip-NeRF360, comparing against Style-NeRF2NeRF and DGE.

• Qualitative Results: The proposed method produces sharper details, fewer visual artifacts, and better adherence to text prompts. It successfully avoids the "oversaturated colors" and incoherent full-scene stylization seen in competitors.
• Quantitative Metrics:
  • CLIP Similarity: Achieved 0.213, outperforming Style-NeRF2NeRF (0.142) and DGE (0.184), indicating better text-image alignment.
  • Warp Error: Achieved 0.050, the lowest among baselines, indicating superior view consistency and temporal coherence.
  • User Preference: In a study with 58 participants, the proposed method was preferred in 58.8% of cases, significantly higher than the baselines.
• Ablation Studies:
  • Removing the tiled depth reference increased warp error (0.061) and reduced CLIP similarity.
  • Removing the multi-region loss resulted in visible color bleeding (styles spilling into unrelated objects).
  • Removing content loss significantly increased warp error, confirming its role in preserving 3D consistency.
• Efficiency: The Importance-Weighted (IW) SWD achieved comparable convergence to vanilla SWD but with 5% of the projection samples, reducing runtime from ~421ms to ~118ms per iteration.

5. Significance and Limitations

Significance:
This work bridges the gap between 2D generative AI and 3D scene editing by solving the critical issues of view consistency and semantic control. The introduction of region-based control allows for complex editing scenarios (e.g., changing the style of a specific object while leaving the rest of the scene untouched), which was previously difficult with NeRF-based or uniform style transfer methods. The efficiency gains via importance weighting make the approach more practical for real-world applications.

Limitations:

• Geometry Constraints: The method relies on depth conditioning, meaning it cannot significantly alter the underlying geometry or shape of the scene; it primarily modifies appearance.
• Dynamic Scenes: The current framework is designed for static scenes; extending it to dynamic scenes (video) remains a future direction.

In conclusion, the paper presents a robust, efficient, and semantically aware framework for 3D scene stylization, leveraging the strengths of Gaussian Splatting and advanced diffusion control mechanisms.