💻 computer science

SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation

SIC3D is a two-stage, image-conditioned text-to-3D generation pipeline utilizing 3D Gaussian Splatting that enhances geometric fidelity and style adherence by introducing a novel Variational Stylized Score Distillation loss to effectively transfer reference image styles while mitigating geometry-appearance conflicts.

Original authors: Ming He, Zhixiang Chen, Steve Maddock

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming He, Zhixiang Chen, Steve Maddock

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a beautiful, detailed 3D statue of a dragon.

In the past, you could only tell a computer, "Make me a dragon." The computer would try its best, but the result often looked like a blurry, shapeless blob, or the texture (the scales, the color) would look muddy and inconsistent.

Or, if you wanted the dragon to look like it was painted by Van Gogh, you'd have to try to describe "swirling blue and yellow brushstrokes" in text. The computer would get confused, and the dragon might end up looking like a blue and yellow mess rather than a Van Gogh painting.

SIC3D is a new, smarter way to do this. Think of it as a two-step construction crew that builds your 3D object and then paints it with a specific artist's style, all while keeping the shape perfect.

Here is how it works, broken down into simple analogies:

Step 1: The "Sculptor" (Building the Shape)

First, the system acts like a master sculptor. You give it a text prompt (e.g., "A dragon").

What it does: It builds a solid, 3D model of the dragon. It focuses entirely on getting the geometry right—the wings, the claws, the curve of the tail.
The Magic: It uses a technique called 3D Gaussian Splatting. Imagine this not as a solid block of clay, but as millions of tiny, glowing, fuzzy dots (like a cloud of glitter) that come together to form the shape. This makes the model look incredibly sharp and real, and it builds much faster than older methods.
The Result: You now have a perfect, neutral-colored 3D dragon. It looks good, but it's just a plain, uncolored statue.

Step 2: The "Painter" (Adding the Style)

Now, you show the system a reference image, like a picture of a Van Gogh painting or a fire texture.

The Problem: If you just tell a computer to "paint this dragon like Van Gogh," it often tries to change the shape of the dragon to match the painting, making the wings look weird or the body twist.
The SIC3D Solution: This is where their secret sauce comes in. They use a special "style injector" (called IP-Adapter) that acts like a ghost painter.
- This ghost painter looks at the reference image and tells the 3D model: "Keep the dragon's shape exactly as it is, but change the colors and textures to look like this painting."
- It uses a new math trick called VSSD (Variational Stylized Score Distillation). Think of this as a quality control manager that ensures the paint doesn't smear or distort the underlying sculpture. It makes sure the "swirls" of the Van Gogh style wrap perfectly around the dragon's 3D curves, no matter which angle you look at it from.

The Secret Ingredient: The "Size Limit" Rule

One of the biggest headaches in 3D generation is "artifacts"—weird, blurry blobs or floating noise that appear when the computer tries to paint.

SIC3D introduces a Scaling Constraint.

The Analogy: Imagine you are painting a wall with a giant, fluffy brush. If the brush is too big, you smear paint everywhere, covering the details and making a mess.
The Fix: SIC3D forces the "brush" (the tiny glowing dots) to stay small and precise. It puts a "size limit" on them. This ensures that the texture stays sharp and crisp, preventing the "blurry mess" effect.

Why is this a big deal?

It's Fast: Because it uses these "glowing dots" (Gaussian Splatting) instead of older, heavier methods, it builds and paints 3D objects much faster.
It's Controllable: You can mix and match. Want a cyberpunk sword? Text: "Sword." Image: "Neon city." Result: A sharp sword that looks like it belongs in a neon city, without the sword melting into a neon blob.
It's Consistent: If you walk around the 3D object, the style looks the same from every angle. Older methods often looked great from the front but weird from the side.

In Summary

SIC3D is like having a sculptor who builds a perfect statue, followed by a painter who applies a specific artistic style to it without ever touching the statue's shape. It solves the problem of "messy 3D art" by separating the shape from the style and using smart rules to keep everything sharp and consistent.

Instead of guessing how to describe a style in words, you just show the computer a picture, and it magically applies that look to your 3D creation.

1. Problem Statement

Recent advancements in text-to-3D generation have leveraged 2D diffusion models and differentiable 3D representations (like 3D Gaussian Splatting, or 3DGS) to synthesize detailed geometry. However, existing approaches face two primary limitations:

Limited Controllability: Relying solely on text prompts often leads to ambiguity in visual style and texture.
Style Transfer Challenges: Existing methods for transferring visual styles from reference images to 3D objects often suffer from:
- Geometric Distortion: Simultaneous optimization of geometry and style leads to unstable gradients and distorted shapes.
- Multi-view Inconsistency: Methods based on NeRF or post-hoc reconstruction often fail to maintain consistent textures across different viewpoints.
- Incompatibility with 3DGS: Many style transfer techniques are designed for NeRF or require slow reconstruction steps, making them inefficient for 3D Gaussian Splatting.

2. Methodology: SIC3D Framework

The authors propose SIC3D, a two-stage framework that decouples geometry construction from appearance refinement to achieve efficient, controllable, and view-consistent stylized 3DGS generation.

Stage 1: Object Generation from Text

Goal: Generate a geometrically consistent base 3D object ( $O_1$ ) from a text prompt.
Technique: The authors utilize Variational Score Distillation (VSD) combined with a trainable LoRA (Low-Rank Adaptation) module.
Mechanism: Unlike standard Score Distillation Sampling (SDS), which can produce noisy gradients, VSD introduces a case-specific LoRA branch that incorporates camera information. This allows the model to learn sharper details and more accurate geometry by jointly optimizing the 3D object parameters and the LoRA weights.
Output: A stable, view-consistent 3D Gaussian Splatting representation serving as the base for stylization.

Stage 2: Style Distillation from Image

Goal: Transfer the visual style of a reference image ( $I_s$ ) onto the base object $O_1$ while preserving the underlying geometry.
Core Innovation: Variational Stylized Score Distillation (VSSD)
- IP-Adapter Integration: The framework uses an IP-Adapter as an image condition processor. It encodes the style image via a CLIP encoder and injects these features into the diffusion model's cross-attention layers alongside the text prompt. This enables multi-modal conditioning (text + style).
- Camera Conditioning: A camera projection layer maps camera parameters into embeddings, ensuring the style transfer is consistent across different viewpoints.
- Loss Function: The VSSD loss minimizes the difference between the noise prediction of the IP-Adapter-augmented diffusion model and a LoRA-augmented branch. This guides the 3DGS parameters to adopt the style while maintaining geometric consistency.
Scaling Regularization ( $L_{surface}$ ):
- Problem: During optimization, individual 3D Gaussians can expand excessively, causing blurry boundaries and artifacts.
- Solution: The authors introduce a regularization term that penalizes the variance of the approximated surface area of each Gaussian (modeled as an ellipsoid). This prevents Gaussians from growing too large, ensuring sharp contours and preventing artifacts without requiring non-differentiable pixel counting.

3. Key Contributions

SIC3D Framework: The first image-conditioned stylization framework specifically designed for 3D Gaussian Splatting, enabling efficient and controllable text-to-3D generation with style transfer.
VSSD Loss: A novel Variational Stylized Score Distillation loss that extends VSD by integrating IP-Adapter for style injection. This ensures stable optimization and multi-view consistent stylization.
Gradient-Based Regularization: A scaling constraint strategy that mitigates artifacts caused by geometry-texture inconsistencies by regulating the projected influence of Gaussian primitives.

4. Experimental Results

The authors evaluated SIC3D against a stylized-prompt baseline and three state-of-the-art 3D style transfer methods (StyleGaussian, G-Style, SGSST).

Quantitative Performance:
- Style Adherence: SIC3D achieved the lowest NNFM score (0.151), indicating the closest alignment with the reference style image.
- Perceptual Quality: It achieved the best LPIPS score (0.877), suggesting superior perceptual plausibility.
- Pixel Accuracy: It maintained competitive RMSE (1.301), showing controlled stylization.
Qualitative Analysis:
- SIC3D successfully transfers complex patterns (e.g., fire, brushstrokes) while preserving the object's geometry, unlike baselines that often distort shapes or produce multi-view inconsistencies.
- GPT-4o Evaluation: In a pairwise comparison using GPT-4o as a proxy for human evaluation, SIC3D achieved the highest ELO scores for both "Object Quality" (1075.1) and "Style Alignment" (1154.9), outperforming all competitors.
Ablation Studies:
- Removing the scaling constraint led to Gaussian expansion and blurry artifacts.
- Removing LoRA resulted in weak style transfer and rough surfaces.
- Fixed camera sampling yielded more consistent textures compared to random sampling.

5. Significance

SIC3D addresses a critical gap in 3D content generation by enabling fine-grained style control without sacrificing geometric fidelity.

Efficiency: By operating directly on 3DGS primitives rather than relying on slow NeRF optimization or post-hoc reconstruction, SIC3D offers a faster and more scalable solution.
Decoupling Strategy: The two-stage approach (Geometry $\to$ Style) effectively solves the entanglement problem where style transfer typically degrades 3D structure.
Generalizability: The framework is backbone-agnostic (tested with both GaussianDreamer and TRELLIS) and compatible with different diffusion models (SD1.5 and SDXL), making it a versatile tool for future 3D stylization research and applications like game asset creation and virtual reality.