💻 computer science

M2StyleGS: Multi-Modality 3D Style Transfer with Gaussian Splatting

M2StyleGS is a novel real-time 3D style transfer framework that leverages 3D Gaussian Splatting and multi-modality CLIP features to flexibly apply artistic styles from text or images to 3D scenes, achieving superior visual quality and consistency through precise feature alignment and specialized observation and suppression losses.

Original authors: Xingyu Miao, Xueqi Qiu, Haoran Duan, Yawen Huang, Xian Wu, Jingjing Deng, Yang Long

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingyu Miao, Xueqi Qiu, Haoran Duan, Yawen Huang, Xian Wu, Jingjing Deng, Yang Long

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 3D model of your living room. Right now, it looks like a standard, boring photograph. You want to turn it into a masterpiece that looks like a Van Gogh painting, or maybe a cyberpunk neon city, but you want to do it instantly and from any angle you walk around in.

That's exactly what the paper M2StyleGS is trying to solve. Here is the breakdown in simple terms, using some fun analogies.

The Big Problem: The "Rough Translator"

Previously, if you wanted to change the style of a 3D scene, you had to use a "reference image" (like a picture of a painting). But what if you wanted to say, "Make it look like a rainy, moody jazz club"? You couldn't just type that; you needed a specific picture.

Even when researchers tried to use text or pictures, the results were often messy. The colors would look weird (like a face turning green), textures would get blurry, and if you walked around the 3D scene, the style would flicker or change inconsistently.

The Analogy: Imagine trying to translate a poem from English to French using a very basic, old dictionary. You get the general idea, but the words are clunky, the grammar is wrong, and the beauty is lost. Previous methods were like that basic dictionary.

The Solution: M2StyleGS

The authors created a new system called M2StyleGS. Think of it as a super-smart, real-time art director that can take a 3D scene and instantly repaint it based on a photo or a sentence you type.

Here is how it works, step-by-step:

1. The Canvas: 3D Gaussian Splatting

Instead of building a 3D scene out of millions of tiny triangles (which is slow), this method uses 3D Gaussian Splatting.

The Analogy: Imagine the 3D world isn't made of bricks, but of millions of tiny, floating, colorful confetti pieces. Each piece has a position, a shape, and a color. To see the scene, the computer just sprays these confetti pieces onto your screen. It's incredibly fast and looks very realistic.

2. The Translator: "Subdivisive Flow"

This is the paper's biggest innovation. The system needs to understand how to turn a "text description" or a "CLIP feature" (a computer's understanding of an image) into the specific "VGG style" (the math that defines an artistic style like Van Gogh).

The Problem: The computer's understanding of text and the math of art styles speak different languages. Trying to jump straight from one to the other is like trying to walk from New York to London in one giant leap—you'd fall off the edge of the world.
The Fix (Subdivisive Flow): Instead of one giant leap, the system builds a bridge with many small stepping stones.
- It uses a mathematical tool called an ODE (Ordinary Differential Equation) to create a smooth, continuous path.
- The Analogy: Imagine you are guiding a river of paint from a "Text Bucket" to a "Style Bucket." Instead of dumping the paint and hoping it lands right, the system slowly steers the river, checking the water's color at every tiny step to make sure it's turning into the right shade of "Van Gogh Blue" before it reaches the destination. This prevents the "weird colors" and "blurry textures" of the past.

3. The Quality Control: Two New "Loss" Functions

Even with a good bridge, the paint might still drip or look wrong. The authors added two safety nets (called "Loss Functions") to keep everything perfect.

Observation Loss (The Art Critic):
- The Analogy: Imagine a strict art critic standing next to you. Every time you paint a new view of the 3D room, the critic compares it to a perfect 2D painting of that same style. If your 3D version looks too different from the "perfect" 2D version, the critic yells, "Fix it!" This forces the 3D scene to look like a genuine piece of art.
Suppression Loss (The Color Police):
- The Analogy: Sometimes, the system gets confused and adds colors that don't belong (like a bright pink sky in a moody jazz scene). This loss function acts like a color police officer who scans the whole scene and says, "Stop! That color doesn't fit the vibe. Remove it." It keeps the colors consistent and true to the reference.

The Result

When you put it all together, M2StyleGS allows you to:

Type a prompt (e.g., "Make this room look like a watercolor painting") OR upload a photo.
Get a 3D scene that looks exactly like that style.
Walk around the scene, and the style stays consistent, sharp, and colorful from every angle.

The Bottom Line:
The authors tested this against the best existing methods and found their system was 32% better at keeping the style consistent across different angles. They fixed the "weird colors" and "blurry mess" of the past, giving us a way to instantly turn any 3D world into a living, breathing work of art, whether you describe it with words or show it with a picture.

1. Problem Statement

Current 3D style transfer methods face significant limitations in flexibility and quality:

Input Rigidity: Most existing approaches rely on a fixed reference image. They struggle to accept diverse inputs like textual descriptions, which are crucial for applications in Virtual Reality (VR) and Augmented Reality (AR).
Feature Misalignment: Previous multi-modality attempts (e.g., ConRF) attempt to map CLIP text-visual features to VGG style features using simple MLP networks and Mean Squared Error (MSE) loss. This results in coarse alignment because CLIP feature distributions are unpredictable and non-Gaussian, unlike the assumptions often made in MSE optimization.
Visual Artifacts: Due to this misalignment, existing methods often produce abnormal colors, blurred textures, and inconsistencies between the generated style and the input description (text or image).
3D Representation: While Neural Radiance Fields (NeRF) have been used for style transfer, they lack real-time rendering capabilities. 3D Gaussian Splatting (3DGS) offers real-time performance but has limited research regarding artistic style transfer.

2. Methodology: M2StyleGS

The authors propose M2StyleGS, a framework that integrates 3D Gaussian Splatting with multi-modality style transfer. The core goal is to transform the color of 3D Gaussians while preserving their geometric structure.

A. Core Architecture

3D Representation: The scene is represented as a set of 3D Gaussian primitives ( $G = \{g_n\}$ ), where each Gaussian has position, covariance, opacity, and color. The method fixes geometry and only transforms the color ( $c_n \to c^s_n$ ).
Multi-Modality Input: The system accepts either an image or text as a style reference.
- CLIP Branch: Extracts features ( $X^c_s$ ) from the input (image or text).
- VGG Branch: Extracts style features ( $X^v_s$ ) from the reference style image.
- Mapping: A mapping network aligns the CLIP features to the VGG feature space.

B. Key Technical Innovations

1. Subdivisive Flow (Fine Feature Alignment)
To solve the coarse alignment issue of previous methods, the authors introduce a Subdivisive Flow mechanism based on Ordinary Differential Equations (ODE).

Concept: Instead of a direct one-step mapping, the method treats the transformation from the CLIP domain to the VGG domain as a continuous flow over time $t \in [0, 1]$ .
Mechanism: It learns a drift force $v$ that steers the source features ( $\hat{X}^c_s$ ) toward the target features ( $X^v_s$ ) along a linear path.
Subdivision: The flow is divided into $r$ rounds. In each round, the output of the previous step becomes the new starting point. This iterative refinement ensures that intermediate feature distributions remain consistent with the target VGG space, significantly reducing color anomalies and texture blurring.

2. Auxiliary Loss Functions
Two new loss terms are introduced to regularize the global 3D scene and ensure consistency:

Observation Loss ( $L_{obs}$ ): Uses a pre-trained 2D style transfer model as a generator to create a "ground truth" 2D stylized image ( $I_g$ ). The 3DGS output ( $I_f$ ) is penalized if its VGG feature representation deviates from $I_g$ . This acts as a prior to guide the 3D scene toward authentic artistic styles.
Suppression Loss ( $L_{sup}$ ): A multi-scale discriminator is used to distinguish between authentic 2D stylized art and the generated 3D output. This loss suppresses misleading color information and offsets during the decoding process, ensuring the 3D scene maintains global color coherence.

3. Training Objective
The total loss function combines content preservation, style transfer, and the new auxiliary terms:
$L_{stylized} = L_{content} + \lambda_{style}L_{style} + \lambda_{obs}L_{obs} + \lambda_{flow}L_{flow}$

3. Key Contributions

Diagnosis of Feature Misalignment: Identified that previous multi-modality 3D style transfer failures (abnormal colors, blurred textures) stem from the rough alignment of distinct feature distributions (CLIP vs. VGG).
Subdivisive Flow: Proposed a novel flow-matching method using ODEs to achieve precise, iterative alignment between CLIP text-visual features and VGG style features, incorporating all intermediate states.
Auxiliary Losses: Introduced Observation Loss (to align with 2D artistic priors) and Suppression Loss (to correct global color offsets), enhancing visual quality and multi-view consistency.
SOTA Performance: Achieved state-of-the-art results by combining explicit multi-modal transfer with 3DGS, surpassing single-input methods in both long-term and short-term consistency.

4. Experimental Results

The method was evaluated on the LLFF and Tanks & Temples datasets against SOTA methods (StyleRF, CLIPStyler, StyleGaussian, ConRF, CLIPNeRF).

Quantitative Metrics:
- Consistency: M2StyleGS achieved a 32.92% reduction in short-range LPIPS compared to ConRF (image-guided) and significantly outperformed text-guided baselines like CLIPStyler.
- Metrics Used: RMSE (Root Mean Square Error) and LPIPS (Learned Perceptual Image Patch Similarity) for both short-range (adjacent views) and long-range (distant views) consistency.
Qualitative Results:
- Image Reference: M2StyleGS produced sharper, clearer results with better color fidelity compared to the blurry outputs of ConRF and the coarse alignment of StyleGaussian.
- Text Reference: Successfully captured semantic richness and color details that CLIPNeRF and CLIPStyler failed to maintain, avoiding content degradation.
Ablation Studies:
- Removing the Subdivisive Flow resulted in poor feature alignment (low cosine similarity, high FID).
- Removing Observation Loss led to a deviation from authentic artistic priors.
- Removing Suppression Loss caused color distribution offsets and reduced multi-view coherence.

5. Significance

Flexibility: M2StyleGS breaks the dependency on fixed reference images, allowing users to style 3D scenes using natural language or arbitrary images, which is critical for interactive AR/VR applications.
Real-Time Capability: By leveraging 3D Gaussian Splatting, the method offers real-time rendering capabilities, unlike NeRF-based style transfer methods.
Quality: It resolves the "abnormal color" and "smoothed texture" issues prevalent in current multi-modality 3D style transfer, setting a new benchmark for consistency and visual fidelity in 3D artistic generation.