💻 computer science

Through Van Gogh's Eyes: Global Style Transfer with Diffusion Mod

This paper introduces Global Style Transfer (GST), a novel artistic image synthesis paradigm that leverages Global Style Guidance and Content Alignment Guidance to aggregate multiple artworks from a target artist into a single content image, thereby overcoming the limitations of existing methods in capturing an artist's global style distribution while preserving semantic structure and reducing text-induced bias.

Original authors: Jeongha Lee, Yujin Kim, Ghazanfar Ali, Suhyun Kim, Jae-In Hwang

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeongha Lee, Yujin Kim, Ghazanfar Ali, Suhyun Kim, Jae-In Hwang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a time-traveling art critic who has just stepped into a room full of digital canvases. You want to know: if a famous painter like Van Gogh were alive today, how would they paint a modern city street or a cat sitting on a fence? This is the heart of "artistic image synthesis," a field where computers try to mimic the unique visual fingerprints of human artists. For a long time, computers had two main ways to do this. The first was like a strict photocopy machine: it would take one specific painting and force a new photo to look exactly like it, but it often missed the bigger picture of the artist's whole career. The second way was like asking a robot to "draw in the style of Van Gogh" using only words. While flexible, this often made the robot rely heavily on the most famous, iconic paintings (like The Starry Night) over and over again, ignoring the rest of the artist's diverse work. The big question scientists are trying to answer is: how can we teach a computer to understand an artist's entire soul—their whole range of styles, colors, and brushstrokes—without just copying one famous picture or relying on a text prompt that might trick the computer?

This paper introduces a clever new solution called Global Style Transfer (GST), which acts like a master art student who studies a whole library of an artist's work before picking up a brush. Instead of looking at just one painting or listening to a text command, the researchers' system gathers hundreds of artworks from a single artist (like Van Gogh, Monet, or Renoir) and learns the "global style" that ties them all together. They call this a "Many-to-One" approach because it takes many source paintings and blends them into one unified style guide for a single new image.

To make this work, the team built two special tools. The first is Global Style Guidance (GSG). Think of this as a secret decoder ring that lives inside the computer's brain. Instead of asking the computer to "be Van Gogh" with words, this tool looks at the visual patterns in hundreds of paintings and calculates a "style offset"—a mathematical nudge that tells the computer how to shift its colors and textures to match the artist's true, broad personality. Crucially, the researchers found that by training this tool with a boring, fixed phrase like "A painting" for every single artwork, they could strip away the influence of text. This means the computer learns purely from what it sees, not from what it reads, avoiding the bias where it only copies the most famous, text-associated works.

The second tool is Content Alignment Guidance (CAG). Imagine you are painting a portrait of your friend in the style of Van Gogh. You want the swirling, thick brushstrokes, but you still want it to look like your friend, not a random stranger. CAG is the safety net that holds the friend's face in place while letting the artist's style warp and twist the rest of the image. It uses a "training-free" method (meaning it doesn't need extra learning time) to constantly check that the new image keeps the same basic structure as the original photo, even while the style changes.

The results are quite promising. When the researchers tested their system on over 80,000 artworks from the WikiArt database, they found that GST produced images that felt much more like the artist's true, diverse self compared to older methods. While other techniques often got stuck on a single famous look or produced blurry, inconsistent results, GST managed to capture the full "spectrum" of the artist. For instance, when generating Van Gogh-style images, the system didn't just copy The Starry Night; it created a variety of swirling, textured scenes that felt authentic to his entire body of work. The data suggests that this method successfully avoids "memorizing" just a few famous patterns and instead learns a richer, more faithful representation of the artist's visual identity, all while keeping the original subject of the photo recognizable.

Technical Summary: Through Van Gogh's Eyes: Global Style Transfer with Diffusion Model

Problem Statement

Artistic image synthesis aims to recreate the expressive visual identity of a target artist. Existing approaches generally fall into two paradigms, both of which suffer from significant limitations:

Conventional Style Transfer (One-to-One): These methods extract style from a single or a few reference artworks and transfer them to a content image. While effective for specific artwork stylization, they fail to capture the broader, coherent stylistic distribution that defines an artist's overall visual identity. They often lead to instance-level overfitting.
Text-to-Image (T2I) Diffusion Synthesis: Methods conditioned on artist names (e.g., "in the style of Van Gogh") offer flexibility but suffer from text-induced bias. These models tend to reproduce textures and compositions from a handful of iconic works (e.g., The Starry Night), collapsing the artist's diverse stylistic spectrum into a narrow, mode-biased distribution.

Neither paradigm reliably captures the global style of an artist, which requires aggregating visual statistics across a large corpus of works rather than relying on single instances or linguistic prompts.

Methodology

The authors propose Global Style Transfer (GST), a "Many-to-One" paradigm that aggregates multiple artworks from a target artist to transfer their shared global style to a single content image. The framework consists of two complementary components operating within a Latent Diffusion Model (LDM):

1. Global Style Guidance (GSG)

GSG operates in the intermediate feature space (h-space) of the diffusion U-Net bottleneck. Its goal is to learn a residual global style offset ( $\Delta h_t$ ) that modulates the model's representation without relying on text prompts.

Training Process: A lightweight Style Extraction Function (SEF), $f_t$ , is trained on hundreds of artworks from a single artist. Crucially, all artworks are paired with a fixed, unified text prompt (e.g., "A painting").
Mechanism: By fixing the prompt, the variance of text conditioning is reduced to near zero. The SEF learns to predict the residual offset $\Delta h_t$ purely from visual statistics via noise reconstruction loss. This ensures the learned style semantics are text-independent and grounded in the artist's visual corpus.
Inference: During the reverse diffusion process, the learned $\Delta h_t$ is added as a residual to the U-Net bottleneck features, steering the denoising trajectory toward the artist's global style distribution.

2. Content Alignment Guidance (CAG)

To preserve the semantic structure of the input content image while allowing for artist-specific geometric deformations, the authors introduce a training-free mechanism called CAG.

Process: The content image is first mapped to a noisy latent vector via DDIM inversion.
Perceptual Guidance: At each diffusion timestep, the method calculates a perceptual loss using the CLIP image encoder. It compares the high-level semantic features of the approximated clean image ( $\tilde{x}_0$ ) derived from the current latent against the features of the generated image ( $x_t$ ).
Optimization: The latent trajectory is adjusted via gradient descent to minimize this perceptual loss. This preserves the abstract structure and semantic composition of the content while permitting the style-driven deformations characteristic of the target artist.

Key Contributions

Global Style Transfer (GST) Paradigm: The authors introduce a novel Many-to-One formulation for artistic synthesis that learns a unified global style distribution from multiple artworks, overcoming the instance-level bias of traditional style transfer and the text-dependent bias of T2I models.
Global Style Guidance (GSG): A guidance mechanism operating in h-space that learns artist-level style semantics purely from visual statistics. By using a fixed prompt during training, it mitigates linguistic dependence and prevents the model from collapsing into iconic visual patterns.
Content Alignment Guidance (CAG): A training-free perceptual guidance mechanism that maintains the semantic integrity of the content image while enabling expressive, style-based geometric deformations during the diffusion process.

Experimental Results

The framework was evaluated on the WikiArt dataset (for style references) and the VanGogh2Photo dataset (for content).

Quantitative Performance: Compared to state-of-the-art style transfer methods (e.g., StyleInjection, StyTR2) and personalization techniques (e.g., Textual Inversion, Custom Diffusion, LoRA fine-tuning), GST achieved superior performance in:
- Stylistic Fidelity: Lower ArtFID scores indicate better alignment with the artist's global style.
- Content Preservation: Competitive CFSD scores demonstrate that content structure is maintained.
- Diversity: GST achieved the highest CLIP-Diversity scores, indicating it captures a broader stylistic distribution rather than memorizing specific patterns. The diversity values closely matched those of the real artist's corpus.
- Memorization Avoidance: High 1-Precision scores suggest GST avoids overfitting to dominant visual motifs found in personalization baselines.
Qualitative Analysis:
- Visual Comparison: GST produces coherent global stylization that reflects the artist's full corpus, whereas baselines relying on single images or averaging multiple outputs result in blurred or inconsistent styles.
- Text Independence: Ablation studies showed that changing the training prompt (e.g., from "A painting" to "An artwork") resulted in nearly identical generated images, confirming the text-independence of the learned style.
- Bias Mitigation: Unlike T2I baselines (e.g., Nano Banana 2, ChatGPT) that repeatedly generate iconic compositions (e.g., swirling patterns for Van Gogh), GST produces diverse outputs reflecting the true global style spectrum.
- Component Analysis: Removing CAG led to significant content degradation (higher ArtFID), confirming its necessity for preserving structure. Additionally, applying CAG at higher CLIP layers (Layer 11) was found to be critical for preserving global composition compared to lower layers.

Significance and Claims

The paper claims that Global Style Transfer provides a more faithful and unbiased method for modeling an artist's coherent visual identity. By shifting from instance-level transfer and text-dependent synthesis to a visual-statistics-driven, Many-to-One approach, GST successfully captures the global stylistic distribution of an artist. The authors emphasize that their method allows for diverse and expressive content re-rendering that reflects the full breadth of an artist's work, rather than being limited to a few iconic exemplars or specific textual descriptions.