💻 computer science

StyleTextGen: Style-Conditioned Multilingual Scene Text Generation

The paper introduces StyleTextGen, a novel framework that achieves state-of-the-art multilingual scene text generation by employing a dual-branch style encoder, a text style consistency loss, and a mask-guided inference strategy to ensure precise style replication across complex backgrounds and diverse writing systems.

Original authors: Zeyu Chen, Fangmin Zhao, Yan Shu, Yichao Liu, Liu Yu, Yu Zhou

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeyu Chen, Fangmin Zhao, Yan Shu, Yichao Liu, Liu Yu, Yu Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic paintbrush that can write words on a photo, but instead of just picking a font from a menu, you want it to copy the exact look of a specific sign you see in a picture. Maybe you want to write "Welcome" in the same messy, neon-green, dripping style as a graffiti tag on a brick wall, or perhaps you want to write a Chinese phrase using the same elegant, brush-stroke style as a calligraphy scroll.

This is the challenge the paper StyleTextGen tackles. While computers have gotten really good at generating images from text, making them write specific words in specific, complex styles (especially mixing different languages like English and Chinese) has been like trying to paint a masterpiece with a blindfold on. The computer often gets the words wrong, or the style looks muddy and inconsistent.

Here is how the authors solved this, explained through simple analogies:

The Problem: The "Cluttered Room" and the "Mismatched Suit"

The authors point out two main headaches:

The Cluttered Room: Real-world photos are messy. If you want to copy the style of a sign, the background (trees, cars, lights) often confuses the computer. It's like trying to learn a specific dance move by watching a dancer in a crowded, chaotic room; you might accidentally copy the crowd's movements instead of the dancer's.
The Mismatched Suit: When switching languages (e.g., from English to Chinese), the "clothing" of the letters changes. English letters are simple loops and lines; Chinese characters are complex blocks of strokes. Existing tools often try to force the English style onto Chinese characters, resulting in a "suit" that doesn't fit the body, looking distorted or broken.

The Solution: A Three-Part Team

To fix this, the team built StyleTextGen, which acts like a specialized art studio with three key tools:

1. The "Dual-Eye" Scanner (Dual-Branch Style Encoder)

Imagine you need to copy a painting. You need two types of vision:

The Detail Eye: This looks only at the text. It ignores the background and focuses on the specific shape of the strokes, the texture of the ink, and the color of the letters. The authors trained this "eye" specifically to understand how text looks in many different languages.
The Big-Picture Eye: This looks at the whole scene to understand the lighting, the mood, and the overall color palette.
The Result: By combining these two views, the computer gets a perfect "style recipe" that knows exactly how the text should look without getting distracted by the messy background.

2. The "Style Police" (Text Style Consistency Loss)

When the computer generates a long sentence, it sometimes gets lazy. The first letter might look perfect, but the last letter might drift and look different.

The Analogy: Think of this as a strict art teacher walking around the classroom. If one student's handwriting looks different from the others, the teacher (the "loss function") steps in and says, "No, that doesn't match the style. Fix it!"
The Result: This ensures that every single letter in the generated sentence looks like it belongs to the same family, maintaining a uniform style from start to finish.

3. The "Magic Transfer" (Mask-Guided Inference)

This is the final polish step. Imagine you have a stencil (a mask) that covers only the letters you want to write.

The Analogy: Instead of just guessing how to paint, the computer takes the "style" from your reference image, puts it inside a special container, and then only pours that style onto the specific spots where the new letters will appear. It's like using a precise spray gun that only paints the letters, ensuring the style transfers perfectly without smearing onto the background.

The New Playground: StyleText-CE

To prove their system works, the authors didn't just test it on easy examples. They built a new "gym" called StyleText-CE.

This is a test set containing both English and Chinese text.
It tests two scenarios: Self-Style (copying the style from the same image) and External-Style (copying a style from a different image).
It also tests Cross-Lingual transfers (taking a style from an English sign and applying it to Chinese text, and vice versa).

The Results

When they ran the tests, StyleTextGen beat all the previous best methods.

Accuracy: The words it wrote were easier to read and spelled correctly more often.
Style: The generated text looked much more like the reference style, whether it was a simple font or a complex artistic brushstroke.
Versatility: It handled the tricky task of mixing languages (like writing Chinese words in an English style) much better than anyone else, without the letters looking broken or distorted.

In short, the paper presents a new way for computers to act like master calligraphers who can look at any sign, understand its unique "soul," and rewrite any message in that exact same soul, regardless of the language or the messy background.

Technical Summary: StyleTextGen

Problem Statement

Style-conditioned scene text generation aims to synthesize text within images that faithfully replicates the visual style (font characteristics, stroke patterns, color gradients, and texture) of a reference example. While text-to-image generation has advanced significantly, generating scene text with precise style control remains a fundamental challenge, particularly in multilingual contexts.

The paper identifies three primary technical hurdles:

Complex Background Interference: Extracting text-specific style features is difficult because reference images often contain cluttered backgrounds, varying lighting, and diverse layouts that confound style perception. Existing methods often struggle to adopt external style references, frequently reusing styles from original in-place text or suffering from style-content entanglement.
Multilingual Consistency: Maintaining fine-grained style consistency across characters is substantially more challenging when dealing with different writing systems (e.g., Latin, Chinese, Arabic) that possess distinct typographic structures and stroke counts. Current methods lack robust cross-lingual transfer mechanisms, leading to inconsistent character appearances and structural distortions.
Style-Content Entanglement: Many existing approaches fail to explicitly separate style from content, resulting in degraded performance when processing complex scenes or generating text in languages different from the reference.

Methodology

The authors propose StyleTextGen, a novel framework built upon a diffusion-based inpainting paradigm (specifically utilizing a DiT backbone similar to TextFlux). The architecture is designed to condition the generation process on arbitrary text-style reference images.

1. Dual-Branch Style Encoder

To robustly encode fine-grained text style cues from complex multilingual scenes, the framework introduces a dual-branch encoder that processes the style reference image ( $I_{style}$ ):

Textual Style Branch: This branch is dedicated to capturing text-specific cues such as glyph structure, stroke texture, and color distribution. It consists of a text-oriented encoder (pretrained on a style-preserving multilingual text segmentation task), a self-attention refinement transformer, and a Q-Former adaptor. This branch ensures the extraction of stable textual features from diverse scripts.
Visual Prior Branch: Built upon a generic vision-language backbone (SigLIP), this branch extracts holistic visual information to provide global appearance and tonal coherence.
Fusion: The outputs of both branches are fused via cross-attention (using the textual representation as queries and the visual prior as keys/values) to produce a robust, text-aware style embedding ( $z_{style}$ ). This embedding is linearly projected into style key-value pairs to modulate the DiT backbone's attention mechanism.

2. Text Style Consistency Loss ( $\mathcal{L}_{tsc}$ )

To address the issue of character style drift and background clutter obscuring style cues, the authors introduce a specific loss function.

Inspired by neural style transfer, this loss computes the Gram matrices of multi-layer features extracted from a pretrained visual encoder.
It calculates the Frobenius distance between the Gram matrices of the text regions in the generated image and the reference image.
This mask-guided loss enforces style alignment specifically within text regions, promoting uniform style attributes across all generated characters.

3. Inference-Time Style Injection

To further refine style alignment during generation, the authors propose a mask-guided inference strategy:

Text-region masks are extracted from both the generated image and the reference style image using a bilingual text segmentation model.
The latent features of the reference image are inverted through the DiT to recover internal key-value features, ensuring only text regions contribute to the style representation.
These style features are injected into the generation process via AdaIN and attention modulation, specifically applied to the text regions defined by the mask.
This injection is performed only during the first 10 denoising steps to preserve scene-text fidelity and background coherence.

Key Contributions

The paper outlines three primary contributions:

Dual-Branch Style Encoder: A novel architecture that combines a text-focused branch (pretrained on multilingual segmentation) with a visual prior branch to generate robust, text-aware style embeddings that remain stable across complex backgrounds and diverse languages.
Text Style Consistency Loss & Inference Strategy: A specialized loss function that enforces style alignment within text regions, coupled with a mask-guided inference strategy that ensures precise style correspondence between reference and generated text across different writing systems.
StyleText-CE Benchmark: The construction of a bilingual (Chinese and English) scene text style benchmark. This benchmark supports four evaluation settings: self-style reference, external-style reference, monolingual generation, and cross-lingual transfer, facilitating systematic evaluation of style consistency and generalization.

Experimental Results

The authors evaluated StyleTextGen on the AnyWord-Eval and StyleText-CE benchmarks, comparing it against state-of-the-art methods including AnyText, Calligrapher, and TextFlux.

Quantitative Performance: StyleTextGen outperformed existing methods across all metrics.
- On AnyWord-Eval, it achieved higher sentence accuracy (Sen.Acc) and Normalized Edit Distance (NED) compared to TextFlux (e.g., +5.6% Sen.Acc on English, +3.4% on Chinese).
- It demonstrated superior style similarity, achieving lower FID and LPIPS scores. For instance, FID scores were reduced by 16.7% (English) and 8.21% (Chinese) compared to TextFlux.
- On StyleText-CE, the model consistently outperformed Calligrapher (the only other method supporting external style references) in both self-style and external-style reference settings, showing significant gains in text accuracy and style similarity across monolingual and cross-lingual configurations.
Ablation Studies: Removing either branch of the style encoder, the text style consistency loss, or the inference-time style injection resulted in measurable degradation in both text accuracy and style similarity, confirming the necessity of each component.
Qualitative Analysis: Visual comparisons showed that StyleTextGen produces text with minimal color deviation from the reference and maintains higher style consistency, even in cross-lingual scenarios and cluttered backgrounds where other methods exhibited style-content entanglement or structural distortions.

Significance

The paper claims that StyleTextGen establishes a new state-of-the-art performance in multilingual style-conditioned text generation. Its significance lies in:

Robust Multilingual Capability: Successfully addressing the challenge of maintaining fine-grained style consistency across different writing systems (e.g., Chinese and English) where previous methods failed.
External Style Adaptation: Enabling the generation of scene text conditioned on arbitrary external style references, overcoming the limitations of methods that rely solely on in-place text or simple font parameters.
Practical Application: Providing a solution crucial for real-world applications such as multilingual signage design and advertising poster creation, while also benefiting downstream tasks like scene text detection and recognition by generating high-fidelity training data.
Benchmarking: Filling a gap in the field by providing a dedicated benchmark (StyleText-CE) for evaluating style consistency and cross-lingual generalization in scene text generation.