💻 computer science

SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Design

SkyReels-Text is a novel, training-free framework that enables precise, font-controllable text editing in poster designs by allowing users to specify custom typography via glyph patches, thereby achieving state-of-the-art performance in both text fidelity and visual realism without requiring font labels or fine-tuning.

Original authors: Yunjie Yu, Jingchen Wu, Junchen Zhu, Chunze Lin, Guibin Chen

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunjie Yu, Jingchen Wu, Junchen Zhu, Chunze Lin, Guibin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a graphic designer working on a movie poster. The title is written in a cool, dripping, neon-green font that looks like slime. Suddenly, the director says, "Change the title to 'Space Slime' but keep that exact same slimy, neon-green look."

In the past, asking a computer to do this was like asking a painter to copy a specific brushstroke perfectly while changing the words on the canvas. The computer would either get the words wrong, make the font look like a different style, or accidentally paint over the background scenery.

SkyReels-Text is a new tool designed to solve this exact problem. Here is how it works, explained simply:

The Problem: The "Copy-Paste" Glitch

Current AI image editors are great at changing big things (like turning a dog into a cat), but they struggle with text. If you ask them to change a word, they often:

Spell it wrong.
Change the font style (making the "slime" look like "helvetica").
Mess up the background around the letters.

It's like trying to replace a single brick in a wall, but the robot keeps knocking down the whole wall or using the wrong color brick.

The Solution: The "Style Photocopier"

SkyReels-Text acts like a master forger who can perfectly copy a font style just by looking at a tiny sample of it.

1. The "Reference Patch" (The Secret Sauce)
Instead of just typing "make it look like this font," you can show the AI a tiny cropped image of the letters you want to copy.

Analogy: Imagine you have a stamp. You don't need to tell the machine the name of the font or the code for the ink. You just hand the machine a piece of paper with the stamp on it. The machine looks at that piece of paper and says, "Ah, I see the style. I will copy that exactly."
Key Point: It works even if the font is weird, handwritten, or doesn't exist in any standard computer library.

2. The "Dual-Stream" Brain
The AI uses two different "eyes" to do the job at the same time:

Eye 1 (The Content): Looks at what you want to write (the new words).
Eye 2 (The Style): Looks at the reference patch to see how it should look.
The Result: It combines these two views to write the new words using the old style, without getting confused.

3. The "Smart Filter" (Data Training)
To teach the AI this skill, the researchers didn't just use boring, standard fonts (like Arial). They built a massive library of 100,000 real-world posters.

They used a special "smart camera" (a Vision-Language Model) to find text in complex, artistic posters, even if the letters were fancy calligraphy or hand-drawn.
They created a system that swaps the words but keeps the style, then double-checks the work to make sure the new words are spelled right and the style hasn't changed.

What It Can Do

Change Multiple Things at Once: You can change three different words on a poster, and each one can have a completely different font style, all in one go.
Handwriting: It can look at a single sample of someone's handwriting and write new sentences in that exact same handwriting, even if it's messy or unique.
Small Text: It can fix tiny, blurry text in images that other AI tools usually can't read.

The Results

The paper tested SkyReels-Text against other top AI tools (like FLUX, Qwen, and Nano Banana).

Accuracy: It spelled words correctly more often than anyone else.
Style: It kept the font style much closer to the original reference than the competition.
Background: It didn't ruin the rest of the picture (the background stayed clean).
Human Vote: When real people looked at the results, they preferred SkyReels-Text over the other tools about 60% to 84% of the time.

In a Nutshell

SkyReels-Text is like a magic editing tool that lets you swap out text in a design while perfectly preserving the "soul" of the original font. It doesn't need you to be a font expert; you just show it a picture of the style you like, and it does the rest, making professional poster design much faster and easier.

1. Problem Statement

Professional poster design requires the ability to rapidly modify textual content (e.g., slogans, dates, brand names) while strictly preserving the original visual harmony, typographic identity, and specific font styles. Current AI-driven image editing models face two primary limitations:

General-Purpose Models: Models like FLUX, Qwen-Image, and Seedream excel at general manipulation but fail at fine-grained, font-aware editing. They often corrupt semantic content or fail to accurately transplant reference typography into target regions.
Specialized Text Models: Existing text-specific models (e.g., FLUX-Text, TextFLUX) focus on spelling accuracy by injecting glyph priors but lack mechanisms to accept arbitrary font styles from user-provided visual patches. They cannot guarantee that edited text mimics a specific reference font without fine-tuning or font labels.

The Gap: There is a critical need for a system that enables zero-shot, font-controllable text editing where users can provide a single reference glyph patch to dictate the style, without requiring font labels or test-time fine-tuning.

2. Methodology

SkyReels-Text introduces a novel framework that decouples textual content from typographic style using a dual-stream visual conditioning mechanism.

A. Data Curation and Content-Style Decoupling

To train the model, the authors curated a high-quality dataset of 100,000 aligned original-edited image pairs using a specialized pipeline:

VLM-Based OCR: A fine-tuned Qwen2.5-VL 7B model is used to detect and recognize text in complex, ornamental, or calligraphic fonts that standard OCR (like PP-OCR) fails to handle.
Content-Style Decoupling Pipeline:
- Divergent Text Generation: An LLM (Qwen3-8B) generates semantically distinct replacement text for detected regions.
- Local Editing: The Nano Banana model replaces the original text with the new content while preserving the original font, color, and alignment.
- Dual-Verification: A rigorous validation block ensures:
  - Content Accuracy: The new text is correctly recognized by the VLM-OCR.
  - Style Consistency: DINO v2 features are extracted to verify that the edited text retains the visual style of the original.
Dataset: The result is a dataset where text content and font style are completely decoupled, allowing the model to learn style transfer independently of semantics.

B. Model Architecture

SkyReels-Text is built upon the Qwen-Image-Edit foundation model with LoRA fine-tuning. It employs a Dual-Stream Visual Conditioning Mechanism:

Input Streams:
1. Plain-Text Reference ( $X_{text}$ ): The target text rendered in a standard font within the image context (providing semantic content and layout).
2. Glyph Map ( $X_{glyph}$ ): A visual patch of the reference font style (providing the typographic style).
Conditioning: Instead of using heavy embedding modules, the model uses a frozen VAE Encoder to extract features from both $X_{text}$ and $X_{glyph}$ . These features are concatenated with the noisy image latent ( $z_t$ ) along the sequence dimension.
Training Objective: A Text-Region Weighted Loss is introduced. Since text occupies a small fraction of the image, the loss is spatially reweighted using a text mask (generated by SAM2) to amplify optimization on glyph regions ( $\lambda=5$ ), ensuring high fidelity without distorting the background.

C. Inference Acceleration

To achieve fast inference, the model is distilled using the Improved Distribution Matching Distillation (DMD2) paradigm, resulting in an 8-step student model that achieves a 10x speedup while maintaining font awareness.

3. Key Contributions

Novel Visual Conditioning Strategy: A framework that uses reference glyph patches for fine-grained, font-controllable editing without requiring font labels or fine-tuning.
Specialized VLM-OCR System: A robust OCR pipeline based on Qwen2.5-VL capable of recognizing ornamental and calligraphic fonts, overcoming limitations of traditional OCR.
Content-Style Decoupled Dataset: A dedicated pipeline generating 100K high-quality, aligned image pairs with maximally divergent content but identical styles.
State-of-the-Art Performance: Comprehensive evaluation showing superior performance in semantic accuracy, layout alignment, and font style fidelity across multiple benchmarks.

4. Experimental Results

The model was evaluated on the SkyReels-Text Benchmark, AnyText Benchmark (English/Chinese), and handwritten text datasets (IAM, CVL).

Quantitative Performance:
- SkyReels-Text Benchmark: Achieved 83.34% Sentence Accuracy and 95.02% NED, outperforming competitors like FLUX-Text and Qwen-Image-Edit. It also achieved the highest DINO similarity (0.8503), indicating superior style preservation.
- AnyText Benchmark: Set new SOTA records for both English (85.36% Sen. Acc) and Chinese (77.10% Sen. Acc), with the lowest FID and LPIPS scores, indicating better distributional and perceptual alignment.
- Handwritten Text: Despite no explicit training on handwritten data, it achieved 1.32 HWD on IAM and 0.89 HWD on CVL, outperforming specialized few-shot and one-shot models like DiffBrush.
Qualitative Results:
- Successfully edits multiple text regions with heterogeneous font styles in a single inference pass.
- Preserves background integrity and complex textures (e.g., metallic gloss, 3D effects) better than commercial models.
- Handles small text (down to ~20 pixels) and paragraph-level edits effectively.
User Study: In a preference study against four commercial models (Qwen-Image-Edit, FLUX-Kontext-Pro, Nano Banana, Seedream 4.0), SkyReels-Text won 71.8% to 84.5% of the time, demonstrating clear user preference for its reliability and style fidelity.

5. Significance

SkyReels-Text bridges the gap between general-purpose image editing and professional-grade typographic design. Its significance lies in:

Zero-Shot Flexibility: It allows designers to use any font style from a single image patch, bypassing the need for font libraries or complex prompt engineering.
Professional Utility: It solves a critical workflow bottleneck in advertising and media design, enabling rapid, precise revisions of poster content without sacrificing aesthetic intent.
Generalization: The ability to generalize to handwritten and multilingual scripts without language-specific training makes it a versatile tool for global design applications.

The code and models are publicly available, fostering further research in reference-driven visual generation.