EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering

This paper introduces EasyText, a controllable Diffusion Transformer framework that leverages character positioning encoding and position interpolation to achieve high-quality, precise multilingual text rendering, supported by a newly constructed large-scale synthetic dataset for pretraining and fine-tuning.

The Gist

Imagine you want to create a movie poster, a magazine cover, or a meme, and you need the text to look perfect. You want the words to say exactly what you mean, in any language (English, Chinese, Japanese, etc.), and you want them to sit naturally on the image—maybe curved around a smile, or slanted on a sign—without looking like a clumsy sticker pasted on top.

For a long time, AI image generators were terrible at this. They could draw a beautiful cat, but if you asked for the word "Cat" on its forehead, the AI would often spell it wrong, use gibberish, or make the letters look like they were floating in a different dimension.

EasyText is a new AI tool that solves this problem. Here is how it works, explained simply:

1. The Core Idea: "Imitation" vs. "Memorization"

Most AI models try to memorize how to write. They try to recall the rules of grammar and spelling for thousands of languages. This is like asking a student to recite a dictionary from memory; it's hard, and they often make mistakes with rare words.

The authors of EasyText realized that humans learn to write differently. When we learn a new language, we often start by imitating the shapes we see, even if we don't understand the meaning yet. We trace the lines.

• The Analogy: Think of EasyText as a master calligraphy student rather than a dictionary. Instead of trying to "remember" how to write 30,000 Chinese characters, it looks at a sample of the character and says, "Okay, I see the shape. I will copy that shape onto the image." It treats writing more like drawing than reading.

2. The Two-Stage Training (The "Apprentice" System)

To teach this AI to be a master calligrapher, the researchers used a clever two-step training process:

• Stage 1: The "Sandbox" (Pre-training)
  Imagine a massive digital playground with 1 million images. In this playground, the AI sees text overlaid on random pictures (like a beach, a city, or a forest).

  • The Trick: The AI sees the text in one font (the "condition") but has to draw it in a different font on the picture.
  • Why? This forces the AI to learn the structure of the letters (the skeleton) rather than just copying the exact style. It learns that a "T" is a "T" whether it's in Times New Roman or Comic Sans. This is like a student practicing the same letter in 30 different handwriting styles so they truly understand the shape.

• Stage 2: The "Fine-Tuning" (Polishing)
  After the sandbox, the AI is shown 20,000 high-quality, professional images where text fits perfectly into the scene.

  • The Goal: Here, the AI learns to blend the text with the background. It learns that if the text is on a wet road, the letters should look a bit shiny; if it's on a brick wall, the letters should look rough. This makes the text look like it was always part of the photo, not added later.

3. The Secret Sauce: "Invisible GPS" for Letters

One of the hardest parts of text rendering is telling the AI where to put the letters.

• Old Way: You had to draw a box around where the text should go, which was rigid and clunky.
• EasyText Way: They invented something called Implicit Character Position Alignment.
  • The Analogy: Imagine you have a transparent sheet with the letters drawn on it. You want to stick this sheet onto a curved, slanted, or wavy surface. Instead of forcing the sheet to stay flat, you use a magical elastic grid.
  • The AI stretches and bends this invisible grid to match the shape of the area where you want the text. Whether you want the text to curve around a coffee cup or slant up a ramp, the AI stretches the "position grid" to fit perfectly, ensuring the letters land exactly where they should.

4. Why This Matters

• Multilingual Magic: It doesn't just do English. It handles Chinese, Japanese, Korean, Thai, Greek, and more, all in one model. It's like a universal translator that can also draw.
• Unseen Characters: Because it learned to "imitate shapes" rather than "memorize words," it can even draw characters it has never seen before, as long as it can see the shape of them.
• Long Text: It can write whole paragraphs, not just single words, and keep the layout neat.

Summary

EasyText is like giving an AI a pair of "mimicry glasses" and a "stretchy ruler."

1. Mimicry Glasses: It learns to copy the shape of letters from any language, treating them like drawings.
2. Stretchy Ruler: It uses a flexible grid to place those letters exactly where you want them, even on weird, curved, or slanted surfaces.

The result? You can type a prompt, say "A sign on a futuristic building that says 'Welcome' in Japanese," and the AI will generate an image where the Japanese text looks perfectly written, perfectly placed, and naturally part of the scene.

Technical Summary

Here is a detailed technical summary of the paper "EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering."

1. Problem Statement

Generating accurate, high-quality text in images using diffusion models has been a persistent challenge, particularly for multilingual scenarios. Existing state-of-the-art models (e.g., FLUX-dev, SD3.5, Ideogram) are predominantly limited to English or struggle with complex layouts. The specific challenges identified are:

• Multilingual Complexity: The character space expands drastically when including logographic systems (e.g., Chinese with 30k+ characters) alongside alphabetic scripts, making joint modeling difficult, especially for rare characters.
• Visual Integration: Rendered text often appears "pasted on," lacking natural integration with the scene background, leading to visual artifacts.
• Generative Prior Degradation: Fine-tuning on large text-image datasets often improves text rendering but degrades the model's general image generation capabilities.
• Control Limitations: Existing methods struggle with precise positioning, irregular regions (curved/slanted text), and long text passages.

2. Methodology

The authors propose EasyText, a framework based on Diffusion Transformers (DiT) that shifts the paradigm from "recalling" text (semantic generation) to "imitating" text (visual glyph generation).

A. Core Architecture & Training Strategy

EasyText employs a two-stage training strategy based on the open-source FLUX model:

1. Stage 1: Large-Scale Pre-training:
   • Trained on a synthetic dataset of 1 million multilingual image-text pairs.
   • Glyph Imitation Strategy: The target text is rendered in diverse fonts, while the condition image uses a standardized font. This forces the model to learn the structure of characters (glyphs) rather than simply copying specific font shapes.
   • Data Efficiency: Text is overlaid randomly on natural images to learn glyph features without requiring massive real-world annotated datasets.
2. Stage 2: Fine-tuning:
   • Fine-tuned on a high-quality dataset of 20,000 annotated multilingual scene text images.
   • Uses LoRA (Low-Rank Adaptation) to enhance visual consistency between text and background while preserving the base model's generative priors.

B. Key Technical Components

• Character Representation (EasyFont):

  • Instead of symbolic tokens, characters are encoded as image patches via a VAE.
  • Alphabetic scripts: Encoded as variable-width images (64px height) to preserve inter-character spacing.
  • Logographic scripts (e.g., Chinese): Encoded as fixed 64×64 pixel squares to respect the isolated nature of ideograms.
  • These "font tokens" are concatenated with denoising latents in the latent space.

• Implicit Character Position Alignment (ICPA):

  • A novel mechanism to control text placement without explicit layout masks. It maps the spatial coordinates of the condition image to the target region using positional encoding interpolation.
  • Linear Alignment: Uses affine transforms for rectangular regions.
  • Non-linear Alignment: Uses Thin-Plate Spline (TPS) interpolation for irregular, curved, or slanted regions.
  • Layout-Free Generation: Supports text generation without specific positioning by injecting positional offsets, allowing the model to place text naturally.

3. Key Contributions

1. The "Imitation" Paradigm: Proposes training AI to mimic glyph morphology rather than recall semantic meaning, significantly improving multilingual generalization.
2. Implicit Character Position Alignment (ICPA): Introduces a flexible control strategy using positional encoding interpolation (including TPS) to handle both precise positioning and irregular text regions.
3. Data-Efficient Pipeline: Demonstrates that high-quality multilingual rendering can be achieved with only 20k fine-tuning samples by leveraging a large-scale synthetic pre-training phase with multi-font decoupling.
4. Comprehensive Capabilities: The model supports:
   • Multilingual rendering (10+ languages including Chinese, Japanese, Thai, Greek).
   • Long text generation (multiple paragraphs).
   • Unseen character generalization.
   • Irregular region rendering (curved/slanted text).

4. Experimental Results

The authors evaluated EasyText against commercial models (FLUX-dev, Jimeng AI, SD3.5) and academic baselines (AnyText, Glyph-ByT5-v2).

• Accuracy:
  • Character-Level Precision: Achieved 99.45% for English and 93.12% for Chinese (fine-tuned), outperforming most baselines.
  • Sentence-Level Precision: High consistency across languages, though slightly behind Jimeng AI in Chinese sentence-level accuracy.
  • Multilingual Performance: Demonstrated strong performance in unsupported languages like Thai and Greek, where other models failed.
• Visual Quality & Integration:
  • CLIPScore: Improved from 0.3318 (pre-trained) to 0.3486 (fine-tuned), surpassing Glyph-ByT5-v2 and approaching SD3.5.
  • OCR Accuracy: Reached 88.72%, significantly higher than FLUX (76.09%) and SD3.5 (79.33%).
  • User Study (GPT-4o): Rated highest in Text-Image Fusing (81.28) and Text Quality (90.66), indicating superior aesthetic coherence and legibility.
• Qualitative Findings: EasyText successfully renders text in complex layouts, curved regions, and handles long passages (4-5 paragraphs) with consistent style, which commercial models often fail to do.

5. Significance

• Bridging the Multilingual Gap: EasyText provides a robust solution for non-English text rendering, a critical bottleneck for global AI applications.
• Efficiency: It challenges the notion that massive real-world annotated datasets are required for text rendering, proving that synthetic pre-training with specific font-decoupling strategies is highly effective.
• Control & Flexibility: By decoupling text content from layout via ICPA, it offers unprecedented control over text placement and shape, enabling applications in design, advertising, and scene generation that were previously difficult to automate.
• Open Source: The release of the model, code, and datasets (1M synthetic + 20K annotated) fosters further research in multilingual generative AI.

In conclusion, EasyText represents a significant advancement in controllable text-to-image generation, successfully balancing high-fidelity multilingual rendering with strong visual integration and precise spatial control.