TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards

Technical Summary: TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards

Problem Statement
Faithful text rendering remains a persistent weakness in large text-to-image generative models. While these models excel at synthesizing high-fidelity images from natural language, they struggle to satisfy the discrete symbolic constraints required for visual text. Unlike generic object generation, text rendering demands that intended words appear, characters are ordered correctly, glyph shapes are recognizable, and the text integrates naturally into the surrounding scene. Existing solutions typically address this through architecture-specific interventions, such as replacing text encoders, augmenting image encoders, or introducing auxiliary glyph-control modules (e.g., segmentation masks). These approaches often complicate deployment, as they are tied to specific model architectures or conditioning interfaces, making it difficult to transfer improvements to new foundation models without significant engineering effort or disruption to pretrained generative priors.

Methodology: TextAlign
The authors propose TextAlign, a non-invasive post-training preference-alignment framework designed to improve text rendering without modifying the generator's architecture. The core innovation lies in a hierarchical vision-language model (VLM)-based reward system that decomposes rendering errors into three distinct levels:

1. Global Level: Detects the presence of readable text and identifies gross structural failures, such as missing text or severely distorted glyph contours (e.g., broken strokes, warped baselines).
2. Word Level: Evaluates semantic faithfulness by identifying dropped, inserted, or substituted whole words.
3. Glyph Level: Assesses fine-grained typographic precision by detecting character-level insertions, deletions, and substitutions.

For each generated image and its reference text, independent VLM calls produce binary defect indicators for each level. These indicators are aggregated into a single scalar reward signal representing the fraction of defect-free categories. This scalar reward is compatible with two major preference optimization paradigms:

• Group Relative Policy Optimization (GRPO): The reward is normalized within a group of samples to compute relative advantages.
• Direct Preference Optimization (DPO): The reward is used to rank samples generated under the same condition to construct winner/loser pairs.

Crucially, the framework requires no additional trainable modules, auxiliary branches, or model-specific supervision signals, preserving the base model's architecture and general generative capabilities.

Key Contributions

• Non-Invasive Alignment: Demonstrates that text rendering can be improved through reward design and post-training alignment rather than architectural redesign, allowing the same "recipe" to be applied to heterogeneous foundation models (e.g., FLUX.1-dev, Z-Image-Turbo, SD3.5, Qwen-Image).
• Hierarchical Reward Design: Introduces a multi-level reward decomposition that distinguishes between global structural failures, semantic word errors, and fine-grained glyph defects, addressing the limitations of single-metric rewards like OCR edit distance.
• Optimizer Agnosticism: Shows that the same hierarchical reward signal effectively drives both GRPO and DPO, offering a general post-training interface for text rendering.

Experimental Results
The authors evaluated TextAlign on FLUX.1-dev and Z-Image-Turbo, comparing against strong foundation models (SD3.5, Qwen-Image) and specialized text-rendering systems (AnyText, TextDiffuser).

• Text Accuracy: Both DPO and GRPO variants consistently improved OCR-based text accuracy metrics (Normalized Edit Score, Precision, Recall, F1-score, and Exact Word-Set Accuracy) compared to their respective base models. On the Z-Image-Turbo backbone, the GRPO-aligned model achieved the best scores across all OCR metrics among all compared methods.
• General Generation Quality: The alignment process did not degrade general image generation quality. Metrics such as CLIPScore, ImageReward, PickScore, and HPSv3 remained competitive, and human studies indicated that aligned models were preferred for both text fidelity and visual integration.
• Robustness and Generalization: The aligned models maintained stable performance across varying text lengths (short to long), spatial placements (front, middle, back), and ten diverse visual categories (e.g., posters, logos, handwriting, academic scenes). The framework also demonstrated transferability, showing gains on an external MARIO-Eval benchmark without retraining on that specific data.
• Ablation Studies: Removing any of the three reward levels (global, word, or glyph) resulted in performance drops, confirming that the hierarchical structure is complementary and necessary for addressing the full spectrum of rendering failures.

Significance and Claims
The paper claims that reliable visual text rendering can be significantly improved through carefully designed reward modeling and post-training alignment, eliminating the need for model-specific architectural redesign. By treating text rendering as a preference-alignment problem, TextAlign offers a scalable, model-agnostic alternative to the complex engineering required by prior architecture-level interventions. The results suggest that the "recipe" of hierarchical reward design can be applied to different image generators to enhance their typographic precision while preserving their broad visual competence.