FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography

This paper introduces FontUse, a data-centric approach that leverages a large-scale, automatically annotated dataset of 70K images with style- and use-case-conditioned prompts to fine-tune text-to-image models, significantly improving their ability to generate typography that accurately reflects requested visual attributes without requiring architectural changes.

The Gist

Imagine you have a super-smart robot artist. You can ask it to draw a "futuristic city" or a "cozy coffee shop," and it does a fantastic job. But if you ask it to write the word "CAFE" on a sign in that coffee shop, the robot usually gets confused. It might spell it wrong, make the letters look like a messy scribble, or ignore your request to make the letters look "elegant" or "handwritten."

The paper "FontUse" is like a new training manual for this robot artist. It teaches the robot how to become a master calligrapher and graphic designer, not just a painter.

Here is the breakdown of how they did it, using some simple analogies:

1. The Problem: The Robot is "Style-Blind"

Think of current AI art generators as a talented chef who can cook a delicious meal but doesn't know how to plate it. If you ask for a "fancy, elegant dessert," the chef might make a great cake but put it on a dirty paper plate.

• The Issue: AI models are great at making pictures, but they are terrible at controlling the style of the text inside those pictures. They don't understand the difference between a "grungy, rock-and-roll font" and a "clean, modern font."

2. The Solution: A Massive "Style Dictionary"

The authors didn't try to rebuild the robot's brain (the AI architecture). Instead, they realized the robot just needed better ingredients (data).

• The Analogy: Imagine you want to teach a child to recognize different types of shoes. You could show them 10 pictures, or you could show them 70,000 pictures, each labeled with exactly what kind of shoe it is, where it's worn, and what it looks like.
• What they did: They built a huge dataset called FontUse containing 70,000 images of text. But they didn't just save the pictures; they added a "smart tag" to every single one.

3. The Secret Sauce: Two-Part Instructions

The magic of this paper is how they labeled the data. They taught the AI to understand text through two lenses, like a pair of glasses:

1. The "Look" (Font Style): Is it fancy? Is it messy? Is it 3D? (e.g., "Elegant," "Handwritten," "Distorted").
2. The "Job" (Use Case): Where is this text supposed to go? (e.g., "A wedding invitation," "A video game logo," "A coffee shop menu").

The Analogy:
If you tell a human designer, "Write the word 'Love'," they might write it in a messy scrawl.
But if you say, "Write the word 'Love' for a wedding invitation," they instantly know to use a fancy, flowing script.
If you say, "Write the word 'Love' for a heavy metal band poster," they will use a jagged, scary font.
FontUse teaches the AI to make that same connection automatically.

4. How They Did It: The "AI Interns"

They didn't hire 100 human designers to label 70,000 images (that would take forever and cost a fortune). Instead, they used other AI models as "Interns" to do the labeling.

• The Process: They took a picture of text, used one AI to find the words, and then used a "Super-Designer AI" (a Multimodal Large Language Model) to look at the picture and write a description like: "This is a playful, bubbly font perfect for a children's toy box."
• The Result: They created a massive library where every image is paired with a clear description of its style and its purpose.

5. The Result: The Robot Gets a Promotion

They took existing AI art generators (like Stable Diffusion) and "fine-tuned" them using this new library.

• Before: You asked for a "futuristic logo," and the AI gave you a generic, boring block of text.
• After: You ask for a "futuristic logo," and the AI generates text that looks sleek, geometric, and sci-fi, exactly matching your request.

Why This Matters

• No More Guessing: You don't have to try 50 different prompts to get the right font. You just tell the AI the "vibe" and the "job," and it gets it right.
• Better Design: It helps non-designers create professional-looking graphics for things like business cards, social media posts, or book covers without needing to know what a "serif" or "sans-serif" font is.
• Legibility: Even with all these fancy styles, the text is still easy to read. The AI didn't just make it look cool; it made sure the letters are spelled correctly.

In a nutshell: The authors realized that the problem wasn't that the AI wasn't smart enough; it just hadn't been taught the language of design. By feeding it a massive, well-organized dictionary of "styles" and "jobs," they taught the AI to finally write text that looks exactly the way you imagine it.

Technical Summary

Here is a detailed technical summary of the paper "FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography."

1. Problem Statement

Recent text-to-image (T2I) models have achieved high visual fidelity but struggle with controllable in-image typography. While users can prompt for text content, current models fail to reliably adhere to specific typographic attributes (e.g., font style, weight, texture) and usage contexts (e.g., "wedding invitation," "tech startup logo").

• The Bottleneck: The primary limitation is not model capacity, but a lack of structured supervision in training data. Existing datasets focus on text detection/recognition (geometric boxes and OCR) or isolated glyph generation, lacking annotations for design intent, style nuances, and application scenarios.
• The Consequence: Models ignore or weakly interpret prompts regarding font aesthetics and use cases, forcing users into trial-and-error prompt engineering.

2. Methodology

The authors propose a data-centric framework that improves typography control by constructing a specialized, large-scale dataset with structured annotations, rather than modifying model architectures.

A. The FontUse Dataset Construction Pipeline

The core contribution is the creation of FontUse, a dataset of ~70,000 typography-focused images with structured annotations. The pipeline involves:

1. Data Collection: Sourcing images from public font-design resources.
2. Region Localization: Using Hi-SAM (Segment Anything Model) to detect word-level text regions and generate bounding boxes.
3. Text Recognition (OCR): Employing a Multimodal Large Language Model (MLLM) (GPT-4o-mini) with a specialized prompt to extract text. This MLLM is more robust to stylized, distorted, or artistic fonts than traditional OCR engines. It outputs the exact string, handling non-Roman characters and failures with specific tokens (# and -).
4. Semantic Annotation: Using an MLLM (GPT-4o) to generate structured JSON annotations for each image, containing:
   • Font Styles: Visual descriptors (e.g., "serif," "distorted," "3D," "elegant").
   • Use Cases: Concrete application scenarios (e.g., "coffee shop menu," "wedding invitation," "tech startup branding").
   • Suitability: A summary of the mood or project type.
   • Colors: Dominant text colors.
   • Constraint: The prompts enforce a strict JSON schema and focus exclusively on the main text's visual attributes to reduce noise.

B. Training Strategy

• Architecture Independence: The method does not require architectural changes to the diffusion model. It fine-tunes existing T2I models (AnyText, TextDiffuser-2, Stable Diffusion 3) using the FontUse dataset.
• Conditioning: During training, the model is conditioned on the target text string, the text region mask, and the concatenated natural language prompts describing the style and use case.
• Inference: Users can specify both the desired text and the design intent (style + use case) to generate or edit images.

C. Evaluation Framework

The authors introduce a specialized evaluation suite:

1. Long-CLIP Metric: A similarity metric based on Long-CLIP, which is fine-tuned on the FontUse dataset (excluding the generation training set). This metric specifically measures the alignment between the generated text image and the style/use-case prompt, outperforming standard CLIP.
2. MLLM-Based Pairwise Preference: Using GPT-4o and Gemini 2.5 Pro to compare generated images against baselines, focusing strictly on typography fidelity (ignoring backgrounds).
3. Legibility: Measured via Character Error Rate (CER) by human assessors transcribing the generated text.

3. Key Contributions

1. FontUse Dataset: A large-scale (70k) dataset of typography images with structured, multi-dimensional annotations (style, use case, color, region) generated via an automated MLLM pipeline.
2. Data-Centric Control Framework: A demonstration that fine-tuning existing diffusion models with style/use-case conditioned supervision significantly improves typography control without architectural modifications.
3. Dual-Axis Conditioning: A novel formulation separating Font Style (visual characteristics) and Use Case (contextual application), which allows for more intuitive and concrete prompting for non-experts.
4. Specialized Evaluation Protocols: The introduction of a fine-tuned Long-CLIP metric and MLLM-based pairwise evaluation specifically tailored for assessing typography fidelity.

4. Experimental Results

The authors evaluated fine-tuned versions of AnyText, TextDiffuser-2, and Stable Diffusion 3 against their base counterparts.

• Alignment & Preference:
  • Fine-tuned Long-CLIP Scores: Fine-tuned models achieved significantly higher alignment scores (e.g., AnyText FT: 0.6368 vs. Base: 0.5858) compared to baselines.
  • MLLM Preference: In pairwise comparisons, fine-tuned models were preferred by GPT-4o and Gemini in ~80% of cases (e.g., 48/60 wins for AnyText FT).
  • Use-Case Adaptation: Qualitative results showed that fine-tuned models successfully adapted glyph characteristics to match specific contexts (e.g., making text look "rugged" for a "creative project" vs. "elegant" for a "boutique"), whereas baselines showed little variation.
• Legibility:
  • Contrary to the fear that stylistic control degrades readability, fine-tuned models improved legibility.
  • CER Reduction: AnyText's average CER dropped from 9.68% (base) to 5.91% (fine-tuned). SD-3 improved from 19.07% to 8.29%.
• OCR Pipeline Robustness:
  • The MLLM-based OCR pipeline achieved a 2.18% CER on stylized fonts, significantly outperforming traditional engines like Tesseract (71.88%) and PaddleOCR (12.44%).
• Metric Validation:
  • The fine-tuned Long-CLIP showed a 16.0% gain in distinguishing correct use-case pairs from non-corresponding pairs, proving its superiority over standard CLIP for this specific task.

5. Significance and Conclusion

• Paradigm Shift: The paper argues that the future of controllable text generation lies in data-centric supervision rather than complex architectural engineering. By providing models with rich, structured examples of "what a font looks like in a specific context," control is achieved naturally.
• Practical Utility: The approach enables novice users to generate professional-grade typography by simply describing the vibe and application (e.g., "dreamy font for a wedding invitation") rather than needing technical font knowledge.
• Limitations: The current system focuses on English typography. There is also a noted trade-off where highly abstract/decorative styles can occasionally compromise legibility, a challenge that persists even in base models.

In summary, FontUse demonstrates that with the right structured data and supervision signals, existing diffusion models can be transformed into powerful, controllable typography engines that respect both aesthetic style and functional context.