💬 NLP

Evaluating Reasoning Fidelity in Visual Text Generation

This paper evaluates the reasoning fidelity of current text-to-image models in visual text generation and finds that, despite producing legible text, they frequently fail to accurately render complex reasoning processes due to semantic errors and logical inconsistencies, revealing a significant gap between visual text generation and procedural reasoning capabilities.

Original authors: Jiajun Hong, Jiawei Zhou

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiajun Hong, Jiawei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two different types of artists.

Artist A (The Text-Only LLM) is a brilliant mathematician and logician. If you ask them, "What is the smallest perfect cube that is the sum of three consecutive integers?" they can instantly write out the solution on a piece of paper. Their logic is perfect, their steps are clear, and the answer is right.

Artist B (The Text-to-Image Model) is a talented painter who has recently learned a new trick: they can paint words directly onto a canvas. They can make the letters look beautiful, crisp, and easy to read.

This paper asks a simple but tricky question: If we ask Artist B to paint the entire step-by-step solution to a math problem, will the logic inside the painting actually make sense, or will they just be painting pretty nonsense?

The Experiment: Painting the Thought Process

The researchers set up a series of challenges to test these "painting" models (Text-to-Image or T2I models). They didn't just ask the models to write a single word; they asked them to paint:

Long documents: Can the model paint a whole page of text without mixing up the letters?
Facts: Can the model paint the correct answer to a science question, along with the reasons why?
Context: Can the model read a long story (painted as text) and answer a question about it?
Math: Can the model paint a multi-step math solution?

The Findings: Pretty Pictures, Broken Logic

The results were surprising and a bit disappointing for the "painting" models.

1. The "Handwriting" Problem (Rendering)
First, the researchers checked if the models could even paint the words clearly.

The Issue: Some models were like a child with a shaky hand. They would blur the letters, cut off the edges of the words, or paint extra words that weren't in the prompt.
The Analogy: Imagine asking someone to copy a recipe onto a card. Some models would write "Add 2 cups of flour" but accidentally paint "Add 2 cups of flour and sugar" or make the "2" look like a "Z".
The Result: The best models (like GPT-Image-2) got much better at this, but others (like older open-source models) failed miserably, producing unreadable scribbles.

2. The "Brain" Problem (Reasoning)
Here is the big discovery. The researchers filtered out the models that couldn't even write clearly. They took the ones that could write perfect, legible text and asked them to solve logic puzzles.

The Analogy: Imagine a student who has perfect handwriting. They write down a math solution that looks beautiful. But if you look closely at the steps, they added 2 + 2 and got 5. Or they wrote, "Because the sky is blue, the answer is 42." The handwriting is perfect, but the thinking is completely wrong.
The Result: Even when the text was perfectly clear, the T2I models frequently made logical errors.
- They would skip steps.
- They would contradict themselves in the middle of the sentence.
- They would hallucinate (make up) facts that weren't true.
- They would repeat the same step over and over like a broken record.

3. The Gap Between "Text-Only" and "Visual Text"
When the researchers compared the "painting" models to the "text-only" models (Artist A), the difference was huge.

Artist A (Text-Only): Got the math right, the logic right, and the steps right.
Artist B (Visual Text): Often got the final answer right by luck, but the steps leading up to it were a mess. Or, they got the steps wrong and the answer wrong.

The paper calls this a "Reasoning Fidelity Gap." It means that just because a model can draw the words, it doesn't mean it understands the words it is drawing.

Why Does This Happen?

The paper suggests that these models are like actors who have memorized the look of a solution but haven't learned the logic of it.

They are great at imitating the surface pattern (e.g., "Math problems usually have numbers and an equals sign").
They are bad at the deep process (e.g., actually calculating the numbers and checking if the logic holds up).

When the task gets harder (like long stories or complex math), the models start to fall apart. They might write a beautiful paragraph that looks like a solution, but if you try to follow the logic, it leads nowhere.

The Bottom Line

The paper concludes that while modern AI models are getting very good at painting words, they are still very unreliable at thinking through problems when those thoughts have to be painted out.

If you need a model to generate a document that requires strict logic (like a legal contract or a math proof), you cannot just rely on the visual text generation yet. The "handwriting" might be perfect, but the "brain" behind the brush is still prone to making mistakes that a text-only model wouldn't make. The gap between "looking smart" and "being smart" is still very wide in the world of visual text generation.

Technical Summary: Evaluating Reasoning Fidelity in Visual Text Generation

Problem Statement

While recent Text-to-Image (T2I) models have achieved significant progress in rendering legible and well-structured text, enabling applications like document and slide generation, a critical gap remains in their ability to perform reasoning fidelity. It is unclear whether these models can faithfully preserve complex reasoning processes when solutions must be expressed directly through rendered text, or if they merely imitate surface-level patterns. Current evaluations primarily focus on visual quality (e.g., OCR accuracy, layout fidelity) rather than the semantic correctness and logical consistency of the generated content. This paper investigates whether T2I models retain the same semantic understanding and logical consistency observed in text-only Large Language Models (LLMs) when multi-step solutions are externalized as visual text.

Methodology

The authors propose a diagnostic evaluation framework designed to disentangle text rendering failures from reasoning failures. The methodology consists of a layered protocol:

Preliminary Text Rendering Task:
- Models are tasked with reproducing input text exactly in image form using passages from WikiText.
- Input lengths are truncated to 64, 128, 256, and 512 words to control difficulty.
- Models failing to produce clear text are excluded from subsequent reasoning evaluations.
- Metrics include Word Error Rate (WER), Character Error Rate (CER), and OCR confidence (ACC).
Advanced Reasoning Tasks:
- Three distinct tasks evaluate reasoning capabilities in image space:
  - Factual Knowledge: Questions from the ARC dataset requiring reasoning steps for multiple-choice answers.
  - Context Understanding: Questions from the DROP dataset requiring discrete reasoning over long passages (e.g., resolving references, counting).
  - Math Reasoning: Competition-style problems from the MATH dataset requiring detailed multi-step derivations.
- Prompts include context, questions, and explicit instructions to generate intermediate reasoning steps and final answers.
Evaluation Pipeline:
- Extraction: Text is extracted from generated images using PaddleOCR (validated against DeepSeek-OCR).
- Scoring: An LLM-based judge (GPT-5.2) evaluates the extracted text.
  - Process Score ( $S_p$ ): Evaluates the logical validity of each intermediate reasoning step.
  - Answer Score ( $S_a$ ): Evaluates the correctness of the final answer.
- Baselines: Performance is compared against text-only LLMs (GPT-5.2, Qwen3-8B) and a diverse set of T2I models (GPT-Image-1.5/2, Gemini-2.5-Flash-Image, Flux.2-Pro, Qwen-Image, SD-XL, TextDiffuser-2).

Key Results

The experiments reveal a substantial gap between visual text generation quality and reasoning fidelity:

Rendering vs. Reasoning: While some models (e.g., GPT-Image-2) achieve high text rendering fidelity, they still struggle with logical consistency. Conversely, models with poor rendering (e.g., TextDiffuser-2, SDXL) are excluded early due to unrecognizable output.
Performance Gap: Text-only LLMs significantly outperform all T2I models across all reasoning tasks. For instance, GPT-5.2 achieves a process score of 0.934 in Math Reasoning, whereas the best T2I model (GPT-Image-2) scores 0.728.
Process vs. Answer Discrepancy: T2I models frequently produce correct final answers ( $S_a$ ) accompanied by flawed, inconsistent, or hallucinated intermediate reasoning steps ( $S_p$ ). This suggests that correct answers do not necessarily imply correct reasoning in visual generation.
Difficulty Scaling: Performance degrades consistently as task difficulty and text length increase. The gap between T2I models and LLMs is most pronounced in long-context understanding and complex mathematical reasoning.
Error Analysis:
- Rendering Errors: Include layout overflow, text blurring, instruction violations, and hallucinated extra text.
- Reasoning Errors: Occur even when text is visually clear. Common modes include logically inconsistent deductions, hallucinated intermediate steps, and repetitive reasoning patterns.
- OCR Impact: Ablation studies confirm that extraction errors (OCR choice) have limited impact on the evaluation results; the observed failures are intrinsic to the generation process.

Key Contributions

Diagnostic Framework: Introduction of a layered evaluation protocol that separates rendering fidelity from reasoning fidelity, allowing for the isolation of specific failure modes in visual text generation.
Comprehensive Benchmarking: Evaluation of state-of-the-art closed-source and open-source T2I models across long-text rendering, factual knowledge, context understanding, and multi-step mathematical reasoning.
Empirical Evidence of the Gap: Demonstration that current T2I models, even those with strong rendering capabilities, fail to faithfully preserve procedural reasoning when externalized as images, often producing logically inconsistent traces despite visually clear text.
Error Characterization: Detailed analysis of failure modes, distinguishing between rendering artifacts (blurring, layout issues) and semantic reasoning failures (logical inconsistencies, hallucinations).

Significance and Claims

The paper claims that while visual text generation has advanced, there remains a substantial gap between rendering quality and reasoning fidelity. The authors argue that:

Current T2I models are unreliable for rigorous multi-step reasoning tasks when the output must be a visual image.
The ability to render text clearly does not equate to the ability to reason correctly within that text.
The performance gap cannot be explained solely by rendering quality or extraction errors; it reflects a fundamental limitation in the models' ability to maintain logical coherence in the visual modality.

The work motivates the need for more reliable reasoning-aware visual text generation and evaluation methods. It highlights that as multimodal systems increasingly generate text-rich visual outputs for applications like document creation and interface agents, ensuring semantic correctness and logical consistency is as critical as visual legibility. The authors suggest that future research should focus on developing specialized models or distilled variants capable of reliable long-text rendering and faithful reasoning externalization.