Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning

1. Problem Statement

While Unified Multimodal Large Language Models (MLLMs) have advanced significantly in combining image understanding and generation, their application to Text-to-Image (T2I) generation via reasoning remains underutilized. Existing approaches suffer from two primary limitations:

• Holistic Judgment Errors: Current reasoning-based methods (e.g., Janus-Pro-R1) rely on global, coarse-grained alignment judgments. They often fail to detect fine-grained semantic mismatches (e.g., incorrect object counts, specific color attributes, or spatial relations), leading to high False Positive rates where misaligned images are incorrectly deemed correct.
• Rationale Bypass & Lack of Localization: When these models do attempt self-correction, they often rely on "regeneration cycles" that regenerate the entire image based on a global "misaligned" signal. Crucially, they often bypass the specific semantic content of the reasoning (rationale), failing to use explicit feedback to target specific errors. This results in inefficient full-image regeneration that may lose correctly generated details.

2. Methodology: FiMR Framework

The authors propose Fine-grained Multimodal Reasoning (FiMR), a framework that enables MLLMs to perform iterative, localized self-correction. The framework operates through a three-step loop:

Step 1: Initial Text-to-Image Generation

The MLLM generates an initial image ($I$) from the input prompt ($P$) using standard generation capabilities.

Step 2: Fine-grained Feedback Generation (Decomposed VQA)

Instead of a holistic judgment, the framework decomposes the prompt into minimal semantic units (tuples) covering entities, attributes, relations, counts, and text.

• Prompt Summarization: The prompt is condensed to remove subjective/flavor text, creating a stable evaluation basis ($P_{summary}$).
• Tuple Decomposition: $P_{summary}$ is broken down into atomic semantic tuples (e.g., entity(oven), attribute(oven, pink)).
• Decomposed VQA: The model performs Visual Question Answering (VQA) for each tuple individually against the generated image.
• Explicit Feedback Synthesis: Based on the binary verification results ("Yes"/"No") for each tuple, the model generates explicit, actionable feedback instructions ($F$) only for the failed units. If all units pass, a termination signal is issued.

Step 3: Localized Image Correction

• If the image is aligned, the process stops.
• If misaligned, the MLLM uses the explicit feedback ($F$) to perform localized edits. Instead of regenerating the whole image, it targets specific erroneous regions to correct the identified mismatches while preserving correctly aligned areas.
• This loop repeats until alignment is achieved.

Training: The authors construct a specialized reasoning dataset involving 200K image-text pairs (from FocusDiff and synthetic composition data). They perform Supervised Fine-Tuning (SFT) on a base MLLM (Janus-Pro) to teach the model the full reasoning path: $\Phi = \{P_{summary}, T, V, F\}$.

3. Key Contributions

1. Iterative Fine-grained Framework: A novel three-step framework (Generation $\to$ Decomposed Feedback $\to$ Localized Correction) that shifts T2I reasoning from global regeneration to precise, targeted editing.
2. Decomposed VQA Mechanism: A methodology to generate explicit feedback by verifying minimal semantic units independently, effectively solving the "False Positive" problem of global judges.
3. Empirical Validation: Extensive experiments demonstrating that FiMR significantly outperforms existing reasoning-based baselines, particularly in compositional tasks requiring high precision (counting, spatial relations, and attribute binding).

4. Experimental Results

The authors evaluated FiMR on three major benchmarks: GenEval, T2I-CompBench, and DPGBench.

• Quantitative Performance:

  • GenEval: FiMR showed progressive improvement across iterations. Notably, it achieved a Counting score of 0.70 (3rd iteration) compared to 0.51 for the baseline Janus-Pro-R1, demonstrating superior handling of discrete object constraints.
  • T2I-CompBench & DPGBench: FiMR consistently outperformed baselines (including T2I-R1 and Janus-Pro-R1) in color, shape, texture, and spatial categories. On DPGBench (long, dense prompts), FiMR achieved the highest overall score (85.36) by the third iteration.
  • Ablation Studies:
    • Global vs. Fine-grained: Replacing the Global Judge with Decomposed VQA reduced the False Positive (FP) rate in counting tasks from 32.50% to 15.94%.
    • Explicit Feedback: Removing explicit instructions (w/o EF) resulted in lower performance (0.85 vs. 0.86 overall), proving that actionable guidance is critical for effective correction.
    • Scalability: The framework proved effective even on smaller models (Janus-Pro 1B), showing consistent gains over the baseline.

• Qualitative Analysis: Visual comparisons showed that while baselines often failed to correct subtle errors or regenerated entire scenes (losing correct details), FiMR successfully pinpointed and fixed specific mismatches (e.g., changing a green motorcycle to a yellow one with a bench) while preserving the rest of the scene.

5. Significance

This work addresses a critical bottleneck in unified MLLMs: the inability to effectively self-correct specific errors in generated images. By moving from holistic, coarse-grained reasoning to fine-grained, decomposed verification, FiMR enables:

• Higher Precision: Drastic reduction in semantic errors regarding object counts, colors, and spatial relations.
• Efficiency: Localized corrections are more computationally efficient and preserve high-quality generated content compared to full-image regeneration.
• Controllability: The explicit feedback mechanism makes the generation process more transparent and controllable, offering a robust path toward reliable, instruction-following image generation for complex, compositional tasks.

The paper concludes that fine-grained multimodal reasoning is essential for the next generation of text-to-image systems, bridging the gap between understanding complex prompts and executing precise visual edits.