Technical Summary: Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

1. Problem Statement

Text-to-Image (T2I) generation has evolved from producing visually plausible images to supporting photorealistic reconstruction and complex creative expression. However, existing evaluation benchmarks remain anchored in foundational criteria such as coarse text-image alignment and generic visual quality, often relying on CLIP similarity or unsupervised Multimodal Large Language Models (MLLMs) as judges. The authors identify three critical limitations in current evaluation paradigms:

1. Saturation on Basic Axes: As models approach the ceiling on common prompt-following, existing benchmarks compress strong models into indistinguishable score bands, lacking the discriminative power to differentiate state-of-the-art systems.
2. Insufficient Characterization of Real-world Fidelity: Current suites fail to distinguish between visually plausible outputs and those that genuinely respect physical commonsense, cultural grounding, or factual structure.
3. Lack of Expert Supervision in Creative Generation: Benchmarks often lack fine-grained taxonomies for creativity (originality, stylistic coherence) and increasingly rely on single MLLM judges, which inherit systematic biases rather than reflecting professional human standards grounded in trained aesthetic intuition.

Consequently, there is a gap between benchmark objectives and the actual demands of professional creators, who require faithful reconstruction of reality and authentic creative expression.

2. Methodology

The authors introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios. The methodology consists of three tightly coupled components:

A. Hierarchical Capability Taxonomy

Instead of a flat checklist, the benchmark employs a top-down taxonomy mirroring professional artistic workflows (ideation, styling, iterative refinement). It is structured into three levels:

• Level 1 (Pillars): Five dimensions: three conventional (Quality, Aesthetics, Text-Image Alignment) and two application-driven (Real-world Fidelity and Creative Generation).
• Level 2 (Sub-capabilities): 23 sub-capabilities (e.g., World Knowledge, Visual Storytelling, Design Applications).
• Level 3 (Facets): 56 verifiable rubrics (e.g., Text Accuracy, Physical Logic, Anatomical Fidelity).
  This structure allows for fine-grained diagnostics, localizing model strengths and weaknesses to specific artistic sub-skills rather than aggregate scores.

B. Expert-in-the-Loop Prompt Construction

The benchmark comprises 1,000 stratified bilingual prompts (500 Chinese, 500 English), balanced between short and long lengths to test robustness. The prompts are generated via a four-stage pipeline:

1. Facet-targeted sampling: Ensuring each prompt exercises at least three pillars and multiple facets.
2. Bilingual drafting: Using LLMs to generate aligned drafts with explicit instantiation notes.
3. Expert review: Professional artists discard ambiguous prompts and rewrite those with weak instantiation.
4. Length-variant expansion: Creating long variants to test scene context richness.
   This process ensures prompts are anchored in concrete creative scenarios (e.g., recipe journals, game design, cinematic storyboards) rather than vague requests.

C. Unified Judge Model (Q-Judger)

To avoid the biases of single MLLM judges and the cost of manual review, the authors trained Q-Judger, a unified model based on Qwen3.6-27B.

• Training Data: Supervised by 80 professional annotators from art academies (photography, directing, fine arts) under blind labeling and triple-review protocols.
• Scoring Protocol: The model scores every prompt-image pair across all 56 third-level facets independently using a rubric-grounded protocol (0: Fail, 1: Pass, 2: Excel, N/A).
• Aggregation: Scores are aggregated bottom-up from L3 to L2, L1, and finally to an overall score, ensuring that aggregate metrics are traceable to specific fine-grained failures.

3. Key Results

The benchmark was used to evaluate 18 leading T2I models (including GPT Image 2, Nano Banana Pro, Seedream 5.0, and Qwen Image 2.0 Pro) under both Chinese and English prompts.

• Overall Performance: GPT Image 2 achieved the highest overall score (64.69 on Chinese prompts), followed by Nano Banana 2.0 and GPT Image 1.5. The models naturally separated into five distinct performance tiers, demonstrating the benchmark's high discriminative power.
• Discriminative Power of New Dimensions: The two application-driven pillars, Creative Generation and Real-world Fidelity, exhibited the largest inter-model variance (over 11× that of Quality). This confirms that these dimensions target capability gaps where existing benchmarks provide little insight.
• Systemic Ceilings: Four Level 3 facets—Physical Logic, Anatomical Fidelity, Animals, and Contact Interaction—emerged as universal bottlenecks. Even the best-performing models scored below 44 on these facets, indicating a "perception-to-cognition" frontier where models struggle with implicit world knowledge and logical reasoning.
• Cross-Lingual Robustness: The findings and tier structures remained consistent across Chinese and English prompts, validating the benchmark's language independence.
• Judge Model Alignment: Q-Judger achieved a Spearman rank correlation of 0.92 with human expert rankings, confirming its ability to track professional judgment while significantly reducing evaluation costs.

4. Significance and Claims

The paper claims that Qwen-Image-Bench represents a shift from "generation" (basic alignment) to "creation" (professional utility). Its significance lies in:

1. Creator-Centric Design: By co-designing with artists, the benchmark evaluates dimensions that matter in professional workflows (e.g., World Knowledge, Text Rendering, Visual Storytelling) rather than just semantic correctness.
2. Granular Diagnostics: The 56-facet granularity provides actionable insights for model improvement, moving beyond opaque aggregate scores to identify specific sub-skill deficits.
3. Revealing the Perception-Cognition Gap: The evaluation exposes that current T2I models have mastered visual surface reproduction (style, aesthetics) but lack the latent understanding of physical laws, biological structure, and spatial reasoning required for true real-world fidelity.
4. Trustworthy Optimization Signal: By relying on human-expert supervision rather than MLLM proxies, the benchmark provides a reliable signal for production-level T2I development, particularly for the application-driven dimensions where current models diverge most sharply.

The authors conclude that closing the gap between current capabilities and professional needs requires deeper integration of world knowledge and structured reasoning into the generation process, a direction explicitly highlighted by the benchmark's diagnostic capabilities.