ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks

1. Problem Statement

Despite rapid advancements in generative image models (diffusion, autoregressive, and hybrid architectures), existing evaluation benchmarks suffer from three critical limitations:

• Fragmentation: Benchmarks often focus on isolated tasks (e.g., only text-to-image or only editing) rather than unified frameworks.
• Narrow Domains: Many datasets are biased toward specific domains like artworks or photorealistic images, neglecting symbolic, text-heavy, or structured content (e.g., screenshots, information graphics).
• Opaque Evaluation: Current metrics provide scalar scores without explaining why a model failed. They lack fine-grained error attribution (e.g., specific object failures vs. global aesthetic issues), making it difficult to diagnose architectural weaknesses.

The paper argues that there is a need for a comprehensive, human-centric benchmark that stress-tests models across diverse real-world tasks and provides explainable diagnostic feedback.

2. Methodology: The ImagenWorld Benchmark

A. Dataset Construction

• Scale: The benchmark consists of 3,600 condition sets (100 samples per task-topic combination).
• Task Taxonomy: It unifies six core tasks under a single instruction-driven framework:
  1. Text-guided Image Generation (TIG)
  2. Single Reference Image Generation (SRIG)
  3. Multiple Reference Image Generation (MRIG)
  4. Text-guided Image Editing (TIE)
  5. Single Reference Image Editing (SRIE)
  6. Multiple Reference Image Editing (MRIE)
• Domain Taxonomy: The data spans six topical domains to ensure diversity:
  • Artworks, Photorealistic Images, Information Graphics, Textual Graphics, Computer Graphics, and Screenshots.
• Annotation: The dataset includes 20,000 fine-grained human annotations.

B. Explainable Evaluation Schema
Unlike traditional benchmarks that rely solely on scalar scores, ImagenWorld introduces a structured evaluation protocol:

1. Four Scoring Criteria: Annotators rate images on a 5-point Likert scale (rescaled to [0,1]) for:
   • Prompt Relevance: Faithfulness to instructions.
   • Aesthetic Quality: Visual appeal and design.
   • Content Coherence: Logical consistency (e.g., correct chart labels).
   • Artifacts: Technical flaws (e.g., distorted text, extra limbs).
2. Error Attribution (The Key Innovation): Annotators must identify specific failure modes:
   • Object-Level Issues: Missing, distorted, or incorrect objects (e.g., "missing chandelier").
   • Segment-Level Issues: Localized visual inconsistencies identified via Set-of-Mark (SoM) partitioning (e.g., "warped edges in the top-left quadrant").
3. VLM-as-a-Judge: The study also employs Vision-Language Models (specifically Gemini-2.5-Flash) to generate automated scores for comparison with human ratings.

C. Models Evaluated
The benchmark evaluated 14 models, including:

• Unified Models: Capable of all six tasks (e.g., GPT-Image-1, Gemini 2.0 Flash, BAGEL, OmniGen2).
• Task-Specific Models: Specialized for generation or editing (e.g., SDXL, Flux, InstructPix2Pix, Qwen-Image).
• Architectures: Diffusion, Autoregressive (AR), and Hybrid (AR + Diffusion) models.

3. Key Results and Insights

A. Task Difficulty: Generation vs. Editing

• Models consistently perform better on generation tasks than editing tasks.
• Editing Failure Modes: Two distinct failure patterns were observed in editing tasks:
  1. Regeneration: The model ignores the source image and generates a completely new image.
  2. No-Op: The model returns the input image unchanged.
• Unified models (especially AR-Diffusion hybrids) showed a higher tendency to "regenerate" rather than edit locally, suggesting a lack of fine-grained control mechanisms.

B. Domain Performance: The "Text-Heavy" Gap

• Strong Performance: Models excel in Artworks and Photorealistic Images (average scores ~0.78–0.90).
• Weak Performance: Models struggle significantly with Screenshots, Information Graphics, and Textual Graphics (average scores ~0.55).
• The Qwen-Image Exception: Qwen-Image outperformed even closed-source leaders in textual graphics. This is attributed to its training pipeline, which utilized synthetic data curation specifically for text-heavy images, proving that targeted data design can overcome architectural limitations.

C. Closed-Source vs. Open-Source

• Closed-source systems (e.g., GPT-Image-1, Gemini 2.0 Flash) lead overall, particularly in editing and multimodal composition.
• Open-source models are competitive in text-to-image generation (driven by community optimization) but lag significantly in editing and complex composition tasks.

D. Human vs. VLM Evaluation

• Ranking Correlation: Modern VLMs achieve high Kendall accuracy (up to 0.79) and Spearman correlation with human rankings, making them reliable for relative model ranking.
• Explainability Gap: VLMs fail to match human performance in fine-grained error attribution. They often miss specific artifacts (e.g., unreadable text, boundary glitches) that humans flag, resulting in a bias where VLMs under-penalize visual defects.

4. Key Contributions

1. ImagenWorld Benchmark: A unified, large-scale dataset covering 6 tasks and 6 domains, designed to stress-test the full spectrum of real-world image generation and editing.
2. Explainable Evaluation Schema: A novel protocol that moves beyond scalar scores to provide object-level and segment-level error tags, enabling interpretable diagnosis of model failures.
3. Comprehensive Analysis: The first large-scale study comparing unified vs. specialized models across diverse domains, revealing that data curation (not just scale) is critical for text-heavy tasks and that editing remains a major bottleneck for current architectures.

5. Significance and Future Impact

• Diagnostic Tool: ImagenWorld serves not just as a leaderboard but as a diagnostic tool to identify specific architectural weaknesses (e.g., the inability of AR-hybrids to preserve source context during editing).
• Data-Centric Insights: The success of Qwen-Image highlights that targeted data augmentation (synthetic text-rich data) is a viable path to closing performance gaps in symbolic domains.
• Evaluation Standards: The paper advocates for moving beyond "black-box" scores toward explainable metrics. It suggests that while VLMs are useful for scaling evaluation, human-in-the-loop fine-grained tagging remains essential for developing robust, controllable image generation systems.
• Safety & Ethics: The authors emphasize the risks of realistic image manipulation (deepfakes, fraud) and commit to releasing only sanitized data to support responsible research.

In summary, ImagenWorld establishes a new standard for evaluating generative AI by combining broad task coverage with deep, explainable human analysis, revealing that while models are getting better at "creating," they still struggle significantly with "controlling" and "editing" in complex, real-world scenarios.