Technical Summary: SANEval

Problem Statement

Despite rapid advancements in Text-to-Image (T2I) models (e.g., Stable Diffusion, Imagen, GPT-4o), their ability to faithfully render complex prompts involving multiple objects, specific attributes, and spatial relationships remains a significant bottleneck. Current evaluation methods suffer from three primary limitations:

1. Closed-Set Vocabularies: Existing benchmarks often rely on object detectors trained on fixed datasets (e.g., MS-COCO's 80 classes), creating a "vocabulary mismatch" where real-world or rare objects cannot be evaluated.
2. Lack of Diagnostic Granularity: Many metrics provide a single, uninterpretable score (e.g., CLIPScore) or binary pass/fail judgments, failing to pinpoint specific compositional failures (e.g., missing objects, incorrect attribute binding, or spatial errors).
3. Scalability and Subjectivity: Human evaluation is costly and difficult to standardize, while automated metrics like FID or IS fail to capture fine-grained compositional errors.

Methodology: The SANEval Framework

The authors introduce SANEval (Spatial, Attribute, and Numeracy Evaluation), an open-vocabulary, compositional benchmark designed to provide scalable, reliable, and interpretable evaluation. The framework consists of a modular pipeline comprising two core technical modules and three evaluation axes.

Core Modules

1. Prompt Understanding Module:

   • Utilizes a Large Language Model (LLM) guided by specific system prompts to parse natural language inputs.
   • Extracts a structured representation including:
     • Objects: Unique entity lists.
     • Attributes: Mapped to specific objects (color, shape, texture).
     • Spatial Relations: Represented as triplets <object_1, relation, object_2>.
     • Numeracy: Expected counts for each object.
   • Designed to be robust against grammatical variations and complex sentence structures.

2. Enhanced Object Detection (OD) Module:

   • Addresses the vocabulary limitation of traditional detectors by integrating an Open-World Detector (specifically YOLO-E) with LLM-driven synonym expansion.
   • Process:
     • The LLM expands expected object names into a set of synonyms and related terms (e.g., "albatross" $\rightarrow$ {albatross, seabird, large bird}).
     • YOLO-E detects objects in the generated image.
     • Detected labels are aligned with the expanded synonym sets to map detections back to the prompt's intended objects.
   • This hybrid approach leverages the structured interpretability of OD with the reasoning flexibility of VLMs.

Evaluation Axes

The framework evaluates three distinct compositional dimensions:

1. Attribute Binding:

   • Process: Detected objects are cropped and passed to a Vision Language Model (VLM) as a "judge" to identify specific attributes (e.g., "What is the color of the bench?").
   • Scoring: An LLM compares the VLM's predicted attribute against the ground truth, assigning a continuous similarity score (0.0 to 1.0) to allow for partial credit (e.g., "turquoise" vs. "blue").
   • Output: Generates structured feedback identifying missing or mismatched attributes.

2. Spatial Relationships:

   • Process: Uses bounding boxes from the Enhanced OD module and spatial triplets from the Prompt Understanding module.
   • Scoring: Applies geometric evaluation functions (e.g., checking if the centroid of Object A is to the left of Object B) to compute adherence scores.
   • Output: Explicitly reports errors such as "Expected cat to the right of dog: Missing object: cat."

3. Numeracy:

   • Process: Compares the count of detected instances against the expected counts extracted from the prompt.
   • Scoring: 1.0 for exact match, 0.5 for presence with incorrect count, 0.0 for missing.
   • Output: Reports over-generation or under-generation (e.g., "two extra suitcases detected").

Key Contributions

1. SANEval Benchmark Suite: A comprehensive evaluation framework for T2I models covering spatial, attribute, and numeracy adherence with open-vocabulary capabilities.
2. Open-Source Dataset: Release of SANEval-Simple (algorithmic, ~5,000 prompts) and SANEval-Hard (human-generated, 250 complex prompts), accompanied by generated images and diagnostic conformity labels.
3. Plug-and-Play Pipeline: A scalable evaluation pipeline that decouples evaluation into distinct axes and provides both quantitative scores and structured, interpretable diagnostic feedback.
4. Robustness Validation: Demonstration that the framework is robust to the choice of the underlying LLM "judge" (tested by swapping Gemini Flash 2.5 with Llama-4) and that the synonym expansion module is critical for detecting rare objects.

Results and Analysis

The authors evaluated six state-of-the-art T2I models (Imagen 3.0/4.0/4.0 Ultra, Nano Banana, Seedream 3.0, GPT Image 1) using the SANEval framework.

• Performance Trends: No single model dominates all categories. Imagen 4.0 Ultra achieved the highest overall averaged score, excelling in numeracy and fine-grained bindings (shape/texture). Seedream 3.0 outperformed in spatial reasoning and color binding.
• Complexity Sensitivity: Performance degrades consistently as prompt complexity (object cardinality) increases. Shape binding was identified as the most brittle metric, while color binding remained the most robust. For example, Imagen 3.0's shape accuracy dropped from ~0.31 on single-object prompts to ~0.06 on prompts with >10 objects.
• Comparison with Existing Benchmarks: A Spearman rank correlation analysis against CompBench++ revealed statistically significant differences (p-values often < $10^{-20}$) in spatial and numeracy tasks. This indicates that SANEval provides a complementary, non-redundant evaluation perspective that captures failures missed by existing methods.
• Ablation Study: Removing the LLM-driven synonym expansion module resulted in substantial performance degradation (e.g., a 75.69% relative drop in spatial scores) and a significant reduction in the number of unique objects detected, validating the necessity of open-vocabulary handling.

Significance and Claims

The paper claims that SANEval addresses the critical gap in T2I evaluation by moving beyond fixed vocabularies and subjective scoring toward feedback-driven, open-world benchmarking.

• Diagnostic Utility: Unlike previous benchmarks that offer opaque scores, SANEval provides actionable, interpretable feedback (e.g., identifying specific missing objects or incorrect attribute bindings), which is essential for diagnosing model behavior.
• Foundation for Improvement: The authors posit that this framework transforms evaluation into a form of supervision, potentially guiding future model improvements and Reinforcement Learning with AI Feedback (RLAIF).
• Modesty: The authors acknowledge that while SANEval improves upon existing methods, it relies on language-based priors (LLMs/VLMs) which may reflect inherent biases. They emphasize that the work is designed for evaluation and analysis, not for adding new generative capabilities, and aims to encourage more transparent and careful evaluation of generative systems.