MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation

1. Problem Statement

Recent advances in text-to-image (T2I) models have enabled multi-reference generation, where models can inherit subjects from multiple reference images and re-render them in new contexts. However, existing evaluation benchmarks suffer from two critical limitations:

• Limited Reference Count: Most benchmarks restrict the number of reference images to 1–4, failing to test model scalability with larger sets (e.g., 5–8 references).
• Vague Task Definitions: Current tasks focus on simple axes like "what to edit" or "how many references," ignoring the inherent complexity of combining heterogeneous references. They do not adequately capture challenges such as domain mismatches (e.g., photo vs. anime), scale differences, rare concepts, or multilingual text rendering.

Consequently, there is no standardized basis to evaluate how well models follow complex, multi-constraint instructions, leading to a lack of understanding regarding model failure modes in realistic, industrial-grade scenarios.

2. Methodology: The MultiBanana Benchmark

The authors introduce MultiBanana, a comprehensive benchmark designed to stress-test T2I models across specific multi-reference challenges.

A. Dataset Construction

• Data Sources: The dataset combines real images (filtered from LAION-5B) and synthetic images (generated by Nano Banana and GPT-Image-1). Synthetic data was crucial to correct distributional biases in real data (e.g., scarcity of specific objects, poses, or multilingual text).
• Filtering & Classification: Images were filtered using YOLOv12, SAM2, and CLIP to ensure object visibility and remove harmful content. A hierarchical classification system (6 major categories: Person, Object, Background, Light, Tone, Style) was applied to ensure task feasibility.
• Instruction Generation: Editing instructions were generated using Gemini, validated for visual plausibility, and manually verified to avoid nonsensical tasks.

B. Task Taxonomy

The benchmark covers 3,769 tasks across 36 task types, categorized by reference count and structure:

1. Single & Two-Reference: Standard editing tasks (e.g., subject addition, replacement, style transfer, text correction).
2. Multi-Reference (3–8 images): Compositional generation tasks defined by four structures:
   • X Objects: Compose $X$ objects from the reference set.
   • X–1 Objects + Background: Place objects into a specific background.
   • X–1 Objects + Local: Apply local attribute changes (e.g., color/texture) to specific objects.
   • X–1 Objects + Global: Arrange objects and modify the overall tone/style.

C. Difficulty Axes (The "Hard" Factors)

MultiBanana explicitly introduces five dimensions of difficulty often overlooked in prior work:

1. Varying Reference Count: Scaling up to 8 references.
2. Cross-Domain Diversity: Mixing domains (e.g., photorealistic subjects in anime backgrounds).
3. Scale & Viewpoint Mismatch: Combining close-ups with wide shots or different perspectives.
4. Rare Concepts: Including uncommon subjects (e.g., a "red banana" or specific fictional creatures).
5. Multilingual Rendering: Requiring text generation in English, Chinese, and Japanese based on reference styles.

D. Evaluation Protocol

• Metrics: Images are scored on a 10-point scale across five criteria:
  1. Text-Instruction Alignment
  2. Reference Consistency (fidelity to source images)
  3. Background-Subject Match (blending)
  4. Physical Realism (no floating objects, etc.)
  5. Visual Quality
• Judges: Evaluation is performed using VLMs (Gemini 2.5 and GPT-5), which showed strong correlation with human judgments (Pearson $r \approx 0.69$). An open-source alternative, Qwen3-VL, is also validated for reproducibility.
• Weighted Score: A total score is computed with weights {3, 3, 1, 1, 1} favoring instruction alignment and reference consistency.

3. Key Results

The benchmark was used to evaluate state-of-the-art closed-source (Nano Banana, GPT-Image-1) and open-source models (Qwen-Image-Edit, DreamOmni2, OmniGen2).

• Performance Degradation with Scale: All models exhibit a decline in performance as the number of references increases from 3 to 8.
• Divergent Failure Modes:
  • Closed-Source Models (e.g., Nano Banana, GPT-Image-1):
    • Strengths: High adherence to instructions and reference consistency (they include all requested subjects).
    • Weaknesses: Suffer from compositional distortion and overcrowding. As they try to satisfy all constraints, visual quality and physical realism degrade significantly (e.g., distorted anatomy, unnatural blending).
  • Open-Source Models (e.g., Qwen-Image-Edit, DreamOmni2):
    • Strengths: Generate visually cleaner, more coherent scenes with better physical realism.
    • Weaknesses: Instruction Ignorance. They frequently omit reference subjects entirely, especially as the number of references grows, leading to low scores in instruction alignment and reference consistency.
• Specific Challenges:
  • Background Replacement: All models struggle significantly with replacing backgrounds while maintaining subject integrity.
  • Cross-Domain & Rare Concepts: These tasks yield the lowest scores, indicating models cannot yet effectively reconcile conflicting visual styles or handle out-of-distribution concepts.
  • Multilingual Text: Performance drops sharply for non-English text rendering, though GPT-Image-1 shows relative resilience.

4. Key Contributions

1. MultiBanana Benchmark: The first benchmark to systematically evaluate multi-reference generation up to 8 references, covering diverse task structures and difficulty axes (domain, scale, rarity, language).
2. Comprehensive Dataset: A curated set of 3,769 high-quality tasks with balanced distribution across categories, augmented by synthetic data to ensure diversity.
3. Empirical Analysis of Failure Modes: The paper identifies a fundamental trade-off in current models: Strict Reference Fidelity vs. Holistic Image Coherence. Closed models prioritize fidelity at the cost of quality; open models prioritize quality at the cost of instruction following.
4. Agentic Framework Evaluation: The authors tested iterative refinement frameworks (IPR, CAFG, SRA), finding that while they can improve visual quality in some cases, they often degrade instruction alignment due to information loss during prompt refinement.

5. Significance

• Standardization: MultiBanana establishes a standardized, rigorous foundation for fair comparison in multi-reference image generation, moving beyond simple single-reference editing.
• Industrial Relevance: The benchmark addresses real-world needs in advertising, fashion, and content creation where combining multiple assets (subjects, backgrounds, styles, text) is routine.
• Future Direction: The results highlight that current diffusion and multimodal LLM architectures are not yet robust enough to handle complex, multi-constraint reasoning. Future research must focus on improving compositional reasoning and constraint satisfaction without sacrificing visual coherence.

The dataset and code are publicly available at https://github.com/matsuolab/multibanana.