Technical Summary: Rethinking Composed Image Retrieval Evaluation

Problem Statement

Composed Image Retrieval (CIR) is a critical task in multimodal understanding, requiring a model to retrieve a target image given a reference image and a natural language description of a desired modification. Despite the proliferation of benchmarks such as CIRR, FashionIQ, and CIRCO, the authors identify two fundamental limitations in current evaluation standards:

1. Coarse-Grained Evaluation: Existing benchmarks focus on a narrow range of modification categories, failing to capture the diverse and fine-grained requirements of real-world scenarios.
2. Limited Query Scale and Ambiguity: Many benchmarks suffer from insufficient query volume and ambiguous category definitions (e.g., overlapping tags like "direct addressing" and "color"). Furthermore, the standard dataset construction methodology—retrieving a target first and annotating the query post-hoc—leads to an absence of certain modification categories and an over-reliance on the retriever's initial output, introducing bias.

These limitations result in benchmarks that may allow models to achieve high scores by exploiting modality biases (over-relying on text signals) rather than demonstrating genuine compositional reasoning.

Methodology

To address these gaps, the authors propose EDIR (Image Editing Derived Benchmark), a novel, fine-grained CIR benchmark constructed through a controllable data synthesis pipeline.

1. Taxonomy Definition

The authors establish a comprehensive taxonomy organizing real-world modification requirements into five main categories and fifteen subcategories:

• Attribute: Color, Material, Shape, Texture.
• Object: Addition, Remove, Replace, Count.
• Relationship: Spatial, Action, Viewpoint.
• Global Environment: Style, Time, Weather.
• Complex: Queries combining multiple modifications from the above categories.

2. Data Synthesis Pipeline

Unlike previous methods that annotate queries post-hoc, EDIR leverages image editing technology to synthesize queries with precise control. The pipeline operates as follows:

• Seed Image Selection: High-quality source images are filtered from the LAION-400M dataset using an MLLM (Qwen2.5-VL-32B) to remove unsuitable images (e.g., text-heavy, low-resolution, or abstract).
• Triplet Construction: For each source image, the MLLM generates specific edit instructions for 5–6 suitable subcategories. An image editing model (Qwen-Image-Edit) applies these instructions to generate target images. To create hard negatives, the pipeline generates a set of related images where some share base modifications while others contain distinctive changes.
• Query Formulation: Raw edit instructions are rewritten into natural language CIR queries. The authors employ both direct rewriting and negation-based rewriting (e.g., "Show me this dress in a different color" instead of "Change to red") to increase linguistic diversity and challenge.
• Quality Control: A two-stage filtering process using an MLLM validates the alignment between the source image, the generated target, and the final query, followed by human validation on a 12% sample.

3. Benchmark Composition

The final EDIR benchmark comprises:

• 5,000 high-quality queries distributed across the 15 subcategories.
• A corpus of 178,645 images, including target images and hard negatives derived from the same source images.

Key Contributions

1. Comprehensive Taxonomy and Pipeline: The proposal of a hierarchical taxonomy for CIR and a controllable data synthesis pipeline that uses image editing to populate it, ensuring precise control over modification types.
2. EDIR Benchmark: The introduction of a large-scale, fine-grained benchmark designed to facilitate comprehensive evaluation, addressing the coverage and granularity gaps in prior works.
3. Diagnostic Evaluation: A thorough analysis of 13 state-of-the-art multimodal embedding models (both MLLM-based and Non-MLLM-based) revealing significant capability gaps, particularly in handling negation, compositional reasoning, and fine-grained details.
4. In-Domain Training Analysis: An experiment training a model (EDIR-MLLM) on the synthesized data to distinguish between challenges solvable with targeted data and those exposing intrinsic architectural limitations.

Experimental Results

The authors evaluated 13 models on EDIR using Recall@1:

• Non-MLLM Models: Models based on CLIP architectures (e.g., PIC2WORD, SEARLE) achieved a low average Recall@1 of 18.4%, struggling to distinguish targets within visually similar groups based on fine-grained text.
• MLLM-Based Models: While outperforming non-MLLM baselines (average Recall@1 of 36.9%), even top-performing models like RzenEmbed and GME failed to perform consistently across all subcategories.
  • Strengths: Models performed relatively well on "Addition," "Replace," and "Action" categories.
  • Weaknesses: Significant performance drops were observed in "Texture," "Remove," "Shape," "Count," "Spatial," and "Viewpoint."
• Error Analysis: The study identified four primary failure modes:
  1. Failure in Handling Negation: Difficulty with removal commands or explicit negative terms.
  2. Deficiencies in Compositional Reasoning: Inability to correctly interpret spatial relationships or global scene transformations.
  3. Struggles with Multiple Constraints: Inability to satisfy multiple conditions simultaneously in complex queries.
  4. Insensitivity to Fine-Grained Details: Overlooking subtle changes in texture, material, or shape.
• In-Domain Training: Training EDIR-MLLM on 225,000 synthesized instances raised the average Recall@1 to 59.9%. This experiment demonstrated that categories requiring fine-grained detail (e.g., color, material) are solvable with sufficient data, whereas categories requiring complex reasoning (e.g., spatial, count) showed only modest gains, indicating intrinsic model limitations.

Significance and Claims

The paper claims that EDIR serves as a rigorous diagnostic tool that exposes evaluation gaps in the current CIR landscape. By comparing EDIR with existing benchmarks (CIRR, CIRCO, FashionIQ, GENECIS), the authors demonstrate that current benchmarks suffer from fine-grained evaluation bias (lack of balanced category coverage) and modality bias (models can succeed by ignoring the image and relying on text).

The authors conclude that EDIR provides the community with a robust framework to drive the development of more genuinely compositional and less biased CIR models. They emphasize that while their benchmark effectively diagnoses weaknesses, the development of scalable training methods to address these specific architectural limitations remains an open research direction. The work is presented primarily as an evaluation tool rather than a universal training solution.