Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

Technical Summary: Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

Problem Statement
Composed Image Retrieval (CIR) is a multimodal task where a query consists of a reference image and a textual modification, with the goal of retrieving a target image that satisfies the combined intent. The field assumes that strong performance on CIR benchmarks (such as CIRR, FashionIQ, LaSCo, and CIRCO) necessitates genuine multimodal composition—the ability to integrate complementary information from both modalities into a unified representation. However, prior work in related domains (e.g., Visual Question Answering) has demonstrated that models often exploit "shortcut signals," solving tasks using a single dominant modality rather than true cross-modal reasoning. This paper investigates whether current CIR benchmarks suffer from similar issues, where high retrieval accuracy may reflect reliance on unimodal shortcuts or dataset noise rather than the intended compositional reasoning.

Methodology
The authors conduct a two-stage audit of four widely used CIR benchmarks using a pool of eleven Generalist Multimodal Embedding (GME) models.

1. Unimodal Shortcut Audit:
   The authors evaluate each query under three conditions: (i) Multimodal (full query), (ii) Text-only (reference image replaced by a black mask), and (iii) Image-only (textual edit removed). A query is classified as admitting a "unimodal shortcut" if any of the eleven tested GMEs can retrieve the ground-truth target within the top-10 results using only a single modality. Queries where no single modality succeeds but the multimodal input succeeds are labeled "Composition-required." Queries where even the multimodal input fails are labeled "Unresolved."

2. Human Validation Audit:
   Recognizing that removing shortcut-solvable queries is insufficient if the remaining set contains ambiguous or ill-posed instances, the authors perform a human validation study on 4,741 shortcut-free queries. Annotators assess the validity of the reference image, textual edit, and target image triplet. They identify specific failure modes:

   • Invalid Text/Reference/Target: Garbled edits, low-quality images, or mismatched targets.
   • Overly Broad Query: Cases where multiple images in the gallery plausibly satisfy the modification, rendering the ground-truth target non-unique.
     This process yields the CIRCUS (Composed Image Retrieval Cleaned of Unimodal Shortcuts) suite, specifically the CIRCUSV (validated) subset containing only well-formed, shortcut-free queries.

3. Re-evaluation:
   The authors re-evaluate the GMEs on the original benchmarks, the shortcut-free subset (CIRCUSSF), and the validated subset (CIRCUSV). They analyze performance drops and compute a "Composition Gap" (normalized difference between multimodal and the strongest unimodal baseline) to quantify the reliance on multimodal integration.

Key Contributions

1. Quantification of Unimodal Shortcuts: The study reveals that a pervasive fraction of queries in standard CIR benchmarks can be solved using a single modality. Specifically, 83.6% of CIRR, 32.2% of FashionIQ, 38.7% of LaSCo, and 74.5% of CIRCO queries are solvable via unimodal shortcuts.
2. Identification of Dataset Noise: Through human validation, the authors demonstrate that even among shortcut-free queries, a significant portion are ill-posed. Only 35.6% of the audited shortcut-free queries (1,689 out of 4,741) were deemed valid CIR instances. The dominant issue is "Overly Broad Query," where the modification does not uniquely identify a target.
3. CIRCUS Benchmark Suite: The authors release CIRCUS, a cleaned evaluation suite comprising shortcut-free and validated subsets. This suite provides a more diagnostic test for multimodal composition, forcing models to combine image and text inputs to succeed.

Results

• Performance Discrepancy: Re-evaluating models on CIRCUS results in a sharp decline in reported performance. For example, the Recall@10 of Qwen3-VL-8B on CIRR drops from 70.1% on the original benchmark to 24.1% on the validated CIRCUSV subset.
• Modality Reliance: While absolute performance decreases, the reliance on multimodal information increases significantly. The Composition Gap (indicating the necessity of combining modalities) rises substantially on CIRCUSV compared to the original benchmarks. For instance, on CIRR, the average Composition Gap increases from 0.137 (Full) to 0.361 (CIRCUSV).
• Shortcut Prevalence: The analysis confirms that shortcut exploitation is a dataset-level property rather than a model-specific artifact, as the pattern holds across diverse model families and architectures.
• Benchmark Specifics: CIRR and CIRCO show high shortcut rates, while LaSCo is dominated by unresolved cases (suggesting general retrieval difficulty or annotation errors). FashionIQ has the lowest shortcut rate but still suffers from significant noise in the shortcut-free residue.

Significance
The paper argues that current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capabilities in multimodal composition. The authors claim that relying on these benchmarks may falsely suggest progress in compositional reasoning when models are merely exploiting unimodal signals. By introducing CIRCUS, the work provides a more faithful assessment of how well models truly combine image and text. The authors conclude that future progress in CIR should be measured by the extent to which retrievers genuinely integrate modalities, necessitating the use of cleaned, validated subsets to avoid inflated performance metrics driven by dataset artifacts.