Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets

Technical Summary: ZeroSight and SC4CIR

Problem Statement
Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image based on a query consisting of a reference image and a relative caption, without relying on task-specific training samples. The authors identify two critical flaws in existing ZS-CIR benchmarks (e.g., CIRR, CIRCO, FashionIQ):

1. Inconsistency: Existing datasets often suffer from visual and semantic inconsistencies between reference and target images. Because these datasets are constructed from noisy, crowdsourced, or abstract image collections (like NLVR2 or MS COCO), the reference and target images may share only abstract concepts (e.g., "a tie") while differing significantly in visual content, making the retrieval task solvable via standard text-to-image techniques rather than true composed retrieval.
2. Spurious Zero-Shot Scenarios: Most existing datasets utilize public image sources (e.g., MS COCO) that were part of the pre-training data for dominant Vision-Language Models (VLMs) like CLIP. Consequently, models are not truly "zero-shot" but are leveraging memorized associations, leading to inflated performance metrics that exaggerate the capabilities of current methods.

Methodology
To address these challenges, the paper introduces ZeroSight, a novel benchmark, and SC4CIR, a training-free retrieval method.

• ZeroSight Dataset Construction:

  • Source: Unlike previous works, ZeroSight extracts frame images from videos published after March 31, 2022 (post-dating the LAION-2B release), ensuring the data was not included in CLIP's pre-training.
  • Consistency: By extracting frames from a single video, the authors guarantee that reference-target pairs are visually and semantically consistent, differing only in the specific modifications described by the caption.
  • Pipeline: A multi-stage, LLM-assisted pipeline (utilizing models like GPT-4o) constructs the dataset. It involves:
    1. Selecting non-redundant reference images from video subsets.
    2. Filtering candidate target images based on visual and semantic similarity thresholds (using ViT and CLIP).
    3. Generating relative captions that describe the transformation from reference to target.
  • Scale & Structure: The dataset comprises 12,048 diverse videos, resulting in a retrieval pool of 197,313 images and 54,740 queries. Uniquely, it includes multiple ground-truth positive targets and hard negative targets (images similar to the target but failing the caption criteria) for each query.
  • Query Categories: Queries are classified into six categories to test specific retrieval capabilities: Addition, Subtraction, Background Change, Viewpoint Change, Attribute Change, and Relative Statement.

• SC4CIR (Symmetric Consistency for CIR):

  • Concept: A training-free, plug-and-play method driven by Multi-modal Large Language Models (MLLMs) to identify and penalize "hard negative" targets that are visually similar to the ground truth but semantically incorrect.
  • Mechanism: It employs three symmetric consistency checks:
    1. Forward Retrieval: $I_r + T \rightarrow I_t$ (Standard retrieval).
    2. Reverse Process 1: $I_t - T \rightarrow I_r$ (Generates an inverse instruction $T'$ to transform the candidate back to the reference).
    3. Reverse Process 2: $I_t - I_r \rightarrow T$ (Generates a description of the difference between the candidate and reference to compare against the original caption $T$).
  • Re-ranking: The final score combines the forward similarity with the two reverse consistency scores ($S = S_1 + S_2 + S_3$), re-ranking results to suppress hard negatives.

Key Contributions

1. ZeroSight Benchmark: The first ZS-CIR dataset ensuring true zero-shot conditions (no CLIP pre-training overlap) and high visual/semantic consistency via video-sourced data. It is the first to include multiple ground truths and hard negative targets.
2. SC4CIR Method: A novel, training-free MLLM-driven approach that improves retrieval performance by verifying symmetric consistency, effectively filtering out hard negatives without requiring model retraining.
3. New Evaluation Metric (PNR-mAP): The authors propose Positive-Negative Ranking mAP, which accounts for the ranking of negative targets relative to positive ones, addressing the limitation of standard mAP which ignores the presence of hard negatives.
4. Comprehensive Evaluation: The paper evaluates 27 state-of-the-art methods, revealing that current benchmarks yield inflated results.

Experimental Results

• Benchmark Performance: Experiments on ZeroSight show that current ZS-CIR methods perform significantly worse than on existing datasets (e.g., CIRCO), confirming the difficulty and validity of the new benchmark. For instance, the mAP@5 for SEIZE dropped from 32.46 on CIRCO to 16.63 on ZeroSight.
• Metric Discrepancy: The average mAP across methods on ZeroSight was 22.93% higher than the PNR-mAP, demonstrating that standard metrics inflate performance by ignoring the ranking of hard negatives.
• SC4CIR Efficacy: Integrating SC4CIR significantly improved performance across all tested methods. For training-free methods, the average PNR-mAP improved by 25.66%. The method is particularly effective for LLM-based ZS-CIR approaches, improving their performance by 9.12%.
• Generalizability: SC4CIR also demonstrated robustness when applied to existing noisy datasets (CIRCO, CIRR, FashionIQ), consistently outperforming baselines and even helping zero-shot methods approach the performance of trained CIR methods on FashionIQ.

Significance
The paper argues that existing ZS-CIR research is hindered by datasets that do not reflect true zero-shot scenarios and lack visual consistency, leading to an overestimation of model capabilities. ZeroSight establishes a new standard for evaluating CIR methods by enforcing strict data consistency and true zero-shot conditions. The introduction of SC4CIR demonstrates that leveraging MLLM reasoning for consistency checks can significantly enhance retrieval accuracy without additional training. The authors conclude that their benchmark and method provide a more reliable framework for developing generalizable and robust composed image retrieval systems, particularly for applications requiring high precision where subtle visual differences matter.