Technical Summary: CIRCLED – A Multi-turn CIR Dataset with Consistent Dialogues across Domains

1. Problem Statement

Existing Multi-Turn Composed Image Retrieval (MTCIR) research faces two critical limitations:

1. Inconsistent Dialogue History: Current datasets, such as Multi-turn FashionIQ, are constructed by simply concatenating single-turn queries. This approach fails to guarantee that intermediate turns progressively approach the ground truth (GT). Consequently, intermediate images and captions may diverge from the target, making it impossible to evaluate retrieval performance at intermediate turns or to model realistic iterative search behavior.
2. Domain Restriction: Prior work is almost exclusively limited to the fashion domain (e.g., dresses, shirts), leaving multi-turn CIR in general domains largely unexplored.

These limitations hinder the development of robust multi-turn CIR systems that can handle evolving user intents and diverse visual contexts.

2. Methodology

Dataset Construction (CIRCLED)

The authors introduce CIRCLED, a new dataset constructed by extending three existing single-turn CIR datasets: FashionIQ, CIRR, and CIRCO. The construction process involves generating multi-turn sessions that simulate a user refining their search based on intermediate results.

• Session Generation: Starting with a ground truth image $Z$, the process generates a sequence of turns $\{(I_1, T_1), \dots, (I_L, T_L)\}$.
  • Turn 1: Uses the original reference image and relative caption from the source dataset.
  • Subsequent Turns: If the GT is not sufficiently ranked, a baseline retrieval algorithm selects the closest image from the top-$K$ results as the new reference image ($I_l$). A Vision-Language Model (VLM) then generates a new relative caption ($T_l$) describing the difference between the selected image and the GT, while explicitly avoiding repetition of previous captions.
• Baseline Retrieval Algorithm: The authors employ an extended version of CIReVL (Karthik et al., 2024) to drive the session generation. This pipeline uses:
  • BLIP for image captioning.
  • GPT-4o-mini for merging captions, generating difference descriptions, and updating queries.
  • CLIP for feature extraction and similarity matching.
  • The query update mechanism aggregates features from the selected image, non-selected images, and the merged text to steer the retrieval toward the GT.

Quality Control Filters

To ensure high-quality, consistent dialogues, four filters are applied to the generated sessions:

1. Retrieval-success Filter: Removes sessions where the GT never appears in the top-10 candidates across 6 turns.
2. Multi-turn Filter: Excludes sessions that succeed in the first turn, ensuring only true multi-turn interactions (2+ turns) are kept.
3. Rank-margin Filter ($\epsilon$-consistency): Enforces that the GT rank improves (or stays within a small margin $\epsilon=30$) at each turn ($r_{l+1} \le r_l + \epsilon$). This ensures monotonic progress toward the target.
4. Text-redundancy Filter ($\tau$-diversity): Ensures that relative captions at each turn provide novel information by limiting the cosine similarity between new captions and previous ones ($\tau=0.8$).

Evaluation Protocol

The paper proposes a turn-wise evaluation framework where a retrieval algorithm $S$ is evaluated cumulatively. For a session of length $L$, the algorithm receives the history $\{(I_1, T_1), \dots, (I_l, T_l)\}$ at turn $l$ and outputs a rank $r_l$ for the GT.

Three metrics are introduced to capture different aspects of performance:

• Hits@10 (Turn-wise): The fraction of queries reaching the top-10 by turn $l$.
• Final Recall@10: Accuracy using all available information at the final turn.
• AUC over Hits@10: The area under the Hits@10 curve, measuring convergence speed and early reachability.

3. Key Contributions

1. CIRCLED Dataset: A large-scale, multi-turn CIR dataset containing 22,608 sessions across nine subsets (covering Fashion and General domains). It includes 202,845 images and supports sessions of 2–6 turns, significantly exceeding the scale and generality of Multi-turn FashionIQ.
2. Consistent Dialogue Structure: Unlike prior datasets, CIRCLED enforces $\epsilon$-consistency and $\tau$-diversity, ensuring that every turn progressively and informatively moves toward the ground truth. This enables the evaluation of retrieval performance at any intermediate turn.
3. Broad Domain Coverage: The dataset extends beyond fashion to include general domains (via CIRR and CIRCO), facilitating research on multi-turn CIR for diverse object types and scene modifications.
4. Evaluation Framework: The introduction of turn-wise metrics (Hits@10, AUC) allows for a granular analysis of how well methods utilize historical context, rather than just measuring final-turn accuracy.

4. Experimental Results

The authors evaluated several baseline methods on CIRCLED, including Text-only, Image-only, Pic2Word, CIReVL, and MagicLens.

• Turn-wise Performance: Results show that retrieval accuracy generally improves with more turns, though it often plateaus. MagicLens performs strongly on general-domain data (CIRCO), while CIReVL excels in the fashion domain.
• Modalities: Text-only baselines consistently outperform Image-only baselines across all subsets, indicating that relative captions carry more discriminative information than reference images alone in multi-turn settings.
• Metric Sensitivity: The AUC metric reveals performance differences that Final Recall@10 might miss. For instance, on the CIRR validation set, CIReVL outperforms Text-only from Turn 2 onward, but its weaker first-turn performance lowers its overall AUC, highlighting the importance of evaluating the entire trajectory.
• Quality Validation: An LLM-as-a-Judge evaluation (using GPT-5-mini) confirmed that CIRCLED sessions score highest in Coherence, Goal Progress, and Overall quality compared to unfiltered sessions or simple concatenations of single-turn datasets.

5. Significance and Claims

The paper claims that CIRCLED provides a practical, high-quality benchmark that addresses the lack of history consistency and domain diversity in prior work. By ensuring that multi-turn dialogues are coherent and progressively converge on the target, the dataset enables the study of iterative search behaviors that were previously unmeasurable.

The authors emphasize that their work facilitates future research on multi-turn CIR by:

• Offering a standardized evaluation protocol that captures both final accuracy and the speed of convergence.
• Providing a dataset that supports research beyond the fashion domain.
• Enabling the development of systems that can effectively aggregate visual and textual cues over multiple interaction turns.

The dataset and code are publicly available to support reproducibility and further advancement in the field.