FashionMV: Product-Level Composed Image Retrieval with Multi-View Fashion Data

1. Problem Definition: View Incompleteness

The paper identifies a fundamental limitation in existing Composed Image Retrieval (CIR) systems, termed View Incompleteness.

• Current State: Standard CIR operates at the image level, taking a single reference image and a modification text to retrieve a single target image.
• The Gap: In real-world e-commerce (especially fashion), products are displayed via multiple viewpoints (front, back, side, detail). Users reason about the product holistically. A single image often cannot capture all defining features (e.g., a deep V-neck is visible only from the front, while an open-back design is visible only from the back).
• Consequence: When a user requests a modification involving unobserved regions (e.g., "make the back open" from a front-view image), single-image CIR models fail because they cannot infer or retrieve based on missing visual data. Existing datasets and methods do not aggregate multi-view data into a unified product-level representation.

2. Methodology

A. The FashionMV Dataset

The authors constructed FashionMV, the first large-scale dataset designed for Product-Level CIR.

• Scale: 127K products, 472K multi-view images, and over 220K CIR triplets.
• Construction Pipeline (Fully Automated):
  1. Multi-View Caption Generation: Uses Kimi K2.5 to generate per-image captions, a comprehensive long caption (synthesizing all views), and a short summary caption for each product.
  2. Directional Hallucination Filtering: Uses Qwen3.5-397B to cross-check directional descriptions (e.g., left vs. right pockets) against visual evidence, filtering out ~4.4% of products with severe hallucinations.
  3. Triplet Construction: Uses Gemini 3.1 Flash Lite to select target products from a candidate pool (visual + caption similarity) and generate modification texts (short and long) that describe differences spanning at least two viewpoints.
• Structure: Each triplet consists of a Source Product (multi-view images), Modification Text, and Target Product (multi-view images).

B. The ProCIR Framework

The authors propose ProCIR (Product-level Composed Image Retrieval), a framework built on a Multimodal Large Language Model (MLLM), specifically Qwen3.5-0.8B. It employs three complementary mechanisms:

1. Two-Stage Dialogue Architecture (MT):

   • Turn 1 (Perception): The model processes only the source multi-view images to generate a pure visual embedding ($s$).
   • Turn 2 (Reasoning): The model receives the modification text and the context from Turn 1 to generate the query embedding ($q$).
   • Benefit: Decouples visual perception from reasoning, preventing the modification text from interfering with the visual encoding. This enables Source-Side Alignment (aligning the pure visual embedding with text), which is impossible in single-turn models.

2. Caption-Based Alignment (Align):

   • Injects product-level semantics by aligning visual embeddings with text captions (long and short) generated in Stage 1.
   • Uses a symmetric InfoNCE loss to align the source visual embedding ($s$) with the source caption ($t_{src}$) and the target visual embedding ($d$) with the target caption ($t_{tgt}$).
   • This anchors the embedding space to structured product knowledge.

3. Chain-of-Thought (CoT) Guidance:

   • Injects the product's long caption into the model's reasoning process (inside a <thought> block) during training.
   • Uses a progressive removal schedule: the caption is fully present at the start of training and gradually removed (ratio decreases from 1.0 to 0.0) to force the model to internalize the semantics rather than relying on the text at inference time.

4. Supervised Fine-Tuning (SFT) (Optional):

   • Before contrastive training, the base model undergoes SFT on caption generation and CIR triplet generation tasks. This injects structured product knowledge directly into the model weights.

Training Objective: A combination of the main CIR loss ($L_{cir}$) and alignment losses ($L_{src}, L_{doc}$).

3. Key Contributions

1. Task Definition: Formally defined Multi-View CIR, generalizing CIR from image-level to product-level retrieval.
2. Dataset (FashionMV): Created the first large-scale, multi-view fashion dataset for this task, featuring 127K products and 220K+ triplets with dual-granularity captions, constructed via an automated pipeline.
3. Model (ProCIR): Proposed a framework that adapts MLLMs for product-level CIR using a two-stage dialogue, caption alignment, and CoT.
4. Empirical Insights:
   • Alignment is Critical: Caption-based alignment is the single most important mechanism.
   • Architecture Prerequisite: The two-stage dialogue is necessary to enable effective source-side alignment.
   • Redundancy: SFT and CoT serve as partially redundant knowledge injection paths. Once the model is fine-tuned (SFT), CoT becomes unnecessary and can even degrade performance.

4. Experimental Results

• Setup: Evaluated on three benchmarks: DeepFashion, Fashion200K, and FashionGen-val.
• Performance:
  • The best 0.8B-parameter ProCIR model outperformed all baselines, including general-purpose embedding models that are 10x larger (e.g., Qwen3-V-8B).
  • On the DeepFashion validation set, ProCIR achieved 89.2% R@5 (vs. 87.4% for the 8B baseline).
  • It significantly outperformed single-image CIR methods (CLIP4CIR, SPRC) and standard multi-view aggregation strategies (MeanPool, MaxSim).
• Ablation Findings:
  • Alignment provided the largest individual gain (e.g., +11.8% R@5 on DeepFashion when added to MT+CoT).
  • SFT vs. CoT: The combination of MT + Align + SFT (without CoT) yielded the best results. Adding CoT to an SFT-initialized model actually reduced performance, confirming that SFT internalizes the knowledge more efficiently than in-context CoT.

5. Significance

• Bridging the Gap: This work bridges the gap between academic CIR research (single-image) and real-world e-commerce needs (multi-view product reasoning).
• Efficiency: Demonstrates that a small, specialized model (0.8B) with the right architecture and data pipeline can outperform massive general-purpose models in specific domains.
• Methodological Insight: Provides a clear blueprint for how to leverage MLLMs for retrieval tasks: decoupling perception and reasoning, using auxiliary text alignment, and prioritizing SFT over complex in-context prompting for knowledge injection.
• Open Source: The dataset, model, and code are publicly available, setting a new standard for product-level retrieval research.