🤖 machine learning

FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image Retrieval

FIRE-CIR is a novel model for fine-grained fashion composed image retrieval that enhances accuracy and interpretability by replacing simple embedding similarity with an explicit, question-driven visual reasoning process that verifies attribute-level modifications between reference and candidate images.

Original authors: François Gardères, Camille-Sovanneary Gauthier, Jean Ponce, Shizhe Chen

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: François Gardères, Camille-Sovanneary Gauthier, Jean Ponce, Shizhe Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are shopping for a new outfit online. You find a shirt you like, but you want to tweak it slightly. You type a search query like: "Show me this shirt, but with longer sleeves and in blue."

This is called Composed Image Retrieval (CIR). It's a tricky task for computers because they have to understand two things at once: what to keep from the original picture (the shirt style) and what to change based on your text (longer sleeves, blue color).

Current AI models are like enthusiastic but clumsy assistants. They see the shirt and the words, mash them together into a "vibe," and show you results. Often, they get the vibe right but miss the details. They might show you a blue shirt, but it's the wrong style, or a long-sleeved shirt that's the wrong color. They struggle to reason why an item fits or doesn't fit.

Enter FIRE-CIR, the new "super-intelligent stylist" introduced in this paper.

The Problem: The "Black Box" Assistant

Imagine asking a friend to find a red dress, but they just guess based on a fuzzy feeling. If they bring you a blue dress, they can't explain why they thought it was red. They just say, "It felt right."

Old AI models work this way. They calculate a "similarity score" (a number) but can't tell you, "I rejected this image because it has short sleeves, and you asked for long ones." This makes them bad at fine details, like fashion, where a tiny difference matters.

The Solution: FIRE-CIR (The Detective Stylist)

FIRE-CIR changes the game. Instead of just guessing a similarity score, it acts like a detective or a quality control inspector.

Here is how it works, step-by-step:

1. Breaking the Request into Questions

When you say, "This shirt, but with longer sleeves," FIRE-CIR doesn't just read the sentence. It breaks it down into a checklist of specific questions, like a detective interrogating a suspect:

"Is the candidate shirt blue?"
"Does the candidate shirt have long sleeves?"
"Is the collar the same as the original shirt?"

2. The "Yes/No" Interrogation

FIRE-CIR has a special brain (a Visual Question Answering model) trained specifically on fashion. It looks at every candidate image in the database and asks itself these questions.

Candidate A: "Is it blue?" -> Yes. "Are sleeves long?" -> No. -> Score drops.
Candidate B: "Is it blue?" -> Yes. "Are sleeves long?" -> Yes. -> Score stays high.

This is the "Fine-grained Reasoning" part. It doesn't just guess; it verifies every single detail you asked for.

3. The "False Negative" Trap (And how they fixed it)

Here is a clever trick the authors used. Usually, when training an AI, you only show it the "perfect" answer. But in fashion, there are many correct answers! If you ask for a "blue long-sleeve shirt," there are hundreds of valid blue long-sleeve shirts, not just one.

If the AI only learns from the "one perfect target," it gets confused when it sees other good options. The authors solved this by creating a massive, automatically generated dataset where the AI learns to say "Yes" to many different correct answers, not just one. This makes the AI robust and less likely to reject a good shirt just because it wasn't the exact one in the training photo.

4. The Final Verdict (Re-ranking)

FIRE-CIR works as a "plug-in" upgrade.

First, a fast, standard AI finds the top 250 shirts that might be right.
Then, FIRE-CIR steps in as the referee. It takes those 250 shirts and runs its "interrogation" (the checklist of questions).
It re-sorts the list. If a shirt failed the "long sleeve" check, it gets pushed to the bottom. If it passed everything, it jumps to the top.

Why This Matters

It's Explainable: Unlike the old "black box" models, FIRE-CIR can tell you exactly why it picked an item. "I picked this because it has long sleeves and is blue, unlike the others."
It's Precise: It handles the tiny details that fashion lovers care about (sleeve length, collar type, fabric texture) much better than before.
It's Fast Enough: It doesn't check every shirt in the world (which would take forever). It checks the top candidates, making it fast enough for real-world use.

The Analogy Summary

Old AI: A friend who squints at a picture and says, "Yeah, that looks like a blue shirt with long sleeves," even if it's actually short-sleeved.
FIRE-CIR: A meticulous tailor who pulls out a measuring tape and a color chart, checks every single seam and stitch against your request, and only hands you the shirt that passes every single test.

In short, FIRE-CIR brings logic and proof to the world of fashion search, ensuring that when you ask for a change, you actually get it.

1. Problem Statement

Composed Image Retrieval (CIR) is the task of retrieving a target image that represents a reference image modified by a textual description (e.g., "this shirt, but with longer sleeves"). While recent Vision-Language Models (VLMs) like CLIP have achieved success in CIR by embedding images and text into a shared space, they face critical limitations in the fine-grained fashion domain:

Incomplete Coverage: Standard methods struggle to detect and leverage intricate details within the text and visual inputs.
Suboptimal Fusion: Fused multimodal queries often retain irrelevant visual information from the reference image that contradicts the modification text. This leads to ranking images that are only partially relevant.
Lack of Interpretability: Existing methods act as "black boxes," failing to explain why a specific image was ranked highly.
Data Limitations: Fine-tuning VLMs on existing CIR datasets (like Fashion IQ) often relies on the assumption that only the ground-truth target image is relevant. In reality, multiple candidates can satisfy a query (e.g., many different long-sleeved black T-shirts), leading to "false negatives" during training and noisy supervision.

2. Methodology: FIRE-CIR

The authors propose FIRE-CIR (FIne-grained REasoning for Composed fashion Image Retrieval), a plug-in framework that replaces simple embedding similarity with question-driven visual reasoning.

A. Visual Question Generation

Instead of treating the modification text as a single block, the model uses a Large Language Model (LLM) to decompose the text into a set of attribute-focused visual questions.

Question Types:
1. Single-image questions: Check if a specific attribute exists in the candidate image (e.g., "Is the dress black?").
2. Dual-image questions: Compare the candidate against the reference image (e.g., "Is the skirt looser than in the reference image?").
Expected Answers: For each question, the system generates the expected answer (Yes/No) based on the modification text.
Bias Mitigation: The generation process explicitly detects whether the reference image is needed, preventing information leakage and improving inference speed when only the candidate image is required.

B. Construction of a Fashion-Specific VQA Dataset

To train a robust reasoning system, the authors automatically construct a large-scale Visual Question Answering (VQA) dataset:

Scale: ~413,848 question-image pairs.
Balancing: To prevent language bias (where models guess "Yes" because most answers are positive), the dataset is balanced with "Yes" and "No" answers.
Annotation Strategy:
- For target images, expected answers are derived from the generation process.
- For non-target (negative) images, a larger VLM (InternVL-3-78B) is used to automatically annotate answers, creating a robust training set without manual labeling.
Composition: Approximately 32% of questions require dual-image analysis.

C. Model Training and Inference

Backbone: A large VLM (InternVL-3-1B) is fine-tuned using LoRA (Low-Rank Adaptation) on the constructed VQA dataset. This adapts the model to fashion vocabulary and reasoning without requiring massive computational resources during inference.
Scoring Mechanism:
1. VQA Score ( $x^{VQA}$ ): For a candidate image, the model answers the generated questions. The score is the average probability of the answers matching the expected text constraints.
2. Fusion: The VQA score is combined with a standard CIR similarity score ( $x^{CIR}$ ) to produce a final ranking score:
  $x_i = \text{norm}(x_i^{CIR}) + \lambda_{VQA} \cdot \sigma_k(x_i^{VQA})$
  Where $\sigma_k$ is a sigmoid-like function that makes high VQA scores more similar, making the ranking robust to minor prediction errors.
Efficiency: The VQA re-ranking is applied only to the top- $n$ candidates (e.g., top 250) from the initial retrieval, balancing accuracy and speed.

3. Key Contributions

Question-Driven Reasoning: A novel approach that decomposes modification text into explicit visual questions, enabling fine-grained comparison between reference and candidate images.
Automated VQA Dataset: A large-scale, automatically annotated dataset specifically for fashion CIR that includes both single- and dual-image analysis, mitigating the "false negative" problem in existing CIR data.
Plug-and-Play Framework: FIRE-CIR acts as a re-ranking module that can be integrated with any existing CIR model (e.g., CLIP4CIR, FashionBLIP-2) to significantly boost performance and provide interpretability.

4. Experimental Results

The method was evaluated on the Fashion IQ benchmark (and its refined variants: enhFashionIQ and Refined-FashionIQ).

State-of-the-Art Performance: FIRE-CIR outperforms existing SOTA methods. When combined with FashionBLIP-2, it achieved:
- Average Recall@10: 58.16% (vs. 55.73% for baseline).
- Average Recall@50: 77.35% (vs. 76.12% for baseline).
- It surpassed the previous best method, DetailFusion, by a significant margin.
Generalizability: The framework improved the performance of three different base CIR models (CLIP4CIR, SPRC, FashionBLIP-2) across all datasets and metrics (Recall and Mean Reciprocal Rank).
Ablation Studies:
- Fine-tuning: Fine-tuning the VQA model on the custom dataset improved accuracy by 14.82% over the pre-trained version.
- Component Analysis: Each component (question decomposition, fine-tuning, score merging) contributed incrementally to the final performance.
- Backbone Efficiency: Using the smaller InternVL-3-1B (1B parameters) with fine-tuning outperformed the larger LLaVA-1.5-13B used in competing methods, while offering 3x faster inference.
Interpretability: The model provides attribute-level insights (e.g., "The dress has shoulder straps, which contradicts the 'strapless' request"), allowing users to understand why an image was ranked lower.

5. Significance

FIRE-CIR addresses the fundamental "black box" nature of current CIR systems by introducing explicit, interpretable reasoning. By shifting from global embedding similarity to local, attribute-level verification, it solves the issue of retaining incompatible visual features from reference images.

The work is particularly significant for the fashion e-commerce industry, where fine-grained distinctions (sleeve length, fabric texture, neckline style) are critical. The ability to automatically generate a robust training dataset without manual annotation makes this approach scalable and adaptable to other fine-grained domains beyond fashion. Furthermore, the plug-in nature of the solution allows it to enhance existing commercial retrieval systems with minimal architectural changes.