FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval

This paper introduces FBCIR, a method to diagnose and address focus imbalances in composed image retrieval models by identifying their tendency to over-attend to one modality, and proposes a data augmentation workflow with curated hard negatives to enforce balanced cross-modal reasoning and improve robustness in challenging scenarios.

The Gist

Imagine you are trying to find a specific photo in a massive library using a very smart, but slightly lazy, librarian.

The Task: You give the librarian a starting photo (say, a picture of a castle) and a note saying, "I want this, but in winter." Your goal is to find a picture of a castle covered in snow.

The Problem:
In the past, the librarian was trained on easy examples.

• Easy Example 1: You show a castle and say "winter." The library only has pictures of castles in summer and pictures of snow-covered mountains (no castles). The librarian looks at the photo, sees "castle," ignores the note, and picks the only castle they see. They got the right answer, but they didn't actually listen to your note!
• Easy Example 2: You show a summer castle and say "winter." The library has pictures of winter castles and summer castles. The librarian reads the note "winter," ignores the photo, and picks the winter castle. Again, right answer, wrong reasoning.

The librarian learned shortcuts. They realized they could just look at the picture or just read the note to get the answer, without doing the hard work of combining both.

The Real Challenge (The "Hard Case"):
Now, imagine a tricky test. You show a summer castle and say "winter." The library has:

1. A winter castle (The correct answer).
2. A winter mountain (Matches the note, wrong picture).
3. A summer castle (Matches the picture, wrong note).

If the librarian is still using their shortcuts, they will fail. They might pick the winter mountain because they only read the note, or the summer castle because they only looked at the photo. To get the right answer, they must balance their attention: they need to look at the castle and the word "winter" at the same time.

Enter FBCIR: The Librarian's "Focus Check"

The authors of this paper, Chenchen Zhao and their team, realized that most AI models (the librarians) are bad at this balancing act. They created a tool called FBCIR to diagnose the problem.

1. The Diagnosis (Focus Interpretation)
Think of FBCIR as a "spotlight" that shines on exactly what the AI is looking at when it makes a decision.

• It takes the photo and breaks it into tiny puzzle pieces.
• It takes the text and breaks it into individual words.
• It then plays a game of "What if I hide this piece?" If hiding a specific word (like "winter") or a specific part of the image (the castle tower) changes the answer, that piece is crucial.

By doing this, they found that most AIs are unbalanced. They are like a person wearing a blindfold over one eye. Sometimes they only look at the image; sometimes they only read the text. They aren't using both eyes together.

2. The Cure (Data Augmentation)
Knowing the problem, the team created a new training method called FBCIR-Data. Instead of giving the AI easy practice tests, they built a "Boot Camp" of difficult scenarios.

• The Trick: They created "fake" wrong answers (negatives) that are designed to trick the lazy shortcuts.
  • Scenario A: They show a picture of a castle and a note saying "winter," but they add a fake option that is a winter mountain. If the AI only looks at the picture, it picks the mountain. If it only reads the text, it picks the mountain. To win, the AI must combine both.
  • Scenario B: They create a fake option that is a summer castle but with a note saying "winter." If the AI only looks at the text, it gets confused.

They used advanced AI tools to generate these tricky examples automatically. It's like a coach who keeps changing the rules of the game so the player can't rely on old tricks and must learn to play the whole game properly.

The Results

When they trained the AI on this new, tougher "Boot Camp" data:

• The Shortcuts Disappeared: The AI stopped ignoring one part of the input. It started looking at both the image and the text equally.
• Better Performance: The AI got much better at solving the hard, tricky puzzles (the "Hard Cases").
• Still Good at the Basics: Interestingly, getting better at the hard stuff didn't make them worse at the easy stuff. They became more robust and reliable overall.

The Big Picture

This paper is like a mechanic realizing that cars are failing on icy roads because they were only tested on dry pavement.

• The Mechanic (FBCIR): Checks the car and realizes the tires are only gripping the left side.
• The Fix (FBCIR-Data): Takes the car to a special training track with ice and mud to force the tires to learn how to grip properly on both sides.
• The Outcome: The car is now safe and reliable on any road, not just the easy ones.

In short, the paper teaches AI models to stop taking shortcuts and start paying attention to everything in the picture and the text, making them much smarter and more reliable for real-world use.

Technical Summary

1. Problem Statement

Composed Image Retrieval (CIR) aims to retrieve a target image based on a reference image and a textual modification instruction. While current state-of-the-art models perform well on standard benchmarks, they often fail in challenging scenarios where negative candidates are semantically aligned with either the query image or the query text, but not both.

The authors identify the root cause of this failure as Focus Imbalance.

• The Shortcut Phenomenon: In standard datasets, negative candidates often have large semantic gaps from the target. Models learn "shortcuts" by over-attending to a single modality (e.g., ignoring the text because the image alone is sufficient to distinguish the target) rather than performing joint reasoning.
• The Consequence: When faced with "hard negatives" (e.g., an image visually similar to the query but semantically inconsistent with the text), models relying on shortcuts fail because they neglect the necessary complementary information from the other modality.

2. Methodology

The paper proposes a two-pronged approach: a diagnostic interpretation method and a data augmentation workflow.

A. FBCIR: Multi-Modal Focus Interpretation

To quantify and diagnose focus imbalances, the authors introduce FBCIR, an iterative token pruning method:

1. Tokenization: The query image is segmented into tokens (using Segment Anything), and the query text is split into word-level tokens.
2. Iterative Pruning: The method iteratively removes tokens (masking image segments or replacing text words with empty strings) while preserving the model's original retrieval ranking.
3. Minimal Focus Identification: The process continues until the minimal set of indispensable tokens (the "focus") is identified.
4. Quantitative Metric: A Focus Balance Ratio ($r_M$) is calculated for each modality ($M \in \{Image, Text\}$). The absolute difference $|r_I - r_T|$ serves as the metric for focus imbalance. A large gap indicates the model is over-relying on one modality.

B. FBCIR-Data: Data Augmentation Workflow

To mitigate these imbalances, the authors propose a data augmentation pipeline that constructs curated hard negatives:

1. Text-Augmented Negatives: The workflow uses a Vision-Language Model (VLM) to alter the query text semantics, then uses an image editing model to generate images that remain visually similar to the query but violate the new text intent. This forces the model to attend to text.
2. Image-Augmented Negatives: The workflow generates a comprehensive text description integrating the query semantics, then uses an image generation model to create images that match the text but differ visually from the query. This forces the model to attend to the image.
3. Positive Refinement: For datasets where original positives are loosely consistent with queries, the workflow synthesizes strictly consistent positive images to ensure high-quality training targets.
4. Outcome: This creates a benchmark and a fine-tuning dataset where successful retrieval requires balanced cross-modal reasoning.

3. Key Contributions

1. Problem Formalization: The paper formally defines and empirically demonstrates the prevalence of "focus imbalance" (shortcut learning) in existing CIR models, showing that it is a primary cause of failure in hard negative scenarios.
2. Diagnostic Tool (FBCIR): They introduce a novel, fine-grained interpretation method to quantitatively measure cross-modal focus imbalances, moving beyond simple retrieval accuracy to analyze how models make decisions.
3. Data-Centric Solution: They develop the FBCIR-Data workflow, which generates targeted hard negatives and refined positives. This workflow provides both a specialized benchmark for evaluation and a dataset for fine-tuning to encourage balanced reasoning.

4. Experimental Results

The authors evaluated various CLIP-based and Vision-Language Model (VLM)-based CIR models (e.g., CLIP4CIR, SEARLE, GME, RzenEmbed) on standard benchmarks (CIRR, FashionIQ, GeneCIS) and the new FBCIR-Data benchmark.

• Pre-trained Performance: Existing models showed significant focus imbalances (high $|r_I - r_T|$) and struggled on the FBCIR-Data benchmark, confirming that standard benchmarks do not adequately test cross-modal reasoning.
• Fine-tuning Results:
  • Models fine-tuned on the FBCIR-Data dataset showed substantial improvements in hard-case performance (FBCIR-Data $R_s@1$), with gains significantly larger than those on standard benchmarks.
  • Focus Balancing: The fine-tuned models exhibited a marked reduction in focus imbalance metrics, indicating they learned to attend to both modalities.
  • Robustness: The improvements were consistent across different model architectures and LoRA ranks, and the models maintained or slightly improved performance on standard benchmarks.
• Zero-Shot Transfer: Models fine-tuned on the augmented data also demonstrated improved zero-shot performance on a distribution-shifted hard-case benchmark (FBCIR-CIRR), proving the generalizability of the approach.

5. Significance

• New Evaluation Perspective: The work shifts the CIR research focus from purely maximizing retrieval accuracy to ensuring robustness and interpretability. It highlights that high accuracy on standard benchmarks can mask fundamental flaws in model reasoning.
• Diagnosis and Remedy: By providing a tool to diagnose why models fail (focus imbalance) and a concrete method to fix it (hard negative augmentation), the paper offers a complete framework for improving CIR model reliability.
• Practical Impact: The proposed workflow is applicable to real-world scenarios where users expect precise, multi-modal understanding (e.g., e-commerce, creative search), ensuring models do not rely on superficial cues but truly understand the relationship between visual content and textual instructions.

In conclusion, FBCIR demonstrates that the path to robust Composed Image Retrieval lies not just in larger models, but in balanced cross-modal reasoning enforced through rigorous data construction and focused diagnostic analysis.