TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

1. Problem Statement

Composed Image Retrieval (CIR) allows users to retrieve a target image using a query consisting of a reference image and a modification text. While existing CIR research has progressed, current setups rely on short, simple modification texts that cover only a limited range of salient changes. This creates two critical limitations for real-world applications:

1. Insufficient Entity Coverage: When multiple entities need modification, training signals tend to focus on salient regions, causing models to ignore less prominent entities. In standard datasets, detailed descriptions of specific entities are often omitted or buried in prepositions/conjunctions.
2. Clause-Entity Misalignment: Real-world queries often involve complex constraints where:
   • Multiple clauses constrain the same entity (e.g., modifying the hem, shoulder, and belt of a single dress).
   • A single clause constrains multiple entities (e.g., changing three dogs in an image to huskies).
   • Existing models struggle to map these "one-to-many" or "many-to-one" relationships effectively, leading to significant performance drops in multi-modification scenarios.

2. Methodology

The authors propose TEMA (Text-oriented Entity Mapping Architecture), the first CIR framework designed to handle both simple and complex multi-modification queries. TEMA operates on two main fronts: data construction and model architecture.

A. Dataset Construction (M-FashionIQ and M-CIRR)

To address the lack of multi-modification data, the authors constructed two new datasets:

• M-FashionIQ: Based on the FashionIQ dataset (fashion domain).
• M-CIRR: Based on the CIRR dataset (open domain).
• Construction Process:
  • MMT Generation: They replaced original short texts with Multi-Modification Texts (MMT) using a Multimodal Large Language Model (MLLM, specifically Llama 3.2). The MLLM was prompted to generate detailed, step-by-step instructions transforming the reference image to the target image, explicitly covering all entities and attributes.
  • Quality Control: A hybrid pipeline involving GPT-4o for hallucination checks and human annotators for consistency, accuracy, diversity, and quality checks ensures the MMTs are faithful to the visual changes.
  • Result: The new datasets feature significantly longer modification texts (avg. 152.7 tokens for M-FashionIQ vs. 24.7 in original) with explicit coverage of Complex Relations (CR), Multiple Entities (ME), and Fine-Grained (FG) attributes.

B. TEMA Architecture

TEMA introduces two novel modules to handle the complexity of MMT:

1. MMT Parsing Assistant (PA) - Training Only:

   • Purpose: To ensure the model does not miss entities buried in long texts.
   • Mechanism: Uses an LLM (e.g., GPT-3.5-turbo) to generate a Summary of the MMT that explicitly lists all to-be-modified entities.
   • Consistency Detector: Verifies that the summary covers all entities in the MMT without hallucinations. If inconsistencies are found, the summary is iteratively refined.
   • Role: The summary acts as a "guide" during training to enhance entity exposure but is disabled during inference to maintain efficiency.

2. MMT-oriented Entity Mapping (EM):

   • Purpose: To align multiple text clauses with specific visual entities (solving Clause-Entity Misalignment).
   • Mechanism:
     • Learnable Queries: Introduces a set of learnable queries ($a_q$) that act as anchors for specific entities.
     • Textual Mapping: Aggregates multiple MMT clauses related to the same entity, guided by the PA-generated summary.
     • Visual Mapping: Aggregates local visual features corresponding to the same entity.
     • Summary-Guided Distillation: A loss function forces the aggregated text tokens to align semantically with the PA summary, ensuring comprehensive entity coverage.
     • Orthogonal Regularization: Ensures that different channels in the learnable queries remain semantically independent to prevent feature confusion.

3. Training Objective:
   The model minimizes a combined loss function:
   $$L = L_{bbc} + \kappa L_{summ} + \mu L_{ortho}$$
   Where $L_{bbc}$ is the batch-based classification loss, $L_{summ}$ is the summary-guided distillation loss, and $L_{ortho}$ is the orthogonal regularization.

3. Key Contributions

1. Identification of Limitations: Demonstrated that existing CIR models fail in practical multi-modification scenarios due to insufficient entity coverage and clause-entity misalignment.
2. New Benchmarks: Constructed M-FashionIQ and M-CIRR, the first instruction-rich multi-modification datasets that provide dense, fine-grained training signals for complex retrieval tasks.
3. TEMA Framework: Proposed the first CIR framework capable of handling multi-modification while remaining robust to simple modifications. It uniquely combines an LLM-based training-time assistant (PA) with a learnable entity mapping module (EM) to bridge the gap between text clauses and visual entities.
4. State-of-the-Art Performance: Achieved superior results on both original and multi-modification benchmarks, proving the model's ability to generalize across different granularities of queries.

4. Experimental Results

• Datasets: Evaluated on M-FashionIQ, M-CIRR, and original FashionIQ/CIRR.
• Performance:
  • Multi-Modification: TEMA significantly outperformed strong baselines (e.g., BLIP4CIR, Candidate, FashionSAP). On M-FashionIQ, it achieved 50.59% R@10 and 72.09% R@50, surpassing the next best model (Candidate) by a clear margin.
  • Original CIR: TEMA also maintained superior performance on traditional CIR benchmarks (FashionIQ and CIRR), demonstrating that the multi-modification training does not degrade performance on simple queries.
• Ablation Studies:
  • Removing the PA module caused the most significant performance drop, highlighting the importance of explicit entity coverage during training.
  • Removing EM components also led to substantial degradation, confirming the necessity of aggregating clauses to entities.
  • The model showed robustness to different LLMs used for the PA module (e.g., Llama-3, Qwen2-VL), proving it does not rely on a specific proprietary model.
• Efficiency: Despite using an LLM during training, TEMA is computationally efficient during inference (PA is disabled) and requires less GPU memory and training time compared to sub-optimal baselines like Candidate.

5. Significance

• Bridging the Gap to Real-World Use: By addressing the "Insufficient Entity Coverage" and "Clause-Entity Misalignment" issues, TEMA moves CIR from controlled, simple experiments to complex, real-world scenarios where users provide detailed, multi-faceted instructions.
• Mitigating False Negatives: The detailed MMT annotations help resolve the "false-negative" problem in CIR datasets, where images that satisfy a query are incorrectly labeled as negatives because the original short text was too vague to distinguish them.
• Scalability: The framework demonstrates that leveraging LLMs for training-time supervision (without inference overhead) is a viable strategy for enhancing multimodal understanding in retrieval tasks.

In conclusion, TEMA represents a significant step forward in Composed Image Retrieval by introducing a data-centric and architecture-centric solution to handle the complexity of multi-entity, multi-clause modification requests.