Technical Summary: TIGER-FG

Problem Definition

The paper addresses Image-to-Multimodal Retrieval (IMMR) in e-commerce, a setting where a user provides a cropped visual query (a specific product region) to retrieve items represented by full item images paired with structured text (titles, categories, attributes). This setting introduces two critical asymmetries that existing methods struggle to handle:

1. Modality Disparity: The query is purely visual, while candidates are multimodal (image + text). Standard image-image or image-text retrieval models assume symmetric modalities.
2. Granularity Disparity: The query is a local, object-centric crop, while candidate images are scene-level, often containing background context, multiple products, or distractors.

Existing solutions fall into two categories with distinct limitations:

• Detection-based pipelines: Use explicit object detection or grounding models (e.g., GroundingDINO) to localize regions before encoding. While they address granularity, they incur high computational costs, suffer from error propagation, and rely on detection quality which is difficult to adapt to dense, attribute-rich e-commerce data.
• CLIP-style encoders: Avoid detection by using global image-text embeddings. However, they are vulnerable to background noise and irrelevant objects in full-item images, failing to focus on the query-relevant region.

Methodology: TIGER-FG

The authors propose TIGER-FG (Text-Guided Implicit Fine-Grained Grounding), a framework that learns target-focused item representations without explicit object detection during indexing or retrieval.

1. Text-Guided Implicit Grounding

Instead of predicting bounding boxes, TIGER-FG uses the structured item text as semantic guidance to interact with visual tokens.

• Architecture: An asymmetric dual-encoder. The query encoder processes the cropped region. The item encoder processes the full image and text.
• Text-Guided Cross-Attention: The item encoder employs learnable query tokens ($Q$) that first attend to textual tokens ($T'$) to form text-conditioned queries, which then attend to visual tokens ($V'$). This produces visual evidence conditioned on the text semantics.
• Complementary Visual Branch: To prevent the model from ignoring visual cues not explicitly mentioned in the text, a parallel branch uses a CLS token and the text-guided feature to weight visual patches via a temperature-controlled softmax.
• Fusion: The final item embedding combines the text-guided semantic features and the appearance-focused features.

2. Regularization and Distillation Objectives

To ensure the implicit grounding is stable and discriminative, the authors introduce three auxiliary objectives:

• Region–Text Embedding Space Alignment ($L_{v2t}$): A contrastive loss that aligns the cropped target region (ground-truth box) with its structured text in a shared space before fusion, establishing explicit region-text correspondence.
• Target-Anchored Fused–Region Alignment ($L_{i2v}$): Uses the ground-truth region feature as a visual anchor to pull the fused item embedding toward the target object. It employs hard negatives by re-fusing the item image with mismatched texts from different categories to prevent false positives.
• Spatial-Relational Distillation ($L_{SRD}$): Distills patch-level spatial relations from a frozen DINOv3 teacher into the student encoder, ensuring the patch-level spatial structure remains consistent with a self-supervised backbone.
• Similarity-Distribution Distillation ($L_{SDD}$): Aligns the query-to-item similarity distribution of the student with a frozen image-to-image retrieval teacher (MoCo-pretrained), preserving the global retrieval structure.

3. Dataset Construction: ECom-RF-IMMR

The authors construct a new benchmark suite to support training and evaluation:

• ECom-RF-IMMR-10M: A large-scale training set (10M pairs) mined from product catalogs and user click logs, utilizing pseudo-boxes generated via GroundingDINO and CLIP filtering.
• ECom-RF-IMMR-Normal: A standard evaluation set with clean, centered item images.
• ECom-RF-IMMR-Mosaic: A challenging evaluation set where candidate images are re-synthesized by compositing the target item with cross-category distractors on random backgrounds. This tests robustness against clutter and center bias.

Key Contributions

1. Text-Guided Implicit Fine-Grained Grounding: A detection-free item encoder that leverages structured text to guide visual token interaction, producing target-focused embeddings directly from full image-text pairs.
2. Distillation-Enhanced Learning: A novel combination of spatial-relational and similarity-distribution distillation objectives that jointly constrain fine-grained visual structure and global retrieval behavior.
3. ECom-RF-IMMR Benchmark: A realistic, large-scale benchmark suite with a 10M-pair training set and evaluation splits covering both clean and cluttered (mosaic) item layouts, addressing the lack of suitable datasets for IMMR.

Experimental Results

Experiments were conducted on the constructed benchmarks and public datasets (eSSPR, LookBench).

• Performance on ECom-RF-IMMR:
  • On the Normal split, TIGER-FG achieved 80.1% Recall@1, outperforming the strongest baseline (Qwen3-VL-Emb) by 6.1 points.
  • On the Mosaic split (cluttered scenes), TIGER-FG achieved 75.2% Recall@1, outperforming the strongest baseline by 34.4 points. Notably, strong baselines like Qwen3-VL-Emb dropped significantly (from 74.0% to 40.8%) in the Mosaic setting, whereas TIGER-FG remained robust.
• Generalization: On public benchmarks like LookBench (noisy, one-to-many scenarios), TIGER-FG achieved the best performance across HitRate, Recall, MRR, and NDCG.
• Efficiency: The model uses only 85.7M query-side parameters and 256-dimensional embeddings, demonstrating high efficiency compared to billion-parameter MLLM-based baselines.
• Ablation Studies:
  • Removing Mosaic augmentation (TIGER-FG-RAW) caused a massive performance drop on the Mosaic split (75.2% $\to$ 47.8%), proving the necessity of clutter-aware training.
  • Removing Distillation (SDD) significantly reduced performance, highlighting the importance of preserving similarity structure.
  • Removing Spatial-Relational Distillation (SRD) had a minor impact on ranking metrics but degraded localization quality in visualizations.

Significance and Claims

The paper claims that TIGER-FG effectively bridges the modality and granularity gaps in e-commerce retrieval without relying on expensive or error-prone object detection pipelines. By leveraging structured item text as semantic guidance and employing specific distillation objectives, the model learns to implicitly ground queries to relevant regions within cluttered full-item images.

The authors emphasize that their approach offers a practical solution for industrial deployment, balancing high retrieval accuracy with computational efficiency. The construction of the ECom-RF-IMMR benchmark is presented as a significant step toward standardizing the evaluation of fine-grained, multimodal retrieval in realistic, noisy e-commerce environments. The results suggest that for IMMR tasks, specialized architectures with text-guided grounding and clutter-aware training are superior to generic large-scale multimodal embeddings or detection-dependent pipelines.