Technical Summary: ZooClaw-FashionSigLIP2

Problem Statement

Adapting foundation vision-language encoders (VLEs) to specialized retrieval tasks presents a fundamental tradeoff: gains on the target distribution often come at the cost of the foundation model's broad out-of-distribution (OOD) generalization. Fashion retrieval is a stringent instance of this problem, requiring models to handle both short, keyword-based user queries (e.g., "red floral maxi dress") and detailed, attribute-rich catalog descriptions. While pre-trained VLEs like SigLIP2 offer strong zero-shot capabilities, they lack the fine-grained visual and textual distinctions necessary for fashion relevance. Conversely, domain-specific fine-tuning often sacrifices OOD robustness, failing to generalize to unseen catalogs or different query styles.

Methodology

The authors propose ZooClaw-FashionSigLIP2, a fashion-specialized SigLIP2-base model designed to resolve the specialization-generalization tension through a three-stage recipe:

1. Full Fine-Tuning with Knowledge Distillation:

   • Architecture: The authors utilize full fine-tuning rather than parameter-efficient methods (like LoRA), arguing that full-rank updates are necessary to absorb multi-task objectives and distillation regularizers.
   • Training Objective: The model is trained using a Generalized Contrastive Loss (GCL) that incorporates graded relevance scores (0–10) to leverage soft labels. This is combined with a multi-task formulation handling both short queries (≤8 words) and long queries (30–60 words).
   • Distillation: To prevent catastrophic forgetting of OOD capabilities, the authors apply Learning without Forgetting (LwF). A frozen copy of the base SigLIP2 model acts as a teacher, minimizing the cosine distance between student and teacher image embeddings.

2. WISE-FT Weight Interpolation:

   • Following the WISE-FT approach, the final model is constructed by linearly interpolating between the base model weights ($\theta_0$) and the fine-tuned weights ($\theta_{ft}$): $\theta_\alpha = (1-\alpha)\theta_0 + \alpha\theta_{ft}$.
   • The interpolation coefficient $\alpha$ is swept to find an operating point that maximizes the minimum margin over baselines across all benchmarks. The authors select $\alpha=0.4$, which balances in-domain specialization with OOD retention.

3. Data Construction:

   • Training: Derived from a proprietary catalog (Gensmo) containing structured attributes (color, material, style, etc.). Short and long queries are generated using Gemma-4-31B, conditioned on sampled attributes or full attribute sets, respectively.
   • Evaluation: The authors introduce ZooClaw-Fashion, a new high-quality benchmark with 2,000 queries and 12,000 images, featuring VLM-scored relevance labels.

Key Contributions

1. Model Performance and Recipe Validation

The paper demonstrates that the proposed recipe (Full FT + Distillation + WISE-FT) outperforms alternative approaches:

• Superiority over LoRA: Extensive ablations show that no LoRA configuration matches full fine-tuning on in-domain retrieval or provides as strong a foundation for model soups.
• Superiority over Scaling: Simply scaling the backbone (up to 1B parameters) yields diminishing returns on OOD benchmarks and may even degrade performance compared to the base model.
• Superiority over External Data: Adding external data (MARQO-FASHION) consistently degraded performance, attributed to distributional mismatch.

2. Benchmark Quality Analysis and Pooled Re-evaluation

A significant contribution is the systematic analysis of the Fashion200k benchmark. The authors identify that the public ground truth is biased toward "caption-source instance recovery" (linking queries only to the image from which the caption was generated), systematically favoring models trained on the same caption distribution.

• Pooled Re-evaluation: To enable fair comparison, the authors apply a TREC-style pooled protocol. They aggregate top-10 retrievals from all systems, grade them using Gemma-4-31B on a 1–5 scale, and re-evaluate metrics.
• Result: Under this fair, pooled evaluation, ZooClaw-FashionSigLIP2 leads or ties all baselines on every metric, whereas the original ground truth favored Marqo-fashionSigLIP.

3. Open-Source Release

The authors release:

• The ZooClaw-FashionSigLIP2 model weights.
• The ZooClaw-Fashion evaluation benchmark.
• The Fashion200k pooled evaluation artifacts (102,494 graded judgments) to facilitate reproducible research.

Results

Under fair evaluation, ZooClaw-FashionSigLIP2 achieves state-of-the-art results across the suite:

• ZooClaw-Fashion: Leads in R@1, R@10, and MRR for both short and long queries.
• H&M: Leads in R@10 and MRR.
• Fashion200k (Pooled): Outperforms Marqo-fashionSigLIP by +0.003 on pooled R@10 and +0.012 on pooled nDCG@10 (at relevance threshold $\ge$ 3).
• Robustness: The model maintains strong performance on OOD benchmarks (Fashion200k, H&M) while excelling on the in-domain ZooClaw-Fashion benchmark, successfully navigating the tradeoff that typically plagues fine-tuned models.

Significance

The paper claims that ZooClaw-FashionSigLIP2 resolves the tension between in-domain performance and OOD generalization through a simple, effective recipe of full fine-tuning with distillation and weight interpolation. It challenges the efficacy of parameter-efficient methods (LoRA) and simple backbone scaling for this specific domain. Furthermore, the work highlights the critical importance of benchmark quality, demonstrating that standard fashion retrieval benchmarks may contain structural biases that obscure true model capabilities. By releasing a corrected evaluation protocol and new benchmarks, the authors aim to provide a more reliable foundation for future research in fashion retrieval.