FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrieval

Technical Summary: FlowCIR

Problem Statement
Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image by editing a reference image based on a natural language instruction, without relying on domain-specific annotated triplets. Existing ZS-CIR methods predominantly rely on textual inversion, which translates the reference image into a small set of pseudo-text tokens and concatenates them with the instruction in the text embedding space. The authors argue this approach is lossy and brittle for fine-grained semantics because compressing a rich image into a few tokens inevitably discards visual cues. Furthermore, learning effective inversion projectors often requires substantial computational resources (e.g., multi-GPU training on web-scale data). Additionally, Vision-Language Models (VLMs) like CLIP struggle with negation (e.g., "no dog" or "remove stripes"), often causing affirmative and negated concepts to collapse geometrically in the embedding space, leading to retrieval failures on negation-heavy queries.

Methodology: FlowCIR
The paper proposes FlowCIR, a new paradigm that reframes ZS-CIR as conditional semantic transport within the embedding space of pre-trained VLMs.

1. Conditional Semantic Transport via Flow Matching:
   Instead of static token manipulation, FlowCIR treats composition as a continuous transport problem. It learns a lightweight conditional velocity field (a neural ODE) that maps the relative instruction embedding ($x_{Tr}$) toward a target-aligned text embedding ($x_{Tt}$), conditioned on the reference image embedding ($x_{Ir}$).

   • Training: The model is trained on synthesized triplets using Conditional Flow Matching (CFM). It minimizes the regression loss between the predicted velocity field and the ground-truth constant velocity ($x_{Tt} - x_{Tr}$) along a linear path.
   • Retrieval Objective: A contrastive retrieval loss (InfoNCE) is added, where the predicted target text embedding is matched against the ground-truth target image embedding, using top-$K$ hard negatives mined from the batch to sharpen discrimination.
   • Efficiency: The model operates on pre-extracted VLM embeddings and trains only a small transport module (a deep residual MLP) without updating the image or text encoders. This allows training in approximately 0.5 hours on a single GPU.

2. Multi-Negative Steering (Inference-Only):
   To address the geometric ambiguity of negation in VLMs, the authors propose an inference-only strategy.

   • Mechanism: Upon receiving an instruction containing negation (e.g., "remove X"), a lightweight parser decomposes the instruction into an affirmative intent and negated concepts.
   • Steering: The method constructs a steered instruction embedding that remains aligned with the affirmative intent but is explicitly repelled from the geometric directions of the negated concepts (modeled as hyper-spherical caps). This pushes the representation away from the "negated collapse" common in CLIP space before the transport process begins.

Key Contributions

1. FlowCIR Paradigm: Introducing conditional semantic transport via flow matching as a mechanism for zero-shot CIR, replacing static textual inversion with a continuous, reference-guided composition process.
2. Multi-Negative Steering: Proposing an inference-only strategy to mitigate the failure mode of VLMs in handling negation and removal instructions, significantly improving robustness on negation-heavy queries.
3. Training Efficiency: Demonstrating a computationally efficient training protocol that achieves competitive performance with only a lightweight transport module trained for ~0.5 hours on a single GPU, contrasting with the multi-day, multi-GPU requirements of prior inversion-based methods.

Experimental Results
Extensive experiments were conducted on three standard benchmarks: CIRR, CIRCO, and Fashion-IQ, using CLIP ViT-B/32 and ViT-L/14 backbones.

• Performance: FlowCIR achieves strong and competitive performance, often surpassing prior textual-inversion-based methods (e.g., SEARLE, Pic2Word) and generative-based methods (e.g., Compo-Diff).
  • On CIRR (ViT-L/14), it improves Recall@1 by 2.3% over the previous best public-data baseline.
  • On CIRCO (ViT-L/14), it improves mAP@5 by 14.6% (from 13.0 to 14.9).
  • On Fashion-IQ, it achieves the best average Recall@10 and matches the best Recall@50.
• Efficiency: As illustrated in the paper's performance-vs-cost analysis, FlowCIR achieves top-tier performance with a fraction of the training time (0.5h on a single RTX 3090) compared to baselines requiring hundreds of GPU hours or massive TPU clusters.
• Ablation Studies:
  • Conditional Flow Matching is identified as the primary driver of performance, proving that modeling composition as semantic transport is superior to direct regression or text-only baselines.
  • Multi-Negative Steering provides consistent, complementary gains, particularly on negation-heavy queries.
  • Target Space: Transporting toward the target text embedding (rather than the image embedding) yields better results, as it provides a more semantics-centered signal and a more homogeneous mapping.

Significance and Claims
The paper claims that FlowCIR offers a principled and training-efficient mechanism for cross-modal composition that avoids the pitfalls of textual inversion and the high costs of generative approaches. By formulating CIR as conditional semantic transport, the method better preserves fine-grained semantics and enables a progressive, reference-guided composition process. The authors position this work as a step toward a new paradigm for cross-modal retrieval that moves beyond static composition, highlighting that flow-based methods can achieve high accuracy with minimal computational overhead. The Multi-Negative Steering strategy is highlighted as a practical solution to a known weakness of CLIP-style models, improving retrieval faithfulness without requiring additional training data or model updates.