Technical Summary: Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval (DeCIR)

1. Problem Definition

Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image given a reference image and a text modification without relying on human-annotated reference–instruction–target triplets. While projection-based methods (which map images to text embeddings using models like CLIP) offer lightweight inference without Large Language Models (LLMs), they suffer from a semantic transition bottleneck.

In these methods, the text encoder implicitly infers the transition from the source (reference) to the target. However, without explicit supervision for this transition, the model often falls into an "endpoint shortcut." Instead of learning a source-conditioned semantic transition (e.g., "change the green bird to two green birds"), the model treats the modification text merely as a target-side attribute cue (e.g., "two birds"), retrieving any image matching the target attribute while ignoring the specific constraints of the reference image.

Furthermore, the authors identify that naively adding semantic transition supervision to the same low-rank text adapter used for endpoint alignment creates an optimization conflict. The objectives of aligning the query to the target endpoint and aligning the instruction to the semantic displacement (transition) compete for the same parameter space, leading to gradient interference that degrades performance.

2. Methodology: DeCIR

The proposed framework, DeCIR, addresses these issues through conflict-aware endpoint–transition decoupling and a novel merging strategy.

2.1 Supervision Construction via LLMs (Offline Only)

To overcome the lack of annotated triplets, DeCIR uses an LLM (specifically glm-4-flash) offline to convert standard image-caption pairs (from CC3M) into structured forward/reverse edit tuples.

• Forward Edit: Generates a target caption and an instruction describing a specific visual change from the source.
• Reverse Edit: Generates an instruction that undoes the forward change.
• This process creates a bidirectional supervision signal $(I_{ref}, c_{src}, t_{fwd}, c_{tgt}, t_{rev})$ without requiring human annotation or LLM usage during inference.

2.2 Decoupled Training Architecture

DeCIR employs two specialized low-rank text adapter branches over the CLIP text encoder, initialized from the same weights but optimized separately:

1. Endpoint Alignment Branch ($E^{end}_T$): Optimized with a contrastive loss ($L_{end}$) to align the composed query embedding with the generated target caption embedding. This branch defines the deployable retrieval pathway.
2. Semantic Transition Branch ($E^{trans}_T$): Optimized with a transition loss ($L_{trans}$) to align the forward instruction with the semantic displacement vector ($\delta = f^{trans}_{tgt} - f_{src}$) and the reverse instruction with $-\delta$. This branch learns to encode the direction of change rather than the final endpoint.

The visual encoder LoRA parameters and the inversion network ($f_\phi$) are updated only by the endpoint branch, ensuring the retrieval pathway remains stable while the transition branch specializes in instruction semantics.

2.3 Low-Rank Directional Merge (LRDM)

To maintain a lightweight inference pipeline, the two branches are merged into a single deployable adapter post-training.

• Shared Basis: Both branches share the low-rank output basis matrix $B_\ell$.
• Coefficient Merging: The branch-specific coefficient matrices ($A^{end}_\ell$ and $A^{trans}_\ell$) are merged linearly: $A^{merge}_\ell = (1-\alpha)A^{end}_\ell + \alpha A^{trans}_\ell$.
• Asymmetry: The endpoint branch preserves the backbone structure, while the transition branch contributes semantic transition coefficients within that shared basis. The transition branch is discarded after merging, resulting in a single text adapter for inference.

3. Key Contributions

1. Identification of the Semantic Transition Bottleneck: The paper demonstrates that projection-based ZS-CIR fails on complex edits because the text encoder collapses the modification into a target-side cue, dropping source-conditioned evidence.
2. Diagnosis of Optimization Conflict: Through layer-wise gradient probing, the authors show that jointly optimizing endpoint alignment and semantic transition alignment in a shared adapter causes significant gradient interference, particularly in deeper transformer layers.
3. Decoupled Learning Framework: DeCIR introduces a training strategy that separates endpoint and transition learning into specialized branches to avoid interference.
4. Low-Rank Directional Merge (LRDM): A merging technique that folds semantic transition sensitivity into the projection-based pipeline without increasing inference complexity or requiring LLMs at runtime.

4. Experimental Results

DeCIR was evaluated on four benchmarks: CIRR (natural image disambiguation), CIRCO (multi-target ranking), FashionIQ (fashion edits), and GeneCIS (conditional retrieval).

• Performance: DeCIR consistently outperforms state-of-the-art projection-based methods (e.g., Pic2Word, DistillCIR, Slerp) and approaches or exceeds LLM-based methods (e.g., CIReVL) in retrieval accuracy.
  • On CIRCO (ViT-L/14), DeCIR achieved 21.3% mAP@5, improving over the previous best (DistillCIR) by 2.6%.
  • On CIRR, DeCIR reached 33.4% R@1 (ViT-L/14), surpassing DistillCIR (32.3%).
  • On FashionIQ, it achieved the best average performance across dress, shirt, and top categories.
• Ablation Studies:
  • Decoupling: Joint training of both objectives in a shared adapter performed worse than retrieval-only training, confirming the gradient conflict. Decoupled training followed by LRDM yielded the best results.
  • Merging: LRDM outperformed generic model merging strategies (Task Arithmetic, TIES, DARE), validating the asymmetric coefficient merging design.
  • Inference Efficiency: DeCIR maintains a lightweight inference profile similar to other projection-based methods (approx. 467M parameters) and does not require LLMs or reasoning loops during inference, unlike heavy LLM-based approaches.

5. Significance and Claims

The paper claims that DeCIR successfully resolves the trade-off between the efficiency of projection-based methods and the reasoning capabilities of LLM-based methods. By explicitly modeling the source-to-target semantic transition through decoupled learning and merging, DeCIR enables lightweight models to perform complex, fine-grained edits without collapsing into endpoint shortcuts.

The authors emphasize that their approach improves the accuracy–efficiency trade-off in ZS-CIR. They do not claim to replace LLM-based reasoning entirely but demonstrate that with proper supervision construction and architectural decoupling, projection-based models can achieve competitive performance while retaining the deployment advantages of being lightweight and inference-time LLM-free. The work is limited to projection-based settings and relies on offline LLM supervision for training data generation.