Technical Summary: GIRCSE – Generative Iterative Refinement for Contrastive Sentence Embeddings

Problem Statement

Current Large Language Model (LLM)-based text embedding methods predominantly rely on an encoder-only paradigm. These models treat LLMs as static feature extractors, performing a single forward pass to generate embeddings via contrastive learning. This approach overlooks the core generative strengths of pretrained LLMs, specifically their ability to reason and iteratively refine representations through autoregressive generation.

The authors identify three primary challenges in transitioning to generative embeddings:

1. Quality Degradation: Naive generation often degrades embedding quality because pretrained LLMs are optimized for fluent, human-readable text rather than tokens aligned with semantic similarity.
2. Ambiguous Targets: Unlike standard language modeling, there is no clear generation target for embeddings; it is unclear what content the model should generate to produce universally useful representations.
3. Framework Limitations: Existing embedding learning frameworks do not support multi-step generative refinement, necessitating new training paradigms that provide meaningful supervision for generative processes.

Methodology: GIRCSE

The authors propose GIRCSE (Generative Iterative Refinement for Contrastive Sentence Embeddings), a framework that bridges the gap between generative LLM capabilities and embedding optimization. The framework consists of two core innovations:

1. Soft Token Generation

To maintain differentiability during training while capturing rich semantics, GIRCSE generates soft tokens rather than discrete tokens.

• Mechanism: At each generation step $k$, the model computes a probability distribution over the vocabulary using the LM head. Instead of sampling a single token, the model computes a soft token $s_k$ as a probability distribution.
• Embedding: The soft token embedding $d_k$ is derived as a weighted sum of all token embeddings in the vocabulary, weighted by their predicted probabilities.
• Benefit: This preserves gradient flow for end-to-end contrastive training and captures the semantic diversity of the full probability distribution, avoiding the information loss inherent in collapsing to a single discrete token.

2. Iterative Contrastive Refinement (ICR)

GIRCSE employs an Iterative Contrastive Refinement objective to guide the generation process. This objective ensures that each step in the generation sequence progressively refines the semantic representation.

• Stepwise Contrastive Loss: Unlike methods that only supervise the final output, ICR applies contrastive supervision at every generation step $k$. The model pools the first $k$ generated soft tokens to form an intermediate embedding $z_k$ and computes a contrastive loss against positive and negative documents. This prevents early steps from drifting into trivial or noisy representations.
• Iterative Refinement Regularization: To address the issue of redundant information where later steps produce highly similar tokens, a regularization term is introduced. This term penalizes cases where later generation steps fail to improve the embedding quality (i.e., reduce the loss) compared to earlier steps, enforcing monotonic improvement.

The total training objective combines the stepwise contrastive loss and the refinement regularization:
$$L_{total} = L_{contrast} + \lambda L_{reg}$$

Key Contributions

1. Novel Embedding Framework: GIRCSE is an end-to-end framework integrating autoregressive generation with contrastive learning. It generates soft refinement tokens without explicit targets, progressively distilling semantics into high-quality embeddings.
2. Superior Performance: The model achieves state-of-the-art or competitive results across diverse benchmarks. It ranks within the top 5–6 on the MTEB benchmark and top 2–3 on instruction-following tasks, outperforming 18 state-of-the-art baselines. Notably, it achieves these results using only 0.2M training examples, significantly fewer than the multi-million-scale datasets used by many competitors.
3. Test-Time Scaling: GIRCSE exhibits an emergent property where embedding quality steadily improves as the number of generated tokens ($K$) increases at inference time. This represents a novel scaling paradigm for embedding models, analogous to test-time compute scaling in reasoning LLMs.

Experimental Results

The authors evaluated GIRCSE on the MTEB (English, v2) benchmark (covering 41 datasets across 7 task types) and instruction-following benchmarks (INTENTEMOTION and NYTCLUSTERING).

• Balanced Performance: Unlike prior methods that often trade off performance between generic tasks and instruction following (e.g., gte-Qwen2 excels on MTEB but lags on instruction following, while Inbedder excels on instruction following but lags on MTEB), GIRCSE delivers consistently strong results in both categories.
• Ablation Studies: Experiments confirm that removing any component (generative embedding, stepwise loss, or iterative refinement) leads to performance degradation. The combination of all three yields the best results.
• Qualitative Analysis: Analysis of generated tokens reveals that the model learns to produce instruction-specific semantic chains. For example, when asked to represent the "emotion" of a text, later generation steps produce tokens like "frustrated" and "struggle," whereas "intention" prompts yield tokens like "seek" and "inquiry."
• Efficiency: While generative processes incur higher computational costs, the authors demonstrate that using KV caching reduces the FLOPs to nearly the same level as standard embedding models (within ~1.0–1.1×), making the approach practical for deployment.

Significance and Claims

The paper claims to establish generative iterative refinement as a new paradigm for representation learning. By enabling LLMs to "speak an embedding language"—generating tokens optimized for semantic representation rather than human readability—GIRCSE moves beyond the limitations of single-pass encoders.

The authors emphasize that their work demonstrates:

• Embeddings can be iteratively refined to capture latent concepts and implicit semantics often missed by encoder-only methods.
• Test-time scaling is a viable and effective strategy for improving embedding quality without retraining.
• Generative approaches can overcome the trade-offs between general-purpose retrieval and instruction-following capabilities, offering a unified solution for text embedding.

The code and pre-trained models are made available to facilitate further research in this direction.