Technical Summary: LightRetriever

Problem Statement

Current Large Language Model (LLM)-based text retrieval systems typically employ symmetric dual-encoder architectures, where both documents and queries are processed by deeply parameterized LLMs to generate dense or sparse representations. While this approach significantly enhances retrieval capabilities through deep contextual modeling, it introduces a critical bottleneck in online deployment. Documents can be pre-encoded and indexed offline, but search queries arrive in real-time and require immediate encoding. Serving a full-sized LLM for query inference on online servers (e.g., A800 GPUs) results in high latency, low throughput, and excessive resource consumption. The paper posits a fundamental tension: while documents benefit from heavy pre-computation, query encoders must operate under strict online constraints, yet existing methods force queries to undergo the same computationally expensive deep modeling as documents.

Methodology: LightRetriever

The authors propose LightRetriever, a novel hybrid retrieval architecture that explicitly breaks the symmetry between document and query encoders. The core insight is that while documents require the full modeling power of LLMs, queries do not need equally heavy processing to achieve strong retrieval performance. The architecture shifts the major modeling cost to the document side while reducing the query side to an extremely lightweight operation.

1. Asymmetric Architecture

• Document Side: Retains a full-sized LLM encoder ($Enc_d$) for both dense and sparse encoding. Documents are pre-computed offline.
• Query Side: Eliminates deep Transformer inference during online serving. The query encoder ($Enc_q$) is reduced to a simple embedding lookup mechanism.

2. Dense Retrieval Mechanism

For dense retrieval, LightRetriever employs a three-stage process:

• Training: A full-sized LLM is used to train the system. For a given query token $t_i$ (prepended with a task instruction), the model generates a token-level representation $v^{den}_{t_i} = Enc_q(Inst; t_i)$. The final dense query vector $v^{den}_q$ is computed by averaging these token vectors:
  $$v^{den}_q = \frac{1}{n} \sum_{i=0}^{n-1} v^{den}_{t_i}$$
  The system is optimized using contrastive loss (Equation 2) to align query and relevant document vectors.
• Caching: Since query tokens do not deeply interact with each other during inference, the token vectors are cacheable. Before serving, the system precomputes token embeddings for the entire vocabulary ($V$) of the LLM and stores them in a single embedding lookup matrix $E$ (shape $[V, H]$).
• Online Serving: At inference time, the query is tokenized, and the dense vector is generated by looking up the cached embeddings for each token and averaging them. This replaces the forward pass of a large Transformer with a single embedding lookup and averaging operation, requiring only a RAM-resident embedding layer.

3. Sparse Retrieval Mechanism

For sparse retrieval, the query side is even lighter:

• Query Encoding: The sparse query vector $v^{spr}_q$ is a term-based, unlearnable vector derived directly from tokenization, where the value for a token $t$ is simply its count in the query. No encoder is needed.
• Document Encoding: The document is processed by the full LLM. The final-layer hidden states are projected to the vocabulary space via the model's language modeling head. To induce sparsity, a ReLU activation and log-saturation function are applied, followed by max-pooling over the sequence length.
• Hybrid Scoring: Dense and sparse scores are linearly interpolated to produce the final retrieval ranking.

Key Contributions

1. Extreme Query Inference Speed: By replacing deep query modeling with an embedding lookup, LightRetriever achieves a 1000x speedup in query encoding compared to serving a full Llama-8b model on an A800 GPU.
2. Throughput Improvement: The architecture enables a 10x increase in end-to-end retrieval throughput (Queries Per Second).
3. Performance Retention: Despite the drastic reduction in query-side computation, the method maintains approximately 95% of the retrieval performance of full symmetric LLM-based retrievers across diverse benchmarks.
4. Generalization: The approach is validated across various foundation models (Llama-1B/3B/8B, Qwen-1.5B/3B/7B) and retrieval tasks (BeIR, CMTEB), demonstrating robustness without requiring task-specific architectural changes.

Experimental Results

The paper evaluates LightRetriever on 15 English tasks from BeIR and 8 Chinese tasks from CMTEB Retrieval.

• Speed: On a benchmark of 65,536 queries against 1M passages, LightRetriever reduced total end-to-end retrieval time from ~119s (Full-Llama8b) to ~9.36s, achieving a throughput of 6,999 QPS compared to 549 QPS for the full model.
• Effectiveness:
  • LightRetriever-Llama3.1-8b achieved an nDCG@10 of 54.4 on BeIR and 63.0 on CMTEB-R.
  • This represents a modest degradation of -2.4 and -4.6 points respectively compared to the full symmetric Llama3.1-8b baseline (56.8 and 67.6), while avoiding the heavy query inference cost.
  • The method outperforms other efficient baselines like BGE-m3 (49.6 on BeIR) and approaches the performance of E5-Mistral and LLM2Vec.
• Ablation Studies:
  • Removing the full document encoder (symmetric lightweight) caused severe performance drops (e.g., -13.8 on BeIR), confirming the necessity of deep document modeling.
  • Using only an embedding bag for query training (without the full Transformer) also resulted in significant performance loss (-11.2 on BeIR), indicating that deep modeling is essential during the training phase to learn effective representations, even if it is discarded at inference.

Significance and Claims

The paper claims that LightRetriever offers a scalable paradigm for latency-sensitive and high-throughput retrieval scenarios. It addresses the practical inefficiency of deploying giant LLMs for real-time query encoding without sacrificing the semantic understanding benefits of LLM-based retrieval. The authors emphasize that their work transfers the computational burden from the online query side to the offline document side, allowing systems to leverage the power of large models while maintaining the responsiveness required for production search engines. The method is presented as the first to explicitly address the online efficiency bottleneck on the query side of LLM-based retrievers through this extreme decoupling.