💬 NLP

Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

The paper introduces Giga-Embeddings, a family of text embedding models featuring a sparse 10B-parameter Mixture-of-Experts encoder that achieves state-of-the-art retrieval quality and high throughput across multiple languages, alongside smaller dense and distilled variants optimized for resource-constrained environments.

Original authors: Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev, Fyodor Minkin

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev, Fyodor Minkin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, computers do not understand words the way humans do. To a machine, a sentence is just a long string of symbols. To make sense of text, researchers have developed a method of turning words into lists of numbers, known as embeddings. Think of these lists as a unique address for every piece of text, placing it in a vast, multi-dimensional space where similar ideas sit close together and different ideas sit far apart. This system powers the tools we use every day, from search engines that find the right document to artificial intelligence that answers questions based on a library of text. However, there is a constant tension in this field: the more complex the system, the better it usually understands language, but the more computer power and memory it requires to run. Making these systems faster and cheaper without losing their intelligence has been a major challenge for scientists.

A team of researchers has introduced a new family of text embedding models called Giga-Embeddings, designed to solve this problem by balancing high quality with high speed. Their most ambitious creation is a massive model containing ten billion parameters, which is a measure of its internal complexity. Instead of using all ten billion parameters to process every single word, this model uses a technique called a mixture of experts. Imagine a large library where, for every book a patron asks for, only a specific, small team of librarians is called in to help, while the rest of the staff remains on standby. In this computer model, only about 1.8 billion parameters are activated for each word, while the full ten billion remain available in memory. This allows the system to carry the knowledge of a huge brain while only doing the work of a smaller one at any given moment.

The researchers tested this large model against several others, including a standard, dense model with three billion parameters and a smaller, distilled version with 480 million parameters. They evaluated the systems on a wide range of tasks involving English, Russian, multilingual text, and computer code. The results showed that the large, sparse model achieved the highest overall performance across all these categories. It not only understood the text better than the smaller models but also processed information faster. When measuring how many words the system could handle per second, the ten-billion-parameter model was 25 percent faster than the three-billion-parameter model and significantly faster than other advanced systems tested in the same environment. This proves that it is possible to build a system that is both incredibly large in its capacity and highly efficient in its operation.

To make these powerful tools accessible to devices with less computing power, the team also created a much smaller version of the model. They used a method called similarity-distribution distillation to teach this compact model. Instead of trying to copy the exact internal numbers of the large teacher model, the small model learned to mimic the large model's final judgments about how similar different pieces of text were to one another. This allowed the small model, which has only 480 million parameters, to perform nearly as well as the much larger three-billion-parameter model. In tests on Russian text, the small model scored slightly higher than a well-known competitor that is almost twice its size, demonstrating that a smaller system can be highly effective if it is taught the right way.

The team released all three versions of their models to the public, allowing others to use them for various applications. They found that the large, sparse model was particularly effective at handling long texts, maintaining its speed even when the input grew longer. While the smaller models required less memory to store, the large model offered the best combination of speed and accuracy for heavy-duty tasks. The researchers noted that their measurements were taken in a specific testing environment, so real-world performance might vary slightly depending on the hardware used. Nevertheless, the work demonstrates a clear path forward: by using smart architectural choices and careful teaching methods, it is possible to create text understanding systems that are both powerful enough to handle complex tasks and efficient enough to run quickly on modern computers.

Technical Summary: Giga-Embeddings

Problem Statement

Text embedding models are critical for semantic search, retrieval-augmented generation, and clustering. While scaling backbone models generally improves representation quality, billion-parameter encoders face significant challenges in storage and serving costs. Existing approaches often rely on knowledge distillation to create smaller models, but these methods frequently require matching hidden dimensions or architectures between teacher and student. Furthermore, while Mixture-of-Experts (MoE) models offer high capacity with sparse activation, their application to bidirectional text embedding encoders for high-throughput inference remains underexplored.

Methodology

The authors introduce Giga-Embeddings, a family of text embedding models comprising three distinct checkpoints: a dense 480M model, a dense 3B model, and a sparse 10B MoE model.

Model Architectures

Base Adaptation: All models adapt decoder-only language models (specifically Qwen3 and DeepSeekMoE families) into bidirectional encoders by replacing causal attention masks with fully visible bidirectional masks. Mean pooling aggregates token states into fixed-dimensional vectors.
Giga-Embeddings-10B-A1.8B: A sparse MoE encoder with 10 billion total parameters. It utilizes 64 routed experts, one shared expert, and top-4 routing, activating approximately 1.8 billion parameters per token. It produces 1536-dimensional embeddings.
Giga-Embeddings-3B: A dense bidirectional Qwen3 encoder with 3 billion parameters and 2048-dimensional outputs.
Giga-Embeddings-480M: A dense bidirectional Qwen3 encoder with 480 million parameters and 1024-dimensional outputs.

Training Pipeline

The models undergo a three-stage contrastive training process:

Contrastive Pre-training: Uses in-batch InfoNCE loss with query-query and document-document contrastive terms.
Retrieval Fine-tuning: Trains with hard negatives mined from a predefined retrieval-rank range.
Multitask Fine-tuning: Optimizes for retrieval, classification, clustering, and semantic similarity tasks.

Similarity-Distribution Distillation:
A key methodological contribution is applied exclusively to the 480M student model during the fine-tuning and multitask stages. Unlike traditional distillation that aligns embedding vectors (requiring equal dimensions), this method aligns the similarity distributions over candidate sets.

The teacher (dense 3B or 10B MoE) and student generate cosine similarities for a set of candidates.
A KL divergence loss ( $L_{KD}$ ) is calculated between the teacher's and student's probability distributions of these similarities.
The total loss combines this distillation term with the standard InfoNCE contrastive loss ( $L_{NCE}$ ): $L = \lambda L_{KD} + (1-\lambda)L_{NCE}$ .
This approach is architecture-agnostic and dimension-agnostic, allowing a dense student to learn from a sparse or differently dimensioned teacher without projecting internal states.

Key Results

Performance on MTEB Benchmarks

Evaluated across English, Russian, Multilingual, and Code suites (MTEB and MMTEB):

10B-A1.8B: Achieves the highest aggregate scores within the family across all four suites. It leads external comparisons on the Russian MTEB benchmark.
3B Model: Serves as the dense baseline, outperforming the 480M model by margins ranging from 2.41 to 6.92 points depending on the suite.
480M Distilled Model: Scores 70.98 on Russian MTEB, outperforming the 823M FRIDA model (70.95) while using 42% fewer parameters. It remains within 2.41–6.92 points of the 3B model.

Throughput and Efficiency

MoE Throughput: The 10B-A1.8B model achieves the highest measured throughput among compared systems. At a 1024-token input length, its throughput is 25% higher than that of the dense 3B model and 1.56–2.65× that of the evaluated external systems (e.g., Qwen3-Embedding 4B, F2LLM-v2-8B).
Storage: The 480M model requires approximately 0.96 GB of weight storage (fp16), roughly 4.8% of the storage required by the 10B MoE model.

Ablation Studies

Distillation ablation on the 480M model shows that similarity-distribution distillation provides consistent gains of 0.09 to 0.22 points across English, Russian, and Code suites compared to a non-distilled version of the same architecture.

Significance and Claims

The paper claims to present the first general-purpose text embedding encoder built around a sparse Mixture-of-Experts architecture. Its primary significance lies in demonstrating that:

MoE for Embeddings: Encoder MoE models can significantly improve inference throughput without sacrificing embedding quality, offering a viable path for high-capacity, high-throughput retrieval systems.
Dimension-Agnostic Distillation: The proposed similarity-distribution distillation method effectively transfers knowledge from large teachers to compact students without requiring architectural or dimensional alignment, enabling efficient compression.
Open Source: The authors release all three model checkpoints (480M, 3B, and 10B-A1.8B) to facilitate further research and deployment.

The authors note limitations, including that benchmark results represent single runs without uncertainty estimates, throughput measurements are specific to a single serving environment (vLLM), and the training mixture includes non-public data, limiting full reproducibility.