💻 computer science

When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression

This paper demonstrates that combining dimensionality reduction and quantization can compress text embeddings to as little as 0.1% of their original size with negligible performance loss, while revealing that the optimal compression strategy varies depending on the specific task.

Original authors: Riku Kisako, Hayato Tsukagoshi, Ryohei Sasano

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Riku Kisako, Hayato Tsukagoshi, Ryohei Sasano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Much Baggage

Imagine you have a massive library of books (text data). To find specific information quickly, you create a "summary card" for every book. These summary cards are called text embeddings.

In the past, these cards were short and simple. But modern AI models create incredibly detailed cards—so detailed that they are huge, heavy, and take up a lot of shelf space (storage) and time to compare (computation). If you have millions of books, these giant cards become a logistical nightmare.

The Two Tools to Shrink the Cards

The paper investigates two ways to make these cards smaller without losing the ability to find the right book:

Quantization (Lowering the Resolution): Imagine your summary card is a high-definition photo. Quantization is like turning that photo into a pixelated, low-resolution image. You keep the same number of pixels (dimensions), but you use fewer colors (bits) to describe each one.
- The Trade-off: You save space, but if you go too low, the image becomes blurry and unrecognizable.
Dimensionality Reduction (Cutting the Size): Imagine your summary card is a long list of 1,000 facts. Dimensionality Reduction is like cutting the list down to just the top 10 facts. You throw away the extra pages.
- The Trade-off: You save a lot of space, but if you cut too much, you might throw away the one fact that actually helps you find the book.

The Big Discovery: Do Both at Once!

The researchers asked: What happens if we do both? Instead of just making the photo pixelated OR just cutting the list, what if we make a short list AND use a low-resolution photo for those few items?

The Answer: It works surprisingly well.
The paper found that by combining these two methods, you can shrink these giant summary cards down to 0.1% of their original size (like shrinking a 100-page document to a single sticky note) while still keeping the AI smart enough to do its job.

It Depends on What You Are Doing

The paper discovered that there is no "one size fits all" strategy. The best way to shrink the card depends on what task the AI is doing:

Classification (Sorting things into boxes): This is like sorting mail into "Junk," "Bills," and "Personal."
- The Finding: This task is very flexible. You can cut the list of facts down to almost nothing (very low dimensions) as long as you keep the "colors" (bits) high enough to see the difference between categories. It's like needing a clear color palette to distinguish a red envelope from a blue one, even if the envelope is tiny.
Retrieval (Finding a specific needle in a haystack): This is like searching for a specific book in a library.
- The Finding: This is the hardest task to shrink. It needs to keep the "shape" of the data intact. If you cut the list of facts too short, you lose the ability to tell similar books apart. It's like trying to find a specific book by only looking at the first letter of the title; you need more details (dimensions) to be accurate.
Clustering & Similarity (Grouping similar items): These tasks fall somewhere in between. They generally prefer keeping more "dimensions" (facts) rather than high "bit-width" (color depth).

The "Magic Trick" of Rotation

The researchers also tested how they cut the list of facts.

Method A (Head-based): Just cutting off the end of the list and keeping the first few items. This is simple and reliable.
Method B (PCA + Rotation): This is like shuffling the deck of cards before cutting. They rearrange the facts so that the most important information is spread out evenly across the list, rather than being concentrated in the first few items.
- The Result: When you are trying to shrink the card a lot (aggressive compression), shuffling the deck first (Method B) works better. However, if you need to keep the card almost perfect (99% accuracy), just cutting the end (Method A) is safer and more reliable.

The "Zero" Trap

One interesting technical finding was about how they stored the numbers.
Text embeddings often have numbers that are very close to zero. If you use a standard "low-bit" format (like a fixed set of numbers), many of these tiny, important numbers get rounded down to zero.

The Analogy: Imagine trying to describe a whisper. If your microphone only has settings for "Loud," "Medium," and "Silent," the whisper gets recorded as "Silent," and you lose the information.
The Fix: The researchers used a custom "dictionary" that matched the specific distribution of the data. This ensured that even the tiny whispers (small numbers) were captured correctly, preventing the AI from going deaf to subtle details.

Summary

The paper proves that you don't need to choose between making your data smaller or keeping it smart. By using a smart combination of cutting the list and lowering the resolution, you can compress text data to a tiny fraction of its size (0.1%) with almost no loss in performance. However, you must choose the right combination based on whether you are sorting mail (Classification) or searching for a needle (Retrieval).

Technical Summary: When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression

1. Problem Statement

Recent high-performing text embedding models, including those based on Large Language Models (LLMs), generate high-dimensional real-valued vectors. While these vectors yield strong downstream performance, their storage and computational costs are substantial, particularly in retrieval and Retrieval-Augmented Generation (RAG) systems where massive collections of embeddings must be stored and compared.

Existing compression strategies typically address this via quantization (reducing bits per coordinate) or dimensionality reduction (reducing the number of coordinates). However, the interaction between these two methods remains insufficiently understood. It is unclear how to optimally balance dimensionality and bit-width under a strict compression budget, or whether combining these methods yields synergistic benefits beyond using them in isolation.

2. Methodology

Experimental Setup

The authors conducted a systematic evaluation across four pretrained embedding models and four task families from the Massive Text Embedding Benchmark (MTEB):

Models: gte-Qwen2-7B-instruct, E5-mistral-7b-instruct (instruction-based), E5-large-v2 (prefix-based), and Qwen3-Embedding-8B (Matryoshka-compatible).
Tasks: Classification, Clustering, Retrieval, and Semantic Textual Similarity (STS).
Compression Budget: Defined as $C(d, b) = d \times b$ , where $d$ is the reduced dimensionality and $b$ is the bit-width.

Compression Techniques

The study evaluated post-hoc compression methods without retraining the embedding models:

Dimensionality Reduction:
- Head: Retains the first $d$ coordinates of the original embedding.
- PCA+ROR: Applies Principal Component Analysis (PCA) followed by a deterministic random orthogonal rotation (ROR). This rotation redistributes variance across coordinates to prevent the imbalance that occurs when PCA-projected vectors (ordered by variance) are subjected to scalar quantization.
Quantization:
- Evaluated bit-widths $b \in \{1, 2, 4, 8, 16, 32\}$ .
- 1-bit: Sign quantization.
- 2, 4, 8-bit: Used a distribution-adaptive equal-count lookup-table quantizer. Instead of using fixed low-precision floating-point formats, the authors constructed quantization tables based on the empirical distribution of the calibration embeddings. This ensures that representable levels are allocated to match the density of the data, avoiding the "zero collapse" common in L2-normalized high-dimensional vectors where values are concentrated near zero.
- Calibration: Parameters for PCA and quantization tables were fitted on unlabeled calibration sets derived from the task inputs, without using task labels or relevance judgments.

3. Key Results

Synergy of Combined Methods

The experiments demonstrate that combining dimensionality reduction and quantization enables significantly stronger compression than either method alone.

Extreme Compression: In specific settings, embeddings were reduced to 0.1% of their original size (e.g., 1/1024) with negligible performance degradation.
Task Dependency: The optimal compression strategy varies significantly by task:
- Classification: Highly tolerant of aggressive compression. It prioritizes bit-width over dimensionality. High-bit settings (8–32 bits) maintained performance even at very low dimensions (e.g., 2 dimensions), whereas low-bit settings failed even at high dimensions.
- Clustering & STS: More dependent on dimensionality. Larger dimensions consistently yielded higher scores at the same compression budget.
- Retrieval: The most sensitive task. It required the largest relative compression budget to maintain performance and showed degradation earlier than other tasks when dimensions were reduced.

Dimensionality Reduction Methods

Head vs. PCA+ROR:
- Head-based truncation was more reliable for preserving near-original performance (99% threshold), successfully reaching this threshold for all model-task combinations.
- PCA+ROR was more efficient for aggressive compression (90% threshold), often achieving the target with a smaller budget, particularly for classification and clustering. However, it failed to reach the 99% threshold in several retrieval and STS settings, suggesting it can distort fine-grained similarity structures required for these tasks.
The Role of Rotation: The random orthogonal rotation (ROR) was critical. Without it (standard PCA), performance was non-monotonic; adding more PCA dimensions sometimes degraded performance due to the introduction of noisy, low-variance coordinates that were poorly handled by scalar quantization. ROR balanced coordinate-wise variance, stabilizing the compression.

Model Characteristics

Original Dimensionality: Models with higher original dimensions (e.g., 3584) could discard or quantize a larger portion of their representation while maintaining performance compared to models with smaller original dimensions (e.g., 1024).
Matryoshka Compatibility: The Qwen3-Embedding-8B, designed for variable-dimensional embeddings, did not exhibit fundamentally different behavior regarding Head-based truncation compared to other high-dimensional models, suggesting that robustness to truncation is a broader property of recent models. However, it showed some efficiency gains in specific retrieval settings.

4. Significance and Claims

The paper claims that the trade-off between bit-width and dimensionality has not been systematically characterized across task families, and this work addresses that gap.

Complementarity: The primary contribution is the demonstration that dimensionality reduction and quantization are complementary axes. By combining them, one can achieve compression ratios (down to 0.1%) that are impossible with single-axis methods without significant performance loss.
Task-Specific Strategies: The authors emphasize that there is no universal "best" compression strategy. Classification benefits from high precision (bit-width), while retrieval and clustering benefit from retaining geometric structure (dimensionality).
Quantization Design: The study highlights that standard fixed low-bit floating-point formats are suboptimal for L2-normalized text embeddings due to value concentration near zero. Distribution-adaptive quantization is necessary to preserve directional information in low-bit regimes.

5. Limitations

The authors acknowledge three main limitations:

Scope of Methods: The study focuses on Head truncation, PCA+ROR, and scalar quantization. It does not exhaustively cover learned projections, product quantization, or vector quantization.
Dataset Scope: Experiments were limited to English datasets within MTEB. Generalizability to other languages, domains, or task types remains unverified.
Calibration Protocol: The compression parameters were fitted using a transductive protocol (accessing the evaluation input distribution during a warm-up pass). This may overestimate performance compared to a deployment scenario where parameters must be fitted on a separate corpus and reused for unseen data.