On the Representational Limits of Quantum-Inspired 1024-D Document Embeddings: An Experimental Evaluation Framework

This paper introduces an experimental framework for evaluating quantum-inspired 1024-dimensional document embeddings and finds that while they offer potential as auxiliary components in hybrid retrieval systems, they currently suffer from structural geometric limitations and instability that prevent them from outperforming strong lexical baselines like BM25 or standalone teacher embeddings.

The Gist

Imagine you are trying to find a specific needle in a massive haystack. In the world of computers, this "needle" is a piece of information you need, and the "haystack" is a giant library of documents.

For years, the best way to find needles has been using Lexical Search (like BM25). Think of this as a librarian who is incredibly good at matching the exact words you say with the words written on the spines of books. If you ask for "apple," they find books with "apple" on the cover. It's fast, reliable, and works great.

Then, we invented AI Embeddings (like the "Teacher" models). This is like a librarian who actually reads the books and understands the meaning. If you ask for "fruit," they know to find the book about "apples" even if you didn't say the word "apple." This is powerful but can be slow and expensive.

This paper is about a third, experimental librarian: The "Quantum-Inspired" Librarian.

The researchers wanted to see if a new type of librarian, inspired by the weird, complex math of quantum physics, could do a better job. They built a system that turns text into 1024-dimensional "maps" (embeddings) using quantum-like tricks.

Here is what they discovered, broken down simply:

1. The "Quantum" Librarian Got Lost

The researchers tested this new librarian on three types of libraries: Technical (manuals), Narrative (stories), and Legal (court documents).

• The Problem: The Quantum Librarian was terrible at ranking. Imagine you ask for "apple." The Quantum Librarian might hand you a book about "apple pie," a book about "Newton," and a book about "red paint," but in a completely random order. It couldn't tell which one was the best match.
• The "Distance Collapse": In a good map, similar things are close together, and different things are far apart. The Quantum Librarian's map was broken; it squashed everything into a tiny, crowded corner. Everything looked "somewhat similar" to everything else, making it impossible to pick the right one.

2. The "Teacher" Tried to Fix It (But Failed)

The researchers tried to teach the Quantum Librarian by having it copy a smart "Teacher" librarian (a standard AI). They hoped the Quantum Librarian would learn to organize the library better.

• The Result: It was a mixed bag. Sometimes the student got slightly better at matching words, but often, the act of trying to copy the teacher actually made the Quantum Librarian worse at its specific job. It's like trying to teach a fish to climb a tree; the fish might learn to jump, but it still can't climb. The "geometry" of the quantum map was just too broken to be fixed by simple copying.

3. The Hybrid Solution: The "Team Effort"

Since the Quantum Librarian couldn't work alone, the researchers tried a team approach. They paired the Quantum Librarian with the old-school "Word-Matching" librarian (BM25).

• The Result: This worked! When they combined the two, the system performed almost as well as the smart "Teacher" librarian.
• The Catch: The Quantum Librarian wasn't doing the heavy lifting. It was just providing a tiny, weird hint that helped the Word-Matching librarian occasionally find a needle it missed. The Word-Matching librarian was still the boss.

4. The "Zoom-In" Test

The researchers did a final test: instead of looking for a whole book, they tried to find a specific paragraph inside the book.

• The Result: The Quantum Librarian completely crashed. It couldn't find the right paragraph at all. This proved that while the system might vaguely understand the "vibe" of a whole document, it completely fails when you need to understand the fine details.

The Big Takeaway

The paper concludes that Quantum-Inspired embeddings are currently not ready to be the main librarian.

• They are too unstable.
• They squash distances so much that everything looks the same.
• They can't rank things in the right order.

However, they aren't useless. They can act as a sidekick in a hybrid team, adding a tiny bit of extra flavor to the search. But for now, the old-school word-matching and the standard AI teachers are still the champions of the library.

In short: The researchers built a fancy, quantum-powered compass. They found out that while the compass spins in interesting ways, it doesn't actually point North very well. It's a cool experiment, but it's not ready to replace the map.

Technical Summary

1. Problem Statement

The paper addresses the growing interest in Quantum-Inspired (QI) models for Information Retrieval (IR) and Retrieval-Augmented Generation (RAG). While Large Language Models (LLMs) dominate current embedding practices, they are computationally expensive and language-specific. Quantum-inspired approaches promise geometric richness (via Hilbert spaces, superposition, and interference) to better capture semantic ambiguity.

However, it remains an open question whether these theoretical geometric properties translate into robust, scalable document retrieval performance. The paper investigates whether a specific pipeline generating fixed 1024-dimensional quantum-inspired embeddings (QEMB) can effectively represent semantic similarity and support retrieval tasks compared to strong lexical (BM25) and dense (LLM-based) baselines.

2. Methodology

A. The QEMB Pipeline

The core of the study is an experimental pipeline designed to construct 1024-dimensional embeddings directly from text without end-to-end gradient training (initially).

• Architecture: Text is segmented into "sub-chunks," which are further divided into overlapping windows.
• Encoding:
  • Angle Projection: Token statistics are mapped to angle parameters. These angles are derived via EigAngle (semantic axes from SVD of term co-occurrence matrices) or lexical fallbacks.
  • Quantum-Inspired Transformation: The angles are processed through a parameterized circuit (simulated on the Aer backend or a Torch surrogate) using a hardware-efficient ansatz (rotations and entangling gates).
  • Feature Extraction: Expectation values of Pauli operators are computed to generate window-level feature vectors.
• Aggregation: Window features are resampled to a fixed number (16 windows) and concatenated to form a 1024-dimensional vector, followed by L2 normalization.
• Distillation: An optional step where a Multi-Layer Perceptron (MLP) or linear projection maps the raw QEMB to align with a "Teacher" model (intfloat/multilingual-e5-large) to improve semantic alignment.

B. Experimental Framework & Evaluation

The authors designed a diagnostic framework rather than a standard benchmark, focusing on controlled corpora:

• Corpora: Three domains in Italian and English: Technical (structured, low ambiguity), Narrative (high semantic variability), and Legal (formal, domain-specific).
• Queries: Synthetic queries (descriptive, questions, keywords) generated to test different retrieval regimes.
• Baselines:
  • BM25: Lexical baseline.
  • Teacher Embeddings: FAISS-indexed dense embeddings (1024-D).
  • QEMB: Raw and Distilled versions.
• Hybrid Fusion: Score-level interpolation controlled by a parameter $\alpha$ ($s = \alpha \cdot s_{embed} + (1-\alpha) \cdot s_{BM25}$).
  • $\alpha=0$: Pure BM25; $\alpha=1$: Pure Embedding.
  • Includes an $\alpha$-oracle (upper bound for score fusion) and Reciprocal Rank Fusion (RRF).
• Metrics: Hit@K, MRR, nDCG, MAP, and pairwise similarity correlations (Pearson/Spearman) against LLM-generated ground truth.

3. Key Contributions

1. Experimental Pipeline: A reproducible framework for constructing 1024-D quantum-inspired embeddings using window-based decomposition, EigAngle projections, and circuit-inspired transformations.
2. Diagnostic Tools: A suite of tools for hybrid retrieval analysis, including dynamic $\alpha$ gating, candidate union strategies, and an $\alpha$-oracle to define the theoretical upper bound of score fusion.
3. Geometric Analysis: A rigorous evaluation of the embedding space geometry, revealing that QEMB suffers from distance compression and pathological inversion of similarity structures (where unrelated items are ranked higher than related ones).
4. Empirical Findings: Comprehensive results showing that while QEMB fails as a standalone retriever, it can act as a complementary signal in hybrid systems, though distillation often degrades hybrid performance.

4. Key Results

A. Geometric Limitations (Pairwise Evaluation)

• Similarity Collapse: QEMB embeddings exhibit distance compression, assigning uniformly high similarity scores to almost all pairs, regardless of semantic relation.
• Negative Correlation: In baseline configurations, QEMB showed negative correlation (Pearson $\approx -0.25$) with reference similarity scores, indicating a complete breakdown of semantic ordering.
• Distillation Failure: While distillation improved alignment with the teacher model in some settings, it did not fully recover the semantic structure. In some cases, it worsened the ranking stability.

B. Retrieval Performance

• Standalone QEMB: Performed significantly worse than BM25 and Teacher embeddings across all domains.
  • Technical Corpus: Hit@1 was 0.30 (QEMB) vs. 0.90 (Teacher) and 0.80 (BM25).
  • Legal Corpus: Hit@1 was 0.11 (QEMB) vs. 1.00 (BM25).
• Hybrid Retrieval:
  • Combining QEMB with BM25 (Hybrid) often recovered competitive performance, suggesting QEMB provides complementary signals to lexical matching.
  • Distillation Paradox: In hybrid settings, using distilled QEMB often performed worse than using raw QEMB. This suggests distillation smooths out the specific (albeit weak) noise patterns that were useful for hybrid fusion.
• Granularity: Performance degraded drastically at the sub-chunk level. While document-level retrieval showed some signal, sub-chunk retrieval for QEMB collapsed (Hit@1 $\approx$ 0.00), indicating an inability to capture local semantic relevance.

C. Domain Variations

• Legal Domain: BM25 achieved perfect scores (1.00), highlighting that for highly structured, terminology-heavy text, lexical matching is superior.
• Narrative Domain: All dense models (including Teacher) struggled more than in technical domains, but QEMB remained the weakest performer.

5. Significance and Conclusion

• Structural Limits: The paper concludes that current quantum-inspired embeddings, as constructed via deterministic window-based pipelines, suffer from intrinsic structural limitations. They fail to preserve the local neighborhood structure required for effective ranking, leading to "pathological inversions" of similarity.
• Role in RAG: QEMB embeddings are insufficient as standalone retrieval representations. Their primary utility lies in acting as auxiliary signals in hybrid pipelines, where they can complement lexical matching (BM25).
• Distillation Warning: The study warns that simply aligning quantum-inspired embeddings with a teacher model via distillation does not guarantee better retrieval performance and may even distort the specific features that make hybrid fusion effective.
• Future Directions: The authors suggest that future work must move beyond static, deterministic mappings toward adaptive, learned transformations and ranking-aware objectives to prevent similarity inversion and preserve geometric structure.

In summary, this work provides a critical, diagnostic perspective on quantum-inspired IR, demonstrating that while the theoretical geometric richness is appealing, the current practical implementations fail to deliver robust retrieval performance without significant architectural and objective function revisions.