HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions

Technical Summary of HAKARI-Bench

Problem Statement
The rapid adoption of Retrieval-Augmented Generation (RAG) and semantic search has made the selection of text embedding models and retrieval configurations critical. However, existing large-scale benchmarks (e.g., MTEB, MMTEB, BEIR) are comprehensive but computationally heavy, making repeated evaluation during development impractical. Furthermore, there is a lack of infrastructure to compare production-time efficiency settings—such as dimensionality reduction, quantization (int8/binary), and reranking—across many models under identical conditions. While these efficiency settings are crucial for balancing retrieval quality against compute, memory, and latency, they are rarely reported consistently per model, and cross-model comparisons under the same conditions are scarce.

Methodology
The paper introduces HAKARI-Bench, a lightweight evaluation infrastructure designed to reconstruct existing retrieval benchmarks into small, manageable datasets ("Nano-sets") while maintaining a unified format.

• Nano-set Construction: The benchmark constructs 551 retrieval tasks across 35 benchmarks and 43 languages. Each task is a "Nano-set" shrunk from original benchmarks to approximately 50–200 queries and 1,000–10,000 documents. This shrinking follows principles from MMTEB (correlation-based downsampling) and NanoBEIR, ensuring the datasets remain representative while reducing evaluation costs.
• Unified Task Format: All tasks adhere to a common format comprising a corpus, queries, relevance labels (qrels), and a fixed top candidate set. The candidate set is a hybrid of the top 500 BM25 results and top 500 dense retrieval results (using a fixed reference model), fused via Reciprocal Rank Fusion (RRF) to create a top-100 set. A safeguard rule ensures every query has at least one relevant document in the candidate set.
• Evaluation Pipeline: The infrastructure evaluates five retrieval families under the same conditions:
  1. Candidate Generation: BM25, dense, sparse, and late interaction models retrieving from the whole corpus.
  2. Reranking: Models re-ordering the fixed candidate set (including cross-encoders and LLM-style rerankers).
  3. Efficiency Variants: Post-encoding transformations applied to dense embeddings, including Matryoshka dimensionality reduction, int8 quantization, binary quantization, and float rescoring. Sparse representations are also evaluated via query-side and document-side pruning.
• Metrics and Aggregation: The primary metric is nDCG@10. Results are aggregated using both micro-averages (equal weight per task) and macro-averages (equal weight per benchmark) to prevent dominance by large benchmark families. The system supports multi-axis filtering by language, domain, and model architecture.

Key Contributions

1. Lightweight Multilingual Infrastructure: HAKARI-Bench provides a unified framework comparing five retrieval architectures (lexical, sparse, dense, late interaction, reranker) across 35 benchmarks and 551 tasks in 43 languages, enabling rapid, repeated measurement.
2. Empirical Validation of Ranking Reproducibility: The authors demonstrate that the Nano-sets act as a high-fidelity ranking proxy. When compared against official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full), the Spearman rank correlations exceed 0.97 (0.983, 0.975, and 0.973, respectively). This confirms that the lightweight evaluation preserves the relative ordering of models found in full-scale evaluations.
3. Cross-Model Efficiency Evaluation: Unlike previous benchmarks where efficiency settings are sporadically reported, HAKARI-Bench systematically applies dimensionality reduction, quantization, and reranking to all supporting models under identical conditions. This allows for the concrete visualization of the quality-efficiency Pareto frontier and reveals that robustness to binary quantization is determined by training characteristics rather than model size alone.

Results

• Ranking Fidelity: The benchmark successfully reproduces the model rankings of large-scale official benchmarks with high correlation, validating its use for model selection and regression detection.
• Efficiency Trade-offs:
  • Quantization: Binary quantization alone causes a significant average drop in performance (−6.50 points), while int8 is milder (−1.95 points). However, applying float rescoring to the top candidates retrieved by quantized vectors nearly restores performance (int8_rescore: −0.09; binary_rescore: −0.93).
  • Sparse Pruning: Aggressive pruning of document-side dimensions has minimal impact on quality, whereas reducing query-side dimensions below 16 significantly degrades performance.
  • Reranking: The effectiveness of rerankers is highly dependent on the task scope. While LLM-style rerankers (e.g., Qwen3-Reranker) outperform dense models across all 551 tasks, traditional cross-encoders excel primarily in short-query, short-document scenarios and struggle with long documents or complex reasoning tasks.
• Domain Specificity: The "best" model varies significantly by domain. For instance, while a specific dense model ranks first overall, a Japanese-specialized model or BM25 may rank first in specific language or long-document scopes, highlighting that a single overall score is insufficient for production selection.

Significance and Claims
The paper positions HAKARI-Bench not as a replacement for full-scale evaluation, but as a high-fidelity ranking proxy that supports rapid model selection, regression detection, and the analysis of quality-efficiency trade-offs. Its primary significance lies in:

• Democratizing Efficiency Comparison: Providing the first systematic, cross-model comparison of efficiency settings (quantization, dimensionality reduction) under identical conditions.
• Enabling Iterative Development: Allowing developers to run evaluations repeatedly during the development cycle without the prohibitive cost of full benchmarks.
• Contextual Model Selection: Shifting the focus from maximizing a single aggregate score to identifying "which domains a model has not learned," thereby facilitating use-appropriate model selection based on specific constraints like language, latency, and document length.

The authors emphasize that while Nano-sets reproduce model rankings with high fidelity, they do not guarantee absolute score agreement with full benchmarks due to the reduced retrieval space and lack of hard negatives. Therefore, the benchmark is intended for relative ranking and setting comparison rather than absolute performance certification.