Technical Summary: One Prompt is Not Enough: Instruction Sensitivity Undermines Embedding Model Evaluation

Problem Statement

Instruction-tuned embedding models have become standard in state-of-the-art NLP applications, including Retrieval-Augmented Generation (RAG), semantic search, and agentic memory. However, current evaluation practices, primarily within frameworks like BEIR and MTEB (Massive Text Embedding Benchmark), rely on a single-point evaluation using a single prompt per task.

This paper identifies a critical flaw in this methodology: instruction-based models are inherently sensitive to the phrasing of the instruction (prompt). By evaluating models on only one prompt, benchmarks fail to capture the distribution of performance across plausible instructions. This single-point approach risks misrepresenting actual model capabilities, potentially leading to prompt deflation (underestimating performance) or prompt inflation (overestimating performance). Furthermore, the current system allows for "prompt hacking," where developers can selectively choose prompts that maximize their model's score, distorting leaderboard rankings without altering model weights or training data.

Methodology

The authors conducted an empirical study to quantify the impact of prompt variation on embedding model performance.

• Scope: The study evaluated 6 open-weight embedding models (including Qwen3-Embedding-0.6B, multilingual-e5-large-instruct, KaLM-embedding-multilingual-mini-instruct-v2.5, and three BGE variants: small, base, and large) across 11 diverse tasks. These tasks spanned retrieval, classification, clustering, and semantic textual similarity (STS).
• Prompt Generation: For each of the 11 tasks, the authors synthesized 15 task-specific prompts using gpt-oss-120b. These prompts were designed to resemble real-world practitioner usage, varying in phrasing while remaining coherent and relevant. The authors validated that synthetic prompts produced score distributions comparable to human-written prompts.
• Evaluation: The models were run on all 11 datasets using all 15 prompts per task, resulting in a total of 990 evaluations (6 models × 11 tasks × 15 prompts).
• Analysis: The authors analyzed:
  1. The distribution of scores across prompts for each model-task pair.
  2. The Coefficient of Variation (CV) to measure sensitivity.
  3. The position of the "default" MTEB prompt relative to the full distribution.
  4. The stability of leaderboard rankings under adversarial prompt selection scenarios.

Key Results

1. Prompt Deflation and Inflation

The study found that reported MTEB scores frequently misrepresent expected performance:

• Prompt Deflation: For several models (e.g., BGE variants on TweetSentimentClassification and STS22.v2), the default prompt yielded scores in the lower tail of the distribution, significantly underestimating the model's potential.
• Prompt Inflation: Conversely, some models (e.g., Qwen3 on MIRACL) achieved reported scores in the upper tail or secondary modes of the distribution, artificially inflating their perceived quality.

2. High Sensitivity (RQ1)

Models exhibit significant sensitivity to prompt variations. While median Coefficients of Variation (CV) were moderate (2–6%), specific model-task combinations showed extreme variance, with individual task CVs reaching 30–45%. This indicates that performance is not robust to prompt choice for many configurations.

3. Leaderboard Instability (RQ2)

The study demonstrated that leaderboard rankings are not robust to prompt selection.

• Adversarial Selection: By assigning a target model its best-performing prompt while assigning competitors their worst (or default) prompts, any model in the study could be promoted to rank 1. For instance, bge-small-en-v1.5, which ranked last under default conditions, could be made the top performer.
• Non-Adversarial Optimization: Even without malicious intent, a single developer optimizing their prompt against default competitor settings could shift rankings significantly (e.g., bge-large moving from rank 5 to rank 3).

4. Prevalence of Prompt Hacking (RQ3)

The analysis of reported scores against observed distributions revealed a systematic pattern where most models report performance well above their median expected score. The authors term this "prompt hacking": the selective reporting of a single favorable prompt that remains compliant with submission policies but exploits undisclosed degrees of freedom. This phenomenon is analogous to p-hacking in statistics.

Significance and Recommendations

The paper argues that single-prompt evaluation is insufficient for instruction-tuned embedding models. The current benchmarking paradigm fails to distinguish between a model that is genuinely robust and one that is merely lucky with a specific prompt.

Key Claims:

• Reported benchmark scores often misrepresent the distribution of performance over plausible prompts.
• Leaderboard rankings are fragile and can be manipulated through prompt selection, even without modifying the model itself.
• The phenomenon of "prompt hacking" allows models to be oversold or undersold based on the specific prompt chosen for evaluation.

Recommendations:
The authors propose that benchmarks must transition from single-point estimates to multi-prompt evaluation. This could be achieved by:

1. Evaluating models over multiple prompts and reporting score distributions or robustness metrics alongside point estimates.
2. Encouraging model developers to optimize for prompt robustness rather than peak performance on a single prompt.
3. Adopting limited, standardized prompt schemes (similar to the Jina v5 series) to reduce variability.

Limitations

The authors acknowledge several limitations:

• Model Scope: The study focused on 6 open-weight models up to 0.6B parameters; larger or closed-source models may exhibit different sensitivity profiles.
• Sample Size: The use of 15 synthetically generated prompts per task, while validated against human prompts, may not capture the full diversity of real-world prompt engineering.
• Language: The analysis was restricted to English; multilingual settings may introduce additional sensitivity due to translation artifacts and language-specific phrasing.
• Practitioner Context: While framed as a threat to benchmark validity, the authors note that prompt sensitivity can be an asset in practice, allowing developers to gain performance improvements with minimal cost through prompt search.