💬 NLP

Closing the Calibration Gap in Semantic Caching

This paper argues that semantic caching model selection is fundamentally a calibration problem rather than a ranking one, demonstrating that standard metrics like PR-AUC lead to poor deployment choices and proposing new metrics (P-CHR AUC and CRR) to bridge the gap between offline evaluation and operational performance.

Original authors: Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a busy coffee shop. You have a "Semantic Cache," which is like a smart menu board that tries to guess what your customers want before they even order. If a customer asks, "How do I reset my password?" and the board sees that someone else just asked, "I forgot my login," it assumes they mean the same thing and instantly shows the answer from its memory. This saves you from having to call the expensive "Head Chef" (the Large Language Model) every single time.

The problem this paper solves is: How do you know if your smart menu board is actually good at its job?

The Old Way: The "Ranking" Trap

Previously, developers tested these smart boards using a metric called PR-AUC. Think of this like judging a chef based on how well they can rank ingredients.

The Test: "Can you put the best tomatoes above the worst tomatoes?"
The Flaw: The test only cares about the order. It doesn't care if the "best" tomato is actually rotten, or if the "worst" tomato is barely edible.
The Result: The paper found that models that were great at ranking (putting good answers above bad ones) were often terrible at deciding when to show an answer. They would confidently serve a wrong answer because their internal "score" was too high, even though the answer was wrong.

It's like hiring a sommelier who is amazing at saying, "This wine is better than that wine," but terrible at knowing when to pour a glass because they can't tell if the wine is actually drinkable at a specific price point.

The New Way: The "Threshold" Reality

In the real world, your smart menu board has a threshold (a line in the sand).

If the confidence score is above the line: Show the cached answer (Save money!).
If the score is below the line: Call the Head Chef (Spend money!).

The paper introduces two new ways to measure success that focus on this line:

P-CHR AUC (Precision–Cache Hit Ratio): This measures: "As we lower the line to show more answers, how many of them are actually correct?" It cares about the trade-off. You want to show many answers (high hit ratio) without showing wrong ones (high precision).
CRR (Calibration Retention Rate): This measures how much of the model's "offline ranking talent" survives when it actually has to make a real-world decision.

The Big Discovery: It's a "Calibration" Problem, Not a "Ranking" Problem

The authors ran a massive experiment with 9 different "searchers" (Retrievers) and 10 different "judges" (Rerankers). They found a shocking inversion:

The "Overconfident" Models (BCE): These models were the kings of the old "Ranking" test. They got the highest scores. But in the real world, they were disasters. They were like a student who memorized the textbook perfectly but fails the exam because they can't apply the knowledge to a specific question. They were "miscalibrated"—their scores were too compressed, making them think they were sure of things they weren't.
The "Underdog" Models (ColBERT): These models looked terrible on the old ranking test. But in the real world, they were the best. They knew exactly when to say "I'm not sure" and when to give an answer.

The Analogy:
Imagine two weather forecasters.

Forecaster A (The BCE model): Always says "It will rain." They are right 50% of the time (because it rains half the time), so they rank well if you just ask "Is rain more likely than sun?" But if you ask, "Should I bring an umbrella right now?", they are useless because they never say "No."
Forecaster B (The ColBERT model): Says "It might rain" or "It will definitely rain" based on the actual clouds. They might rank lower on a "who predicts rain best" test, but if you ask, "Should I bring an umbrella?", they are the one you trust.

The "Gap" Explained

The paper breaks down the difference between the "Offline Test" and "Real Life" into two parts:

The Structural Gap (The Unfixable Floor): This is just the nature of your data. If 45% of your customers ask the same question, there's a mathematical limit to how perfect your system can be. You can't fix this; it's just the rules of the game.
The Calibration Gap (The Fixable Mistake): This is the part caused by bad training. The paper found that how you train the model matters more than how much data you feed it.
- Training with a specific method (Binary Cross-Entropy) created a "calibration gap" that made the model useless in real life, even if you gave it 40 million examples.
- Training with a different method (MNRL) kept the model usable, even with less data.

The Takeaway for Practitioners

Stop using the old ruler: Don't pick your AI models based on PR-AUC (ranking). It will trick you into picking models that fail in production.
Use the new ruler: Pick models based on P-CHR AUC or CRR. These tell you if the model can actually make the "Go/No-Go" decision correctly.
Reranking isn't always free: The paper found that adding a second "judge" (a reranker) often made things worse because it introduced more calibration errors. Sometimes, the first "searcher" is actually the best choice.
Calibration is key: The problem isn't that the models can't find the right answer; it's that they can't assign the right confidence score to that answer.

In short: Don't just look for the model that ranks the answers best. Look for the model that knows when to stop guessing and start answering.

Technical Summary: Closing the Calibration Gap in Semantic Caching

Problem Statement

Semantic caching reduces Large Language Model (LLM) inference costs by serving cached responses for semantically similar queries. Standard practice evaluates the models powering these systems using Precision-Recall Area Under the Curve (PR-AUC). However, the authors identify a critical mismatch between this evaluation metric and the actual deployment objective.

In deployment, a semantic cache fires (returns a cached response) when a similarity score exceeds a fixed threshold $\tau$ . The operational goal is to maintain high precision (the fraction of correct cache hits) across varying levels of cache utilization (the fraction of queries served from the cache). PR-AUC measures relative ranking quality but is agnostic to score magnitudes; it does not indicate whether scores are calibrated to a specific threshold. The authors demonstrate that models with the highest PR-AUC often perform poorly in operation because their scores are miscalibrated, leading to "precision collapse" when the threshold is set to a practical utilization level.

Methodology

Dataset Curation

To address the lack of large-scale, diverse data for semantic caching, the authors curated the LangCache SentencePairs collection. This dataset aggregates sentence pairs from diverse sources (paraphrase detection, QA, adversarial examples, etc.) and includes three progressively scaled versions:

v1: ~1.05M pairs (high-quality baseline).
v2: ~8M pairs (augmented with synthetic paraphrases).
v3: ~40M pairs (incorporating large-scale back-translated and synthetic corpora).
All evaluations were conducted on the v3 test split (74,265 pairs, 45% positive rate).

Model Training

The study evaluated 9 bi-encoder retrievers and 10 rerankers (cross-encoders and multi-vector models).

Retrievers: Fine-tuned on LangCache SentencePairs v2 using an in-batch contrastive objective (ArcFace loss).
Rerankers: Fine-tuned on v1 and v3 datasets using two distinct training objectives:
1. Binary Cross-Entropy (BCE): Treats reranking as binary classification, minimizing cross-entropy against hard labels.
2. Multiple Negatives Ranking Loss (MNRL): A contrastive objective learning relative ordering rather than absolute probability boundaries.

Proposed Metrics

The authors introduce cache-aware metrics to replace or supplement PR-AUC:

Precision–Cache Hit Ratio AUC (P-CHR AUC): Measures the tradeoff between precision and cache utilization (CHR) as the threshold $\tau$ varies. It quantifies how well a model maintains precision as cache utilization grows.
Calibration Retention Rate (CRR): The ratio of P-CHR AUC to PR-AUC, measuring how much offline ranking quality survives at deployment.
Operational Gap Decomposition: The total gap between PR-AUC and P-CHR AUC ( $\Delta_{op}$ $Δ_{o p}$ ) is decomposed into:
- Structural Gap ( $\Delta_{str}$ ): An irreducible component fixed by the dataset's positive rate ( $p$ ). Even a perfect ranker cannot exceed a P-CHR AUC of $p(1 - \ln p)$ .
- Calibration Gap ( $\Delta_{cal}$ ): A recoverable component arising from score miscalibration.

Key Results

The Calibration Gap

Experiments across 9 retrievers and 10 rerankers revealed a systematic "calibration gap" where high PR-AUC models often have low P-CHR AUC.

Training Objective Dominance: The calibration gap is governed by the training objective, not data scale. Scaling training data from 1M to 40M pairs did not improve P-CHR AUC for either BCE or MNRL objectives.
BCE Failure: BCE-trained models achieved high PR-AUC (e.g., 0.816) but suffered severe precision collapse in deployment (P-CHR AUC ~0.199). Their scores were compressed near the decision boundary, causing false positives to fire at any useful threshold.
MNRL Improvement: MNRL-trained models performed significantly better than BCE in deployment (P-CHR AUC ~0.353 vs. 0.199) but still lagged behind the retriever baseline. They learned relative ordering but did not fully solve the threshold calibration problem.
The ColBERT Inversion: ColBERT-family models (multi-vector) presented a striking inversion. They had the lowest PR-AUC (~~0.515) but the highest P-CHR AUC (~~0.402) and the lowest operational gap. Their softmax-normalized scores over the candidate pool provided natural spread, making them the best deployment choice despite poor offline ranking metrics.

Post-Hoc Calibration

The authors tested post-hoc calibration methods (Temperature Scaling and Platt Scaling) on BCE models.

Partial Recovery: Temperature scaling partially recovered P-CHR AUC for BCE models (e.g., +0.032 gain), but the models remained significantly worse than MNRL counterparts.
Ineffectiveness for MNRL: Calibration methods provided negligible or negative gains for MNRL models, confirming they do not suffer from the same score compression issues.
Conclusion: Post-hoc calibration is a partial fix; the training objective is the primary determinant of deployment quality.

Reranking Overhead

The study found that reranking rarely improves deployment quality over a strong domain-specific retriever.

Most rerankers (including BCE and MNRL variants) lowered the P-CHR AUC compared to using the retriever alone.
Only ColBERTv2.0 improved upon the average retriever baseline, yet it still fell short of the best retriever (LangCache-Embed-v3) used without reranking.
Reranking adds latency (17–66 ms) without guaranteeing a gain in the metric that matters for production (precision at a fixed threshold).

Significance and Claims

The paper reframes semantic cache model selection as a calibration problem rather than a ranking problem. The authors claim that:

PR-AUC is Misleading: Relying on PR-AUC leads to systematic errors in model selection, favoring models that rank well but fail at fixed thresholds.
Metrics Matter: Measuring P-CHR AUC and CRR is the first step to closing the gap between offline promise and production behavior.
Objective Over Scale: The training objective (BCE vs. MNRL vs. Multi-vector) dictates score placement and deployment viability more than dataset scale.
Practical Guidance: Practitioners should prioritize cache-aware metrics, prefer objectives that naturally spread scores (like contrastive or multi-vector approaches), and critically evaluate whether reranking is necessary, as it often degrades performance for strong retrievers.

The authors release their datasets (LangCache SentencePairs) and models (LangCache-Embed-v3, LangCache Rerankers) to facilitate cache-aware evaluation, emphasizing that closing the calibration gap requires shifting focus from relative ranking to absolute score usability.