📊 statistics

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

This paper introduces Sem-ECE, a novel framework that evaluates calibration in open-ended question answering by sampling model outputs and grouping them into semantic classes to provide an unbiased, robust metric that outperforms existing verbalized and sampling-based methods, particularly when internal model probabilities are unavailable.

Original authors: Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang, Bojian Hou, Li Shen

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang, Bojian Hou, Li Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of experts to answer difficult trivia questions. You want to know not just what they answer, but how sure they are about their answers. If an expert says, "I'm 90% sure the capital is Paris," you want to know if they are actually right 90% of the time. This alignment between confidence and accuracy is called calibration.

In the world of AI (Large Language Models), this is tricky. When a model gives a short, multiple-choice answer, we can easily check its math. But when it writes a long, open-ended essay or a creative story, checking its confidence is like trying to measure the temperature of a cloud.

This paper introduces a new way to measure that "temperature" called Sem-ECE. Here is how it works, broken down into simple analogies.

The Problem: The "Overconfident Student"

Current ways to check AI confidence are flawed:

Asking the AI directly: If you ask an AI, "How confident are you?" it often lies or guesses, usually saying it's more sure than it actually is. It's like a student who raises their hand and shouts, "I'm 100% sure!" even when they are just guessing.
Looking at the code (Logits): Sometimes, we can peek inside the AI's brain to see its internal math. But most commercial AI services (like the ones used by doctors or lawyers) don't let us see this. It's like trying to judge a magician's trick by looking at their hands, but they are wearing gloves.
Repeating the question: If you ask an AI the same question 50 times, it might give 50 slightly different answers. If 49 of them say "Paris" and 1 says "London," we might guess it's pretty sure about Paris. But existing methods for doing this are messy and rely on rigid rules that break when the AI uses different words to say the same thing.

The Solution: The "Sem-ECE" Framework

The authors propose a new method called Sem-ECE (Semantic-Sampling Expected Calibration Error). Think of it as a Taste-Test Panel.

Instead of asking the AI once, you ask it 50 times.

The Sampling: You get 50 different answers.
The Grouping (Semantic Clustering): You don't just count exact word matches. You use a smart judge (another AI) to group answers that mean the same thing.
- Example: "The capital is Paris," "It's Paris," and "Paris, France" all get put into the same "Paris" bucket.
The Frequency: If 40 out of 50 answers end up in the "Paris" bucket, the AI's confidence is 80%.

The Two New Estimators: "Same-Sample" vs. "Held-Out"

The paper introduces two specific ways to calculate this confidence, comparing them to two different ways of grading a test.

1. Sem1-ECE: The "Same-Sample" Score (The Biased Judge)

This method picks the most popular answer from the 50 samples and uses the same 50 samples to calculate the confidence.

The Flaw: This is like a teacher grading a test using the same students who took the test to determine the passing score. Because the teacher picked the "winner" from that specific group, they are likely to overestimate how good that winner is. In statistics, this is called the "Winner's Curse." The AI looks more confident than it really is because it's judging itself on the very data it used to pick the answer.

2. Sem2-ECE: The "Held-Out" Score (The Fair Judge)

This method splits the 50 samples into two groups:

Group A (25 samples): Used to pick the "winning" answer.
Group B (25 samples): Used only to count how often that winning answer appears.
The Advantage: This is like a teacher picking the best essay from the first half of the class, then grading that specific essay using the second half of the class as a reference. Because Group B didn't help pick the winner, the confidence score is fairer and more accurate. It avoids the "Winner's Curse."

The Big Discovery: Easy vs. Hard Questions

The paper proves mathematically that:

On Easy Questions: Both methods agree. The AI is so sure that the "Winner's Curse" doesn't matter much.
On Hard Questions: The methods diverge. The "Same-Sample" method (Sem1) stays overly optimistic, while the "Held-Out" method (Sem2) correctly lowers the confidence score.
The Gap as a Diagnostic: The difference between the two scores acts as a difficulty meter. If the two scores are far apart, the question is hard and the AI is struggling. If they are close, the question is easy.

The Results: What They Found

The authors tested this on five major AI models (like GPT-4, Claude, Gemini) across three difficult question sets (ranging from simple facts to expert-level science).

Sem2-ECE won: In almost every case, the "Held-Out" method (Sem2) gave a more accurate picture of the AI's true reliability than asking the AI directly or using the "Same-Sample" method.
It works without "gloves": This method works even if you can't see the AI's internal math (logits). You just need to ask it questions and read the answers.
It catches overconfidence: The study showed that when models are wrong, they often think they are right. Sem-ECE exposes this by showing that the AI's "confidence" (how often it repeats an answer) doesn't match its actual "accuracy" (how often that answer is correct).

Summary

Imagine you are a pilot flying a plane. You need to know if your navigation system is trustworthy.

Old way: Ask the system, "Are you sure?" (It says "Yes!" but might be wrong).
New way (Sem-ECE): Ask the system to plot the course 50 times. Group the similar paths. If 40 paths go left and 10 go right, you know it's 80% sure. But to be really sure, you check those 40 paths against a fresh set of 25 new simulations (Sem2) to make sure the system isn't just fooling itself.

This paper provides the toolkit to do exactly that for AI, ensuring that when a model says it's confident, it actually is.

Technical Summary: A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

1. Problem Statement

Calibration—the alignment between a model's predicted confidence and its empirical accuracy—is a prerequisite for the reliable deployment of Large Language Models (LLMs) in high-stakes domains like medicine and law. While recent work focuses on improving calibration, the question of how to evaluate it in realistic, open-ended settings remains underdeveloped.

Existing evaluation methods fail in open-ended Question Answering (QA) due to three primary limitations:

Logit-based metrics: Require restricted output formats and access to internal probabilities (logits), which are frequently unavailable in commercial APIs.
Verbalized confidence: Relies on the model self-reporting its uncertainty. This approach is format-agnostic but often suffers from overconfidence and depends on the model's ability to accurately introspect.
Sampling-based methods: Derive confidence from the consistency of repeated generations. However, these typically rely on task-specific extraction rules and heuristic frequency scores without a clear finite-sample statistical target or rigorous theoretical guarantees.

The core challenge is that open-ended QA involves an unbounded answer space where correctness is semantic rather than lexical, and two answers worded differently can be equally correct.

2. Methodology: The Sem-ECE Framework

The authors introduce Sem-ECE (Semantic-Sampling Expected Calibration Error), a framework that evaluates calibration by sampling answers, grouping them into semantic classes, and using the resulting frequencies as confidence estimates.

Core Components

Semantic Answer Space: Instead of treating raw strings as distinct answers, the framework uses an LLM judge to cluster free-form generations into semantic equivalence classes ( $Z_q$ ).
Oracle Confidence: Defined as the population probability of the most likely semantic answer class ( $c^*_q = \max_k \pi_{q,k}$ ).
Two Estimators: The framework studies two estimators for this target, both asymptotically unbiased but with distinct finite-sample properties:
1. Sem1-ECE (Same-Sample): The standard self-consistency score. It selects the most frequent semantic answer from a block of $n$ samples and uses that same frequency as the confidence estimate. This estimator suffers from the "winner's curse" (Jensen's inequality bias), where the empirical maximum is biased upward.
2. Sem2-ECE (Held-Out): A variant that decouples selection from evaluation. It selects the most frequent answer from a selection block ( $N$ ) of size $n$ but calculates the confidence based on the frequency of that specific answer in a disjoint held-out block ( $E$ ) of size $m$ . This eliminates the upward Jensen bias, though it introduces a downward selection bias.

Theoretical Analysis

The paper provides a rigorous statistical analysis of these estimators:

Asymptotic Unbiasedness: Both Sem1 and Sem2 are proven to be asymptotically unbiased as sample sizes $n, m \to \infty$ .
Regime Analysis: The behavior of the estimators depends on the "standardized margin" ( $\tilde{m}_q$ $\tilde{m}_{q}$ ), which measures the gap between the top two semantic classes relative to the variance.
- Large-Margin Regime: On easy questions (large margin), both estimators converge rapidly and are indistinguishable.
- Low-Margin Regime: On hard questions (small margin), the estimators diverge. Sem1 overshoots the true confidence (positive bias), while Sem2 undershoots (negative bias).
Key Theoretical Result: In the low-margin regime, Sem2-ECE achieves a strictly smaller calibration error than Sem1-ECE. Furthermore, the gap between Sem1 and Sem2 serves as a diagnostic for question difficulty; a large gap indicates a hard, low-margin question.

3. Experimental Results

The framework was evaluated on three open-ended QA benchmarks (SimpleQA, Humanity's Last Exam (HLE), and PopQA) across five leading commercial LLMs (OpenAI, Anthropic, Google, xAI, Mistral).

Convergence: Empirical results confirm the theoretical predictions. As sample size increases, Sem1-ECE and Sem2-ECE converge to a common limit from opposite sides (Sem1 from above, Sem2 from below).
Performance: Sem2-ECE consistently outperforms verbalized confidence and Sem1-ECE. Across 15 model-benchmark pairs, Sem2-ECE achieved the lowest calibration error in 12 pairs.
Regime Structure: Stratification by question margin shows that the performance gap between Sem1 and Sem2 is largest in the low-margin (hard) regime, matching the theoretical "Jensen-dominated regime" (JDR).
Comparison to Verbalized Confidence: While verbalized confidence occasionally outperforms Sem-ECE in specific cases (e.g., Anthropic on SimpleQA), this occurs when models exhibit high semantic agreement but low factual accuracy. Sem-ECE provides an external, self-report-independent reference that detects such miscalibration.

4. Key Contributions

Framework: Introduced Sem-ECE, a calibration evaluation framework for open-ended QA that requires no logits, multiple-choice options, or hand-crafted extraction rules.
Estimators: Defined and analyzed two estimators (Sem1 and Sem2), proving their asymptotic unbiasedness and characterizing their bias behavior in low-margin regimes.
Theoretical Insight: Demonstrated that separating answer selection from confidence evaluation (Sem2) yields strictly smaller calibration error on hard questions compared to same-sample self-consistency (Sem1).
Diagnostic Tool: Showed that the gap between Sem1 and Sem2 serves as a simple, observable diagnostic for question difficulty.
Empirical Validation: Validated theoretical predictions across diverse models and benchmarks, showing Sem-ECE outperforms existing black-box methods.

5. Significance and Claims

The paper claims that Sem-ECE places sampling-based calibration evaluation on a "principled statistical footing" by replacing heuristic frequency scores with estimators that have explicit population targets and provable guarantees.

Complementarity: The authors position Sem-ECE not as a replacement for verbalized confidence but as a complement. It provides an external audit of model behavior that does not rely on the model's self-reporting, which is crucial when internal probabilities are unavailable.
Practical Utility: By identifying that Sem2-ECE is strictly superior on hard questions, the framework offers a more reliable metric for assessing model trustworthiness in high-stakes, difficult scenarios.
Diagnostic Value: The divergence between the two estimators provides a new signal for identifying questions where the model is uncertain or where semantic agreement does not align with factual correctness.

The work concludes that while improving calibration remains a critical area of research, Sem-ECE provides a necessary, statistically rigorous tool for evaluating it in the dominant deployment setting of modern LLMs: open-ended generation.