💬 NLP

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models

This paper argues that single-prompt accuracy benchmarks are insufficient for assessing language model reliability, demonstrating through a multi-variant audit that evaluation design choices, fragile confidence signals, and inconsistent prompt robustness can drastically alter conclusions and obscure critical failures in both models and evaluators.

Original authors: Ranit Karmakar, Jayita Chatterjee

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ranit Karmakar, Jayita Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of junior detectives (the AI models) to solve a series of puzzles. The standard way to judge them is to ask one question, see if they get the answer right, and give them a score. If they get 8 out of 10 right, they get an 80% grade.

This paper argues that this single score is a lie. It misses the most important parts of whether a detective is actually reliable in the real world. The authors ran a "multi-variant audit" on 15 different small AI detectives to show that how you ask the question, how you read the answer, and how you measure their confidence matters just as much as the detective's intelligence.

Here are the three big discoveries, explained with simple analogies:

1. The "First-Word" Trap (Evaluation Design Can Fake Failure)

The Analogy: Imagine you ask a detective to solve a mystery, but you tell them, "First, write a long story about how you thought about it, then write the answer on the very last line."
The detective writes a brilliant 3-page story, solves the case perfectly, and writes "The butler did it" at the end.
However, your grading system is a robot that only looks at the very first word of the response to decide if they are right. Since the first word was "The," the robot marks the answer as wrong.

The Paper's Finding:
The researchers found that when they asked AI models to use "Chain of Thought" (think step-by-step), the models actually got smarter. But because the standard grading system only looked at the first letter of the output, the models' scores crashed by 72% to 88%.

The Fix: They didn't need to upgrade the models. They just needed to change the "robot grader" to look for the answer at the end of the text instead of the beginning. Once they fixed the grader, the models' scores bounced back to nearly 100%.
The Lesson: A model can look terrible just because the person grading it is looking in the wrong place.

2. The "Confident Liar" (Confidence Signals are Fragile)

The Analogy: Imagine a student taking a hard test. They get the answer wrong, but when asked, "How sure are you?" they say, "I am 90% sure I got this right!"
In the real world, we need to know if a student is actually sure or just sounding sure.

The Paper's Finding:
On a very hard test (MMLU-Pro), the AI models were consistently overconfident liars.

They got the answers right only about 20–30% of the time.
But when asked to say how confident they were, they verbally claimed to be 60–78% sure.
Even worse, sometimes the models would ramble so much or use such weird phrasing that the computer couldn't even find their confidence number (a "parse failure"). It's like a student shouting their confidence in a language the teacher doesn't speak.
The Lesson: Just because an AI says "I'm sure!" doesn't mean it's right. And sometimes, you can't even tell if it's confident because it's talking in a way you can't understand.

3. Bigger Isn't Always Steadier (Size Doesn't Mean Robustness)

The Analogy: Imagine you have a small, sturdy wooden chair and a giant, fancy marble throne. You might assume the giant throne is more stable. But if you wiggle the legs of the marble throne, it might wobble and break, while the small wooden chair stays rock solid.

The Paper's Finding:
People often assume that a bigger AI model (with more "brain power" or parameters) is more stable and won't get confused if you change the wording of the question slightly.
The researchers tested this by asking the same questions in five different ways (changing the order of words, adding examples, changing the format).

The Result: There was no clear link between the size of the model and how well it handled these changes.
Some tiny models were incredibly steady (they got the same score no matter how the question was asked).
Some huge models were very fragile (their scores swung wildly depending on the wording).
The Lesson: You cannot judge how "robust" or reliable a model is just by looking at its size. A small model can be much more reliable than a big one.

The Bottom Line

The paper concludes that we need to stop treating AI reliability like a simple high score on a test.

If you want to know if an AI is ready for the real world, you can't just look at the final grade. You need to know:

How was the test graded? (Did the grader look at the first word or the last?)
Did the model actually understand the question? (Or did it just guess?)
Can we trust its confidence? (Is it a confident liar?)
Is it sturdy? (Does it break if you change the wording slightly?)

The authors argue that before we trust these models, we need to publish the entire recipe of how we tested them, not just the final number. Otherwise, we might be hiring a detective who looks great on paper but fails the moment the real work starts.

Technical Summary: What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models

Problem Statement

Single-prompt accuracy is the dominant metric for benchmarking language models (LMs), yet it provides an incomplete picture of reliability, particularly for small language models (SLMs) in the 1–8B parameter range intended for local or on-device deployment. A deployed model requires not only average correctness but also robust confidence signals, parseable outputs for downstream systems, and stability under prompt variations. Current evaluations often obscure critical failure modes by relying on a single accuracy number, fixed prompt templates, and opaque evaluation pipelines. This paper argues that reliability conclusions depend as much on the evaluation pipeline (metric definitions, evaluator logic, and prompt variants) as on the model itself.

Methodology

The authors conducted a multi-variant reliability audit of 15 open-weight SLMs (1B–8B parameters), with primary analyses focused on 10 instruct models. The evaluation covered five benchmarks: three classification tasks (SST-2, MNLI, AG News) and two reasoning tasks (ARC-Challenge, MMLU-Pro).

Experimental Design:

Prompt Variants: Each model-dataset pair was evaluated under five distinct prompt variants: surface_paraphrase, instruction_reorder, fewshot_3, format_change, and implicit_framing. These perturb phrasing, instruction order, in-context conditioning, output format, and framing while keeping the underlying task fixed.
Metrics: For every (model × dataset × variant) cell, the study recorded:
- Accuracy: Scored from the first generated character for multiple-choice tasks.
- Token-Probability Calibration: Measured via Expected Calibration Error (ECE) using both raw first-token probabilities and label-set-normalized probabilities.
- Verbal-Confidence Calibration: Measured via ECE of verbally reported confidence, collected via follow-up prompts.
- Verbal Parse Rate (VPR): The fraction of responses from which a numerical confidence can be extracted.
- Prompt-Perturbation Spread: The difference between maximum and minimum accuracy across non-format-change variants.
Infrastructure: All runs utilized vLLM on a single NVIDIA L4 GPU with bfloat16 precision, greedy decoding, and a fixed random seed (42).

Key Findings

1. Evaluation Choices Can Create or Hide Failures

The study identifies two evaluation-side failure modes that significantly alter reliability conclusions without changing the model:

Evaluator-Prompt Mismatch: On the ARC-Challenge benchmark, switching to a format_change variant that elicits chain-of-thought (CoT) reasoning caused a catastrophic drop in reported accuracy (72–88% relative drop) when scored by a standard "first-character" evaluator. The first token of CoT output is often a reasoning word (e.g., "The...", "Let...") rather than the answer label. Two independent repair procedures (regex re-parsing and constrained decoding) recovered 93.8% and 102.7% of the lost performance, respectively. This indicates the failure was in the evaluator logic, not the model's reasoning capability.
Calibration Definition Sensitivity: The definition of token-probability calibration significantly impacts ECE values. Switching from raw first-token probability to label-set-normalized probability (dividing by the sum of probabilities over valid labels) changed the per-cell ECE by a mean absolute 0.149. In extreme cases (e.g., Mistral-7B on SST-2 format_change), the shift was 0.825.

2. Confidence Signals Are Fragile

Verbal Overconfidence: On MMLU-Pro, all primary instruct models verbally reported confidence levels substantially higher than both their actual accuracy and their token-probability confidence. The gap between verbal ECE and token ECE was consistently positive (+0.286 to +0.582), indicating systematic overconfidence.
Parseability Collapse: Verbal confidence is not always extractable. For example, Mistral-7B-Instruct failed the 0.80 VPR threshold on the surface_paraphrase variant across all datasets (e.g., VPR = 0.002 on SST-2). This failure is variant-specific; the same model passed the threshold on other variants. Aggregating these cells into a single scalar would mask the fact that 20–25% of outputs were unparseable.

3. Parameter Count Is a Poor Proxy for Robustness

The study tested the hypothesis that larger models are more robust to prompt perturbations. Across 10 instruct models (1–8B), the correlation between parameter count and prompt-perturbation spread was weak and inconsistent (Spearman $\rho$ ranging from -0.244 to 0.474, all $p > 0.10$ ).

Non-Monotonicity: Smaller models were sometimes more robust than larger ones. For instance, Phi-4-mini (3.8B) had a SST-2 spread of 0.084, while Mistral-7B (7.2B) had a spread of 0.500.
Family Clustering: Robustness patterns appeared more structured within model families (e.g., both Qwen models were robust on SST-2) than across parameter counts, though the authors note this is descriptive given the limited sample size per family.

Contributions

The paper makes three primary contributions:

Identification of Evaluation-Side Failure Modes: It demonstrates that calibration-normalization sensitivity and prompt-evaluator mismatches can drastically alter reliability conclusions independently of model performance.
Joint Reporting of Confidence and Parseability: It introduces a framework for reporting verbal-confidence direction and verbal parse rate per cell, revealing that confidence can be both overconfident and selectively unparseable.
Empirical Evidence on Robustness: It provides data showing that parameter count is a poor proxy for prompt robustness in the 1–8B range, challenging the assumption that scaling guarantees stability.

Significance and Recommendations

The authors argue that reliability conclusions for SLMs are not intrinsic properties of the model but are contingent on the evaluation pipeline. Consequently, they propose a reporting checklist for SLM reliability audits to ensure reproducibility and deployment readiness:

Disclose ECE Normalization: Explicitly state whether token-probability ECE uses raw or label-set-normalized definitions.
Publish Evaluator Logic: Specify the exact rules used to score generated text (e.g., first-character match vs. regex).
Retain Full Generations: Store raw output text to allow re-scoring under alternative evaluators.
Report Verbal Parse Rate: Report VPR per cell with explicit inclusion thresholds, rather than as a model-level scalar.
Replicate Confidence Claims: Test verbal confidence under multiple elicitation phrasings (e.g., decimal vs. percentage).
Report Prompt-Perturbation Spread: Include spread metrics alongside accuracy when comparing robustness, rather than relying on parameter count.

The paper concludes that for SLM benchmarks to guide real-world model selection, they must describe the evaluation assumptions that make a conclusion possible, moving beyond a single leaderboard score to a comprehensive "deployment-readiness profile."