💬 NLP

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

This paper demonstrates that asking Large Language Models to self-report their confidence is a more cost-effective and well-calibrated method than self-consistency voting for identifying reliable automated grading predictions, enabling effective selective automation where high-confidence outputs are processed automatically while uncertain cases are flagged for human review.

Original authors: Robinson Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Robinson Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very smart, but occasionally overconfident, robot tutor to grade your students' essays. The robot is fast and cheap, but sometimes it gets things wrong. The big question isn't just "How do we make the robot smarter?" but rather, "When can we actually trust the robot, and when should we call a human teacher to double-check?"

This paper is like a guidebook for figuring out exactly that. The researchers tested three different ways to ask the robot, "How sure are you about this grade?" to see which method gives the most honest answer.

Here is the breakdown using simple analogies:

The Three Ways to Ask "Are You Sure?"

The researchers tried three different "confidence checks" on the robot:

The "Self-Reported" Method (Asking the Robot):
- How it works: You simply ask the robot, "On a scale of 1 to 10, how confident are you that this answer is correct?" and it gives you a number.
- The Analogy: It's like asking a student, "How sure are you about this math problem?" and taking their word for it.
- The Result: This was the winner. Surprisingly, just asking the robot to state its confidence level was the most accurate way to predict if it was right or wrong. It was honest, cheap, and fast.
The "Self-Consistency" Method (Asking the Robot to Think Twice):
- How it works: You ask the robot to solve the same problem five different times. If it gives the same answer four or five times, you assume it's confident. If it changes its mind, it's unsure.
- The Analogy: It's like asking a student to solve the same math problem five times. If they get the same answer every time, you trust them. If they keep changing their answer, you know they are confused.
- The Result: This method was expensive and slow (it took 5 times longer to run) but didn't actually give better results than just asking the robot once. It was like paying for a double-check that didn't really help.
The "Token Probability" Method (Reading the Robot's Mind):
- How it works: This looks at the internal math the robot uses to decide between "Yes" and "No" before it even speaks. It calculates the odds based on its internal code.
- The Analogy: It's like trying to guess how sure a student is by measuring how fast their heart beats or how sweaty their palms are, without them saying a word.
- The Result: This was okay, but not as good as just asking the robot directly.

The Big Surprises

1. Bigger isn't always "more honest."
The researchers tested robots of different sizes (from small 4-billion-parameter models to huge 120-billion-parameter ones).

The Good News: The bigger robots were definitely smarter (they got more answers right).
The Bad News: Being bigger didn't automatically make them honest about their confidence. A huge robot could still be confidently wrong. However, the biggest robot tested (GPT-OSS-120B) did the best job of both being smart and knowing when it was right.

2. The "Confidence Floor" (The Overconfidence Trap)
The researchers noticed something weird: The robots were almost always very confident.

The Analogy: Imagine a weather forecaster who says "It will rain" 90% of the time, even when the sky is clear. They rarely say, "I'm only 40% sure."
The Problem: Because the robots are so "top-skewed" (they always give high confidence scores), you can't just set a rule like "Trust anything above 50%." That rule would fail because the robot is rarely below 50%. You have to calibrate your trust based on the specific robot's habits.

The Practical Takeaway: "Selective Automation"

So, how do we use this in the real world?

Think of it like a security checkpoint at an airport.

High Confidence: If the robot says, "I'm 95% sure this student passed," and our tests show the robot is usually right at that level, we let the grade go through automatically.
Low/Uncertain Confidence: If the robot says, "I'm only 60% sure," or if the robot is known to be overconfident on tricky questions, we flag it. We don't throw the grade away; we just send it to a human teacher for a quick look.

The Bottom Line

You don't need to build a perfect robot to automate grading. You just need a robot that can tell you when it's unsure.

The paper concludes that the simplest approach works best: Just ask the robot how sure it is. If you set your "trust threshold" correctly based on how that specific robot behaves, you can automate the easy grades and save human teachers for the tricky ones, making the whole system faster and fairer.

1. Problem Statement

Large Language Models (LLMs) show promise for automated grading but suffer from reliability issues, including arithmetic errors, inconsistent scoring, and overconfidence. Directly improving grading accuracy is difficult; therefore, the authors address a complementary problem: predicting when an LLM grader is likely to be correct.

The goal is to enable selective automation: a workflow where high-confidence predictions are processed automatically, while low-confidence (uncertain) cases are flagged for human review. The core challenge is confidence calibration—ensuring that a reported confidence score (e.g., 0.8) accurately reflects the actual probability of correctness (80% accuracy). Current LLMs often exhibit systematic overconfidence, making raw confidence scores unreliable for decision-making.

2. Methodology

The authors systematically evaluate three distinct methods for estimating confidence in binary grading tasks (determining if a student response satisfies a rubric item).

A. Confidence Estimation Methods

Token Probability:
- Mechanism: Aggregates the probability mass of "yes" and "no" tokens from the model's output distribution.
- Cost: 1 LLM call.
- Logic: Uses the internal softmax probabilities of the verdict tokens.
Self-Reported Confidence:
- Mechanism: Prompts the model to explicitly output a numeric confidence score (1–10) alongside its grading decision.
- Cost: 1 LLM call.
- Logic: Relies on the model's metacognitive ability to verbalize uncertainty.
Self-Consistency:
- Mechanism: Samples $n$ outputs (where $n=5$ ) from the model at a non-zero temperature and calculates the fraction of the majority vote.
- Cost: $n$ LLM calls (5x inference cost).
- Logic: Assumes that higher agreement across stochastic samples indicates higher confidence.

B. Experimental Setup

Datasets: Three educational datasets covering different domains and answer lengths:
- RiceChem: Long-answer chemistry (8,392 responses).
- SciEntsBank: Short-answer science (10,804 responses).
- Beetle: Short-answer electronics/circuits (3,559 responses).
Models: Seven LLMs spanning two orders of magnitude in scale (4B to 120B parameters):
- Small: Llama-3.1-8B, Qwen3-4B, GPT-OSS-20B.
- Large: GPT-OSS-120B, Llama-3.3-70B, Qwen3-80B (both "Thinking" and "Instruct" variants).
Evaluation Metrics:
- Expected Calibration Error (ECE): Measures the average difference between confidence and accuracy across bins (lower is better).
- Brier Score: Mean squared error between confidence and binary correctness.
- AUC-ROC: Measures ranking quality (does higher confidence correlate with higher accuracy?).
- Coverage at Risk: The fraction of predictions that can be automated while maintaining accuracy above a target threshold.
Statistical Rigor: Used student-block bootstrapping (1,000 draws) to preserve within-student dependencies and paired comparisons to test statistical significance.

3. Key Contributions & Results

Finding 1: Self-Reported Confidence is Superior

Contrary to the intuition that sampling-based methods (Self-Consistency) are more robust, Self-Reported Confidence consistently achieved the best calibration across all conditions.

Performance: Self-reported achieved an average ECE of 0.166, compared to 0.229 for Self-Consistency and 0.235 for Token Probability.
Efficiency: Self-Consistency was 38% worse in calibration despite requiring 5x the inference cost.
Granularity: Self-Consistency suffers from discrete confidence levels (e.g., 3/5, 4/5, 5/5), whereas Self-Reported provides continuous scores, allowing for finer threshold tuning.

Finding 2: Model Scale Boosts Accuracy More Than Calibration

Accuracy: Larger models (≥70B) were consistently more accurate across all datasets.
Calibration: Calibration gains from scaling were mixed and dataset-dependent.
- SciEntsBank showed the largest calibration improvement with scale (ECE dropped from 0.208 to 0.121 for self-reported).
- RiceChem showed minimal improvement.
- Token Probability actually worsened in calibration as model size increased for some datasets.
Best Performer: GPT-OSS-120B with self-reported confidence achieved the best overall calibration (Avg ECE: 0.100) and strong discrimination (Avg AUC: 0.668).

Finding 3: The "Confidence Floor" (Top-Skewed Distributions)

Confidence scores are heavily top-skewed across all methods.

Observation: 86% of predictions exceed a confidence of 0.8; fewer than 1% fall below 0.2.
Implication: Practitioners cannot use intuitive midpoints (e.g., 0.5) as thresholds. Thresholds must be set relative to the specific model's observed distribution (e.g., setting a threshold at the 80th percentile of the model's output) to effectively filter out uncertain cases.

Additional Insights

Reasoning Modes: For Qwen3-80B, the "Thinking" mode improved accuracy and calibration on complex long-answer tasks (RiceChem) but degraded calibration on short-answer tasks, likely due to overconfidence on simpler cues.
Stability: GPT-OSS-120B showed remarkable stability across different reasoning depths (High/Medium/Low), making it a reliable choice for deployment without extensive hyperparameter tuning.

4. Significance and Practical Implications

Practical Deployment: The study validates that simply asking an LLM to report its confidence is the most practical and cost-effective method for identifying reliable grading predictions.
Selective Automation Limits: Even with the best-performing model (GPT-OSS-120B) and strict reliability targets (5% risk), only 11%–20% of predictions qualify for full automation. This highlights that while LLMs can assist, they cannot yet fully replace human graders for the majority of cases without compromising quality.
Human-AI Collaboration: The findings support a hybrid workflow where the system automatically grades high-confidence cases and routes the remaining 80%+ to human reviewers, significantly reducing instructor workload while preserving grading integrity.
Methodological Shift: The paper argues against the heavy computational cost of self-consistency for calibration purposes, suggesting that well-tuned prompting for self-reported confidence is sufficient and superior.

Conclusion

The paper demonstrates that Self-Reported Confidence is the optimal signal for calibrating LLM graders in educational settings. While larger models improve accuracy, calibration remains dataset-dependent. The "confidence floor" phenomenon necessitates data-driven threshold setting rather than heuristic defaults. These findings provide a robust framework for deploying trustworthy, selective automation in automated assessment.