💬 NLP

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

This paper introduces an Item Response Theory (IRT) framework to evaluate LLM-based automatic short answer grading, revealing that models with similar aggregate scores exhibit distinct robustness profiles as response difficulty increases and identifying specific linguistic and semantic characteristics that correlate with grading errors.

Original authors: Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of short essays. Some answers are obvious: "The sky is blue" is clearly correct. Others are tricky: a student writes a sentence that is half-right, half-wrong, and uses confusing words.

For a long time, when researchers tested AI (Large Language Models or LLMs) to see if they could grade these essays automatically, they only looked at the final grade. They asked, "Did the AI get 80% right overall?" This is like saying a basketball player is "good" just because they made 80% of their shots, without noticing that they made all the easy layups but missed every single difficult three-pointer.

This paper argues that this "average score" approach hides a lot of important details. Instead, the authors use a tool from psychology called Item Response Theory (IRT). Think of IRT as a special microscope that lets you see two things at once:

How skilled the grader is (the AI).
How hard the specific question is (the student's answer).

Here is a breakdown of what they found, using simple analogies:

1. The "Difficulty Curve" Test

The researchers tested 17 different AI models on two sets of science questions. They didn't just count the total correct answers; they sorted the student answers from "Easy" to "Hard" based on how tricky they were.

The Finding: Even if two AIs have the same overall score, they behave very differently when things get tough.

Analogy: Imagine two runners. Runner A and Runner B both finish a 5-mile race in the same total time. But, Runner A runs fast on flat ground and slows down to a crawl on hills. Runner B runs at a steady pace the whole time.
The Result: Some AIs are like Runner A. They are great at grading easy answers but fall apart completely when the student's answer is confusing or ambiguous. Other AIs are more like Runner B; they might not be the absolute fastest on easy tasks, but they stay steady and reliable even when the answers get difficult.

2. The "Safe Middle" Trap

When the AI models encountered very difficult answers, they didn't just start guessing randomly. They developed a specific bad habit.

The Finding: When an answer was hard to understand, the AI tended to default to a "middle-ground" label called partially_correct_incomplete.

Analogy: Imagine a judge in a courtroom. When the evidence is confusing and the lawyer's argument is messy, instead of saying "Guilty" or "Not Guilty," the judge keeps saying, "Well, it's sort of both, let's call it 'Maybe Guilty'."
The Result: The AI stops making sharp distinctions. It collapses all the confusing answers into this one "safe" middle category. It becomes too optimistic, thinking a messy answer is "partially right" when it might actually be wrong or irrelevant.

3. Why Are Some Answers So Hard?

The authors also looked at what makes an answer difficult for the AI to grade. They analyzed the words and meaning of the student responses.

The Finding: Hard answers usually have three specific traits:

They don't match the "textbook" answer: The meaning is far away from the correct reference answer.
They contain contradictions: The student says things that fight against each other or the facts.
They are "isolated": In the world of language, these answers are like islands. They don't look like other common answers. They are unique or strange in a way that confuses the AI.

The Big Takeaway

The paper concludes that we shouldn't just ask, "Which AI is the best grader?" We need to ask, "Which AI stays calm and accurate when the answers get messy?"

By using this new "microscope" (IRT), we can see that some AIs are fragile—they break under pressure—while others are robust. This helps us understand that grading isn't just about getting the right number; it's about understanding where and why an AI might fail, especially when dealing with the tricky, real-world answers that students actually write.

1. Problem Statement

Automatic Short Answer Grading (ASAG) using Large Language Models (LLMs) is typically evaluated using aggregate metrics such as macro-F1, accuracy, and Cohen's kappa. The authors argue that these metrics provide a coarse view of system performance, failing to reveal:

How grading performance varies across student responses of differing grading difficulty.
Whether models with similar overall scores differ in their robustness when handling difficult, ambiguous, or semantically complex responses.
The specific failure modes (e.g., confusion patterns) that occur as difficulty increases.

Current evaluation methods obscure the fact that two models may perform similarly on average but have vastly different degradation rates as response difficulty increases.

2. Methodology

The authors propose an evaluation framework based on Item Response Theory (IRT), a psychometric model traditionally used in educational testing to separate latent ability from item difficulty.

A. Experimental Setup

Datasets: SciEntsBank and Beetle (two widely used ASAG benchmarks with a 5-way label space: correct, contradictory, partially_correct_incomplete, irrelevant, non_domain).
Models: 17 open-weight LLMs (ranging from 0.8B to 14B parameters) across various families (Google, Qwen, Meta, Mistral, etc.).
Task: Each LLM acts as a "grader" assigning a label to a student response. Correctness is defined by agreement with the gold standard label.

B. IRT Formulation

The authors model the probability of a grader $i$ correctly grading a response $j$ using a 1PL Testlet Response Theory model:
$P(y_{ij} = 1) = \sigma(\theta_i - b_j + u_{i, t(j)})$
Where:

$\theta_i$ : Latent grader ability of LLM $i$ .
$b_j$ : Latent grading difficulty of student response $j$ .
$u_{i, t(j)}$ : Testlet effect (grader-specific deviation) associated with the specific question $t(j)$ to account for question-level dependencies.
$\sigma(\cdot)$ : Logistic sigmoid function.

Parameters are estimated by minimizing a regularized negative log-likelihood function.

C. Validation and Analysis

Parameter Recovery: Synthetic data generated from the fitted model is used to verify if the estimation procedure can recover the original latent parameters (high correlation indicates stability).
Split-Half Stability: The model is re-estimated on disjoint subsets of data to ensure the inferred difficulty and ability scales are reproducible.
Difficulty-Conditioned Analysis: Responses are binned by estimated difficulty ( $b_j$ $b_{j}$ ). The authors analyze:
- Accuracy trends across difficulty bins.
- Confusion matrices to identify how error patterns shift.
Feature Correlation: Semantic and linguistic features (e.g., NLI entailment, embedding distance, lexical overlap) are correlated with estimated difficulty to characterize why certain responses are hard to grade.

3. Key Contributions

IRT-Based Framework: Introduction of an IRT perspective for ASAG that decouples grader ability from response difficulty, enabling fine-grained analysis beyond aggregate scores.
Robustness Characterization: Demonstration that LLMs with similar aggregate scores exhibit substantial differences in how sharply their accuracy declines as response difficulty increases.
Error Pattern Discovery: Identification of a specific failure mode where errors on difficult responses are not random but concentrate disproportionately on the partially_correct_incomplete label, suggesting an "intermediate-label collapse" under ambiguity.
Semantic Correlates of Difficulty: Empirical evidence linking high grading difficulty to:
- Weaker semantic alignment with reference answers.
- Stronger contradiction signals.
- Greater isolation in embedding space (semantic atypicality).

4. Key Results

A. Stability and Recovery

Parameter Recovery: The IRT model showed near-perfect recovery for grader ability ( $r \approx 0.999$ ) and strong recovery for response difficulty ( $r \approx 0.896$ ), confirming the latent estimates are reliable.
Split-Half Stability: Grader ability estimates were highly stable across data splits. Response difficulty estimates showed moderate-to-strong stability, indicating the difficulty structure is reproducible.

B. Performance Degradation

Differential Robustness: As difficulty increased (from Bin B1 to B5), accuracy dropped monotonically for all models, but the rate of decline varied significantly.
- Example: On SciEntsBank, Gemma-3-12B retained 16.6% accuracy on the hardest bin, while Llama-3.1-8B dropped to 2.1%.
- The slope of performance degradation is a stable characteristic of the model, consistent across both datasets.

C. Confusion Patterns

Intermediate Collapse: On high-difficulty responses, correct, contradictory, and irrelevant answers are increasingly misclassified as partially_correct_incomplete.
Over-optimism: Difficult partially_correct_incomplete responses are often misclassified as correct, indicating models struggle to distinguish between ambiguous partial answers and fully correct ones.

D. Correlates of Difficulty

Negative Correlates (Easier to grade): High NLI entailment, high semantic similarity, and high lexical overlap with the reference answer.
Positive Correlates (Harder to grade): High NLI contradiction scores, large embedding-space distances (k-NN distance), and high "missing reference segments."
Conclusion: Difficult responses are not random outliers; they are semantically ambiguous, incomplete, or contradictory relative to the reference.

5. Significance and Implications

Beyond Aggregate Metrics: The study argues that evaluating LLMs solely on average scores is insufficient. A model that degrades gracefully on difficult items is qualitatively superior to one that collapses, even if their overall F1 scores are similar.
Human-in-the-Loop Deployment: The ability to estimate response difficulty allows for selective grading pipelines. Easy responses can be graded automatically, while high-difficulty responses (identified by low semantic alignment or high embedding isolation) can be routed to human reviewers, improving efficiency and reliability.
Rubric and Prompt Design: The tendency to collapse into the "intermediate" label suggests current prompting strategies lack clear decision boundaries. The authors suggest scaffolded evaluation (sequential sub-decisions on relevance, correctness, contradiction, and completeness) to mitigate this collapse.
Educational Fairness: Understanding why a response is difficult (e.g., semantic isolation vs. contradiction) provides interpretable feedback for educators, moving beyond simple "right/wrong" scores.

6. Limitations

Model Scope: Limited to 17 specific open-weight models; results may vary with proprietary or larger models.
Dataset Generalization: Findings are based on two science-domain datasets; generalizability to other domains or label schemas is unproven.
Correlational Nature: The feature analysis identifies associations with difficulty but does not establish causal mechanisms.