💻 computer science

Estimating Exam Item Difficulty with LLMs: A Benchmark on Brazil's ENEM Corpus

This paper benchmarks ten LLMs against Brazil's ENEM exam to reveal that while they can moderately rank question difficulty, they systematically underestimate item hardness, struggle with multimodal content, and lack the contextual plasticity needed for personalized assessment, suggesting they are best suited as calibrated screeners rather than authoritative generators.

Original authors: Thiago Brant, Julien Kühn, Jun Pang

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thiago Brant, Julien Kühn, Jun Pang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to build a new math test. You have a super-smart robot assistant (an AI) that can write questions for you. But before you give the test to students, you need to know: Is this question too easy? Is it too hard?

Traditionally, teachers answer this by giving the test to thousands of students first, grading the results, and doing complex math to figure out the difficulty. This takes a long time and costs a lot of money.

This paper asks a simple question: Can we just ask the AI, "How hard is this question?" and trust its answer?

The researchers tested this idea using Brazil's massive national high school exam (ENEM), which has over 1,000 real questions. They asked ten different AI models to guess the difficulty of these questions and compared their guesses to the "official" difficulty scores calculated by human experts.

Here is what they found, explained with some everyday analogies:

1. The "Good at Ranking, Bad at Numbers" Problem

The AI models are like a student who is great at ordering books by size but terrible at measuring them with a ruler.

The Good News: If you ask the AI to sort 10 questions from "easiest" to "hardest," it does a decent job. It can tell you that Question A is harder than Question B.
The Bad News: If you ask the AI, "On a scale of 1 to 10, how hard is this?", it is consistently wrong. It almost always thinks the questions are easier than they actually are. It's like a person who thinks a marathon is just a "light jog" because they've never run one.

2. The "Blindfold" Effect (Visuals vs. Text)

Many exam questions have pictures, graphs, or charts. Since some AIs can't "see" images, the researchers had to describe the pictures in words (like a blind person describing a painting to a friend).

The Result: When the AI had to guess the difficulty based only on the text description of a picture, it got confused. It was much worse at judging these questions than the ones that were just text.
The Analogy: Imagine trying to guess how hard a puzzle is by reading a description of the pieces, rather than looking at the puzzle itself. You might miss a crucial clue, making the puzzle seem easier or harder than it really is.

3. The "One-Size-Fits-All" Trap (Student Backgrounds)

The researchers wanted to see if the AI could adjust its answer based on who is taking the test. They asked the AI: "How hard is this for a student from Finland?" vs. "How hard is this for a student from Brazil?"

The Result: The AI barely changed its mind. It was mostly "tone-deaf" to these changes.
The Analogy: Imagine a chef who cooks a soup. If you ask, "Is this soup too salty for a baby?" or "Is it too salty for a bodybuilder?", the chef just shrugs and says, "It's the same soup." The AI didn't really understand that different students have different backgrounds and knowledge levels. It couldn't adapt its "difficulty meter" to fit the person.

4. The "Prompt" Game (How you ask matters)

The researchers tried asking the AI in different ways. Sometimes they just said, "Rate this." Other times, they said, "Act like a math teacher, think step-by-step, and then rate this."

The Result: The way you ask the question changed the answer significantly. Some "thinking" prompts helped the AI get the ranking right, but they didn't fix the problem of it thinking everything was too easy.
The Analogy: It's like asking a friend for a movie recommendation. If you ask, "What's a good movie?", they might say a comedy. If you ask, "What's a good movie for a rainy Tuesday night?", they might say a thriller. The AI's answer depends heavily on how you phrase the question.

The Big Conclusion: The "Screening" Tool

The paper concludes that we shouldn't treat these AI models as Oracles (all-knowing gods who give the perfect answer). Instead, we should treat them as Screeners (a first filter).

The Recommendation: Use the AI to quickly scan a pile of new questions and say, "Hey, this one looks like it might be too easy, and this one looks like it might be too hard."
The Catch: You cannot trust the AI's exact number. You must fix its "bias" (teach it that it's underestimating difficulty) and you must have a human check the questions, especially the ones with pictures.

In short: AI is a helpful assistant that can help you sort the "easy" questions from the "hard" ones, but it is not ready to be the final judge of exactly how hard a question is, nor is it ready to understand the specific needs of different types of students. We need to calibrate it and keep a human in the loop.

Technical Summary: Estimating Exam Item Difficulty with LLMs

Problem Statement

As Large Language Models (LLMs) are increasingly deployed to generate educational content, a critical safety gap exists: the inability to reliably estimate the difficulty of the questions they produce before deployment. Traditional difficulty estimation relies on Item Response Theory (IRT), which requires expensive, large-scale pilot testing with representative student samples. While LLMs offer the potential to approximate item difficulty directly from text, several challenges hinder their reliability:

Domain Dependency: Models behave inconsistently across subjects (e.g., Mathematics vs. Humanities).
Multimodality: Real-world exam items often rely on diagrams and tables; converting these to text risks losing spatial cues essential for reasoning.
Calibration vs. Ranking: Models may achieve low absolute error by predicting the mean difficulty (regression to the mean) while failing to preserve the relative ordering (rank fidelity) of items.
Context Sensitivity (Plasticity): It is unclear whether models can meaningfully adjust difficulty estimates based on learner demographics (e.g., country or educational background) or if such cues induce noisy, biased shifts.

This study addresses these gaps by benchmarking LLMs against high-stakes, IRT-calibrated data to determine if they can serve as reliable pre-screening tools for assessment design.

Methodology

Dataset

The study utilizes a corpus of 1,031 multiple-choice questions from Brazil's Exame Nacional do Ensino Médio (ENEM), covering the years 2017–2022. The dataset spans four subjects: Mathematics (MT), Natural Sciences (CN), Human Sciences (CH), and Languages and Codes (LC).

Ground Truth: Official IRT difficulty parameters ( $b_i$ ) derived from large-scale student performance, mapped to a 1–10 scale.
Modality: 38.1% of items contain visual elements (diagrams, graphs, tables). These were transcribed into text using vision-capable LLMs (GPT-4o, Gemini-2.0-Flash) to simulate text-only pipelines, preserving minor transcription errors to reflect realistic deployment conditions.

Experimental Protocol

Models: Ten heterogeneous LLMs were evaluated, including four proprietary APIs (GPT-4o, o3, Claude-4-Sonnet, Gemini-2.0/2.5-Flash) and six open-weight models (Phi-4, Mistral Large, DeepSeek-R1, Qwen-3, Llama-3.2).
Prompt Strategies: Eight prompt families were tested, ranging from minimal instructions to structured reasoning scaffolds:
- Zero-shot and Enhanced Zero-shot.
- Few-shot and Persona+Few-shot.
- Chain of Thought (CoT), Point-based (rubric decomposition), and Tree of Thoughts (ToT).
- Synthetic/Playground (structured JSON output).
Bias Probe: A "plasticity" test injected a single country token (e.g., "students in [Country]") into prompts for four representative models across seven nations (Brazil, China, Finland, etc.) to measure shifts in difficulty estimates.

Metrics

Performance was evaluated using:

Absolute Calibration: Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) against IRT targets.
Rank Fidelity: Spearman's rank correlation ( $\rho$ ).
Stability: Prompt sensitivity ( $\sigma_{prompt}$ ) and distributional alignment (KL Divergence).
Bias: Signed residuals and Plasticity Index (PI) to measure responsiveness to demographic cues.
Calibration: Post-hoc additive shift and affine mapping were applied to correct global location biases.

Key Results

1. Trade-off Between Accuracy and Ranking

A significant trade-off exists between absolute calibration and rank fidelity.

Low RMSE is Misleading: Models like Llama-3 achieved low RMSE (1.40) by regressing toward the mean difficulty, resulting in near-zero rank correlation ( $\rho \approx 0.10$ ).
Best Performers: Phi-4 (open-weight) and Gemini-2.5-Flash offered the most balanced performance, achieving competitive RMSE with strong rank fidelity ( $\rho \approx 0.31–0.37$ ).
Prompt Sensitivity: Switching prompt families often shifted performance more than switching models. For open-weight models, reasoning scaffolds (Persona, Point-based) significantly improved rank fidelity. Proprietary models showed less sensitivity to prompt changes.

2. Subject-Specific Performance

Mathematics: Exhibited the highest error and strongest prompt sensitivity. "Point-based" prompts (step decomposition) were most effective, suggesting rubric-style reasoning aids multi-step STEM items.
Languages: Proprietary models (Claude-4-Sonnet) excelled here.
Human Sciences: DeepSeek-R1 performed best, likely due to reasoning-centric pretraining for socio-historical contexts.
Natural Sciences: Showed wide error variance; Gemini-2.5-Flash achieved the highest rank correlation.

3. The Modality Penalty

Transcribing visual items to text incurred a consistent performance penalty.

Visual items increased RMSE by an average of 1.26 points compared to non-visual items.
Models systematically failed to capture the human difficulty trend for visuals: while humans find visual items easier in Natural Sciences, models often overestimated their difficulty or remained neutral.
Concise, criterion-driven prompts (Point-based, Enhanced Zero-shot) were less sensitive to modality shifts than complex reasoning templates.

4. Systematic Bias and Calibration

Global Underestimation: All models systematically underestimated item difficulty (predicted items as easier than IRT). The mean residual was positive for every system, with o3 showing the largest shift ( $\approx +3$ points).
Calibration Efficacy: A simple additive shift (post-hoc calibration) significantly reduced RMSE and MAE for all models without altering rank correlations ( $\rho$ ). This indicates that a large portion of the error is a location bias correctable via linear adjustment.

5. Plasticity and Context Sensitivity

When prompted with country-specific cues:

Limited Plasticity: Models exhibited small but systematic shifts in difficulty estimates. However, these shifts were noisy and inconsistent with real-world performance gaps (e.g., PISA rankings).
Model Variance: Only Llama-3 showed a significant monotonic association with PISA reading ranks. Other models (GPT-4o, o3) showed weak or negative trends.
Risk: The lack of consistent, meaningful adaptation suggests current LLMs are not yet ready for context-adaptive personalization and risk encoding a "one-size-fits-all" notion of difficulty or amplifying biases if used for adaptive testing.

Significance and Claims

The paper concludes that while LLMs are not yet authoritative oracles for item difficulty, they possess actionable signal for screening and triage. The authors advocate for an "evaluation-before-generation" pipeline:

LLMs as Screeners: LLMs can serve as low-cost, pre-deployment filters to prioritize candidate items by predicted difficulty, provided that:
- Simple post-hoc calibration is applied to correct global location biases.
- Evaluation emphasizes both absolute alignment (via calibration) and rank fidelity.
- Visual content is handled with caution, as text-only transcription degrades performance.
Human-in-the-Loop: Items passing the LLM screening gate must undergo human curation and limited pilot testing (IRT validation) before operational deployment.
Fairness Guardrails: The study demonstrates that current models lack the "plasticity" to reliably adapt to learner demographics. Therefore, demographic tokens should be avoided in production prompts, and fairness diagnostics must be integrated into the assessment design pipeline to prevent the reinforcement of inequalities.

The work establishes that difficulty estimation is a prerequisite capability that must be calibrated and audited before LLMs can be responsibly used to generate or select high-stakes assessment content.