📊 statistics

An Interpretable and Scalable Framework for Evaluating Large Language Models

This paper proposes a scalable and interpretable framework based on the majorization-minimization principle to overcome the computational and stability limitations of traditional Item Response Theory methods, enabling efficient and accurate evaluation of large language models across diverse benchmarks.

Original authors: Xinhao Qu, Qiang Heng, Hao Zeng, Xiaoqian Liu

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinhao Qu, Qiang Heng, Hao Zeng, Xiaoqian Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to grade a class of 2,000 students (Large Language Models) on a massive test with thousands of questions (Benchmark Items).

The Old Way: The "Average Score" Trap
Currently, most people grade these AI models by simply counting how many questions they got right and dividing by the total. It's like saying, "Student A got 85% right, Student B got 84% right, so Student A is better."

But the authors of this paper point out two big problems with this simple math:

The AI is a bit jittery: If you ask the same AI the same question twice, it might get it right the first time and wrong the second time, just because of how it generates text. It's like a student who is brilliant but sometimes has a bad day or misreads a word. The old method treats these flukes as real facts.
Not all questions are equal: In a standard test, some questions are easy "freebies," and some are incredibly hard. The old method treats a question about "1+1" the same as a question about "quantum physics." It assumes every question is worth exactly one point, which hides the true difficulty of the test and the true skill of the student.

The New Idea: The "Psychologist's Report Card"
The authors propose using a method borrowed from human psychology called Item Response Theory (IRT). Think of this not as a simple score, but as a detailed diagnostic report.

Instead of just a number, this method tries to figure out three hidden things:

The Student's Ability: How smart is the AI really?
The Question's Difficulty: How hard was this specific question?
The Question's Discrimination: How good was this question at telling the difference between a smart AI and a dumb one? (Some questions are so easy that even a dumb AI gets them right, so they aren't very useful for ranking).

The Problem with the Old "Psychologist" Tools
The problem is that the traditional math used to calculate these hidden traits is incredibly slow and unstable. It's like trying to solve a giant jigsaw puzzle where the pieces keep changing shape. If you try to use it on 2,000 students and 10,000 questions, the computer might crash or take weeks to finish. It also gets confused easily if the data is messy (like if an AI fails to answer a question due to a timeout).

The Solution: The "Smart Assembly Line" (cBMM)
The authors built a new, super-fast framework called cBMM (Constrained Block Majorization-Minimization).

Here is a creative analogy for how it works:
Imagine you are trying to assemble a massive, complex piece of furniture (the evaluation) from a pile of parts.

The Old Way: You try to hold all the parts in your hands at once, trying to figure out where every single screw goes simultaneously. You get tired, drop pieces, and it takes forever.
The New Way (cBMM): You break the job down into tiny, manageable steps. You first assemble just the legs, then just the tabletop, then just the drawers. You do this in a loop, getting a little better with each pass. Because each step is simple and follows a strict rule, you never get stuck, and you finish the job in a fraction of the time.

What They Found
The authors tested this new "assembly line" on real-world data, including the famous MATH-500 benchmark and the Hugging Face Open LLM Leaderboard (which tracks thousands of models).

It's Lightning Fast: Their method was 40 to 80 times faster than the next best methods. In some cases, while other methods crashed or gave up, their method kept running smoothly.
It's More Accurate: Because it's so stable, it didn't get confused by messy data. It recovered the "true" abilities of the models better than the old methods.
It Reveals Hidden Truths:
- Scaling Laws: They confirmed that bigger models generally perform better, matching what scientists already suspected.
- Question Quality: They found that some questions in popular benchmarks are actually "junk"—they don't help distinguish between good and bad models. Their method can spot these useless questions automatically.
- Ranking Shifts: When they used their new method, the ranking of the top AI models changed slightly compared to the old "average score" method. Some models that looked average suddenly looked much smarter once you accounted for the difficulty of the questions they answered.

In a Nutshell
This paper gives us a new, faster, and smarter way to grade AI. Instead of just counting right answers, it acts like a skilled teacher who understands that some questions are harder than others and that students (AIs) have good days and bad days. It does this without breaking the computer, allowing us to evaluate thousands of models on massive tests in minutes rather than weeks.

Technical Summary: An Interpretable and Scalable Framework for Evaluating Large Language Models

Problem Statement

The evaluation of Large Language Models (LLMs) currently relies heavily on standard benchmarking methods that calculate average accuracy across benchmark items. The authors argue this approach is fundamentally flawed due to two overlooked challenges:

Stochasticity of LLM Outputs: LLM outputs can vary significantly under minor perturbations (e.g., prompt order, typos, synonyms) or even with identical prompts under greedy decoding, leading to fragile evaluations.
Heterogeneity of Benchmark Items: Standard methods implicitly assume all items carry equal weight. However, items vary in their discriminative power (ability to differentiate model capabilities) and integrity (some may be compromised by training data contamination or redundancy).

While Item Response Theory (IRT) offers a principled framework to model latent model abilities and item characteristics (discrimination and difficulty), conventional IRT implementations (e.g., Expectation-Maximization algorithms) suffer from computational inefficiency and numerical instability. These limitations restrict IRT to small-scale applications, making it ill-suited for the rapidly growing ecosystem of LLMs and massive benchmark datasets.

Methodology

To address these limitations, the paper proposes a Majorization-Minimization (MM) based framework, termed constrained Block MM (cBMM).

1. Probabilistic Formulation

The evaluation process is modeled as a latent probabilistic framework using a two-parameter IRT (2PL-IRT) model.

Observations: A binary response matrix $Y \in \{-1, 1\}^{N \times J}$ where $y_{ij}$ indicates if the $i$ -th LLM correctly answered the $j$ -th item.
Latent Score: The probability of a correct response is governed by a logit link function: $\Phi(x_{ij}) = \frac{1}{1 + \exp(-x_{ij}/\sigma)}$ , where $\sigma$ is a scaling parameter (analogous to temperature).
Reparameterization: The pre-activation score matrix $X$ $X$ is decomposed as $X = \theta a^\top + \mathbf{1}b^\top$ $X = θ a^{⊤} + 1 b^{⊤}$ , where:
- $\theta$ : Model ability vector.
- $a$ : Item discrimination vector (constrained to be non-negative, $a \ge 0$ ).
- $b$ : Item easiness vector.

2. Optimization via Majorization-Minimization

The goal is to minimize the binary cross-entropy loss $\ell(X)$ . Direct minimization is challenging. The authors employ the MM principle to convert this into a sequence of simpler surrogate problems:

Majorization: A quadratic surrogate function $g(X|\tilde{X})$ is constructed that upper-bounds the original loss and touches it at the current estimate $\tilde{X}$ .
Minimization: The problem reduces to a constrained matrix factorization subproblem:
$\min_{U, V} \| UV^\top - \tilde{Y} \|_{F(\Omega)}^2 \quad \text{subject to } a \ge 0$
where $U = (\theta, \mathbf{1})$ and $V = (a, b)$ .
Algorithm (cBMM): The algorithm alternates between updating $a$ (solved via non-negative least squares), $b$ (closed-form update), and $\theta$ (closed-form update). This block-wise optimization ensures monotonic decrease of the objective function.

3. Theoretical Guarantees

Identifiability: The authors prove that the model is identifiable up to an equivalence class (preserving rank ordering of abilities and discrimination) under the conditions that $\theta$ is not a constant vector and $a \ge 0$ with at least one non-zero entry.
Convergence: Under assumptions of boundedness and non-degeneracy, the sequence generated by cBMM is proven to converge to a stationary point satisfying KKT conditions. If the objective satisfies the Kurdyka–Łojasiewicz (KL) property with exponent $1/2$ , the convergence is linear.

Key Contributions

Novel MM-based Framework: A scalable computational framework for LLM evaluation that reformulates the problem as constrained matrix factorization, offering reliable interpretations of model abilities and item features.
Theoretical Guarantees: Formal proofs for parameter identifiability (up to equivalence class) and algorithmic convergence.
Computational Efficiency: The method achieves orders-of-magnitude speedups over competing methods while maintaining or improving estimation accuracy.
Empirical Insights: Application to real-world benchmarks reveals fine-grained heterogeneity in item difficulty and discrimination that average accuracy fails to capture.

Experimental Results

The framework was evaluated on synthetic data and seven real-world benchmark suites (MATH-500, IFEval, MuSR, GPQA, MATH, BBH, MMLU-Pro).

Scalability and Efficiency

Speed: cBMM demonstrated 41x to 86x speedups over the fastest baselines (L-BFGS-B, Manopt) across large-scale benchmarks. In some synthetic settings, speedups reached 200x.
Stability: Traditional IRT methods (e.g., mirt in R) frequently failed to converge (reported as 'NA') on large or ill-conditioned datasets (e.g., GPQA, MMLU-Pro) due to numerical instability. cBMM remained stable across all tested configurations, including varying temperatures, missing data patterns (MCAR, MAR, MNAR), and item redundancy.

Interpretability and Validity

Model Abilities: Recovered ability scores ( $\hat{\theta}$ ) aligned with established parametric scaling laws, showing clear upward trends over time and higher capabilities for larger model tiers.
Item Difficulty: Recovered difficulty parameters ( $\hat{b}$ ) showed strong consistency with human-annotated difficulty levels in the MATH-500 benchmark, though some deviations suggested latent misalignments between model-perceived and human-judged difficulty.
Item Discrimination: The study found an inverse relationship between the heterogeneity of item discrimination estimates and the ranking consistency between cBMM results and average accuracy. This confirms that average accuracy rankings can be misleading when item discrimination varies significantly.
Ranking Shifts: Fine-grained analysis revealed that standard accuracy-based evaluations may underestimate the capabilities of certain models (e.g., specific Qwen variants) compared to the cBMM-recovered abilities.

Significance and Claims

The paper claims to present a substantial advancement over conventional IRT-based approaches by providing a rigorous, scalable, and interpretable alternative for LLM evaluation.

Fairer Ranking: By accounting for stochasticity and item heterogeneity, the recovered parameters offer more reliable capability rankings than simple average accuracy.
Benchmark Design: The method provides data-driven signals for identifying redundant or non-informative benchmark items (via sparse discrimination estimates), guiding more efficient benchmark construction.
Automation: The recovered item difficulty parameters offer automated assessments that complement human annotations, enabling scalable benchmark analysis.

The authors note limitations, specifically treating model capability as a single-dimensional trait (suggesting future work on multidimensional IRT) and the need for statistical error bounds to quantify uncertainty. However, the current framework is presented as a robust solution for the immediate challenges of scaling LLM evaluation.