💬 NLP

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

The paper proposes RankLLM, a novel framework that quantifies both question difficulty and model competency through bidirectional score propagation to enable fine-grained, stable, and efficient evaluation of large language models that outperforms existing benchmarks and baselines like IRT.

Original authors: Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge the skill of 30 different chefs. You have a menu of 35,000 different dishes, ranging from "toast a slice of bread" to "create a 10-course molecular gastronomy feast."

The Old Way (Traditional Benchmarks):
Currently, most people judge these chefs by simply counting how many dishes they got right. If Chef A gets 80% of the dishes right and Chef B gets 80% right, they are considered equal.

The Problem: This is unfair. If Chef A only made toast and got it right, but Chef B made the complex 10-course feast and got it right, they are treated the same. The old method doesn't care how hard the dish was; it just counts the "correct" answers.

The New Way (The EIP Paper):
The authors of this paper, Ziqian Zhang and team, propose a new system called EIP (Empirical Interaction Propagation). Think of EIP as a smart, self-correcting rating system that figures out two things at the same time:

How hard each dish actually is.
How skilled each chef really is.

How It Works: The "Ping-Pong" of Difficulty

Imagine a giant game of ping-pong between the Chefs (Models) and the Dishes (Questions).

The Setup: You have a graph connecting every chef to every dish they tried.
The Rules:
- If a chef solves a hard dish, they get a huge boost in their "Skill Score."
- If a chef fails an easy dish, it's a big red flag that the dish might actually be tricky (or the chef is struggling).
- If a dish is solved by almost everyone, its "Difficulty Score" goes down.
- If a dish stumps even the best chefs, its "Difficulty Score" goes up.
The Magic Loop: The system runs a loop (like a ripple effect). It asks: "Who solved this hard dish? They must be good." Then it asks: "Who failed this easy dish? That dish must be harder than we thought." It passes these scores back and forth until the numbers settle into a stable, fair ranking.

What They Found (The Results)

The researchers tested this on 30 different AI models (from tiny ones to massive ones) across 35,550 questions. Here is what they discovered:

It Matches Human Intuition: When they asked humans to guess which questions were harder, EIP agreed with them 90% of the time. Other methods (like standard math models used in education) were much less accurate.
It Spots Hidden Talent: EIP found that some smaller AI models were actually better at solving hard problems than larger models, even if the larger models had a higher overall "correct answer" count. The old method missed this; EIP gave the smaller models a higher rank because they tackled the tough stuff.
It's Super Fast: Calculating these rankings used to take hours or days with old methods. EIP did it in 0.006 seconds on a regular laptop. It's like switching from a horse-drawn carriage to a rocket ship.
It's Stable: Even if you remove half the chefs from the competition, the ranking of the dishes and the remaining chefs stays almost exactly the same. It doesn't get confused by who is in the room.
The "Crowd" Effect: The system works best when you have a mix of small, medium, and large models. If you only use tiny models, they might think everything is impossible. If you only use giant models, they might think everything is too easy. Mixing them all together gives the most accurate picture of reality.

The Bottom Line

The paper argues that we need to stop treating all questions as equal. Just because an AI got a question right doesn't mean it's smart if the question was easy. EIP is a tool that weighs the difficulty of the question against the skill of the model, creating a much fairer and more detailed leaderboard for Artificial Intelligence.

It's not just about who got the most points; it's about who conquered the hardest mountains.

Technical Summary: Empirical Interaction Propagation (EIP)

1. Problem Statement

Current benchmarks for Large Language Models (LLMs) rely heavily on aggregate accuracy metrics within topical categories (e.g., MMLU-Pro, MATH). This approach implicitly treats all evaluation items as equally informative, failing to differentiate between question difficulties. Consequently, benchmarks cannot effectively distinguish between routine pattern matching and advanced reasoning. A single arithmetic fact and a multi-step calculus derivation are often weighted equally, leading to rankings that may flip when the mixture of easy versus hard items shifts. The authors posit that difficulty must be modeled explicitly at the item level to enable fine-grained, difficulty-aware comparisons of model capabilities.

2. Methodology: Empirical Interaction Propagation (EIP)

EIP is a non-parametric framework designed to jointly estimate question difficulty and model competency through an iterative, bidirectional score propagation process.

Core Intuition

The framework operates on the principle that:

A model earns a competency score when it correctly answers a question.
A question's difficulty score increases when it challenges (i.e., causes failure in) a model.
Solving a hard question carries more evidential weight for competency, while failing an easy question contributes more weight to the question's difficulty.

Graph Formulation

EIP constructs a directed bipartite graph $G = (V, E)$ where vertices represent Models ( $M$ ) and Questions ( $Q$ ).

Competency Edges ( $E_{Comp}$ ): Directed edges from questions to models ( $q \to m$ ) indicate a correct answer.
Difficulty Edges ( $E_{Fail}$ ): Directed edges from models to questions ( $m \to q$ ) indicate a failure.

Score Propagation Mechanism

The system models interactions as an ergodic Markov chain. It defines two row-stochastic transition matrices:

Competency Transition ( $P_{Q \to M}$ ): Normalized by the number of models that solved a question ( $S(q)$ ).
Difficulty Transition ( $P_{M \to Q}$ ): Normalized by the number of questions a model failed ( $F(m)$ ).

Scores are updated iteratively using a damping factor $\alpha \in (0, 1)$ to ensure convergence to a unique stationary distribution (guaranteed by the Perron-Frobenius theorem). The update equations are:
$\pi^{(t+1)}_Q = \alpha P^\top_{M \to Q} \pi^{(t)}_M + (1 - \alpha)\frac{1_Q}{Q}$
$\pi^{(t+1)}_M = \alpha P^\top_{Q \to M} \pi^{(t+1)}_Q + (1 - \alpha)\frac{1_M}{M}$
Where $\pi_Q$ represents question difficulty and $\pi_M$ represents model competency. The process converges asymptotically to unique scores.

Extensions

Continuous Scores: EIP extends to benchmarks with partial credit by allowing the response matrix to take values in $[0, 1]$ rather than binary $\{0, 1\}$ .
Preprocessing: Questions universally solved or universally failed are excluded to ensure graph connectivity and avoid trivial solutions.

3. Key Contributions

The paper makes three primary contributions:

A New Evaluation Framework: EIP introduces a difficulty-aware, non-parametric approach that jointly estimates question difficulty and model competency, moving beyond flat accuracy metrics.
Comprehensive Empirical Validation: The framework was evaluated on 30 models across 35,550 questions spanning six benchmarks (BBH, GPQA, GSM8k, HellaSwag, MATH, MMLU-Pro).
New Insights: The study reveals distinct dataset difficulty profiles, consistent family-level patterns across model scales, and the reliability of open-weight and mixed-scale pools for producing stable difficulty estimates.

4. Experimental Results

Alignment with Human Judgment

EIP achieved 90% agreement with human judgments on question difficulty across 20 evaluators.
This significantly outperformed baselines including Simple Rank and various Item Response Theory (IRT) models (1PL, 2PL, Multi-IRT), which ranged from 50% to 62.9% agreement.
A diverse model pool (mixed scales) was found to be crucial; homogeneous pools (e.g., only small or only large models) resulted in lower consensus (38.6%–64.3%) and higher bias.

Ranking Differentiation

While EIP scores correlate with accuracy (Kendall's $\tau = 0.8492$ ), it frequently reorders adjacent models.
Example: Qwen2-0.5B (20.2% accuracy) was ranked higher than DeepSeek-Chat-Lite (30.49% accuracy) because Qwen2-0.5B solved a higher proportion of hard questions (5.5% vs. 2.4%).
Simulation: In a controlled simulation with ground-truth rankings, EIP was the only method to correctly recover the true ranking order, whereas accuracy-based and IRT-based methods failed to distinguish models with identical overall accuracy but different difficulty profiles.

Stability and Robustness

Model Perturbation: Removing up to 50% of the model pool (15 out of 30 models) resulted in a Spearman correlation of 0.9382 for question difficulty rankings and 0.9934 for model competency rankings compared to the full pool.
Dataset Perturbation: Rankings remained stable under dataset perturbations ( $\rho > 0.99$ ).

Efficiency and Scalability

Time Complexity: $O(tQM)$, where $t$ is the number of iterations (typically constant).
Performance: On consumer hardware (Intel i7), EIP converged in 0.00597 seconds for 30 models and 35,550 questions.
Comparison: This is orders of magnitude faster than IRT baselines (e.g., 1PL-IRT took ~1782 seconds).
Scalability: Linear scaling observed with interaction counts up to $2 \times 10^9$ interactions.

Dataset and Model Insights

Dataset Profiles: MATH and MMLU-Pro showed broader difficulty tails suitable for advanced reasoning, while GSM8k and HellaSwag were skewed toward easier questions.
Model Families: Scaling primarily shifts absolute accuracy rather than altering the relative difficulty structure; models within a family (e.g., Llama-3 variants) preserved similar difficulty-specific answering distributions.
Open-Weight Models: Difficulty estimates derived from open-weight models alone showed strong correlation with full-pool results (Spearman 0.96), rivaling proprietary pools.

5. Significance

The paper claims that EIP provides a practical, scalable, and human-aligned alternative for LLM evaluation. By quantifying question difficulty, it enables:

Finer-grained comparisons: Distinguishing models that excel at hard reasoning versus those that rely on pattern matching.
Robustness: Maintaining stable rankings even as model pools or datasets change dynamically.
Efficiency: Offering a computationally lightweight solution suitable for large-scale, continuous evaluation (e.g., leaderboards) without the heavy calibration costs of parametric models like IRT.

The authors emphasize that EIP does not replace broader alignment assessments but complements them by providing a more nuanced view of model capabilities relative to task difficulty.