Technical Summary: DualEval – Joint Model-Item Calibration for Unified LLM Evaluation

Problem Statement

Current Large Language Model (LLM) evaluation relies on two complementary but often disconnected signals:

1. Static Benchmarks: Provide objective correctness labels and standardized grading but suffer from saturation, contamination, and a lack of nuance for open-ended tasks.
2. Arena-Style Evaluations: Reflect open-ended user interactions through preference rankings (e.g., Bradley-Terry models) but introduce subjectivity, annotator disagreement, and systematic evaluator biases.

Existing approaches often treat these settings as separate or aggregate item-level outcomes into overall model scores, ignoring the critical insight that ranking signals depend on model-item interactions. Items that are trivially easy or impossibly hard provide limited separation among models. Furthermore, current methods are largely model-centric, failing to diagnose the properties of the evaluation items themselves (e.g., difficulty, sharpness, or saturation).

Methodology: DualEval

The authors propose DualEval, a unified framework that jointly estimates model abilities and item properties within a shared latent space, inspired by Item Response Theory (IRT).

Core Framework

DualEval represents each model $i$ with an ability parameter $\theta_i$ and each task $q$ with a difficulty parameter $b_q$ and a sharpness parameter $a_q$ (where $a_q = \exp(k_q)$). The framework integrates two distinct data sources:

1. Static IRT (Binary Correctness):
   For static benchmarks with binary labels ($y_{i,q} \in \{0, 1\}$), DualEval employs a standard two-parameter logistic (2PL) IRT model:
   $$p_{i,q} = P(y_{i,q} = 1) = \sigma(a_q(\theta_i - b_q))$$
   where $\sigma$ is the sigmoid function. The loss is the binary cross-entropy between predicted probabilities and observed labels.

2. Reward-Distilled Pairwise IRT (Open-Ended Preferences):
   For arena-style data, a reward model (RM) assigns scalar scores $r_{i,q}$ to responses. These are standardized to $z_{i,q}$.

   • Soft Preference Targets: Pairwise preferences are constructed as soft targets $p^*_{ijq} = \sigma(z_{i,q} - z_{j,q})$, converting reward gaps into calibrated preference strengths.
   • Latent Prediction: The model predicts the probability of model $i$ beating $j$ on question $q$ as $\hat{P}(i \succ j | q) = \sigma(\gamma(p_{i,q} - p_{j,q}))$, where $\gamma$ is a learned temperature parameter.
   • Filtering: The framework filters out "ties" (small reward gaps) and "both-bad" pairs (where both models receive low rewards) to avoid noisy gradients. "Both-bad" pairs are treated as absolute failure evidence, pushing both models toward low success probabilities for that item.

Joint Optimization

The static and arena components share the same latent parameters $(\theta, b, k)$. The total objective function combines the static loss, the arena preference loss, and a "both-bad" penalty, regularized to ensure identifiability (centering abilities to zero mean after each step).

Key Contributions

1. Unified Evaluation Paradigm: DualEval bridges static correctness and open-ended preferences on a shared latent scale, allowing each signal source to compensate for the other's weaknesses (e.g., static labels stabilizing noisy reward signals).
2. Item-Level Diagnostics: Unlike traditional leaderboards, DualEval learns item profiles (difficulty, sharpness, and Fisher information), enabling the identification of saturated, discriminative, or low-signal items.
3. Diagnostic Applications:
   • Benchmark Compression: Using Fisher information to select high-signal subsets of items that recover full-data rankings with minimal data.
   • Anomaly Detection: Using residuals to flag suspicious (model, item) pairs that deviate from expectations, aiding in contamination detection.

Experimental Results

The framework was evaluated across four domains (Coding, Math, Miscellaneous, Generic) using 18 frontier LLMs.

• Signal Reconstruction: DualEval achieved 88–92% accuracy on static labels and 68–81% decisive-pair agreement on arena comparisons.
• Robustness: In static-anchored domains, DualEval maintained high agreement with static references even when the reward model was swapped for a public alternative (Skywork-Reward-V2-Qwen3-8B). In contrast, arena-only baselines showed significant ranking divergence ($\rho \approx 0.25$) under the same conditions.
• Benchmark Compression: Selecting the top 10% of items by Fisher information recovered the full-data ranking with Spearman $\rho \ge 0.95$ across static-anchored domains, demonstrating that ranking signals are concentrated in a small subset of high-informativeness items.
• Anomaly Detection: The residual-based detector achieved AUROC $\ge 0.995$ for static-label contamination and AUROC 0.93–0.98 for arena-reward contamination when using a clean-reference fit.

Significance and Claims

The paper reframes LLM evaluation as a joint model-item calibration problem rather than a purely model-centric ranking task. The authors claim that this approach:

• Produces more balanced and stable rankings by integrating static and arena signals.
• Transforms evaluation data into an object of analysis, allowing for the identification of saturated items and the construction of sample-efficient benchmarks.
• Provides auditable, interpretable diagnostics (via item profiles and residuals) that support more rigorous and cost-effective evaluation pipelines.

The authors acknowledge limitations, including reliance on a proprietary reward model (mitigated by public RM validation), the use of single-response evaluations (ignoring sampling variability), and the use of scalar ability parameters (future work suggests multidimensional profiles). However, they assert that DualEval offers a principled step toward more efficient, interpretable, and grounded evaluation systems.