💬 NLP

JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

JudgeArena is an open-source framework that unifies major LLM-judge benchmarks under a single interface to enhance reproducibility, enable systematic studies of evaluation design choices, and provide high-accuracy Elo score simulations using tuned open models as an alternative to closed-model judges and costly human annotation.

Original authors: Erlis Lushtaku, Bora Kargi, Ali Elganzory, Fabio Ferreira, Alejandro R. Salamanca, Julia Kreutzer, David Salinas

Published 2026-08-05

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Erlis Lushtaku, Bora Kargi, Ali Elganzory, Fabio Ferreira, Alejandro R. Salamanca, Julia Kreutzer, David Salinas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which of two robots is better at telling jokes, writing poems, or solving riddles. In the world of artificial intelligence, these robots are called Large Language Models (LLMs). For a long time, the only way to know who was the "funniest" or "smartest" was to ask a bunch of real humans to read the answers and vote. But asking humans is slow, expensive, and sometimes the humans get tired or argue about what "funny" even means. So, scientists started using a clever shortcut: they let one super-smart AI robot act as a judge to grade the other robots. This is called "LLM-as-a-judge." It's like hiring a robot referee to blow the whistle on a game between two other robots.

However, the playground for these robot referees has been a mess. Every time someone wanted to test a new robot, they had to build their own tiny, separate playground with its own rules, its own referee, and its own scoring system. Some referees were secret, closed-off robots that no one could peek inside, and if the company that owned them changed their mind about how they worked, all the old scores would suddenly become useless. It was like trying to compare the speed of two cars when one was measured on a track in 2020 and the other on a different track in 2024, using different stopwatches. This made it incredibly hard to know if a robot was actually getting better or if the rules just changed.

Enter JudgeArena, a new open-source toolkit that acts like a universal translator and a giant, fair arena for these robot tests. The authors of this paper built a single, unified framework that brings together the most popular ways of testing AI models into one place. Instead of building a new playground every time, researchers can now use this one tool to swap out different judges, try different types of questions, and run tests on their own computers or through various cloud services.

The paper finds that by using this unified system, they can create "tuned" judges using open-source models (robots whose brains are open for anyone to see) that perform just as well as, or even better than, the expensive, secret closed-model judges. They tested these setups across 33 different languages and found that while some languages are harder for AI judges to evaluate than others, the system can reliably tell us which models are winning. Furthermore, they discovered that by combining these AI judges with a few human votes, they can simulate the famous "Elo rating" system (the same one used to rank chess players and video game gamers) with high accuracy. This means developers can now estimate how good their new AI model is without having to wait for a massive, expensive human voting campaign.

In short, the paper suggests that the chaotic, fragmented world of AI testing can be organized into a clean, reproducible, and transparent system. It argues against the idea that we must rely on opaque, closed-source judges to get good results, showing instead that with the right setup, open and cheaper alternatives can do the job just fine. The authors measured these results using simulations and comparisons against human preferences, finding that their new method reduces errors significantly compared to older, less flexible ways of testing. It's a step toward making the AI world less like a collection of secret clubs and more like a fair, open sports league where everyone knows the rules and the scores.

Technical Summary: JudgeArena

Problem Statement

The current ecosystem for LLM-as-a-judge evaluation is fragmented and lacks reproducibility. While LLM judges have become a dominant paradigm for ranking language models due to the high cost and slow turnaround of human annotation, existing benchmarks (e.g., AlpacaEval, Arena-Hard, MT-Bench) operate in isolation. Key issues include:

Code Fragmentation: Most benchmarks ship with independent codebases, making it difficult to compare results across different studies.
Hardcoded Judges: Benchmarks often hardcode specific closed-weight models (e.g., GPT-4) as judges, introducing opaque biases and vulnerability to silent model updates or deprecations.
Limited Reproducibility: Many setups lack comprehensive metadata logging, preventing the exact replication of evaluation runs.
Inflexibility: Existing frameworks often struggle to support diverse inference backends (local vs. hosted) or allow for systematic studies of how design choices (judge model, prompt, backend) affect conclusions.

Methodology

The authors introduce JudgeArena, an open-source framework designed to unify major LLM-judge benchmarks under a single, reproducible interface.

Architecture and Design

JudgeArena operates via a unified command-line interface (CLI) where the benchmark, candidate models, and judge model are selected as orthogonal flags rather than being baked into scripts.

Unified Interface: It supports a common instruction schema, allowing the pipeline to remain agnostic to the underlying benchmark. This enables the addition of new benchmarks without extending new pipelines.
Inference Backends: The framework abstracts model inference behind a uniform backend, supporting:
- Local Inference: vLLM (GPU) and llama.cpp (CPU/Apple Silicon) for in-process execution.
- Hosted APIs: Providers like OpenRouter and Together AI via LangChain.
- This allows both candidate models and judges to run on different scales or precisions (e.g., a large judge on the same GPU as smaller candidates via tensor parallelism).
Reproducibility Infrastructure: Every run generates a self-describing artifact bundle containing:
- A snapshot of the full configuration (CLI arguments, prompt presets, truncation budgets).
- Per-battle annotation tables and aggregate results.
- Versioned metadata (package version, dependencies, run date).

Tuned Judge Configurations

Instead of relying on default or proprietary judges, JudgeArena integrates tuned open-weight judge configurations derived from multi-objective optimization.

Scoring Mechanism: The default preset asks the judge for two numerical scores ( $s_A, s_B \in [0, 10]$ ) rather than a binary verdict. These are converted into a winning probability using a sigmoid function with a sharpness factor $\beta = 0.3$ .
Bias Mitigation: The system includes bias-mitigating system prompts, optional chain-of-thought (CoT) explanations, and position-swapping (evaluating both A–B and B–A orderings) to reduce positional bias.
Flexibility: Users can toggle CoT, enable swap modes, and configure sampling parameters independently for the judge and battle models.

Supported Benchmarks and Arenas

JudgeArena unifies four primary pairwise instruction benchmarks:

AlpacaEval: Standard instructions with pre-shipped completions for baselines.
Arena-Hard: Supports v0.1 and v2.0 (curated hard prompts).
MT-Bench: 80 multi-turn questions across eight categories.
m-Arena-Hard: Multilingual coverage across 23 languages (v0.1 and v2.0).

Additionally, it incorporates crowd-sourced battle datasets (ComparIA, LMArena splits) for meta-analysis and Elo estimation.

Key Results

1. Meta-Evaluations

The authors conducted meta-evaluations on English and multilingual instructions to assess judge quality against human preferences.

Open vs. Closed Models: Tuned open-weight configurations (e.g., gemma-4-31b-it, qwen3.5 series) were found to be competitive with, and in some cases cheaper than, proprietary judges.
Performance Metrics:
- Cohen-Kappa: Tuned open models achieved competitive agreement scores. For instance, gemma-4-31b-it provided the best Cohen-Kappa agreement while being the cheapest option.
- Spearman Correlation: While noisy, the framework allows for systematic study of ranking correlations.
Multilingual Reliability: The framework evaluates judge performance per language. It found that gemma-4-31b-it generally outperforms qwen3.5-27b across most languages. The authors note that languages where the judge's agreement with humans does not significantly outperform random chance are excluded by default to ensure reliability.

2. Elo Approximation

JudgeArena demonstrates the ability to simulate LMArena Elo scores using LLM judges instead of human annotators.

Method: By replacing human preferences ( $\Phi_{Human}$ ) with LLM judge probabilities ( $\Phi_{LLM}$ ) in the Bradley-Terry model, the framework estimates Elo ratings.
Accuracy: Using continuous probability outputs (capturing judge confidence) rather than hard labels significantly reduced the Mean Absolute Error (MAE) in Elo estimation.
- Standard hard-label approaches (like AlpacaEval/Arena-Hard defaults) showed MAE errors around 86–112.
- The JudgeArena default configuration (using continuous probabilities) achieved an average MAE of 54 per model, offering a high-accuracy, low-cost alternative to large-scale human annotation campaigns.

3. Case Study: Open Model Benchmarking

The framework was used to benchmark several open-weight models (e.g., Qwen3.5, Olmo-3, EuroLLM) across all supported benchmarks.

Ranking Stability: The ordering of top models (e.g., Qwen3.5-9B ranking first) remained stable across instruction sampling levels.
Difficulty Variance: Arena-Hard v2.0 proved the most difficult English benchmark, while translated multilingual benchmarks (m-Arena-Hard) showed performance drops compared to English-only benchmarks, highlighting the challenges of cross-lingual evaluation.
Scaling: Performance was observed to scale monotonically with model size for most benchmarks.

Significance and Claims

The paper claims that JudgeArena addresses the critical lack of unification and reproducibility in the LLM-judge ecosystem. Its primary contributions are:

Standardization: Providing a unified API for major benchmarks with swappable judges and broad backend support.
Transparency: Offering tuned, open-weight judge configurations that match or outperform closed-model judges while reducing reliance on opaque closed models.
Reproducibility: Ensuring every evaluation run is fully reproducible through comprehensive metadata logging and artifact bundling.
Practical Utility: Enabling the estimation of Elo ratings with high accuracy, allowing model developers to gauge performance without participating in live, costly human arenas.

By decoupling instruction sets from evaluator configurations, JudgeArena facilitates controlled studies of meta-evaluation components and supports the continuous improvement of LLM evaluation practices.