Technical Summary: A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

Problem Statement

Traditional Large Language Model (LLM) benchmarks rely heavily on static datasets and objective scoring metrics (e.g., exact match against ground truth). While effective for binary correctness, these methods fail to capture qualitative differences in response quality when multiple valid answers exist. In real-world scenarios, the distinction between responses often lies in a spectrum of clarity, completeness, and usefulness rather than a strict correct/incorrect binary. Existing "LLM-as-a-Judge" approaches attempt to address this but often suffer from systematic biases (positional, verbosity, stylistic) and introduce interaction effects when models engage in debates or iterative discussions, potentially skewing judgments.

Methodology

The paper proposes a Consensus-Based Framework grounded in blind peer ranking to measure relative preference among model-generated responses. The core objective is to derive a Relative Intelligence Index (RII) that reflects how frequently a model's responses are preferred by a panel of diverse peer models, rather than measuring objective correctness.

Experimental Design

• Participants: Five state-of-the-art LLMs (Anthropic Claude Opus 4.6, OpenAI GPT-5.2, Google Gemini Pro 3.1, Grok-fast 4.1, and Mistral Large 2512) served dual roles as both Generators and Judges.
• Domains: Evaluation spanned five domains: Programming, General Knowledge, Safety, Logical Reasoning (Puzzles), and Mathematics.
• Scale: The study utilized 25 unique prompts (5 per domain) across 5 independent experimental runs, resulting in 3,125 total evaluations ($25 \times 5 \times 5 \times 5$).
• Configuration: All models operated at a temperature of 0.3 to balance stochasticity and diversity.

Evaluation Pipeline

The process followed five sequential phases:

1. Response Generation: Each model generated a response to a standardized prompt.
2. Anonymization & Randomization: Responses were stripped of model identifiers and assigned anonymous labels. Ordering was randomized per judge to mitigate positional bias.
3. Independent Ranking: Each model acted as a judge, independently ranking all anonymized responses for a given prompt based on relevance, correctness (where applicable), clarity, and conciseness. No interaction or debate between judges was permitted.
4. De-anonymization: Labels were mapped back to originating models.
5. Aggregation: Scores were aggregated to compute the RII.

Metric Definition

The Relative Intelligence Index (RII) for a model $i$ is defined as the average score assigned to its responses by peer judges across all prompts and runs:
$$RII_i = \frac{\sum_j S_{i,j}}{\sum_j 1}$$
Where $S_{i,j}$ is the score assigned by judge $j$ to model $i$. The metric is invariant to linear scaling and reflects ordinal preference.

Key Results

The study revealed consistent patterns of inter-model preference, though with notable domain-specific variations and stability trade-offs.

• Aggregate Preference: Claude achieved the highest mean RII (3.491), followed by Grok (3.382). Mistral scored the lowest (2.328).
• Domain Specificity:
  • Mathematics & Logic: Claude was consistently preferred, showing resilience across runs.
  • Programming: Claude maintained high average preference, though ChatGPT showed a significant spike in specific runs.
  • Safety: Scores clustered tightly among top models (Claude, ChatGPT, Gemini), suggesting shared heuristics for safety evaluation. Mistral consistently underperformed.
  • General Knowledge & Puzzles: Grok emerged as the most preferred model, particularly in lateral thinking tasks, outperforming Claude in several runs.
• Stability vs. Performance: A trade-off was observed between consistency and peak preference. Mistral and Gemini exhibited the highest consistency (lowest standard deviation), while ChatGPT showed the lowest consistency (highest variability).
• Latency Trade-offs: A practical trade-off was noted between preference scores and response speed. Mistral was the fastest (4,300ms), while Gemini and Claude exhibited significantly higher latency (14,600ms and 11,200ms, respectively).

Key Contributions

1. Blind Peer Ranking Framework: Introduces a scalable, model-driven method for comparative evaluation that minimizes interaction effects and brand bias by relying on independent, blind judgments.
2. Relative Intelligence Index (RII): Proposes a metric that quantifies inter-model preference alignment, offering a complementary perspective to traditional accuracy-based benchmarks.
3. Empirical Analysis of Inter-Model Dynamics: Provides data on how diverse LLMs rank one another across multiple domains, highlighting that "best" is often context-dependent and relative.

Significance and Claims

The paper positions this framework not as a replacement for human evaluation or a measure of objective intelligence, but as a proxy signal for perceived response quality in scenarios where multiple valid answers exist.

• Modest Claims: The authors explicitly state that RII reflects "inter-model preference alignment" influenced by shared training data and stylistic biases, rather than objective correctness or human judgment.
• Utility: The framework offers a reproducible, automated approach for rapid benchmarking in "gray area" tasks where correctness is not binary.
• Future Directions: The paper suggests future work should focus on integrating correctness filtering, expanding model diversity, and calibrating against human preferences to better understand the correlation between inter-model consensus and real-world user satisfaction.

The study concludes that while consensus-based evaluation does not solve the alignment problem, it provides a valuable, stable signal for understanding how models perceive the quality of their peers' outputs.