Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards

1. Problem Statement

Large Language Model (LLM) leaderboards (e.g., LMArena/Chatbot Arena) are the primary mechanism for comparing models and guiding deployment decisions. However, the authors identify a critical asymmetry:

• Fixed Evaluation Priorities: Rankings are determined by the dataset composition and aggregation methods chosen by a small group of benchmark designers, not by the diverse goals of actual users.
• The "Aggregate" Fallacy: A single aggregate score (e.g., Elo rating) obscures significant performance variations across different prompt types, domains, and task complexities.
• Contextual Mismatch: Real-world users often care about specific subsets of data (e.g., math reasoning for an education app) rather than global averages. Current leaderboards fail to support this context-specific evaluation, potentially leading to suboptimal model selection.

The core question posed is: Who defines "best"? The paper argues that evaluation should shift from a passive consumption of fixed rankings to an active, interactive sensemaking process where users define their own evaluation priorities.

2. Methodology

The research employs a three-pronged methodology: Dataset Analysis, Interface Design, and Qualitative User Study.

A. Dataset Analysis (LMArena)

The authors conducted an in-depth analysis of the Human Preference 140K dataset (LMArena), comprising 135,634 pairwise judgments across 53 LLMs.

• Topic Hierarchy Construction: They built a three-level semantic topic hierarchy (8 top-level, 53 mid-level, 400 fine-grained categories) using a pipeline of GPT-5 mini for initial clustering and GPT-5 for higher-level organization, refined manually.
• Disaggregated Performance Analysis: They computed win rates for models across specific slices and calculated Spearman rank correlations between slice-specific rankings and the global ranking.
• Task-Type Investigation:
  • Deterministic Tasks (Math): Analyzed 2,773 math problems to see if human preference aligns with correctness. They found that even when both models were correct, humans chose a winner 56% of the time, heavily influenced by explanation styles (conciseness, structure, derivation) rather than just accuracy.
  • Value-Dependent Tasks (Politics/History): Analyzed 1,158 prompts involving political disputes. They found that "pluralistic" (neutral) vs. "non-pluralistic" (position-taking) responses did not show a statistically significant preference difference, yet rankings fluctuated wildly based on the evaluator composition, revealing structural bias in aggregated preference signals.

B. Interactive Visualization Interface (Design Probe)

To address the findings, the authors designed a web-based interactive tool (React frontend, Python Flask backend) with two coordinated views:

1. Category/Slice Selection (Left Panel): A hierarchical tree allowing users to select, exclude, and weight specific prompt categories. Users can inspect individual prompt examples to verify category relevance.
2. Model Ranking View (Right Panel): A dynamic heatmap table showing model performance (win rates) across the selected slices.

• Interaction: Users can adjust weights and observe immediate changes in rankings. The interface highlights outliers (e.g., a model performing exceptionally well in a specific slice but poorly globally) and allows drilling down into specific model-category intersections.

C. Qualitative User Study

• Participants: 10 practitioners (AI engineers, data analysts, researchers) with an average of 5.5 years of experience.
• Procedure: Participants used the interface to select the best model for a specific scenario (e.g., an educational chatbot) from a pool of 10 models.
• Protocol: Think-aloud sessions followed by interviews to understand how they interpreted the data and made decisions.

3. Key Contributions

1. Reframing Evaluation: Proposes shifting LLM leaderboard evaluation from a static, aggregate metric to an interactive, user-defined sensemaking process.
2. Empirical Evidence of Instability: Demonstrates through LMArena data that:
   • Dataset composition is heavily skewed (30% toward coding/AI topics).
   • Model rankings vary significantly across slices (e.g., Claude models rank low globally but top 5 in "Data Processing & Analysis").
   • Preference-based judgments often fail to capture correctness in deterministic tasks or handle value-laden disagreements in political tasks.
3. Interactive Design Probe: Introduces a visualization interface that enables users to construct custom evaluation datasets by weighting prompt slices, making the "black box" of leaderboard aggregation transparent.

4. Key Results & Findings

From Dataset Analysis

• Skewed Composition: The dataset is dominated by developer-centric topics, potentially misrepresenting performance for general users.
• Ranking Volatility: Global rankings are poor proxies for specific domains. For instance, minimax-m1 ranked 19th globally but 1st in math due to its superior handling of derivational reasoning, a trait underweighted in the global aggregate.
• Style over Substance: In math tasks, human preference correlated more with explanation style (structure, elaboration) than with raw correctness.
• Value Conflicts: In political/historical tasks, there is no consensus on whether "pluralistic" or "position-taking" responses are better. Aggregated rankings in these areas reflect the dominant values of the specific evaluator group rather than objective quality.

From User Study

• Challenging Assumptions: Participants frequently discovered that their prior beliefs about model superiority (based on global leaderboards) were incorrect when viewed through their specific context (e.g., a smaller model outperforming a flagship in a niche task).
• Granular Relevance: Users found value in drilling down to subcategories to exclude irrelevant data (e.g., removing advanced physics for a high-school tutoring bot).
• Grounding Decisions: Users relied on inspecting concrete prompt examples to validate why a model performed well, moving beyond abstract scores.
• Active Calibration: Participants actively negotiated weights and sample sizes, treating the tool as a decision aid rather than an oracle. They recognized that small sample sizes in specific slices required caution.

5. Significance and Implications

• Transparency and Accountability: The work highlights that "best" is a value-laden concept dependent on data composition. Interactive tools make these dependencies visible, allowing stakeholders to audit why a model is ranked highly.
• Responsible AI Deployment: By enabling context-specific evaluation, organizations can avoid deploying models that are globally "good" but fail in their specific, high-stakes use cases.
• Limitations of Aggregation: The paper argues that a single metric cannot capture the nuance of LLM capabilities. Future leaderboards should support disaggregated, multi-metric, and user-weighted evaluations.
• Future Directions: The authors suggest extending this paradigm to support multiple metrics (e.g., switching from win-rates to accuracy for math tasks) and handling dynamic, continuously evolving datasets.

In conclusion, the paper argues that the era of the "single global leaderboard" is insufficient for the diverse needs of the AI ecosystem. Instead, evaluation must become an interactive, iterative process where users actively define the scope and weight of their evaluation criteria to make informed, responsible deployment decisions.