💻 computer science

The Trust Paradox: How CS Researchers Engage LLM Leaderboards

Through interviews with eight CS researchers, this paper reveals a paradox where practitioners distrust LLM leaderboards yet rely on them as rough guides, ultimately finding that peer networks and arena-based rankings drive model selection more than static benchmarks while highlighting a critical demand for cost transparency.

Original authors: Pouya Sadeghi, Anamaria Crisan, Jimmy Lin

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pouya Sadeghi, Anamaria Crisan, Jimmy Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence research as a massive, high-stakes sports league. In this league, Leaderboards are the official scoreboards. They rank AI models (the "athletes") based on how well they perform on specific tests, like a standardized math exam or a reading comprehension quiz.

The common assumption is simple: If a model is at the top of the scoreboard, it's the best athlete.

However, a new study from the University of Waterloo reveals that the researchers actually playing the game don't believe the scoreboard tells the whole truth. In fact, they have a strange, contradictory relationship with it.

Here is the breakdown of their findings, using everyday analogies:

1. The "Trust Paradox": Knowing the Scoreboard is Flawed, But Checking It Anyway

The researchers interviewed eight computer scientists from different fields. They found a universal pattern called "Pragmatic Skepticism."

The Analogy: Imagine you are buying a car. You know the official "Car of the Year" magazine rankings are flawed because the magazine might be biased, or the tests don't reflect real driving conditions. You know the rankings are shaky.
The Reality: Even though these researchers deeply distrust the leaderboards, they still look at them. Why? Not to make the final decision, but to narrow down the list.
The Result: They treat the leaderboard like a "rough filter." It helps them eliminate the clearly bad options so they don't have to test 100 models, but they never trust the #1 spot as the absolute truth.

2. The Real "Coach": Peer Networks Over Scoreboards

If the researchers don't trust the scoreboard, who do they trust? Their friends and colleagues.

The Analogy: Think of it like choosing a restaurant. You could look at the official "Michelin Star" guide (the leaderboard), but you are much more likely to ask your trusted friend, "Hey, where did you eat last night that was actually good?"
The Finding: For 7 out of 8 researchers, the most important factor in choosing an AI model was a recommendation from a peer. They trust a colleague's personal experience ("I tried this, and it worked for my specific project") far more than a company's marketing or a static test score.
The Hierarchy of Trust:
1. Peers/Colleagues (Most trusted)
2. Third-party experts
3. The Companies making the AI (Least trusted, seen as biased advertisers)

3. The "Arena" vs. The "Written Exam"

The study found that researchers prefer one type of leaderboard over the other.

Static Benchmarks (The Written Exam): These are fixed tests where models answer the same questions over and over. Researchers are suspicious of these because companies can "study for the test" and cheat, or the test might be outdated.
Arena-Based (The Live Show): These are platforms where real humans vote on which AI response they like better in real-time (like a popularity contest).
The Verdict: Everyone preferred the "Arena." They felt it was more honest because it reflected how humans actually interact with the AI, rather than how well the AI memorized a specific test.

4. Not Everyone Plays the Same Game

The study revealed that the pressure to chase the top of the leaderboard depends entirely on what "team" (subfield) you are on.

The NLP Team (Natural Language Processing): These researchers are under immense pressure. It's like a high school sports league where if you don't have the highest GPA, you can't get into college. They must compare their work to the top-ranked models to get published.
The HCI & Privacy Teams: These researchers are like people playing a different sport entirely. They don't care about the leaderboard. They care about user experience, safety, or privacy. Being "top-ranked" on a math test doesn't matter if their AI is unsafe or confusing to use.
The Surprise: Some researchers in these other fields didn't even know the leaderboards existed! They were navigating the world of AI without ever looking at the scoreboard.

5. The Missing Piece: The Price Tag

If the researchers could redesign these leaderboards to make them actually useful, what would they add?

The Big Request: Cost Transparency.
The Analogy: Imagine a leaderboard that tells you a car gets 50 miles per gallon but doesn't tell you the car costs $200,000. It's useless information.
The Reality: Researchers need to know how much it costs to run these models. A model might be "smart," but if it costs a fortune to use, it's not practical. Seven out of eight researchers said this is the most important missing feature. They also wanted to know who was voting in the "Arena" to ensure it wasn't just a bunch of people from one specific country or culture.

Summary

The paper concludes that leaderboards are broken as "truth-tellers," but they are useful as "conversation starters."

Researchers don't blindly follow the rankings. They use them as a starting point, but they rely on their human network, personal testing, and cost calculations to make the real decisions. The study suggests that if we want these scoreboards to be more helpful, they need to stop pretending to be perfect and start showing the full picture: the cost, the specific strengths, and the human voters behind the scores.

Technical Summary: The Trust Paradox: How CS Researchers Engage LLM Leaderboards

Problem Statement

Large Language Model (LLM) leaderboards have become central artifacts in modern AI research, aggregating model scores on standardized tasks to provide rankings. These platforms generally take two forms: static benchmark leaderboards (e.g., MMLU, Open LLM Leaderboard) which rank models on fixed test sets, and arena-based leaderboards (e.g., Chatbot Arena) which aggregate human preference votes.

Despite their visibility, technical literature has established significant limitations regarding their reliability and robustness. Studies indicate that leaderboard rankings are sensitive to minor perturbations (e.g., tweaking evaluation prompts or removing small fractions of data), susceptible to strategic manipulation (e.g., vote rigging or targeted testing), and that different ranking algorithms (e.g., Elo vs. Glicko) can produce divergent orderings from the same data.

However, a critical gap remains: while the technical fragility of leaderboards is documented, it is empirically uncharted how these platforms actually shape researchers' daily practices, decision-making processes, and trust judgments. It is unclear whether researchers treat leaderboard positions as definitive proxies for progress or if they distrust and selectively use these signals. This study addresses this gap by examining the socio-technical role of leaderboards across different computer science subfields.

Methodology

The study employs a qualitative, interpretivist approach grounded in reflexive thematic analysis (RTA).

Participants: Eight computer science researchers were recruited across four subfields: Natural Language Processing (NLP, $n=3$ ), Human-Computer Interaction (HCI, $n=2$ ), Systems/Privacy ( $n=2$ ), and an interdisciplinary "Others" category ( $n=1$ ). The sample included PhD students, a master's student, and postdoctoral researchers.
Data Collection: Data was gathered through semi-structured interviews (30–45 minutes) preceded by a background questionnaire and followed by a member-check process where participants reviewed summaries of their interviews for factual accuracy.
Procedure: Participants were shown a visual typology of leaderboard designs to ground discussions. Interviews explored model selection strategies, attitudes toward leaderboard trustworthiness, and the influence of rankings on research framing.
Analysis: Transcripts were analyzed using Braun and Clarke's six-phase RTA process. The primary researcher generated 210 coded segments across 71 unique codes, organized into seven analytical dimensions and six themes. Inter-coder reliability was not pursued; instead, reflexivity was maintained through analytic memo-writing and participant member-checks.

Key Contributions

The paper makes three primary contributions:

Empirical Evidence of the "Pragmatic Skepticism" Paradox: The study identifies a near-universal stance where researchers express deep distrust of leaderboard rankings yet continue to utilize them as rough decision-making aids.
Cross-Subfield Comparison of Influence: The research demonstrates that leaderboard influence is mediated by disciplinary culture rather than individual attitudes. For instance, NLP researchers face significant state-of-the-art (SoTA) comparison pressure, whereas HCI and Systems/Privacy researchers report little to no such expectation.
Actionable Design Recommendations: The study translates researcher needs into concrete infrastructure improvements, moving beyond assumptions about evaluation to address actual usage patterns.

Key Results

1. The Trust Paradox and Usage Patterns

Contrary to a spectrum of enthusiastic adoption to total rejection, all eight participants exhibited pragmatic skepticism.

Distrust: Participants acknowledged that static benchmarks can be "learned" or gamed, and that arena rankings can be manipulated. They recognized that rankings are not definitive measures of quality.
Usage: Despite this distrust, participants used leaderboards as "weak priors" to narrow their search space. They did not treat rankings as ground truth but as a starting point for further investigation.
Primary Selection Mechanism: Peer networks, not leaderboards, emerged as the dominant mechanism for model selection. Seven of eight participants relied primarily on peer recommendations (lab conversations, social media, trusted experts). One NLP researcher noted, "90% of my usage is based on recommendation."
Experience Over Evidence: Personal, hands-on trial-and-error testing was prioritized over quantitative rankings. Participants often engaged in "satisficing," accepting models that were "good enough" rather than seeking the top-ranked option.

2. Disciplinary Differences in Engagement

The influence of leaderboards varied sharply based on subfield:

NLP: Researchers faced high pressure to compare against SoTA models due to publication norms. They were heavily engaged with leaderboards but remained skeptical of their stability.
HCI and Systems/Privacy: Researchers in these fields reported no leaderboard-driven publication pressure. Their evaluation criteria (e.g., latency, user experience, formal privacy guarantees) often fell outside standard benchmark metrics. Notably, five of the eight participants (including both HCI and Systems/Privacy researchers) had little to no prior awareness of specific leaderboard platforms, highlighting a discoverability failure for non-NLP audiences.

3. Trustworthiness and Missing Metrics

Arena vs. Static: There was universal consensus (8/8) across all subfields that arena-based (human-voting) leaderboards were preferred over static benchmark leaderboards. Participants found arena results more interpretable and aligned with actual user experience.
Cost Transparency: The most demanded missing feature, raised by seven of eight participants, was cost transparency. Researchers criticized the lack of pricing or efficiency data, noting that cost-per-query is often a binding constraint for academic researchers.
Granularity: Participants requested task-specific score breakdowns (e.g., separate scores for math, coding, multilingual) rather than single aggregate numbers.
Transparency: There were calls for voter demographic disclosure to assess bias, as well as information on model provenance and institutional conflicts of interest.

Significance and Claims

The paper argues that LLM leaderboards should be understood not merely as technical infrastructure but as socio-technical artifacts embedded in research culture.

Social Embedding: The study challenges the view that better metrics alone will improve adoption. It posits that model selection is deeply social, driven by peer trust networks that filter and interpret (or ignore) leaderboard signals.
The Paradox of Utility: The persistence of leaderboard use despite widespread distrust suggests their value lies not in accuracy, but in their function as coordination devices that reduce an overwhelming search space.
Ethical and Structural Concerns: The authors highlight that current leaderboard culture encodes resource inequality, penalizing researchers without industry-scale compute budgets who cannot compete on SoTA benchmarks. Furthermore, the lack of transparency in governance and the potential for benchmark gaming pose ethical risks, particularly for practitioners outside NLP who may lack the expertise to critically evaluate benchmark methodologies.

The study concludes that evaluation infrastructure must be redesigned to align with how researchers actually use it, incorporating cost data, task-specific granularity, and transparency regarding voting demographics and governance, rather than relying on single-score rankings that fail to capture the heterogeneous utilities researchers require.