🤖 AI

LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation

This paper introduces LivingArena, an automated, contamination-resistant evaluation framework where LLMs iteratively probe each other's knowledge boundaries through adversarial question generation and answering, yielding a stable leaderboard that reveals specific failure modes and higher-order probing abilities distinct from static benchmarks and human preferences.

Original authors: Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the smartest computers on the planet are constantly taking tests to see who is the "best." For a long time, scientists have used static exams—like a giant, unchanging multiple-choice quiz that everyone takes. But there's a problem: these quizzes get old fast. Once a computer learns the answers, it just memorizes them, and the test stops being useful. It's like trying to measure a runner's speed by having them run on a treadmill that never gets faster; eventually, they all just hit the top speed of the machine, and you can't tell who is actually the fastest.

To fix this, researchers are looking for a new way to test these "Large Language Models" (LLMs)—the super-smart AI brains that write stories, solve math problems, and write code. Instead of giving them a fixed test, what if we let the computers test each other? This paper asks a fascinating question: Do these AI brains know things that other AI brains don't? If one AI can figure out exactly where another AI is weak and ask a question that trips it up, we might finally have a way to see who is truly the smartest, without needing humans to write thousands of questions or worry about the computers cheating by memorizing the test bank.

The Living Arena: A Never-Ending Game of "Gotcha!"

Meet LivingArena. Think of it not as a classroom, but as a high-stakes, automated debate club where ten of the world's smartest AI models are locked in a tournament. Instead of a human teacher handing out worksheets, the AIs take turns being the Questioner and the Answerer.

Here's how the game works:

The Trap: One AI (the Questioner) has to invent a tricky question, provide the correct answer, and explain why that answer is right.
The Safety Check: Before the other AI even sees the question, a panel of three "Judge" AIs checks to make sure the question isn't nonsense, the answer is actually correct, and the logic holds up. If the Questioner makes a mistake or tries to trick the judges, they get penalized. It's like a referee blowing a whistle for a foul.
The Showdown: If the question passes the check, the second AI (the Answerer) tries to solve it.
The Score: If the Answerer gets it wrong, the Questioner wins a point. If the Answerer gets it right, they win a point.

The goal isn't just to be smart; it's to be a detective. The smartest players aren't just the ones who know the most facts; they are the ones who can figure out exactly what their opponent doesn't know and then hit them with a question about that specific weakness.

The Big Discovery: "I Know Where You're Weak"

The researchers ran this tournament with 10 different top-tier AI models, creating 360 matches and 3,600 rounds of play. They found something amazing: The AIs really do know what the others don't.

When the models played against each other, they didn't just ask random hard questions. They started to "read" their opponents. If an AI stumbled on a logic puzzle, the next time they played, the other AI would immediately ask another logic puzzle. They were essentially saying, "Hey, you failed at this last time; let's see if you can do it now."

The paper shows that these models can localize weaknesses. For example, one top model (GPT-5.5) was so good at this that after it figured out an opponent was bad at a specific type of reasoning, it doubled down on that topic, hitting the opponent's weak spot 52% of the time in subsequent rounds. It's like a chess player noticing their opponent always blunders when a knight is on the board, and then immediately moving their knight there every single time.

However, there is a limit. The paper found that an AI cannot ask a question that is harder than its own brain can handle. If an AI doesn't understand a concept, it can't invent a tricky question about it. It's like a student who doesn't know calculus can't write a tricky calculus exam to stump their teacher. The questions are always bounded by the Questioner's own knowledge.

The Scoreboard: Who Won?

The tournament produced a stable ranking (called an Elo leaderboard) that separated the models into five distinct tiers.

The Champion: GPT-5.5 took the top spot. It was perfect at answering questions (100% accuracy) and very aggressive at finding weaknesses (26.3% hit rate).
The Passive Players: Some models, like the Claude family, were great at answering questions but very shy about asking them. They rarely tried to stump their opponents, which kept their overall score in the middle.
The "Self-Harm" Penalty: This was a crucial part of the game. If a Questioner asked a bad question (one with a wrong answer or a logical flaw), they lost a point. This forced the AIs to be honest and careful. Models that were overconfident or "hallucinated" facts (made things up) got punished heavily. For instance, GPT-5.2 had a very high "self-harm" rate (42.7%), meaning it kept asking bad questions and hurting its own score, which dropped it to the bottom of the leaderboard.

Why This Matters

This paper suggests that peer probing—letting AIs test each other—is a better way to measure "real" intelligence than old-fashioned tests.

The researchers found that this new method measures something different than what humans usually prefer. In human-voting arenas, models that sound polite and write long, fancy answers often win. But in LivingArena, those fancy styles didn't help. Instead, the winners were the models that were factually precise, could admit when they didn't know something, and were aggressive about finding the truth.

The study concludes that while we can't build a test that is harder than the smartest AI in the room, we can build a system that gets harder as the AIs get smarter. As the models improve, they will naturally come up with harder and harder questions for each other, creating a "living" benchmark that never gets boring and never gets saturated. It's a self-updating race where the finish line keeps moving, ensuring we can always tell who is truly the fastest runner.

Technical Summary: LivingArena – Peer-Probing as Scalable Evaluation

1. Problem Statement

The evaluation of frontier Large Language Models (LLMs) faces three compounding challenges:

Contamination and Saturation: Static benchmarks (e.g., MMLU, MMLU-Pro) suffer from data leakage into training sets and performance saturation, where top models cluster near ceiling scores, rendering them unable to distinguish between leading systems or identify specific failure modes.
Subjectivity of Human Preference: Human preference arenas (e.g., LMArena) bypass static authoring but measure subjective helpfulness rather than objective truth. Crowd raters often struggle with highly specialized or technical answers.
High Cost and Latency: Manual benchmark creation is expensive and slow to refresh, often becoming obsolete before the next model generation is released.

The authors posit a fundamental question: Do LLMs know what other LLMs don't? If models can spontaneously uncover and exploit each other's weaknesses, this dynamic could replace laborious human test suite construction with a self-adaptive, living benchmark.

2. Methodology: The LivingArena Framework

LivingArena is a fully automated, contamination-resistant evaluation framework that instantiates a peer-probing paradigm. It operates as a zero-sum adversarial game where models take turns acting as a Questioner and an Answerer.

Match Structure

Tournament Format: A round-robin tournament involving 10 frontier models (from OpenAI, Anthropic, Google, and DeepSeek) results in 45 unique pairs. Each pair plays 8 games (4 bidirectional matchups), totaling 360 matches and 3,600 rounds.
Round Flow (10 rounds per match):
1. Question Generation: The Questioner generates a question ( $q_t$ ), a gold standard answer ( $a^*_t$ ), and verification logic ( $v^*_t$ ) within a strict JSON schema.
2. Independent Validation: A panel of three distinct model judges (GPT-5.5, Gemini-3.5-Flash, Claude-Sonnet-4.6) blindly validates the tuple. If any judge votes "INVALID" (due to logical flaws, calculation errors, or ambiguity), the question is rejected.
  - Penalty: The Questioner receives a self-harm penalty ($-1.0$) for invalid questions.
3. Answering: If validated, the Answerer responds to the question.
4. Grading: The same judge panel grades the Answerer's response against the gold standard using a binary vote (CORRECT/WRONG).
  - Scoring: If the Answerer is correct, they gain $+1.0$ . If incorrect, the Questioner gains a "hit" point.

Scoring Mechanisms

Diminishing Returns: To prevent models from exploiting a single weakness repeatedly, a decay function $w(C)$ reduces the reward for hitting the same capability dimension ( $D_1$ – $D_6$ ) multiple times. Switching dimensions resets the multiplier.
Bonus Rounds: Rounds 9 and 10 are unconstrained bonus rounds with no decay, allowing models to concentrate fire on discovered weaknesses.
Elo Aggregation: Net scores from bidirectional matches are mapped to pairwise outcomes ($0.0, 0.5, 1.0$) and used to update global Elo ratings using a standard logistic curve ( $K=32$ ).

Capability Dimensions

The framework evaluates models across six dimensions:

Quantitative Reasoning
Code & Algorithms
Natural Sciences
Humanities & Social Sciences
Language & Linguistics
Logic & Abstract Reasoning

3. Key Contributions

LivingArena Framework: An automated, self-adaptive evaluation system that requires no static question bank and remains discriminative as models improve.
Industrial Utility: A continuous, low-cost regression harness (a 10-model tournament runs in ~100 minutes) that transforms peer probing into actionable, per-dimension failure diagnostics.
Empirical Evaluation: A comprehensive tournament of 10 frontier LLMs yielding a stable Elo leaderboard that cleanly separates near-saturated systems.
Open Source: Release of complete evaluation code and raw logs.

4. Results and Behavioral Analysis

Frontier Ranking

GPT-5.5 dominates the leaderboard (Elo 1743), exhibiting a perfect balance of answering accuracy (100%) and aggressive, effective questioning (26.3% hit rate).
Tiered Performance: The models separate into five distinct tiers. Notably, the Claude family (Tier 4) shows high answering accuracy (88–89%) but is a passive questioner (2–4% hit rate), resulting in a mid-tier ranking despite strong answering capabilities.
Calibration: Models with high self-harm rates (e.g., GPT-5.2 at 42.7%, Claude family at 31–38%) correlate with lower rankings, indicating that self-calibration is a critical factor in this paradigm.

Behavioral Validity (Do LLMs know what others don't?)

Boundary Localization: Models cannot generate questions beyond their own knowledge boundaries (self-play accuracy is ~98%), but they successfully identify and exploit peers' weaknesses.
Exploit-After-Hit: After a successful hit in a specific dimension, the probability of the Questioner targeting the same dimension in the next round increases significantly (+16.3% over baseline). Strong models (e.g., GPT-5.5) double down on discovered vulnerabilities, increasing their conditional hit rate from 25% to 52% after the first hit.
Strategic Bonus Rounds: In bonus rounds (9–10), models deliberately concentrate on previously discovered weak dimensions (41.4% probability vs. 18.2% baseline), achieving a 38.8% hit rate on these targeted attacks compared to 4.5% on non-targeted dimensions.

Comparison with Human Preference

Weak Correlation: LivingArena Elo ratings correlate weakly with human preference rankings (Spearman's $\rho = 0.358$ ).
Divergent Axes: Human preference favors polite, verbose formatting (benefiting Claude models), whereas LivingArena rewards objective rigor, factual precision, and aggressive probing. Peer probing measures a distinct axis of adversarial calibration that human voting misses.

Judge Reliability

The judge panel (composed of active contestants) shows high agreement (Fleiss' $\kappa = 0.915$ for grading).
Analysis indicates that self-bias is negligible when controlling for item difficulty, suggesting that multi-judge consensus effectively mitigates strategic favoritism.

5. Significance and Claims

The paper claims that LivingArena offers a scalable, low-cost alternative to static benchmarks by leveraging the dynamic interaction between models.

Diagnostic Power: It reveals that "asking" and "answering" are distinct capabilities. A model can be a strong answerer but a weak questioner (and vice versa), a nuance hidden by aggregate accuracy metrics.
Self-Calibration: The framework introduces "self-harm rate" as an interpretable metric for model calibration, which static accuracy overlooks.
Dynamic Difficulty: Unlike frozen question banks, the difficulty of LivingArena regenerates as models improve; as models become stronger, they naturally pose harder questions to one another.
Limitations: The authors acknowledge that peer probing is bounded by the participating models (a shared blind spot cannot be probed) and that finer resolution within tiers requires more games, trading off against API costs.

In conclusion, the paper argues that peer probing provides a robust signal for objective rigor and adversarial calibration, offering a path forward for evaluating frontier models in an era where static benchmarks are saturated and human preference is subjective.