🤖 AI

Knowledge Index of Noah's Ark

The paper introduces KINA, a rigorously designed 899-item knowledge benchmark spanning 261 disciplines that employs greedy approximation for representativeness and a bonus-on-bar tournament for annotation quality, revealing a tiered performance landscape among 42 leading models with significant room for improvement below saturation.

Original authors: Sheng Jin, Minghao Liu, Yunze Xiao, Zeqi Zhou, Heli Qi, Yifan Yao, Meishu Song, Kaijing Ma, Xuan Zhang, Sicong Jiang, Yizhe Li, Ningshan Ma, Jie Wei, Ziniu Li, Minglai Yang, Bangya Liu, Yiming Liang

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sheng Jin, Minghao Liu, Yunze Xiao, Zeqi Zhou, Heli Qi, Yifan Yao, Meishu Song, Kaijing Ma, Xuan Zhang, Sicong Jiang, Yizhe Li, Ningshan Ma, Jie Wei, Ziniu Li, Minglai Yang, Bangya Liu, Yiming Liang, Xiao Fang, Qingcheng Zeng, Jiarui Liu, Rui Yang, Shen Yan, Wenhao Huang, Jiaheng Liu, Zihan Wang, Weihao Xuan, Ge Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to test how smart a group of students is, but instead of giving them a standard math test, you want to see if they truly understand the essence of 261 different subjects, from advanced engineering to obscure history.

The paper introduces KINA (Knowledge Index of Noah's Ark), a new, highly sophisticated "exam" designed to test Large Language Models (AI). The authors argue that previous exams were flawed in three main ways: they were too broad to be fair, they paid reviewers too little to care, and the results were too shaky to trust. KINA fixes these problems with a new design.

Here is how KINA works, explained through simple analogies:

1. The Problem: The "Lazy Survey" vs. The "Expert Map"

The Old Way: Imagine trying to map a continent by just throwing darts at a wall and seeing where they land. Previous AI tests did this: they grabbed thousands of questions from everywhere. Sometimes they hit a deep, important concept; other times, they hit a trivial fact. The result was a messy map that didn't show where the AI was actually weak.
The KINA Way: The authors treated the exam like building a Noah's Ark. They didn't just grab random animals; they carefully selected exactly 899 items to represent 261 specific "disciplines" (like a tiny, perfect ecosystem).

The Analogy: Instead of a dart throw, they used a "greedy selection" algorithm. Imagine you have a limited budget of space on a boat. You want to pick the animals that represent the most unique and important parts of the forest. KINA picks questions that act as "anchors" for each subject, ensuring the test covers the core of every field, not just the easy parts.

2. The Problem: The "Flat Wage" vs. The "Tournament"

The Old Way: Imagine hiring people to grade these exams and paying them $10 for every paper they check, regardless of how hard they work. A rational worker would do the bare minimum to get the $10, leading to "lazy consensus" where bad questions slip through.
The KINA Way: The authors introduced a "Bonus-on-Bar Tournament."

The Analogy: Imagine two judges grading the same question. They both get a base salary, but only the one who gives the higher score (provided it passes a strict quality threshold) gets a big bonus.
The Result: This creates a race to be the most careful and thorough. If a judge is lazy, they lose the bonus to their competitor. The paper proves mathematically that this system forces reviewers to work harder and catch more errors than the old "flat pay" system.

3. The Problem: The "One-Off Snapshot" vs. The "Stable Leaderboard"

The Old Way: If you test an AI on 100 questions, a 5-point difference in score might just be luck (like rolling dice). You can't be sure if Model A is truly better than Model B.
The KINA Way: The authors ran a "stress test" on their own results. They used a statistical technique called bootstrapping (imagine taking a photo of the leaderboard, shuffling the questions, and taking 1,000 new photos to see if the rankings hold up).

The Result: They found that the top rankings are very stable (like a sturdy ship), but the middle rankings are wobbly. They warn us not to obsess over tiny differences between models in the middle tier, as those gaps might just be noise.

What Did They Find? (The Race Results)

They tested 42 different AI models on this new exam. Here is the summary:

The Winners: The top AI (Gemini-3.1-Pro-Preview) got about 53% correct. The next two (Claude and GPT) were close behind at roughly 50%.
The Gap: Even the best AI is failing nearly half the questions. The exam is far from "solved."
The Surprise: The biggest differences between the smartest AIs weren't in math or science (where they all do okay). The real battleground was Humanities and Social Sciences (like History, Sociology, and Philosophy).
- Analogy: It's like a race where everyone runs fast on the paved road (Science), but the real winners are decided by who can navigate the muddy, tricky forest paths (Humanities).
The Tools: Giving the AIs access to a search engine helped, but not equally. It helped the weaker models fill in memory gaps and helped the stronger models verify facts, but it didn't magically make everyone perfect.

The Takeaway

KINA is not just a harder test; it's a better-designed test.

It ensures the questions actually represent the heart of a subject.
It pays reviewers in a way that forces them to be honest and thorough.
It admits that some rankings are shaky and tells us to stop over-interpreting small differences.

The paper concludes that while AI is getting smarter, there is still a massive amount of "headroom" for improvement, especially in understanding the complex, nuanced world of human culture and history.

Technical Summary: Knowledge Index of Noah's Ark (KINA)

Problem Statement

Current knowledge benchmarks for Large Language Models (LLMs) suffer from three primary design flaws:

Lack of Disciplinary Representativeness: Existing benchmarks often scale test sets by quantity (hundreds of subfields) or difficulty (research-frontier items) without formalizing whether items actually probe the core theoretical pivots of a discipline. This leads to uneven coverage of disciplinary competencies.
Incentive Misalignment in Annotation: Most pipelines use flat-payment per-item rates, which encourages "lazy consensus" among reviewers. Rational reviewers minimize effort when accepting borderline items incurs no penalty, leading to lower-quality annotations.
Ranking Instability: Compact test sets (approx. $10^3$ items) are cost-effective but prone to variance. Small performance gaps between frontier models (e.g., 5 points) may not survive resampling, yet benchmarks rarely report stability statistics, leading to over-interpretation of adjacent ranks.

Methodology

The authors introduce KINA (Knowledge Index of Noah's Ark), an 899-item benchmark spanning 261 fine-grained disciplines, designed around two formal theoretical guarantees and a rigorous construction pipeline.

1. Formal Guarantees

Representativeness via Greedy Coverage: The authors formalize disciplinary representativeness as "budgeted support centrality." Domain experts elicit a compact prototype $\Sigma_d$ $Σ_{d}$ of anchors (methods, theorems, concepts). Items are scored against these anchors, and a selection objective $F_d^{sp}(S)$ $F_{d}^{s p} (S)$ is defined to maximize coverage.
- Theoretical Result: The objective is proven to be monotone submodular. Consequently, a greedy selection algorithm under cardinality constraints achieves a $(1 - 1/e)$ approximation of the proxy optimum (Proposition 1). The authors clarify this is a guarantee on the proxy (operational coverage), not a global guarantee on population representativeness.
Incentive-Aligned Review via Tournament: To replace flat payment, the authors implement a bonus-on-bar tournament. Two reviewers evaluate each item; the reviewer with the higher validated score receives a bonus $B$ $B$ only if the score clears a threshold $\tau$ $τ$ .
- Theoretical Result: Under assumptions of effort-induced First-Order Stochastic Dominance (FOSD), score-noise independence, and monotone aggregation, the authors prove that this mechanism strictly FOSD-dominates the flat-payment baseline in terms of released review quality (Theorem 1). A closed-form condition for the bonus is derived: $B > \Delta C / \Delta p_{min}$ , where $\Delta C$ is the cost of high effort and $\Delta p_{min}$ is the minimum gain in winning probability.

2. Data Construction Pipeline

The pipeline consists of four stages:

Rule-Based Screening: Items must pass uniqueness checks (cosine similarity < 0.8), LaTeX formatting, and a difficulty filter requiring at least 3 of 5 flagship LLMs to fail.
Double-Blind Expert Review: Reviewers (graduate students and industry experts) are recruited via a two-round exam. They score items on six rubrics (representativeness, factuality, etc.) under the bonus-on-bar tournament with stochastic audits (5–10% of items) to deter collusion.
LLM-as-Judge Consensus: Three independent LLM judges score items on multidimensional features (coverage, uniqueness, impact) and analyze failure patterns. A majority vote determines admission.
Agentic Refinement: A "Diagnosis Agent" searches for counter-evidence to flagged items, and a "Refinement Agent" revises stems or premises. This loop addresses boundary defects (e.g., context drift) before final acceptance.

3. Evaluation Protocol

Format: Pseudo-multiple-choice questions with 10 combinatorial options (reducing chance accuracy to 10%).
Models: 42 frontier models from 13 labs (closed-source APIs, open-source dense/MoE, and reasoning models).
Metrics: AVG@4 accuracy, per-discipline breakdowns, tool-use augmentation (web search), and bootstrap-based ranking stability statistics.

Key Results

Performance Ceiling: The top model, Gemini-3.1-Pro-Preview, achieved 53.17% accuracy. Claude-Opus-4.6 (49.92%) and GPT-5.4 (48.55%) followed. The leaderboard is far from saturation, with a clear tiered structure: a frontier tier (>48%), a dense strong-model tier (38–45%), and low-performing models hovering just above the 10% chance baseline.
Ranking Stability: Bootstrap analysis (1,000 replications with 50% subsampling) showed high stability for the top-10 ranking (Kendall $\tau \approx 0.89$ ) and a 94% rank-1 retention rate. However, pairwise gaps below ~2 percentage points are not statistically resolvable.
Tool-Use Impact: Web search augmentation yielded positive but non-uniform gains (+1.50 to +5.17 points). The strongest and weakest models gained the most, suggesting retrieval serves distinct functions (filling parametric gaps vs. grounding evidence).
Disciplinary Discrimination: Analysis of the top-10 models revealed that humanities and social sciences (e.g., Sociology, Management, Literature & Arts) drive differentiation at the frontier, showing spreads of 30–38 points. In contrast, STEM fields (Science, Engineering) showed much smaller spreads (~10–14 points), suggesting frontier models have converged on these domains.
Scaling: Parameter scaling curves for Qwen families showed generational improvements (Qwen3.5 scaling at ~14.8 points/decade vs. 7.6 for Qwen3). However, log-linear saturation was observed at the upper end, indicating that scale alone may not resolve the humanities/social science gap.

Significance and Claims

The paper positions KINA not merely as a difficult test but as a diagnostic instrument. Its significance lies in:

Formalizing Design Principles: It provides the first formal guarantees for disciplinary representativeness (via submodular coverage) and incentive-aligned annotation (via tournament theory) in the context of LLM benchmarks.
Empirical Diagnostics: It reveals that current frontier models are not uniformly superior; their differentiation is heavily concentrated in humanities and social sciences, while they have largely converged on STEM tasks.
Methodological Transparency: By releasing bootstrap stability statistics, the paper explicitly discourages over-interpretation of fine-grained rank differences in compact benchmarks.
Resource Availability: The authors release the dataset, annotation manuals, LLM-judge rubrics, and evaluation code to enable replication and extension.

The authors explicitly state that the representativeness guarantee applies to the operational proxy, not the underlying population, and that the tournament mechanism's empirical validation is observational rather than a randomized causal verification. They commit to refreshing the benchmark (KINA-vx.0) when the strongest model exceeds 70% accuracy to prevent saturation.