Are Large Language Models Truly Smarter Than Humans?

This paper presents a rigorous multi-method audit of six frontier large language models, revealing that their apparent superiority over humans on public benchmarks is significantly inflated by data contamination, with memorization rates varying by domain and causing substantial performance drops when tested on indirect or paraphrased questions.

The Gist

Imagine a group of students taking a very important exam to prove they are geniuses. The test is called the "MMLU," and it covers everything from law and medicine to physics and philosophy.

Recently, AI models (like the super-smart chatbots you might have heard of) have been getting near-perfect scores on this exam. People are saying, "Wow, these AIs are smarter than human experts!"

But this paper asks a very simple, nagging question: "Did these students actually study the material, or did they just memorize the answer key?"

The authors of this paper decided to investigate three different ways to find out if the AI is truly "smart" or just "cheating" by having seen the test questions before. Here is what they found, explained simply.

The Three Investigations

1. The "Google Search" Test (Experiment 1)

The Idea: If a student memorized the test, the questions must be somewhere on the internet.
The Method: The researchers took 513 questions from the exam and Googled them. They checked if the exact questions and answers were floating around online in textbooks, lecture notes, or blog posts.
The Result: They found that 13.8% of the questions were definitely "contaminated" (meaning the AI likely saw them during training).

• The Analogy: It's like finding out that 1 in 7 questions on a final exam were posted on a public forum where the teacher accidentally left the answer key.
• The Surprise: The "STEM" subjects (Science, Tech, Engineering, Math) were the most contaminated. Even worse, in subjects like Philosophy, up to 66% of the questions were found online. It seems the AI didn't just learn the concepts; it memorized the specific questions.

2. The "Rewrite" Test (Experiment 2)

The Idea: If a student truly understands a concept, they can answer it even if the teacher rewrites the question. If they just memorized the answer, they will get confused if the words change.
The Method: The researchers took 100 questions and rewrote them.

• Original: "What is the capital of France?"
• Rewritten: "Which city serves as the seat of government for the French Republic?"
  They asked the AI the same questions but with different wording.
  The Result: When the words changed, the AI's scores dropped significantly.
• The Analogy: Imagine a student who can recite the poem "The Road Not Taken" perfectly. But if you ask them, "What is the poem about the traveler who took the less traveled path?" they freeze. They memorized the words, not the meaning.
• The Big Drop: In Law and Ethics, the AI's score dropped by nearly 20%. This suggests that in these critical fields, the AI was mostly just recognizing the specific phrasing of the question, not actually understanding the legal or moral reasoning.

3. The "Fill-in-the-Blank" Test (Experiment 3)

The Idea: If the AI has memorized the test, it should be able to fill in the blanks of a question it has seen before, even if you hide a word.
The Method: They took the questions and covered up a key word or a wrong answer choice. They asked the AI to guess what was hidden.
The Result: The AI was able to guess the hidden parts 72.5% of the time.

• The Analogy: It's like showing a student a sentence with a blank: "The capital of France is [____]." If they say "Paris" without thinking, they didn't just guess; they remembered the sentence structure.
• The Weird Anomaly: One model, DeepSeek-R1, was strange. It couldn't remember the exact words (it couldn't fill in the blank perfectly), but it could guess the idea of the missing word. It was like a student who forgot the exact vocabulary but remembered the general story. This explained why it did poorly on the "Rewrite" test but still seemed to know the material.

The Big Picture: What Does This Mean?

The paper concludes that AI is not necessarily "smarter" than humans yet; it is just very good at recognizing patterns it has seen before.

1. The "Exam" is Leaked: The tests we use to measure AI intelligence are like open-book exams where the answers are posted on the internet. The AI has read the internet, so it has seen the test.
2. Memorization vs. Understanding: High scores on these tests often mean the AI has memorized the "script" of the question, not that it has the deep, flexible reasoning of a human expert.
3. The Danger: If we trust these AI models for real-world jobs (like being a lawyer or a doctor) based on these scores, we are in trouble.
   • The Metaphor: Imagine hiring a pilot who has scored 100% on a simulator test, but only because they memorized the specific flight path of the test. If the wind changes or a new obstacle appears (a "paraphrased" question), they might crash.

The Takeaway

The authors aren't saying AI is useless. They are saying we need to stop pretending these public tests are a fair measure of "human-level intelligence."

To truly know if AI is smart, we need to give it a test it has never seen before, with questions written in a way it has never encountered. Until we do that, the high scores we see on leaderboards might just be the sound of a very good parrot repeating what it heard, rather than a genius thinking for itself.

Technical Summary

1. Problem Statement

Public leaderboards and benchmarks (e.g., MMLU) increasingly claim that Large Language Models (LLMs) surpass human experts in domains like law, medicine, and STEM. However, these benchmarks are static and fully public, meaning their questions and solutions are widely available on the internet. This creates a high risk of evaluation data contamination, where models are trained on the very data used to evaluate them.

The core problem is that high benchmark scores may reflect memorization and surface-pattern recognition rather than genuine general intelligence or causal reasoning. Current evaluation practices often fail to distinguish between a model that has "seen the exam" and one that truly understands the underlying concepts, leading to inflated performance claims and unsafe deployment in high-stakes fields.

2. Methodology

The authors conducted a rigorous, multi-method contamination audit on six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. The study utilized three complementary experiments, all reproducible via public APIs and open benchmarks.

Experiment 1: Lexical Contamination Detection (External)

• Goal: Detect if MMLU questions exist in the public web (a proxy for pretraining data).
• Method: Sampled 513 MMLU questions (9 per subject across 57 categories). Used Tavily web search to retrieve top-5 snippets for the first 150 characters of each question.
• Criteria: A question was flagged as contaminated if:

  1. 30% of its 8-grams appeared in the search snippets.

  2. The correct answer appeared verbatim in the snippets.
• Metric: Calculated Estimated Performance Gain (EPG) using the ConTAM methodology to estimate how much contamination artificially inflated scores.

Experiment 2: Paraphrase and Indirect-Reference Diagnostic (Behavioral)

• Goal: Test if models rely on surface-form familiarity rather than domain knowledge.
• Method: Selected 100 questions across 6 subjects (spanning high and low contamination categories). Generated two variants for each:
  1. Paraphrased: Identical knowledge requirement but entirely different wording.
  2. Indirect-Reference: Key entities described via properties/events rather than direct names.
• Metric: Measured the accuracy drop (Original vs. Variants). A purely reasoning-based model should show zero drop; a memorization-based model should show significant degradation.

Experiment 3: TS-Guessing Behavioral Probe (Internal)

• Goal: Detect internal memorization without relying on external web indices.
• Method: Applied Testset Slot Guessing (TS-Guessing) to all 513 questions.
  • Task A (Option Mask): Masked a wrong answer option and asked the model to reconstruct the original text. (Random baseline for exact match is ~0%).
  • Task B (Word Mask): Masked a content word in the question stem and asked for reconstruction.
• Criteria: Flagged as contaminated if the model achieved >50% token overlap on Option Mask or exact match on Word Mask.

3. Key Contributions

1. Triangulated Evidence: The paper provides the first study to converge three independent detection methods (lexical search, accuracy degradation, and internal reconstruction) to prove contamination is pervasive.
2. Quantification of Inflation: Estimates that contamination artificially inflates accuracy by +0.030 to +0.054 points depending on the category, with STEM showing the highest gains.
3. Discovery of "Distributed Memorization": Identified a novel memorization signature in DeepSeek-R1, where the model reconstructs the semantic structure of wrong answers (high partial overlap) without retaining exact verbatim text (zero exact recall).
4. Subject-Level Mapping: Demonstrated that contamination is not uniform; STEM and Professional domains are significantly more contaminated than Humanities or Social Sciences.

4. Key Results

Experiment 1 Results (Lexical)

• Overall Contamination Rate: 13.8% of sampled MMLU questions.
• Category Breakdown:
  • STEM: 18.1% (Highest)
  • Professional: 12.7%
  • Social Sciences: 12.1%
  • Humanities: 10.3%
• Extreme Cases: Philosophy reached 66.7% contamination; Anatomy 55.6%.
• Impact: Removing contaminated items would reduce reported STEM accuracy by ~5.4 percentage points.

Experiment 2 Results (Behavioral Degradation)

• Average Accuracy Drop: Models dropped 7.0 percentage points on indirect-reference variants.
• Domain Sensitivity:
  • Law & Ethics: Dropped 19.8 pp (matching their high contamination in Exp 1).
  • History & Math: Dropped only ~2–5 pp.
• Model Anomaly: DeepSeek-R1 had low baseline accuracy (0.292) but minimal drop on indirect variants (-0.032), suggesting it does not rely on verbatim surface matching.

Experiment 3 Results (Internal Memorization)

• Overall Flagging Rate: 72.5% of questions triggered memorization signals across all models.
• DeepSeek-R1 Anomaly:
  • Option Mask (Partial): 76.6% (Highest among all models).
  • Option Mask (Exact): 0.0%.
  • Word Mask (Exact): 0.0%.
  • Interpretation: DeepSeek-R1 stores the conceptual structure of MMLU questions (knowing what wrong answers should look like) but not the exact text. This explains its resilience to paraphrasing in Experiment 2.
• Convergence: Both Exp 1 (Web) and Exp 3 (Internal) independently ranked STEM as the most contaminated category.

5. Significance and Implications

Scientific Validity

The study challenges the validity of current "human-level" or "superhuman" claims. If 72.5% of benchmark questions trigger memorization signals and accuracy drops significantly when surface forms change, current scores reflect familiarity with the test set, not generalizable intelligence.

Policy and Regulation

• Disclosure: Model providers have an obligation to disclose contamination analyses alongside performance claims.
• High-Stakes Deployment: In fields like Law and Medicine, where the paper found the largest accuracy drops (19.8 pp) under indirect referencing, reliance on contaminated benchmarks is dangerous. Models may hallucinate case law or drug interactions when faced with novel phrasing.
• New Standards: Regulatory bodies should mandate:
  • Evaluation on decontaminated/proprietary benchmarks.
  • Behavioral contamination audits (like TS-Guessing).
  • Robustness testing across paraphrased and indirect-reference variants.

Future Directions

The paper calls for the creation of private, difficulty-calibrated benchmarks authored after model training cutoffs. It also highlights the need to investigate the "distributed memorization" signature (seen in DeepSeek-R1) to understand if it is an architectural feature or a training artifact, as it represents a harder-to-detect form of contamination.

Conclusion

The paper concludes that LLMs are not necessarily "smarter" than humans on these benchmarks; they have simply "seen the exam." The convergence of three independent methods provides robust evidence that benchmark contamination is a structural, pervasive, and practically consequential issue that invalidates many current claims of AI superiority.