SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

The paper introduces SimpleQA Verified, a rigorously filtered 1,000-prompt benchmark designed to overcome the limitations of OpenAI's original SimpleQA by providing a more reliable evaluation of LLM factuality, on which Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6.

The Gist

Imagine you are trying to test how well a student knows the world without letting them use a textbook or the internet. You want to see what's actually stored in their brain (their "parametric knowledge").

For a while, the AI community used a test called SimpleQA to do this. But, much like a high school exam that was written by a single teacher who had a weird obsession with 19th-century poets and kept asking the same question in three different ways, the original test had some major flaws. It was noisy, biased, and sometimes the answer key was just wrong.

Enter SimpleQA Verified. Think of this as the "Gold Standard" re-examination of that test, created by Google DeepMind to fix the mess.

Here is the story of how they cleaned it up, explained with some everyday analogies:

1. The "Garbage In, Garbage Out" Problem

The original SimpleQA was like a bucket of fruit that had some rotten apples, some double-counted oranges, and a whole section dedicated only to pineapples.

• The Issue: The original questions were too similar (redundant), came from a narrow set of sources (biased), and sometimes had the wrong answers (noisy labels).
• The Fix: The researchers acted like a team of strict but fair librarians. They went through the bucket and:
  • Removed Duplicates: If two questions were basically the same (e.g., "When was City A founded?" and "When was City B founded?" when they were just variations of the same fact), they kept only the hardest one.
  • Respected the Owners: Some websites had put up "Do Not Disturb" signs (robots.txt) saying, "Hey AI, don't use our content to train your brain." The researchers respected that and removed questions relying on those sites.
  • Balanced the Menu: They made sure the test wasn't just about Science and Math. They added questions about Art, Sports, and Geography so the test measured general knowledge, not just one specific skill.

2. The "Fuzzy Math" Problem

One of the biggest headaches in grading AI is numbers.

• The Old Way: If the answer was "8,282" and the AI said "8,333," the old grading system would mark it Wrong. That's like failing a student because they estimated the distance to the moon as 238,900 miles instead of 238,855 miles.
• The New Way: The researchers introduced a "Margin of Error" rule. Now, if the answer is a number, they give the AI a little wiggle room (like a target with a bullseye and a surrounding ring). If the AI hits the ring, it gets credit. This makes the test fairer and less about memorizing exact digits and more about knowing the right ballpark.

3. The "AI Teacher" Upgrade

To grade these tests, they use another AI (an "autorater") to act as the teacher.

• The Problem: The old teacher was easily confused. If the AI student gave the right answer but added a long, polite explanation or said, "I think it's X, but maybe Y," the old teacher would get confused and mark it wrong.
• The Fix: They gave the teacher a new rulebook. Now, the teacher is told: "Ignore the fluff. Just look for the core answer. If the student hedges their bets (says 'maybe'), mark it as 'not attempted' so they can't game the system by guessing wildly."

The Results: Who Passed?

They ran the new, stricter, fairer test on the world's smartest AI models (like GPT-5, Claude Opus, and Gemini).

• The Winner: Gemini 2.5 Pro took the top spot with a score of 55.6%.
• The Context: Don't panic if 55% sounds low! Remember, this is a hard test. It's like a PhD qualifying exam. Getting 55% means the AI knows a lot, but it still hallucinates (makes things up) about 45% of the time.
• The Takeaway: The other models (like GPT-5 and Claude) did well, but Gemini 2.5 Pro was the only one to consistently beat them on this specific, cleaned-up test.

Why Should You Care?

Think of SimpleQA Verified as a new, high-quality speedometer for AI.

• Before, the speedometer was broken; it told you the car was going 100 mph when it was actually going 60, or it got confused by rain.
• Now, we have a reliable speedometer. This helps researchers see real progress. Are AI models actually getting smarter at remembering facts, or are they just memorizing the answers to the old, flawed test?

By releasing this new benchmark, Google is saying: "Here is a clean, honest ruler. Let's measure the future of AI with it, so we can build systems that are truly trustworthy and don't just make things up."

Technical Summary

Here is a detailed technical summary of the paper "SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge."

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in critical applications where factual accuracy is paramount. However, existing benchmarks for evaluating parametric factuality (knowledge stored directly in model weights) suffer from significant limitations that obscure genuine progress:

• Noisy and Incorrect Labels: Original datasets often contain ground truth errors or ambiguous answers.
• Topical and Source Biases: Benchmarks like OpenAI's SimpleQA (2024) were derived from a narrow distribution of sources, leading to over-representation of specific topics (e.g., science/tech) and answer types (e.g., dates).
• Redundancy: High levels of semantic and lexical overlap between questions allow models to "overfit" to specific patterns rather than demonstrating true knowledge retrieval.
• Evaluation Instability: Automated graders (autoraters) struggle with nuanced model responses, such as those containing extra context, approximations, or hedging, leading to inconsistent scoring.

These issues create a noisy evaluation signal, making it difficult to distinguish between genuine improvements in factual recall and models simply memorizing benchmark quirks.

2. Methodology: Constructing SimpleQA Verified

The authors curated SimpleQA Verified, a 1,000-prompt benchmark derived from the original 4,326-question SimpleQA dataset. The curation process involved a rigorous, multi-stage pipeline designed to eliminate bias, redundancy, and errors while increasing difficulty.

A. Data Curation Pipeline

1. Unique Source Documents: Removed questions sharing the same reference URL to prevent bias toward specific human raters. This reduced the dataset by 28.5%.
2. De-duplication:
   • Semantic De-duplication: Used Gemini Embeddings (cosine similarity > 0.77) to remove semantically identical questions.
   • TF-IDF De-duplication: Used TF-IDF vectors (cosine similarity > 0.4) to remove lexically overlapping questions.
   • Result: Reduced dataset by an additional 13.6%.
3. Respecting Web Publisher Choices: Removed questions referencing URLs from publishers that explicitly block AI training data via robots.txt (e.g., Google-Extended, OpenAI, Anthropic controls). This ensured ethical compliance and reduced the dataset by 30.4%.
4. Diversity Balancing: Re-balanced the distribution of Answer Types (e.g., Date, Person, Number) and Topics (e.g., Art, Geography, Sports) to prevent skewing results toward specific model strengths. This reduced the dataset by 34.3%.
5. Source Reconciliation:
   • Non-Numeric: Removed ambiguous questions where sources conflicted or pointed to multiple distinct answers.
   • Numeric: Used code execution to verify if sources fell within a 5% margin of error of the ground truth.
6. Increasing Headroom (Difficulty Filtering): To ensure the benchmark remained challenging, the authors filtered out questions that leading frontier models (GPT-4o, Gemini 2.0 Flash, Claude 3.7 Sonnet) could answer correctly. They retained only the most difficult samples.
7. Manual Review & Metadata: Corrected invalid URLs, refined date precision, and enriched metadata to classify questions requiring reasoning (3.7%) or multi-step logic (7.3%).

B. Autorater Improvements

The paper identifies that standard automated grading fails on complex response patterns (e.g., adding extra info, hedging, or providing approximate numbers). The authors improved the autorater prompt by:

• Explicit Numeric Ranges: Instead of requiring exact matches, ground truth answers for numeric questions are rewritten with acceptable ranges (e.g., ±1% or ±5%) based on a specific taxonomy (Exact Match, Small Margin, Large Margin).
• Hedging Guidelines: Clarified that if a model hedges (offers multiple candidates), it must commit to one answer to be graded; otherwise, it is marked as "not attempted."
• Context Ignorance: Instructed the grader to ignore extra information provided by the model if the core answer is correct.

3. Key Contributions

• SimpleQA Verified Dataset: A high-fidelity, 1,000-question benchmark that eliminates redundancy, balances topics, and ensures ethical source usage.
• Robust Evaluation Protocol: A refined autorater prompt that handles numeric approximations, hedging, and verbose responses more reliably than previous standards.
• Open Resources: The dataset, evaluation code, and a public leaderboard are released on Kaggle to foster reproducibility and community tracking of factuality progress.

4. Results

The benchmark was evaluated on a suite of frontier models (Gemini, GPT, Claude, DeepSeek) without tool use (search).

• State-of-the-Art Performance: Gemini 2.5 Pro achieved the highest F1-score of 55.6%, outperforming other top models including GPT-5 (52.3%) and o3 (51.9%).
• Benchmark Sensitivity:
  • Models like GPT-4o and Claude Opus 4 showed statistically significant performance drops on SimpleQA Verified compared to the original SimpleQA, indicating the original benchmark may have been easier or biased in their favor.
  • Conversely, some models (e.g., o4-mini) showed slight improvements, suggesting the cleaning process removed noise that previously hindered them.
• Difficulty: The "headroom" filtering ensured that even top models struggled, with the best model answering only ~55% of questions correctly, demonstrating the benchmark's effectiveness in differentiating capabilities.

5. Significance

• Reliability: By removing incorrect labels and redundant questions, SimpleQA Verified provides a cleaner signal for measuring true parametric knowledge, reducing the risk of models overfitting to benchmark artifacts.
• Ethical Alignment: The exclusion of sources that opt-out of AI training respects publisher rights, setting a precedent for future dataset curation.
• Future-Proofing: The improved autorater methodology offers a more robust framework for evaluating LLMs as they become more verbose and nuanced in their generation.
• Community Impact: The release of this benchmark allows the research community to track genuine progress in reducing hallucinations and improving factual recall, a critical step toward deploying trustworthy AI in enterprise and critical sectors.