🤖 AI

The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

This paper introduces RiDiC, a configurable multilingual dataset of 3,000 entities across varying popularity tiers designed to evaluate and reveal hallucinations in long-form factuality generation by frontier LLMs.

Original authors: Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko, Andrey Sakhovskiy, Vasily Konovalov, Elena Tutubalina, Alexander Panchenko

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko, Andrey Sakhovskiy, Vasily Konovalov, Elena Tutubalina, Alexander Panchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart librarian (an AI) who can answer any question you ask. If you ask, "Who is the President of the United States?" or "What is the capital of France?", the librarian answers instantly and correctly. These are the "famous" questions, the ones everyone knows.

But what happens if you ask, "Tell me everything about the Little River of Gloom in a remote valley in Peru," or "Describe the specific features of a 1998 car model that only 50 people ever bought"?

This is where the story of RIDIC begins.

The Problem: The Librarian's "Famous Person" Bias

The researchers behind this paper noticed something tricky about AI. When an AI talks about famous things (like the Nile River or the Toyota Camry), it's usually accurate. But when it talks about obscure, "long-tail" things (like a tiny river or a rare car), it starts to make things up. It hallucinates. It sounds confident, but it's lying.

Most tests for AI only check if it knows the famous stuff. It's like testing a chef only on how well they can make a grilled cheese sandwich. You never know if they can actually cook a complex, rare dish until you ask them to.

The Solution: Building a "Popularity Menu"

The authors created a special tool (a pipeline) to build a new testing ground called RIDIC. Think of this as a menu designed specifically to test the AI's knowledge across the entire spectrum of popularity.

They didn't just pick random things. They carefully selected 3,000 items from three specific categories:

Rivers (Nature's highways)
Natural Disasters (Earth's tantrums)
Car Models (Human engineering)

For each category, they picked items in three "tiers" of fame:

The Head (The Stars): Famous things everyone knows (e.g., The Amazon River, Hurricane Katrina, the Honda Civic).
The Torso (The Middle Class): Moderately known things (e.g., a specific river in Michigan, a flood in Spain, a specific Cadillac model).
The Tail (The Ghosts): Extremely obscure things that very few people know about (e.g., a tiny creek in Russia, a minor landslide in 1977, a rare French car).

They did this in two languages: English and Chinese, to see if the AI struggled more in one language than the other.

The Experiment: Putting the AI to the Test

Once they built this menu, they asked three different AI "librarians" (Llama, Qwen, and GPT-5) to write a detailed story about each item on the list.

Then, they acted as strict fact-checkers. They compared the AI's stories against the "source of truth" (Wikipedia and other databases). They broke the stories down into tiny facts and checked each one: Is this true? Is this made up?

The Findings: What They Discovered

The results were like a report card for the AI, revealing some surprising grades:

The "Long-Tail" Trap: As the items got less famous, the AI got worse. It's like a student who aces the math test but fails the history test because they only studied the chapters everyone else studied. The AI struggled significantly with the "Tail" items, often inventing facts to fill the silence.
The Language Gap: The AI was much more accurate in English than in Chinese. It's as if the librarian has a massive, well-organized English library but only a few scattered, dusty pamphlets in Chinese. When the AI tried to write about obscure Chinese topics, it hallucinated even more.
The "Big Brain" Advantage: The biggest AI (GPT-5) was the most accurate, but even it wasn't perfect. The smaller AIs (Llama and Qwen) struggled much more with the rare items, showing that having a bigger "brain" (more parameters) helps with remembering obscure facts.
The Domain Difference: The AI was surprisingly good at talking about cars and disasters but terrible at talking about rivers. It seems the AI just knows more about cars and storms than about waterways!

Why This Matters

This paper is like giving the AI industry a new kind of mirror. Instead of just seeing how the AI looks when it's wearing its "famous" hat, we can now see how it looks when it's dealing with the obscure, the rare, and the difficult.

By releasing this dataset and the tools to create it, the authors are saying: "Don't just test your AI on the easy stuff. If you want to trust it with real-world problems, you need to know if it can handle the obscure stuff without making things up."

In short, RIDIC is a spotlight that shines on the dark corners of an AI's knowledge, ensuring that when the lights go down and the famous topics fade away, the AI doesn't start making up stories in the dark.

1. Problem Statement

Large Language Models (LLMs) have become primary tools for information retrieval, yet they frequently produce factual hallucinations (plausible but incorrect information). While short-form Question Answering (QA) benchmarks exist, they fail to capture the complexity of long-form generation, which is more common in real-world user interactions.

Key challenges identified in the paper include:

Popularity Bias: LLMs perform significantly better on "head" (popular) entities than on "tail" (rare/long-tail) entities, but existing datasets often lack controlled popularity distributions to systematically study this gap.
Multilingual Gaps: Evaluation of factuality in non-English languages (specifically low-resource or less-resourced languages) is under-researched and often unreliable due to a lack of high-quality reference data.
Evaluation Limitations: Most existing datasets are static, QA-based, or limited to specific domains (e.g., biographies), making them unsuitable for evaluating comprehensive, narrative responses across diverse domains and popularity tiers.

2. Methodology: The RIDIC Pipeline

The authors propose a configurable, multilingual pipeline to generate datasets with controlled entity popularity distributions. The process involves four main stages:

Entity Extraction (Wikidata):
- Target classes are defined (e.g., rivers, natural disasters, car models).
- SPARQL queries extract entities from Wikidata.
- Entities are filtered to ensure they have corresponding Wikipedia pages in target languages (English and Chinese) and are not "stubs" (too short/incomplete).
Popularity Metric Calculation:
- Popularity is proxied using Wikipedia pageview statistics (specifically 2024 data).
- Alternative metrics (page edits, incoming links, Wikidata triple counts) are also calculated to analyze correlations.
- Entities are stratified into three tiers based on cumulative pageviews: Head (top 1/3 of views), Torso (middle 1/3), and Tail (bottom 1/3).
Stratified Sampling:
- The pipeline samples entities to achieve a specific distribution across popularity tiers and geographic regions (Africa, Americas, Asia/Australia/Oceania, Europe).
- RIDIC Dataset Construction: The authors generated a dataset of 3,000 entities (1,000 per domain: Rivers, Natural Disasters, Car Models).
- Evidence Collection: For each entity, the pipeline collects the main Wikipedia page, top-10 Wikipedia search results, and linked pages (specifically for tail entities) to serve as ground-truth evidence.
Generation and Evaluation:
- Generation: Three LLMs (Llama-3-8B, Qwen2.5-7B, and GPT-5) generated long-form responses about the entities in both English and Chinese.
- Evaluation: A modified FActScore pipeline was used:
  - Atomic Fact Extraction: Breaking down responses into verifiable claims.
  - Evidence Retrieval: Matching facts against the collected Wikipedia evidence.
  - Verification: Using an LLM-as-a-judge to label facts as "supported" or "not supported."
- Metrics: Factuality precision (ratio of supported facts to total facts) and error analysis (e.g., river length accuracy).

3. Key Contributions

Configurable Pipeline: A flexible framework for generating multilingual datasets with controlled popularity distributions, allowing systematic evaluation of long-tail factuality.
The RIDIC Dataset: A new benchmark containing 3,000 entities across three distinct domains (Rivers, Disasters, Cars) in English and Chinese, explicitly stratified by popularity and geography.
Multilingual & Long-Form Analysis: The first large-scale study comparing long-form factuality across English and Chinese, revealing significant performance gaps and evaluation challenges in non-English contexts.
Open Resources: Release of code, data, and generation/evaluation scripts to facilitate reproducibility.

4. Key Results

The evaluation of LLMs on the RIDIC dataset yielded several critical insights:

Popularity Correlation: Factuality scores are strongly correlated with entity popularity.
- Head > Torso > Tail: All models showed a steep decline in accuracy for tail entities.
- Magnitude of Drop: Smaller models (Llama, Qwen) showed a ~2x drop in factuality from Head to Tail entities (e.g., Llama English Rivers: 0.58 $\to$ 0.25). GPT-5 was more robust but still showed a decline.
Multilingual Disparity:
- Chinese Performance: All models performed significantly worse in Chinese than in English (Total factuality: English 0.31–0.67 vs. Chinese 0.21–0.42).
- Evaluation Reliability: The FActScore evaluation pipeline itself was found to be less reliable in Chinese, with a 7.8% lower precision compared to human judgments, likely due to poorer evidence ranking and scarcer reference data.
Domain Sensitivity: Factuality varied by domain. Disasters and Cars generally received higher scores than Rivers, suggesting domain complexity impacts hallucination rates regardless of model size.
Geographic Bias: No clear continental bias was found, though performance varied by specific country/region.
Evidence Impact:
- English: Adding search results and linked pages improved performance for tail entities but sometimes degraded performance for head entities (likely due to noise).
- Chinese: Adding extra evidence reduced performance across all tiers, suggesting that retrieval mechanisms struggle to find relevant context in Chinese Wikipedia, introducing misleading information.
Lexical Diversity: Generations about rare entities exhibited lower lexical diversity (vocabulary growth), particularly for Disasters and Rivers.

5. Significance and Future Directions

Advancing Trustworthy AI: RIDIC provides a critical tool for diagnosing the "long-tail" failure modes of LLMs, which are often overlooked in standard benchmarks.
Highlighting Multilingual Gaps: The study underscores that current factuality evaluation methods and LLM capabilities are heavily skewed toward English and high-resource data, necessitating better multilingual evidence collection and evaluation frameworks.
Methodological Shift: The paper advocates for moving away from static, QA-only benchmarks toward dynamic, entity-driven, long-form evaluation pipelines that can adapt to different domains and popularity distributions.

Limitations Noted:

The dataset is biased toward entities with non-stub Wikipedia pages, potentially underrepresenting the extreme long tail.
Evaluation relies heavily on Wikipedia, which may not be the most up-to-date source for all domains (e.g., car models).
Experiments were limited to three models and two languages.

In conclusion, the RIDIC framework and dataset offer a rigorous, reproducible approach to understanding and improving the factual reliability of LLMs, particularly for rare entities and non-English languages.