GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledge
This paper introduces GPTKB v1.5, a cost-effective, 100-million-triple knowledge base constructed from GPT-4.1 via massive-recursive materialization, which enables systematic exploration, structured querying, and comparative analysis of factual LLM knowledge.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart robot that has read almost everything on the internet. You know it knows a lot of facts, but if you ask it, "What do you know about the Eiffel Tower?" it might tell you a few things. If you ask, "What about the Louvre?" it tells you more. But you have no idea what it doesn't know, or how deep its knowledge really goes, because you can only ask it one question at a time. It's like trying to map a continent by only looking at the spots you happen to walk past.
GPTKB v1.5 is a project that changes the game. Instead of asking the robot questions one by one, the researchers built a massive, interconnected map of everything the robot (specifically GPT-4.1) knows. Think of it as turning a scattered pile of trivia cards into a giant, searchable library where every fact is linked to every other fact.
Here is how they did it and what they found, explained simply:
1. The "Snowball" Method (How they built it)
The researchers didn't just ask the robot for random facts. They used a clever "snowball" technique:
- The Seed: They started with one specific thing, like "Vannevar Bush" (a famous engineer).
- The Expansion: They asked the robot, "Tell me everything you know about Vannevar Bush." The robot gave them a list of facts, like "He worked at X" or "He invented Y."
- The Chain Reaction: The robot mentioned new names in those answers (like "X" and "Y"). The researchers then immediately asked the robot about those new names.
- The Result: They kept doing this recursively, like a chain reaction, for 10 layers deep. This created a massive web of 100 million facts (called "triples") connecting 6.1 million different things.
It cost about $14,000 in computer time to build this entire library, which took about 18 days to complete.
2. Cleaning Up the Mess (The "Consolidation")
When you ask a robot the same question in different ways, it sometimes gives you slightly different answers or uses different words for the same thing (e.g., calling a person a "human" in one sentence and a "person" in another).
To fix this, the researchers ran a "cleaning" process. They grouped similar words together and merged duplicate facts. This turned a messy, redundant pile of data into a clean, organized database with clear categories.
3. What Can You Do With This Map?
The paper introduces a website where anyone can explore this massive knowledge base in three main ways:
- The "Wikipedia" Style Explorer: You can click on any item (like the "Merlion statue" in Singapore) and see a list of facts the robot knows about it. You can then click on the people or places mentioned in those facts to jump to their pages, exploring the robot's mind like walking through a museum.
- The "Detective" Query Tool: You can ask complex questions using a special language called SPARQL. For example, you could ask, "Show me the top 100 most common types of things this robot knows about." The robot's map revealed it knows the most about people, followed by movies, companies, and books.
- A Fun Discovery: The researchers checked if the robot's knowledge is fair. They asked, "If the robot knows 'A is married to B', does it also know 'B is married to A'?" They found that 84% of the time, it does not. The robot's knowledge is often one-sided, remembering one side of a relationship but forgetting the other.
- The "Taste Test" (Comparison): You can put different AI models side-by-side. The researchers compared GPT-4.1 against other models (like Llama and DeepSeek).
- Example: When asked about a famous AI researcher named Ilya Sutskever, the GPT model spitted out 40 facts, while the Llama model only gave 14.
- The Catch: Both models got his birth date wrong! This shows that even though one model knows more facts, they can both make the same silly mistakes.
4. How Accurate Is It?
The researchers didn't just trust the robot; they checked its work.
- They randomly picked 1,000 facts and checked them against the real world (using web searches).
- The Score: About 75.5% of the facts were true.
- The Problem: About 19.5% were false, and some were just too confusing to verify.
- The Bias: Because they started with specific "seed" names, the map is heavily weighted toward things the researchers thought to ask about. It's a massive map, but it's not a complete map of the entire universe of knowledge.
The Bottom Line
GPTKB v1.5 is a massive, $14,000 experiment that turned a "black box" AI into a transparent, searchable library. It proves that we can systematically map out what an AI knows, revealing that while these models are incredibly knowledgeable (holding 100 million facts), they are also prone to one-sided thinking and specific errors. It's a tool for researchers to stop guessing what the AI knows and start actually seeing it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.