💬 NLP

REAP: Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction from LLMs

The REAP system, built on a 24B-parameter Mistral model without fine-tuning, achieves a macro-F1 score of 0.62 in the AKBC Shared Task 2026 by employing structured chain-of-thought reasoning, relation-specific queries, and a reasoning-based empty-set gate to elicit and parse parametric knowledge into valid JSON arrays.

Original authors: Thanh-Dan Bui, Thanh-Trung Do, Tuan-Phong Nguyen

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thanh-Dan Bui, Thanh-Trung Do, Tuan-Phong Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a friend who has read almost every book in the world and memorized a staggering amount of facts, from the capital of France to the number of stars in a specific galaxy. This friend is a "Large Language Model" (LLM). Usually, when you ask them a simple question like "Who won the Nobel Prize?", they give you a quick answer. But what if you asked them to list every single person who has ever won that prize, or to tell you exactly which countries share a land border with a tiny island nation? Suddenly, the task gets much harder. The friend might forget a few names, make up a fake winner, or get confused about whether an island even has neighbors.

This is the challenge of "Knowledge Base Construction." Scientists want to turn these giant, fact-filled brains into organized databases that computers can use reliably. The tricky part is that the friend (the AI) doesn't have a library to look up; they have to rely entirely on what's inside their head (their "parameters"). If the answer is "nobody," the friend needs to know to say "empty" instead of guessing. If the answer is a long list, they need to get every single item right without dropping any. This paper tackles the problem of how to ask these AI friends the right questions to get perfect, organized lists without letting them hallucinate or get confused.

The REAP System: A Detective's Guide to AI Memory

Meet REAP (Relation-Aware Elicitation and Parsing). Think of REAP not as a single question-asker, but as a clever detective team working with a very smart, but sometimes scatterbrained, witness (the AI model). The goal? To build a perfect, organized list of facts from the witness's memory without letting them make things up.

The researchers found that if you just ask the AI, "List all the countries bordering X," it might get overwhelmed, forget a few, or accidentally invent a border that doesn't exist. It's like asking a student to recite a whole chapter of a textbook in one breath; they might stumble. Instead, REAP breaks the job down into two distinct stages, like a two-step interview process.

Stage 1: The Detective's Interrogation (Reasoning)
In the first stage, REAP doesn't just ask for the answer. It asks the AI to think before it speaks. Depending on the type of question, the detective uses a special strategy:

For Borders: Instead of asking "Who borders this?", the AI is told to act like a geography expert and scan the map in four directions (North, South, East, West) to make sure no small neighbor is missed.
For Awards: If the AI needs to list award winners, it doesn't just guess. It breaks time down into decades, asking, "Who won in the 1970s? Who won in the 1980s?" This ensures the AI doesn't skip a whole era.
The "Empty Set" Gate: This is a crucial safety net. Sometimes the answer is "nobody" (like a living person who hasn't died yet, or a private company that isn't traded on the stock market). REAP teaches the AI to recognize when a property doesn't exist and confidently say, "The list is empty," rather than making up a fake name.

Stage 2: The Scribe's Cleanup (Parsing)
Once the AI has finished its thinking and generated a messy list of facts, Stage 2 kicks in. This is the scribe who cleans up the notes. The system uses strict rules to grab the answers and format them perfectly into a JSON list (a standard computer format). If the AI gets confused and writes a sentence instead of a list, the scribe tries to fix it. If it can't be fixed, the system tries the question again with a simpler prompt.

What They Found

The researchers tested this system using a specific AI model called Mistral-Small-24B-Instruct-2501, which has about 24 billion "brain cells" (parameters). They were working under strict rules: no looking up information on the internet, no changing the AI's brain (no fine-tuning), and using a model with no more than 32 billion parameters.

The results were quite promising. On a test set of difficult questions, the REAP system achieved a macro-F1 score of 0.62. To put that in perspective, this was significantly better than the organizers' baseline system (which scored 0.30) and better than using other popular AI models like Llama or Gemma with the same method.

The system shined brightest on specific types of questions:

Country Borders: It got a score of 0.95, meaning it was almost perfect at listing neighboring countries.
Stock Exchanges: It scored 0.73 for finding where companies are traded.
Area: It scored 0.77 for calculating the size of geographical areas.

However, the system wasn't perfect everywhere. It struggled a bit more with venue capacity (scoring 0.23), often confusing a small local stadium with a famous one that has a similar name. The researchers suggest this is because the AI's internal memory isn't always precise enough for very specific numbers or rare, long-tail facts.

What This Means (and What It Doesn't)

The paper suggests that the key to getting better answers from AI isn't just having a bigger brain, but asking the right questions in the right way. By forcing the AI to "think step-by-step" (using Chain-of-Thought reasoning) and giving it specific strategies for different types of facts, the system can pull out more accurate information.

However, the authors are careful to note that this isn't a magic bullet. The system still relies entirely on what the AI has memorized. If the AI doesn't know a fact about a very rare entity (like a small island or a minor award), the system can't invent it. Also, because the AI is running on specific computer chips (TPUs), there is a tiny bit of randomness in the results, though it's small enough not to change the big picture.

In short, REAP shows that with a little bit of structure and a lot of patience, we can coax much more reliable facts out of our AI friends, turning their chaotic memories into neat, usable lists.

Technical Summary: REAP – Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction

Problem Definition

The paper addresses the AKBC Shared Task 2026, which focuses on constructing knowledge bases from Large Language Models (LLMs) under strict constraints. The core challenge is Closed-Book Knowledge Base Construction (KBC): systems must operate without retrieval-augmented generation (RAG) or external knowledge, using models with at most 32 billion parameters and no fine-tuning.

Unlike standard factual probing which often assumes a single correct answer, this task requires handling variable cardinality for subject-relation pairs $(s, r)$ . The output must be a valid JSON array representing:

Empty sets ( $\emptyset$ ): e.g., a country with no land borders (islands) or a living person.
Single-valued objects: e.g., the city of death for a deceased person.
Multi-valued objects: e.g., all stock exchanges a company trades on or all recipients of an award.

The difficulty lies in simultaneously recalling potentially long lists of facts, handling precise quantitative values, and adhering to a strict JSON schema without hallucinating entities or omitting valid answers.

Methodology: The REAP System

The authors propose REAP (Relation-Aware Elicitation and Parsing), a two-stage pipeline designed to decouple factual elicitation from answer serialization.

1. Architecture Overview

Stage 1 (Generation & Reasoning): The LLM generates evidence or reasoning chains tailored to specific relations. If the output is unparsable, the system employs retry mechanisms or fallback prompts.
Stage 2 (Parsing & Extraction): The system extracts the final answer into a JSON array. This is primarily done via deterministic regular expression parsing (handling ~80% of records) to ensure format compliance. LLM-based extraction is reserved as a fallback for complex outputs.

2. Relation-Specific Prompt Design

REAP employs specialized strategies for the six target relations, moving beyond generic prompting:

awardWonBy (Chronological Multi-Pass): To improve recall for awards with hundreds of recipients, the system uses six distinct prompts: one general pass and five auxiliary passes covering specific decades (inception–1970s, 1980s–1990s, 2000s–2010s, 2020+, and a pass for non-Western/less prominent recipients).
countryLandBordersCountry (Four-Direction Scan): The model acts as a geography expert, explicitly scanning the subject's East, West, South, and North boundaries to capture small neighbors and detached territories, while explicitly excluding maritime borders.
companyTradesAtStockExchange (Four-Step Check): Includes a "Strict Public Check" (Empty-Set Gate) to identify private companies or subsidiaries, returning an empty set if the entity is not publicly traded.
hasArea & hasCapacity: These use multi-step CoT reasoning to disambiguate entities (e.g., distinguishing a specific venue from a city-wide namesake), normalize units (converting to $km^2$ ), and verify values against plausible ranges.
personHasCityOfDeath: A five-step reasoning process that first checks if the subject is alive (triggering an empty set) before resolving name ambiguity and retrieving the death city.

3. Key Components

Reasoning-Based Empty-Set Gate: Inspired by selective prediction, the system uses CoT to explicitly decide if an answer exists. If evidence is insufficient or the property does not apply (e.g., a living person), the model outputs FINAL_ANSWER: [] rather than hallucinating a value.
Hybrid Parsing: The system prioritizes deterministic regex extraction over LLM generation for the final JSON format to minimize schema violations.
Post-processing: Includes robust array extraction, numeric normalization (removing thousands separators), and noise filtering (removing honorifics like "Dr." or "Sir" and parenthetical qualifiers).

Experimental Setup

Models Evaluated: Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-Small-24B-Instruct-2501 (selected as the primary model).
Hardware: TPU v5e-8 with vLLM serving.
Data: Training, validation, and test splits provided by the task organizers, covering all three cardinality cases (0, 1, N).
Metrics: Macro-F1, Micro-F1, Precision, and Recall. Quantitative relations use a 5% relative tolerance.

Results

Performance on Validation and Test Sets

Zero-Shot Baseline: Without the REAP pipeline, models struggled significantly (Macro-F1 $\approx$ 0.38–0.43), highlighting the difficulty of simultaneous fact recall and format adherence.
REAP Pipeline Impact: The two-stage pipeline significantly improved performance across all models.
- Mistral-24B achieved the best results, improving from 0.43 (zero-shot) to 0.65 on the validation set.
- On the official test set, the Mistral-24B system achieved an overall Macro-F1 of 0.62.
Relation-Specific Performance:
- Strongest: countryLandBordersCountry (F1 = 0.95), companyTradesAtStockExchange (F1 = 0.73), and hasArea (F1 = 0.77).
- Weakest: hasCapacity (F1 = 0.23), where confusion between similarly named venues and strict tolerance thresholds led to errors.
Comparison: The system more than doubled the organizer's baseline (Qwen3.5-9B, F1 = 0.30) and outperformed Llama-3.1-8B and Gemma-2-9B pipelines.

Ablation Study

The study confirmed the necessity of each component:

Removing Task Decomposition (multi-pass queries) caused a drop of 0.06 in Macro-F1, with severe recall loss on multi-valued relations.
Removing the Empty-Set Gate dropped Macro-F1 by 0.08 due to hallucinations on unanswerable entities.
Removing CoT Reasoning entirely caused the largest drop (0.22), reducing performance to zero-shot levels, proving that structured reasoning is the primary driver of success.

Significance and Claims

The paper claims that structured reasoning is essential for reliably eliciting parametric knowledge in closed-book KBC settings. Key takeaways include:

Decoupling is Effective: Separating the "thinking" (elicitation) from the "formatting" (parsing) significantly reduces hallucinations and schema violations.
Relation-Awareness Matters: Generic prompting fails on variable-cardinality tasks; specialized strategies (geographical scans, chronological passes) are required to maximize recall.
Model Capacity Limits: While reasoning prompts help, there is a clear performance gap between 9B and 24B models, suggesting that larger models benefit more from structured pipelines, particularly for long-tail entities.
Limits of Parametric Knowledge: The system acknowledges that even with advanced prompting, knowledge for very rare entities remains incomplete, and quantitative precision is limited by the model's internal representation.

The authors conclude that REAP demonstrates a viable path for constructing knowledge bases from LLMs without fine-tuning or external retrieval, provided that the system employs rigorous, relation-specific reasoning and deterministic parsing.