💬 NLP

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

KNIGHT is a knowledge graph-driven framework that leverages large language models to efficiently generate high-quality, difficulty-controlled multiple-choice question datasets from external sources, thereby overcoming the cost and time bottlenecks of manual assessment creation for evaluating RAG systems.

Original authors: Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori, Farhan Farsi, Behnam Bahrak

Published 2026-06-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori, Farhan Farsi, Behnam Bahrak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher who needs to create a quiz for your students. Usually, you have to spend hours reading textbooks, picking out facts, and writing questions that are just the right difficulty. If you want to make a harder quiz, you have to start all over again.

The paper introduces a new tool called KNIGHT (Knowledge Graph-Driven Natural Item Generation with Adaptive Hardness Tuning). Think of KNIGHT not as a robot that writes questions from scratch, but as a smart librarian who builds a custom map before writing the quiz.

Here is how it works, broken down into simple steps:

1. The "Map" (The Knowledge Graph)

Instead of asking the AI to read a whole book every time it needs to write a question, KNIGHT first builds a simplified map of the topic.

The Analogy: Imagine you want to teach someone about "Biology." Instead of handing them a 500-page encyclopedia, the librarian draws a single sheet of paper with dots (entities like "cells," "DNA," "plants") and lines connecting them (relationships like "cells make up DNA").
Why this helps: This map is small, fast to read, and can be reused. Once the map is drawn for "Biology," the librarian can use it to make easy questions, hard questions, or tricky questions without ever needing to look at the big encyclopedia again.

2. Drawing the Path (Multi-Hop Questions)

To make a question, KNIGHT doesn't just pick one dot; it traces a path along the lines of the map.

Easy Question (Level 1): The path is short. It goes from Dot A to Dot B.
- Example: "What connects Biology to Cells?"
Hard Question (Level 3): The path is long and winding. It goes from Dot A to Dot B, then to Dot C, then to Dot D.
- Example: "If Biology leads to Cells, and Cells lead to DNA, and DNA leads to Genetics, which field studies the final step?"
The Magic: The system controls the difficulty simply by deciding how many "hops" (steps) to take on the map.

3. The "Quality Control" Inspector

Just because a robot can write a sentence doesn't mean it's a good question. KNIGHT has a built-in inspector (another AI) that checks every question against five strict rules before it's allowed into the final quiz:

Grammar: Is the English correct?
One Answer: Is there only one right answer? (No trick questions with two correct options).
Unique Options: Are the wrong answers (distractors) actually different from each other?
Truthful: Can the answer be found only using the map and the source text? (This stops the AI from making things up or "hallucinating").
On Topic: Does the question actually fit the subject?

4. Why It's Better Than Old Methods

The paper compares KNIGHT to other ways of making quizzes:

The "Plain" Way: Just asking an AI to "Write a biology question." This often leads to made-up facts or questions that are too easy.
The "RAG" Way: Giving the AI a long document to read every time. This is slow, expensive, and the AI still might get confused.
The KNIGHT Way: By using the Map, KNIGHT is cheaper and faster. It builds the map once, then generates hundreds of questions from that small map. It also produces questions that are harder and more logical because they are based on the actual connections in the map, not just random guessing.

The Results

The researchers tested KNIGHT on three subjects: History, Biology, and Math. They found that:

The questions were grammatically perfect and clear.
The "hard" questions were actually harder (students and AI models got them wrong more often), and the "easy" ones were easy.
The system made very few mistakes where the answer couldn't be found in the source material (which means it didn't "hallucinate" or lie).
It ranked AI models in the same order as famous, expensive benchmarks, proving it's a reliable way to test intelligence.

In short: KNIGHT is a system that draws a small, reusable map of knowledge and then uses that map to generate high-quality, difficulty-controlled quizzes quickly and cheaply, without needing to re-read entire books for every single question.

Technical Summary: KNIGHT

Problem Statement

The rapid advancement of Large Language Models (LLMs) has made them central to applications like Retrieval-Augmented Generation (RAG). However, evaluating these systems faces a significant bottleneck: the time, cost, and expertise required to build specialized, high-quality assessment datasets. Existing public evaluation datasets are often scarce due to proprietary restrictions, while standard benchmarks like MMLU are largely static, difficult to update, and lack fine-grained instructor control over per-topic difficulty or multi-hop reasoning structures. Furthermore, prior automated dataset generation efforts lack widely adopted, reproducible, open-source frameworks that can generate questions with controlled difficulty without repeatedly re-ingesting long source texts.

Methodology

The authors introduce KNIGHT (Knowledge-graph-driven Natural Item Generation with Adaptive Hardness Tuning), a fully automated framework designed to synthesize large-scale, topic-specific multiple-choice question (MCQ) datasets from external document collections and ontologies. The framework operates through a four-stage pipeline:

Knowledge Graph (KG) Construction: Given a user-defined topic $\tau$ , the system constructs a directed property graph $G=(V, E, R)$ via a retrieve-generate-filter loop.
- Retrieval & Synthesis: It retrieves ranked evidence from sources (e.g., Wikipedia) and uses an LLM to generate a structured, eight-point gloss (description) for the seed entity.
- Triple Induction: The system extracts subject-predicate-object triples from these descriptions, filtering for near-duplicates.
- Curation & Expansion: A curator module applies type checks (instantiated with Wikidata), Natural Language Inference (NLI) consistency checks, and content-policy screening. The graph expands breadth-first up to a user-defined depth limit $d_{max}$ , creating a compact, reusable representation of entities and relations.
MCQ Synthesis: The system generates questions by traversing multi-hop paths within the constructed KG.
- It samples paths of configurable length (depth) and orientation (forward or reverse).
- Node glosses and relation labels are verbalized into a context template, which prompts an LLM to generate a question stem, a single correct answer, and three semantically proximate distractors.
Difficulty Calibration: Difficulty is explicitly controlled via the path length (number of hops) and abstraction level within the KG. The system validates that longer paths correlate with increased model uncertainty (entropy) and human-perceived difficulty.
Validation & Filtering: A validator LLM scores candidate items against five criteria adapted from item-writing best practices:
- Grammatical fluency.
- Single-correct-answer unambiguity.
- Option uniqueness (non-overlapping distractors).
- Answerability from the provided evidence (serving as a proxy for hallucination).
- Topic relevance.
  Items failing any criterion are discarded.

The framework is instantiated using Wikipedia/Wikidata as the ontology but is designed to be domain- and ontology-agnostic. It utilizes GPT-4o-mini for all LLM calls to maintain a cost- and token-aware evaluation setting.

Key Contributions

Reusable KG Representation: KNIGHT introduces a method to build a topic-specific KG once and cache it, allowing for the generation of multiple question variants (different difficulty levels, formats, and targets) without repeatedly re-feeding long source documents. This significantly reduces token usage and cost compared to naive prompting or standard RAG baselines.
Adaptive Hardness Control: By leveraging graph depth and path structure, KNIGHT enables explicit control over question difficulty, facilitating the creation of curriculum-aligned evaluations with multi-hop reasoning requirements.
Comprehensive Evaluation Pipeline: The framework integrates RAG-based extraction, KG-guided generation, and LLM-based validation into a modular pipeline that mitigates hallucination and ensures pedagogical validity.
Open-Source Implementation: The authors provide a publicly available package on PyPI and GitHub, offering a reproducible, token-efficient alternative to static benchmarks.

Experimental Results

The authors evaluated KNIGHT across six datasets in History, Biology, and Mathematics, covering two difficulty levels (Level 1 and Level 3). They compared full KNIGHT against ablations: Plain (no evidence), RAG (evidence only), RAG+KG (evidence + KG, no validator), and RAG+Val (evidence + validator, no KG).

Quality Metrics: KNIGHT achieved high scores in linguistic quality (fluency, grammar) and significantly outperformed baselines in validity metrics. It produced the lowest rates of "unanswerable" items (hallucination proxy), ambiguous keys, and duplicate options.
Difficulty Calibration: Predictive entropy analysis using a fixed probe model (LLaMA 3.2-3B) showed that KNIGHT successfully generated harder items at Level 3 (higher entropy, lower probe accuracy) compared to Level 1. Crucially, increasing difficulty did not degrade item validity.
Distractor Quality: The KG-guided approach (RAG+KG) was identified as the primary driver for generating competitive distractors (higher entropy), whereas RAG alone or validation alone failed to induce meaningful difficulty separation.
Benchmark Utility: Model rankings on KNIGHT datasets aligned closely with established benchmarks like MMLU, ARC, and CSQA, suggesting the generated datasets capture general QA competence factors.
Efficiency: The framework demonstrated token efficiency, averaging ~600 tokens per question by reusing the cached KG, compared to the higher costs of re-ingesting long contexts in standard RAG approaches. Generation times on commodity hardware (Google Colab T4) were practical (minutes to ~40 minutes per dataset).

Significance and Claims

The paper positions KNIGHT not merely as a new dataset but as a scalable, low-cost benchmark generator. Its primary significance lies in addressing the bottleneck of dataset curation for LLM evaluation. By decoupling the expensive knowledge extraction phase (KG construction) from the generation phase, KNIGHT enables:

Rapid Refreshability: Unlike static benchmarks that require substantial expert effort to update, KNIGHT allows for frequent, topic-specific updates.
Instructor Control: It provides explicit control over difficulty and multi-hop structure, which is often lacking in broad-coverage static suites.
Cost-Effectiveness: It offers a token-efficient alternative to naive prompting pipelines, making high-quality evaluation accessible without prohibitive computational costs.

The authors conclude that KNIGHT complements existing broad benchmarks by offering a flexible, reproducible, and domain-adaptable tool for creating rigorous, fact-grounded evaluation sets. They acknowledge limitations regarding generalization to calculation-heavy domains (e.g., physics) and the residual presence of hallucinations, but assert that the framework substantially improves validity over retrieval-only approaches.