🤖 AI

Verifiable Knowledge Expansion through Retrieval-Grounded Formal Concept Analysis

This paper proposes a retrieval-augmented small language model framework that integrates Formal Concept Analysis as a symbolic verification loop to validate and expand ontological knowledge, demonstrating improved relation and implication accuracy in a rare ataxia domain through iterative seed-based exploration and counterexample detection.

Original authors: Yujin Yang, Heejung Lee

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yujin Yang, Heejung Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a massive, perfect library of medical knowledge about rare diseases. Specifically, you want to organize a collection of "Ataxia" diseases (conditions that cause uncoordinated movement) and list exactly which symptoms (like tremors or speech issues) belong to which disease.

Doing this manually is like trying to write a dictionary by hand while blindfolded; it takes forever, and experts make mistakes. Using a standard AI chatbot to do it is risky because the AI might "hallucinate" (make up facts) or confidently state something that isn't true.

This paper proposes a clever, three-part system to build this library safely and verifiably. Think of it as a construction crew with three distinct roles:

1. The Architect (Formal Concept Analysis - FCA)

Imagine a strict, logical architect who doesn't know anything about medicine but is an expert in rules and patterns.

What they do: They look at the list of diseases and symptoms you have so far and ask, "If a patient has Symptom A and Symptom B, do they always have Symptom C?"
The Catch: The architect doesn't just guess. They treat every new rule as a "hypothesis" that must be proven. If the architect suggests a rule, they demand a counterexample. They ask, "Show me one single disease that has A and B but does not have C." If you can't show one, the rule is accepted. If you can, the rule is rejected, and the architect learns from that mistake.

2. The Librarian (Retrieval-Augmented Generation - RAG)

The architect needs facts, but they can't just rely on their own memory. Enter the Librarian.

What they do: When the Architect asks a question (e.g., "Does Disease X have Symptom Y?"), the Librarian goes to the source material (medical definitions and records) and pulls out the exact text that answers the question.
Why it matters: This stops the AI from making things up. The Librarian ensures that every answer is grounded in actual evidence found in the documents, not just a guess.

3. The Junior Assistant (Small Language Model - SLM)

The Librarian has the text, but they need someone to read it and make a quick "Yes" or "No" decision.

What they do: This is a smaller, cheaper, and faster AI. Its only job is to look at the text the Librarian found and say, "Yes, the text confirms this symptom," or "No, the text doesn't support this."
Why it matters: Using a giant, expensive AI for every single check would be too slow and costly. This "Junior Assistant" is efficient enough to handle thousands of these tiny "Yes/No" checks quickly.

How They Work Together (The Loop)

The paper describes a cycle that repeats 20 times:

Start Small: They begin with a few known symptoms (seeds).
Ask a Question: The Architect (FCA) looks at the current list and proposes a new rule (e.g., "All diseases with Tremors also have Speech Issues").
Check the Evidence: The Librarian (RAG) finds the medical text for the diseases involved.
Make a Decision: The Junior Assistant (SLM) reads the text and decides:
- Yes: The rule is true. It gets added to the library.
- No: The rule is false. The Assistant finds a specific disease that breaks the rule (a counterexample) and adds it to the list so the Architect knows not to make that mistake again.
Expand: Once the current rules are settled, the system looks for new symptoms to add to the list and starts the cycle again.

What They Found (The Results)

The researchers tested this on a dataset of rare ataxia diseases. Here is what happened:

It Works, But It's Not Perfect: The system successfully built a "partial" library. It found many correct connections between diseases and symptoms.
The "Seed" Matters: Starting with a larger list of known symptoms (20 seeds instead of 10) helped the system find more rules and make fewer mistakes.
The Hard Part: Even with the best tools, the system struggled to find every single connection. Sometimes the medical text was too vague to say for sure if a symptom belonged to a disease.
The "Black Box" is Open: Unlike other AI systems where you just get a final answer, this system keeps a logbook. You can see exactly which rules were accepted, which were rejected, and why (which specific disease broke the rule). This makes the process "inspectable" and trustworthy.

The Bottom Line

This paper doesn't claim to have built the perfect medical encyclopedia overnight. Instead, it claims to have built a verifiable construction process.

Think of it like a scaffolding system for building knowledge. It uses a strict logic engine to propose ideas, a search engine to find proof, and a fast AI to check the proof. The result isn't a finished building, but a very clear, auditable record of what has been built, what was rejected, and exactly where the builders need human experts to step in and double-check the work.

Technical Summary: Verifiable Knowledge Expansion through Retrieval-Grounded Formal Concept Analysis

Problem Statement
The construction of biomedical ontologies, such as mapping rare diseases to phenotype attributes, traditionally requires costly manual curation by domain experts. While Large Language Models (LLMs) can propose structural knowledge from text, their outputs often lack verifiability, leading to unsupported or inconsistent relations. The core technical challenge is to verify proposed knowledge structures before they become permanent commitments. Specifically, the system must be able to ask targeted structural questions and incorporate counterexamples when a proposed regularity (e.g., "Disease X with symptoms A and B must have symptom C") fails.

Methodology
The paper proposes a hybrid framework combining Formal Concept Analysis (FCA) as a symbolic verification loop with a Retrieval-Augmented Generation (RAG) system powered by Small Language Models (SLMs). The process operates as an iterative ontology exploration loop:

Initialization: The system starts with a finite, controlled set of phenotype attributes ( $M$ ) and a seed subset ( $M_0$ ).
FCA Attribute Exploration: Using the current formal context (observed objects $E_\tau$ and active attributes $M_\tau$ ), the FCA module generates candidate implications ( $A \rightarrow B$ ). An implication suggests that any disease possessing attributes in set $A$ must also possess attributes in set $B$ .
RAG-Grounded SLM Oracle: For each candidate implication, the system retrieves disease-level text evidence relevant to the premise and conclusion. An SLM oracle then performs constrained, evidence-based binary judgments:
- Acceptance: If the evidence supports the implication, it is added to the implication base.
- Rejection: If the evidence contradicts the implication, the oracle must return a specific counterexample object (a disease that has $A$ but lacks at least one attribute in $B$ ).
- Incidence Judgment: The oracle also validates specific object-attribute pairs (YES/NO) based on retrieved snippets.
Context Update: Accepted implications refine the knowledge base. Rejected implications introduce counterexample objects into the observed context, preventing the system from over-generalizing.
Attribute Expansion: The system proposes new attributes for the next round based on the closure of accepted implications and retrieved evidence, gradually expanding the active attribute set $M_\tau$ .

The framework utilizes three different SLMs (Gemma2:2B, Llama3.2:3B, Qwen2.5:1.5B) to evaluate the cost-efficiency and performance trade-offs of using smaller models for repeated, evidence-conditioned decisions.

Key Contributions
The paper makes three primary contributions:

FCA-Based Verification Loop: A mechanism that tests object-attribute structures through implications and counterexamples, ensuring that structural commitments are not made without evidence.
Retrieval-Grounded SLM Oracle: A low-cost interface that grounds local judgments in external, retrievable evidence rather than relying solely on parametric memory.
Symbolic-Subsymbolic Hybrid: A system where FCA controls the global construction logic (proposing queries and managing the context), while the language model handles evidence-based local decisions.

Experimental Results
The framework was evaluated on a dataset of rare ataxia diseases derived from Orphadata and the Human Phenotype Ontology (HPO). The experiments ran for 20 rounds starting from 10 seed attributes, with a secondary set of experiments using 20 seeds.

Performance Metrics:
- Relation Scores (F1): Ranged from 0.29 to 0.52 across the 10-seed runs. Llama3.2:3B achieved the highest relation F1 (0.52), while Gemma2:2B showed the highest recall (0.87) but lower precision.
- Implication Scores (F1): Ranged from 0.22 to 0.30 for the 10-seed runs. Qwen2.5:1.5B achieved the highest implication F1 (0.30).
- Impact of Seed Size: Increasing the seed set to 20 attributes improved implication F1 scores for all models (e.g., Gemma rose from 0.22 to 0.36; Qwen reached 0.41), though it did not fully resolve the bottleneck of evaluating generated attributes against the gold standard.
Ablation Studies:
- LLM-Only (No Retrieval): Removing retrieval resulted in smaller explored contexts that terminated early. While these runs showed high implication F1 (up to 0.67), they failed to expand the object and attribute sets effectively, highlighting that retrieval is necessary for context growth.
- Fixed Object-Attribute Setting: Even when the set of diseases and attributes was fixed (removing the discovery difficulty), the best relation F1 achieved was only 0.31. This indicates that the primary bottleneck is not just discovering entities, but accurately judging specific disease-phenotype incidences from the available text evidence.

Significance and Claims
The paper modestly positions its method not as a complete, automatic ontology generator, but as a verifiable construction procedure.

Inspectability: The system produces a trace of accepted implications, rejected implications, counterexamples, and corrections. This allows domain experts to audit why a structural commitment was made or rejected, rather than accepting a "black box" generation.
Partial Construction: The authors claim the output is an "explored implication set" over an evidence-accessible context, not a canonical basis for the full controlled phenotype attribute set.
Limitations: The paper acknowledges that current results are limited by the sparsity of disease definition text (which may not explicitly state all curated relations) and the difficulty of cell-level incidence recovery. The framework is presented as an "auditable pre-curation artifact" that requires expert review before becoming a clinical ontology.

In summary, the work demonstrates that combining FCA's rigorous structural logic with RAG-grounded SLMs creates a scalable, inspectable loop for expanding biomedical knowledge, provided that the limitations of evidence retrieval and the need for expert oversight are recognized.