AOP-Smart: A RAG-Enhanced Large Language Model Framework for Adverse Outcome Pathway Analysis

This paper introduces AOP-Smart, a Retrieval-Augmented Generation framework that leverages official AOP-Wiki data to significantly reduce hallucinations and boost answer accuracy in Adverse Outcome Pathway analysis tasks across multiple large language models.

The Gist

Imagine you are trying to solve a complex medical mystery: "How does chemical X cause liver failure?"

In the world of toxicology, scientists use a roadmap called an Adverse Outcome Pathway (AOP). Think of this roadmap like a giant, intricate "Choose Your Own Adventure" book, but instead of pages, it has thousands of biological events (like a cell getting stressed, a protein breaking, or an organ shrinking) all connected by cause-and-effect chains.

The Problem: The "Confident but Wrong" Expert

Recently, we've had access to super-smart AI assistants (Large Language Models, or LLMs) that can read this book and answer questions. But there's a catch: Hallucinations.

Imagine asking a very confident tour guide who has memorized a lot of books but hasn't read the specific one you are holding. If you ask a tricky question about a rare path, the guide might make up a story that sounds perfect and flows beautifully, but it's completely made up. In science, this is dangerous. If an AI says a chemical causes a disease when it doesn't, it could lead to bad safety decisions.

The Solution: AOP-Smart (The "Fact-Checker" System)

The authors of this paper built a tool called AOP-Smart. Think of it as giving that confident tour guide a real-time, digital library card and a magnifying glass before they answer your question.

Here is how it works, using a simple analogy:

1. The "Index Card" System (Retrieval)

Instead of letting the AI guess from its memory, AOP-Smart first looks at the official "AOP-Wiki" (the master database of all toxicology facts).

• The Old Way: You ask, "What happens after Event A?" The AI guesses based on what it learned during training.
• The AOP-Smart Way: The system first finds the specific "Index Card" for Event A. It then pulls out the cards for everything before Event A (Upstream) and everything after Event A (Downstream). It also grabs the full story of the entire "Adventure Path" (the AOP) that connects them.

2. The "Context Sandwich" (Augmentation)

Now, the AI doesn't just get your question. It gets a sandwich:

• Top Bun: Your Question.
• The Filling: The exact, verified facts, connections, and rules from the official database that AOP-Smart just found.
• Bottom Bun: The AI's instructions to answer only based on this filling.

3. The Result: No More Guessing

Because the AI is now reading the facts right in front of it, it can't make things up. It has to follow the map.

The Magic Numbers

The researchers tested this on three famous AI models (Gemini, DeepSeek, and ChatGPT) with 20 tricky toxicology questions.

• Without the "Library Card" (No RAG): The AIs were like students guessing on a test they didn't study for.

  • One got 15% right.
  • One got 20% right.
  • One got 35% right.
  • Result: Mostly wrong, full of made-up facts.

• With "AOP-Smart" (With RAG): The AIs were like students who had the textbook open right in front of them.

  • One got 95% right.
  • One got 100% right.
  • One got 95% right.
  • Result: Almost perfect accuracy.

Why This Matters

This isn't just about getting a better grade on a quiz. In toxicology, getting the answer right means knowing which chemicals are safe for humans and which ones are dangerous.

AOP-Smart is like upgrading from a "Guessing Game" to a "Fact-Checking Machine." It takes the creative power of AI and anchors it firmly to the ground of scientific truth, ensuring that when we ask about the dangers of chemicals, the answer is reliable, evidence-based, and safe.

In short: It stops the AI from "making things up" by forcing it to look at the source code of reality before it speaks.

Technical Summary

1. Problem Statement

Adverse Outcome Pathways (AOPs) are critical conceptual frameworks in toxicology that link Molecular Initiating Events (MIEs) to Adverse Outcomes (AOs) through a series of Key Events (KEs) and Key Event Relationships (KERs). While Large Language Models (LLMs) show promise in analyzing these pathways, they face significant challenges:

• Hallucinations: LLMs often generate fluent but factually incorrect information when dealing with highly specialized domain knowledge, lacking evidence or consistency with established facts.
• Knowledge Limitations: Standard LLMs rely on internal training parameters, which may not cover the rapidly expanding and structured nature of AOP data (e.g., specific species, biological levels, and complex causal chains).
• Retrieval Gaps: Existing Retrieval-Augmented Generation (RAG) methods are typically based on vector databases and semantic similarity, which fail to capture the specific hierarchical, causal, and network characteristics of AOP structures (KEs, KERs, and AOP chains).

2. Methodology: AOP-Smart Framework

The authors propose AOP-Smart, a domain-specific RAG framework designed to integrate structured AOP knowledge into the LLM reasoning process.

A. Data Source and Preparation

• Source: Official XML data from AOP-Wiki (specifically the stable snapshot from 2026-01-01).
• Structure: The data is parsed into two structured resources:
  1. Index.txt: A lightweight index (~20k tokens) containing KE IDs and titles for efficient preliminary candidate retrieval.
  2. AOP-Smart.json: A detailed repository storing structured information for KEs, KERs, and full AOP chains.

B. The Retrieval and Expansion Process

The framework operates in a closed-loop process:

1. Preliminary Retrieval: The user query and Index.txt are fed to the LLM. Through prompt engineering, the LLM ranks all KEs and selects the Top-N most relevant KE IDs (where N is a tunable parameter, set to 5 in experiments).
2. Knowledge Expansion (The Core Innovation): Based on the selected Top-N KE IDs, the system performs a three-step expansion to reconstruct the causal context:
   • KE Expansion: Retrieves direct upstream and downstream KEs for the selected IDs.
   • KER Reconstruction: Traverses the knowledge base to find all KERs connecting any two KEs within the expanded set, recovering causal relationships.
   • AOP Retrieval: Identifies full AOP chains where at least two elements (MIE, KE, or AO) match the expanded KE set. This reduces noise from accidental single-event matches.
3. Context Integration: The expanded KE, KER, and AOP details are formatted into a structured context.
4. Generation: The structured context and the original user query are jointly input into the LLM to generate a factually grounded answer.

C. Implementation

• Built as a Python-based software system.
• Allows configuration of model endpoints, API keys, temperature (set to 0 for determinism), and the Top-N KE parameter.
• Supports integration with various LLMs (e.g., DeepSeek, Gemini, ChatGPT).

3. Key Contributions

1. Domain-Specific RAG Framework: Proposed the first RAG framework specifically tailored for the AOP domain, moving beyond generic semantic search to utilize causal and hierarchical structural information.
2. KE-Based Expansion Strategy: Designed a novel mechanism to expand retrieval from single Key Events to their causal neighbors (upstream/downstream KEs), KERs, and complete AOP chains, ensuring the LLM receives comprehensive mechanistic context.
3. Empirical Validation: Conducted rigorous comparative experiments demonstrating that structured retrieval significantly mitigates hallucinations in toxicological reasoning.

4. Experimental Results

The study evaluated the framework on a test set of 20 AOP-related tasks across four categories: KE Identification, Downstream KE Retrieval, Upstream KE Retrieval, and Complex AOP Queries. Three models were tested: DeepSeek, Gemini, and GPT.

Performance Comparison (Accuracy):

Model	Without RAG (Baseline)	With AOP-Smart (RAG)	Improvement
DeepSeek	35.0%	100.0%	+65%
Gemini	20.0%	95.0%	+75%
GPT	15.0%	95.0%	+80%
Overall Avg	23.33%	96.67%	+73.34%

• Task-Specific Findings:
  • Simple Retrieval (KE ID/Downstream): All models achieved near 100% accuracy with RAG.
  • Complex Reasoning (Upstream/Full Chains): While baseline performance was near zero for some models, RAG boosted performance to 80–100%, proving its efficacy in multi-hop structural relationship queries.
• Conclusion: The framework effectively eliminates the hallucination problem, transforming LLMs from unreliable generators into accurate, evidence-based reasoning engines for toxicology.

5. Significance and Future Work

Significance:

• Reliability: AOP-Smart provides a robust solution for integrating scientific knowledge into LLMs, ensuring outputs are consistent with the official AOP-Wiki database.
• Usability: It bridges the gap between unstructured natural language queries and structured, causal scientific data, making complex toxicological mechanisms accessible to researchers.
• Scalability: The system is designed to be updated easily by replacing XML files, ensuring it remains current with the evolving AOP knowledge base.

Limitations & Future Directions:

• Current Limitations: The system does not yet weight information by evidence strength or confidence levels; text truncation for context limits may lose some semantic nuance; and the evaluation set (20 questions) is relatively small.
• Future Work:
  • Expand the evaluation benchmark to include larger-scale, multi-hop reasoning tasks.
  • Integrate evidence levels and confidence scores into the retrieval weighting mechanism to enhance interpretability.
  • Explore applications in automated literature extraction and AOP knowledge graph completion.