VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

Technical Summary: VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

1. Problem Statement

While Large Language Models (LLMs) show significant potential in medical diagnosis, their deployment is hindered by the high stakes of clinical decision-making and the lack of transparency in their reasoning processes. Current debugging practices for medical LLMs face three primary challenges:

• Domain Expertise Gap: Developers often lack the clinical knowledge required to interpret whether a model's reasoning is medically sound.
• Scalability and Prioritization: Models fail across vast, diverse datasets, making it impossible for developers to manually inspect every error to find critical failures.
• Instance-Centric Limitations: Existing debugging tools focus on individual text outputs or token-level behaviors, making it difficult to identify recurrent structural error patterns that suggest systemic model weaknesses.

2. Methodology

The authors propose VeriLLMed, a visual analytics system that shifts the debugging focus from raw text to structured reasoning paths grounded in a Biomedical Knowledge Graph (Bio-KG). The technical pipeline consists of four main stages:

• Reasoning Path Construction:
  • Model Paths: The system uses a reasoning model (GPT-5-mini) to generate structured reasoning paths (entities and relations) alongside the diagnosis. These entities are aligned to the iBKH (Biomedical Knowledge Hub) using a three-stage strategy: exact matching, semantic embedding similarity (via MedEmbed), and LLM-based disambiguation.
  • Reference Paths: The system constructs "gold standard" reasoning paths by finding the shortest directed paths in the Bio-KG between entities extracted from the clinical question and the correct diagnosis.
• Error Taxonomy & Detection: By comparing observed model paths to KG-grounded reference paths, the system identifies three distinct structural error types:
  1. Relation Errors: The model traverses a transition between two entities that is medically invalid according to the KG.
  2. Branch Errors: The model follows a valid medical path that leads away from the correct diagnosis toward an incorrect one.
  3. Missing Errors: The model omits a key clinical concept present in the reference path that is essential for reaching the correct diagnosis.
• Pattern Extraction: To move beyond single instances, the system expands local errors into "error families" by retrieving neighboring erroneous and reference transitions, using an LLM to summarize the semantic differences between these groups.
• Visual Analytics Interface: A multi-level coordinated interface allows users to:
  • Overview: View error distributions (Dataset Overview).
  • Localize: Identify error-dense medical concept regions (Projection View).
  • Inspect Structure: Compare reasoning neighborhoods using specialized glyphs (Path View).
  • Analyze Patterns: Compare erroneous vs. correct reasoning flows via a Sankey layout (Error View).
  • Validate: Verify findings against specific clinical cases (Detail and Instance Views).

3. Key Contributions

• A KG-Grounded Debugging Framework: A novel approach that transforms unstructured LLM reasoning into a structured, comparable, and medically grounded format.
• A Structural Error Taxonomy: The definition of relation, branch, and missing errors specifically tailored for diagnostic reasoning.
• VeriLLMed System: An interactive visual analytics tool that supports a workflow from global error summaries to localized, recurrent pattern discovery and individual case validation.
• Actionable Debugging Workflow: A methodology that enables developers to translate visual insights into concrete model refinement strategies (e.g., prompt engineering).

4. Results and Evaluation

The system was evaluated through two expert case studies and a user study with six experts:

• Case Study 1 (Missing Errors): An expert identified that the model consistently failed to integrate nonspecific symptoms (like fever) into complex inflammatory reasoning, instead defaulting to infectious disease patterns. This insight led to a prompt-based intervention that improved accuracy on the analyzed set from 81.5% to 84.3%.
• Case Study 2 (Relation Bias): An expert discovered a bias where the model incorrectly linked dermatologic symptoms (like pruritus) to localized skin issues rather than systemic autoimmune conditions. A targeted prompt intervention improved accuracy from 81.5% to 85.5%.
• Expert Evaluation: Participants reported that VeriLLMed was significantly more efficient than manual text review, provided essential medical grounding for non-experts, and successfully helped them move from "noticing an error" to "identifying a systemic pattern."

5. Significance

VeriLLMed represents a significant step toward reliable and interpretable medical AI. By leveraging Knowledge Graphs as an "external auditing substrate," the system bridges the gap between machine learning development and clinical expertise. Its methodology—converting unstructured reasoning into structured, graph-based paths—is highly generalizable and can be applied to any domain where reasoning-intensive tasks can be mapped to a structured knowledge base.