Technical Summary: ClinicBot

Problem Statement

The deployment of Large Language Models (LLMs) in high-stakes clinical domains is hindered by their tendency to hallucinate and their lack of explicit grounding in authoritative guidelines. While standard Retrieval-Augmented Generation (RAG) systems can access external sources, they often treat all evidence indiscriminately based on textual similarity. This approach produces noisy contexts and generic answers that fail to respect clinical evidence hierarchies, obscuring the relationship between specific recommendations and their supporting data. Consequently, clinicians struggle to verify if an answer faithfully reflects official guidelines, undermining trust and utility in medical decision support.

Methodology

ClinicBot addresses these limitations through a structured, guideline-grounded architecture that decouples retrieval into two complementary stages: Semantic Knowledge Extraction and Prioritized Evidence Retrieval.

1. Structured Knowledge Base Construction

Unlike generic RAG systems that process unstructured text (e.g., PubMed abstracts), ClinicBot grounds its operations in a single authoritative source: the American Diabetes Association (ADA) Standards of Care in Diabetes (2025). The system converts this document into a structured JSON knowledge base using semi-automatic parsing and manual verification. Content is explicitly classified into three distinct types, each assigned a specific retrieval priority:

• Recommendations: Structured records containing IDs and evidence grades (e.g., Grade A).
• Criteria Tables: Diagnostic and screening criteria with numeric thresholds and classification ranges.
• Narrative Context: Rationale, population-specific guidance, and definitions.

2. Two-Stage Retrieval Pipeline

The retrieval process employs a "routing-then-retrieval" strategy to ensure clinical relevance and authority:

• Routing: An LLM-based few-shot classifier maps the clinician's query to a specific section and subsection of the guideline (e.g., "Diagnosis & Screening"). This narrows the search scope to a conceptually coherent segment, improving specificity over full-document searches.
• Prioritized Retrieval: Within the routed section, the system enforces a strict retrieval order based on clinical authority rather than semantic similarity:
  1. Recommendations (Highest priority)
  2. Diagnostic/Screening Criteria Tables
  3. Supporting Narrative
     This structural filtering ensures that actionable, citable content dominates the context passed to the generator.

3. Answer Generation and Validation

The generation phase uses an LLM constrained by a prompt template that forbids the injection of external medical knowledge. The output is structured into:

• A Concise Answer (2–3 sentences) that is direct and actionable.
• Verifiable Citations linking claims to specific recommendation IDs or table rows.
• Evidence Details including precise numeric thresholds.

Crucially, a Validation Module operates before the answer is presented. It enforces three constraints to prevent hallucinations:

1. Citation Requirement: Every claim must be supported by retrieved evidence with a specific citation; uncited claims are rejected.
2. Numeric Integrity: Numeric thresholds (e.g., mg/dL values) must match retrieved values exactly via string matching to prevent drift.
3. Graceful Refusal: If the routed section lacks sufficient evidence, the system declines to generate an answer rather than hallucinating.

Key Contributions

• Guideline-Grounded Architecture: A system that translates complex clinical guidelines into a structured, prioritized knowledge base rather than treating them as undifferentiated text.
• Evidence Prioritization: A retrieval mechanism that ranks content by clinical significance (Recommendations > Tables > Narrative) rather than textual similarity, aligning with clinical evidence hierarchies.
• Verifiable Citations and Traceability: A design that enforces full traceability from answer to source material, including a validation layer that rejects unsupported claims and ensures numeric accuracy.
• Dual-Mode Interface: A web-based demonstration featuring both a guideline-grounded question-answering interface and a diabetes risk assessment tool faithful to the ADA framework.

Evaluation Results

The system was evaluated against 30 manually curated diabetes-related questions sourced from the Hugging Face Diabetes QA dataset and the diabetes.co.uk medical webboard. The evaluation covered diagnostic criteria, medication management, lifestyle interventions, and patient education.

• Overall Accuracy: ClinicBot achieved a 96% combined accuracy.
• Breakdown:
  • 63% (19/30): Fully correct responses.
  • 33% (10/30): Correct understanding with minor omissions of non-critical details (e.g., secondary medication options).
  • 3% (1/30): Incorrect response.
• Performance: The results indicate that the guideline-grounded architecture effectively operationalizes complex ADA recommendations into clinically accurate and actionable responses for practicing clinicians.

Significance

The paper positions ClinicBot as a step toward trustworthy clinical AI that prioritizes explainability and evidence justification over raw generative capability. By combining semantic knowledge extraction with hierarchical evidence ranking, the system demonstrates that complex clinical guidelines can be processed at scale in a multi-agent setting while maintaining rigorous adherence to authoritative sources. The work highlights that for high-assurance domains like medicine, the ability to verify decisions against specific guideline sections and evidence grades is as critical as the accuracy of the answer itself. The system is designed to support, not replace, clinicians by providing transparent, traceable, and guideline-faithful support.