Multi-agent Self-triage System with Medical Flowcharts

1. Problem Statement

The paper addresses the growing reliance on Large Language Models (LLMs) and online resources for medical self-triage, highlighting three critical limitations in current approaches:

• Reliability & Accuracy: Standard symptom checkers often lack accuracy and fail to provide definitive, tailored actions. LLMs, while flexible, suffer from "hallucinations" (fabricating information) and provide vague, probabilistic advice.
• Lack of Transparency: LLMs operate as "black boxes," making it difficult for clinicians or patients to audit the decision-making process or trust the output.
• Resource Strain: The inability of patients to effectively self-triage contributes to overcrowded emergency departments (EDs), with nearly 40% of visits being non-urgent.

The core challenge is to create a system that combines the conversational flexibility of LLMs with the rigor, safety, and auditability of standardized clinical protocols.

2. Methodology

The authors propose TriageMD, a proof-of-concept conversational self-triage system guided by 100 clinically validated flowcharts from the American Medical Association (AMA). The system employs a Multi-Agent Framework powered by LLMs, structured as follows:

A. Knowledge Base: Graph-Represented Flowcharts

• Source: 100 flowcharts from the AMA Family Medical Guide were preprocessed to focus on triage actions rather than just diagnosis.
• Representation: Flowcharts were converted from text/PDFs into directed graphs using Python (NetworkX).
  • Nodes: Categorized into four types:
    • N (Node): Yes/No questions.
    • F (Flow): Redirects to a linked flowchart.
    • A (Action): Endpoints providing recommendations (e.g., "See a doctor").
    • I (Info): Explanatory text preceding a redirect.
  • Edges: Directed edges with conditions (Yes/No) connecting nodes.

B. Multi-Agent Architecture

The system utilizes three specialized agents working sequentially:

1. Retrieval Agent:
   • Input: Patient demographics (age, sex) and main concern (opening statement).
   • Mechanism: Uses Retrieval-Augmented Generation (RAG). It performs a cosine similarity search (via FAISS) against vectorized flowchart descriptions to retrieve top candidates, then uses an LLM to select the single most appropriate flowchart based on semantic analysis.
2. Decision Agent:
   • Input: The selected flowchart's current question node and the patient's natural language response.
   • Mechanism: Classifies the response along four axes to determine the next step:
     • isOnTopic: Is the response relevant?
     • isAnswered: Did the patient provide a Yes/No?
     • actualAnswer: What is the specific Yes/No?
     • isUncertain: Is the response ambiguous?
   • Logic: The system only advances to the next node if the response is certain and on-topic. Otherwise, it triggers clarification or redirection.
3. Chat Agent:
   • Input: The decision agent's structured output and the flowchart content.
   • Mechanism: Generates empathetic, concise, and patient-friendly responses. It adapts tone based on whether the patient answered clearly, was off-topic, or expressed uncertainty.

C. Evaluation Strategy

Due to the lack of large-scale real-world patient conversation datasets, the authors generated synthetic datasets using four different LLMs (GPT-4o, Claude 3 Haiku, Gemini 2.0 Flash-Lite, DeepSeek-Chat) to simulate diverse patient interactions.

• Dataset 1 (Retrieval): 2,000 synthetic opening statements (20 per flowchart).
• Dataset 2 (Navigation): 37,200 synthetic patient responses across 100 flowcharts, categorized into five conversational patterns: Brief, Descriptive, Weak (vague), Uncertain, and Off-topic.

3. Key Contributions

• Shared Representation: Introduces a novel approach of using clinically validated flowcharts as a shared, auditable representation between human experts and LLMs, bridging the gap between unstructured AI and structured medical protocols.
• Multi-Agent Triage System: Demonstrates a modular architecture (Retrieval + Decision + Chat) that separates concerns, allowing for specific optimization of flowchart selection, logic navigation, and user interaction.
• Handling Ambiguity: The system explicitly handles uncertain and off-topic patient responses by pausing the triage process to seek clarification, rather than forcing a potentially incorrect path forward.
• Scalable Synthetic Evaluation: Establishes a rigorous framework for evaluating medical AI using diverse synthetic data generated by multiple LLMs to test robustness across different conversational styles.

4. Results

The system was evaluated on two primary tasks:

A. Flowchart Retrieval Accuracy

• Top-3 Accuracy: The system achieved 95.29% accuracy in retrieving the correct flowchart within the top 3 candidates from a database of 100.
• Agent Performance: The combined Retrieval Agent (Similarity Search + LLM) outperformed using either component alone.
  • LLM-only: 78.40%
  • Similarity Search-only: 82.14%
  • Combined Agent: 84.66% (Top-1), rising to 95.29% (Top-3).
• Failure Analysis: Most errors occurred due to symptom overlap (e.g., chest pain vs. shortness of breath) or profile-influenced biases (e.g., age/gender leading to the wrong specific flowchart).

B. Flowchart Navigation Accuracy

• Overall Accuracy: The system achieved 99.10% accuracy in correctly navigating the flowchart based on patient responses.
• Pattern Performance:
  • Brief & Descriptive: High accuracy (>98%) in identifying correct answers.
  • Weak (Ambiguous): The system successfully detected uncertainty in 55.38% of cases (classifying them as "Uncertain & Correct"), preventing premature progression.
  • Uncertain & Off-topic: The system correctly identified these as non-answers 99.53% and 98.26% of the time, respectively, and refrained from advancing the triage.
• Critical Safety Metric: The "Certain and Incorrect" category (where the system confidently makes a wrong decision) was extremely low (<0.5% for most patterns), indicating high safety.

5. Significance and Future Work

• Clinical Impact: The system offers a transparent, auditable alternative to "black box" medical chatbots. Clinicians can review the exact decision path taken, enhancing trust and accountability.
• Resource Optimization: By accurately guiding patients to self-care, primary care, or emergency services, the tool has the potential to reduce unnecessary ED visits and improve healthcare resource allocation.
• Limitations & Future Directions:
  • Current Scope: Limited to English text and polar (Yes/No) questions.
  • Future Work: Expanding to multilingual/multimodal inputs (e.g., image analysis), integrating dynamic flowchart updates via human-AI collaboration, and conducting real-world clinical trials to validate safety and efficacy in live settings.
  • Error Handling: Developing mechanisms for confidence estimation and "human-in-the-loop" interventions for edge cases.

In conclusion, TriageMD demonstrates that combining structured clinical knowledge (flowcharts) with flexible LLM agents creates a viable, safe, and transparent framework for AI-assisted medical self-triage.