💻 computer science

CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs

CareGuardAI is a context-aware, multi-agent framework that ensures clinical safety and mitigates hallucinations in patient-facing LLMs by employing a multi-stage inference pipeline with dual risk assessments (SRA and HRA) to filter and refine responses before release.

Original authors: Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui, Niyousha HosseiniChimeh

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui, Niyousha HosseiniChimeh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant that knows a lot about medicine. You ask it, "Can I take aspirin while I'm pregnant?" The robot, eager to be helpful, might say, "Yes, low doses are often okay."

While that answer might be technically true for some people, it's dangerously incomplete for a pregnant woman without a doctor's specific approval. The robot missed the context: pregnancy is a special, high-risk situation.

This is the problem CareGuardAI solves. It's a new "safety net" system designed to catch medical AI before it gives advice that could hurt a patient. Think of it not as a single doctor, but as a high-tech triage team working together to ensure every answer is safe and true.

Here is how the system works, using simple analogies:

1. The Triage Nurse (The Controller)

When a patient asks a question, the first thing CareGuardAI does is send it to a "Triage Nurse" (a small, fast AI).

The Job: This nurse doesn't answer the question yet. Instead, they act like a hospital intake clerk. They scan the question to see if anything is missing.
The Analogy: Imagine you walk into a doctor's office. The nurse doesn't just hand you a prescription; they ask, "Are you pregnant? Do you have allergies? How old are you?"
What it does: If the patient asks about aspirin, the Triage Nurse spots the word "pregnant" and flags it as High Risk. It tells the next part of the system: "Be very careful! This is a sensitive situation. Do not give specific medical orders."

2. The Writer (The Generator)

Once the Triage Nurse has set the rules, the "Writer" (a powerful AI) drafts the response.

The Job: The Writer tries to answer the question, but they are wearing "safety goggles" provided by the Triage Nurse.
The Analogy: If the Triage Nurse said, "This is a high-risk pregnancy," the Writer is told, "You cannot say 'Take this pill.' You must say, 'Go see your doctor.'"
The Result: Instead of prescribing aspirin, the Writer says, "Please consult your OB-GYN, as aspirin can be risky during pregnancy."

3. The Double-Check Inspectors (The Evaluators)

Before the answer is ever shown to the patient, it goes through two strict inspectors working in parallel.

Inspector A (Safety Check): This inspector looks at the answer to see if it's medically dangerous. They use a scale from 1 to 5 (like a hurricane warning).
- Level 1: Just information.
- Level 5: "This could kill someone."
- The Rule: If the score is higher than 2, the answer is rejected.
Inspector B (Truth Check): This inspector looks for hallucinations (lies or made-up facts).
- Level 1: Totally true.
- Level 5: A dangerous lie.
- The Rule: If the score is higher than 2, the answer is rejected.

4. The "Red Light, Green Light" Gate (The Decision Layer)

This is the final boss. It looks at the scores from both inspectors.

Green Light: If both scores are low (Safe ≤ 2 AND Truthful ≤ 2), the answer is released to the patient.
Red Light: If either score is too high, the answer is blocked.
The "Do-Over" Loop: If the answer is almost safe but has a small mistake, the system doesn't just delete it. It sends it back to the Writer with a note: "You said X, but that's risky. Try again." The Writer rewrites it, and the inspectors check it again. They do this up to three times. If it's still unsafe, the system blocks it entirely and tells the patient to see a human doctor.

Why is this different from what we have now?

Most current medical AIs are like a student who memorized a textbook. They can pass a written exam perfectly, but if you ask them a messy, real-life question (like "I'm pregnant and have a headache"), they might give a textbook answer that ignores the real-world danger.

CareGuardAI is like a team of professionals (a nurse, a writer, and two inspectors) who stop to think, "Wait, is this safe for this specific person?" before letting the answer out.

The Results (What the paper found)

The researchers tested this system on thousands of tricky medical questions.

Safety: Without this system, the AI gave dangerous advice about 20% of the time. With CareGuardAI, that dropped to less than 3%.
Truth: The system stopped the AI from making up fake medical facts.
Speed: It takes about 14 seconds to process a question. That's a little slower than a chatbot, but the paper argues it's worth the wait to ensure the answer won't hurt anyone.

In short: CareGuardAI doesn't just try to make the AI smarter; it builds a safety cage around the AI to make sure it doesn't give dangerous or fake advice, especially when the patient's situation is complicated or unclear.

1. Problem Statement

The integration of Large Language Models (LLMs) into patient-facing healthcare systems faces two critical, interrelated failure modes that existing benchmarks and mitigation strategies fail to address adequately:

Clinical Safety Risk: LLMs often generate conditionally correct but medically inappropriate responses (e.g., prescribing medication to a pregnant patient) because they lack the contextual awareness to infer risk from incomplete information, unlike human clinicians.
Hallucination Risk: Models frequently produce unsupported, inconsistent, or fabricated medical claims.
The Gap: Current approaches typically treat safety and hallucination as separate problems or rely on static benchmarks (like USMLE exams) that do not reflect real-world, open-ended, and underspecified patient queries. Furthermore, training-time alignment (fine-tuning) is insufficient for handling dynamic, context-dependent risks at inference time.

2. Methodology: The CareGuardAI Framework

CareGuardAI is a risk-aware, inference-time safety framework designed as a multi-agent pipeline. It operates by integrating a controller, a generator, dual evaluators, and a decision layer to enforce safety before a response is released.

Core Components

Controller Agent (Triage & Screening):
- Model: Lightweight Small Language Model (SLM), specifically Phi-3.5-mini.
- Function: Performs risk-aware query classification and vulnerability screening. It identifies missing patient context (e.g., pregnancy status, age, symptom severity) using keyword rules and model detection.
- Mechanism: If a query is underspecified, the controller generates targeted multiple-choice screening questions to elicit critical context before generation. It then produces deterministic safety instructions (e.g., "do not prescribe") based on the identified risk category.
Response Generator:
- Model: Large Language Model (GPT-4o-mini).
- Function: Generates the patient-facing response.
- Constraint: The generation is conditioned on the safety instructions and context inserts provided by the Controller. It enforces behavioral guardrails (e.g., refusing to diagnose or prescribe) tailored to the specific risk profile.
Dual Risk Evaluators (Parallel Assessment):
- Model: Quantized LLaMA-3.1-8B-Instruct (running locally).
- Clinical Safety Risk Assessment (SRA): Inspired by ISO 14971, this agent scores the response on a 1–5 scale (1=Minimal, 5=Critical) based on actionable medical risk (diagnosis, prescription, validation of harmful actions).
- Hallucination Risk Assessment (HRA): Inspired by HalluGuard, this agent scores factual reliability on a 1–5 scale, distinguishing between data-driven hallucinations (factual errors) and reasoning-driven hallucinations (logical inconsistencies).
Decision Layer & Refinement Loop:
- Logic: A response is released only if both $SRA \le 2$ and $HRA \le 2$ .
- Refinement: If thresholds are exceeded, the system triggers an iterative refinement loop (max 3 iterations). The evaluators provide structured feedback to the generator to revise the response.
- Fallback: If the response remains unsafe after 3 iterations, it is blocked, and a safe refusal message is returned.

System Architecture

Hybrid Design: Combines local SLMs for fast, cost-efficient triage and evaluation with a cloud-based LLM for high-quality generation.
Latency: Average latency is approximately 13.8 seconds per query, making it feasible for real-time deployment.

3. Key Contributions

Dual-Axis Safety Framework: Introduces a novel approach that jointly models Clinical Safety Risk (SRA) and Hallucination Risk (HRA) as complementary failure modes, addressing the gap where a response can be factually correct but clinically unsafe, or vice versa.
Inference-Time Control Architecture: Proposes a structured pipeline integrating triage-based context recovery (screening for missing patient data) with safety-constrained generation and parallel post-generation evaluation.
Risk-Bounded Decision Mechanism: Implements a threshold-based strategy ( $SRA \le 2, HRA \le 2$ ) with iterative refinement, ensuring only clinically acceptable outputs are deployed.
Comprehensive Evaluation: Validates the framework across three benchmarks (PatientSafeBench, MedSafetyBench, MedHallu), demonstrating significant improvements over strong baselines like GPT-4o-mini.

4. Experimental Results

The framework was evaluated on PatientSafeBench (466 samples), MedSafetyBench (450 samples), and MedHallu (200 samples).

Safety & Reliability:
- Deployable Rate: CareGuardAI achieved a 98.7% – 99.8% deployable rate (responses meeting both safety thresholds) across benchmarks, compared to a baseline of 60–86%.
- Risk Reduction: The average SRA dropped from ~3.2 (baseline) to 2.02, and HRA dropped from ~2.9 to 1.23 on the MedHallu dataset.
- Violation Elimination: Safety violations (e.g., prescribing medication without context) dropped from 19.7% (baseline) to 2.8%. Misdiagnosis errors were reduced from 30.3% to 0.0%.
Refinement Efficiency:
- The system converges rapidly, with an average of ~1.0 iteration per query.
- 100% of initially unsafe responses on MedSafetyBench and MedHallu were successfully downgraded to safe levels after refinement.
- The system successfully eliminated the "Safe but Hallucinated" failure mode (0% occurrence in results).
Ablation Study:
- Removing the Controller caused the deployable rate to plummet to 39.0%.
- Removing the HRA agent led to a 42.0% rate of hallucinated outputs.
- Removing the SRA agent resulted in a high deployable rate (98%) but a 2.0% rate of clinically unsafe responses, proving that hallucination control alone is insufficient for medical safety.

5. Significance and Implications

Context-Awareness: CareGuardAI addresses the critical limitation of static benchmarks by simulating clinical triage, allowing the system to detect vulnerability (e.g., pregnancy) and request missing context before generating advice.
Regulatory Alignment: The framework aligns with ISO 14971 risk management principles and FDA Good Machine Learning Practice by providing transparent, auditable, and deterministic decision paths (release, refine, or block).
Practical Deployment: By using a hybrid architecture (local SLMs + cloud LLM), it achieves a balance between high-quality generation and low-latency, cost-effective safety enforcement, making it viable for real-world patient-facing applications like chatbots and telehealth.
Paradigm Shift: The paper argues that reliable medical AI requires moving beyond static, exam-style evaluations toward dynamic, inference-time oversight that accounts for the uncertainty and context-dependency of real-world patient interactions.

In conclusion, CareGuardAI demonstrates that a multi-agent, risk-aware framework can effectively mitigate both clinical safety risks and hallucinations in patient-facing LLMs, enabling safer and more reliable deployment in healthcare settings.