ClinNoteAgents: An LLM Multi-Agent System for Predicting and Interpreting Heart Failure 30-Day Readmission from Clinical Notes

ClinNoteAgents is a novel LLM-based multi-agent system that effectively predicts and interprets 30-day heart failure readmission risks by transforming unstructured clinical notes into structured risk factors and clinician-style abstractions, offering a scalable and interpretable solution for data-limited healthcare settings.

The Gist

Imagine a hospital is like a massive, chaotic library. Inside, there are millions of books (patient records). Most of these books have a neat, organized index card at the front with checkboxes for things like "Age," "Blood Pressure," and "Diagnosis." These are the structured data computers love.

But here's the problem: The real story of a patient's health is often written in the free-text notes at the back of the book. These are handwritten or typed paragraphs by doctors, full of abbreviations, slang, messy handwriting, and personal details like "patient lives alone" or "struggles to afford food." Computers usually can't read these stories; they just see a wall of text.

This paper introduces ClinNoteAgents, a team of AI "librarians" designed to read those messy stories, understand them, and turn them into useful information to predict if a heart failure patient will end up back in the hospital within 30 days.

Here is how the system works, broken down into simple concepts:

1. The Problem: The "Lost in Translation" Library

Heart failure is a serious condition where the heart struggles to pump blood. A major issue is that many patients get sick again and have to be readmitted to the hospital within a month. This is expensive and stressful.

Doctors write down why a patient might return in their notes, but these notes are unstructured. They might say, "Patient is a retired teacher who lives in a drafty apartment and has trouble walking to the bus stop." A traditional computer program sees this as gibberish. It doesn't know that "retired," "drafty apartment," and "trouble with the bus" are actually huge red flags for readmission.

2. The Solution: A Team of AI Specialists (The Agents)

Instead of one giant robot trying to do everything, the authors built a team of three specialized AI agents (using a smart model called Qwen3) to work together like a production line:

• Agent 1: The Detective (Risk Factor Extractor)
  This agent reads the messy notes and acts like a detective looking for clues. It pulls out specific facts:

  • Clinical Clues: "Blood pressure was 120/80," "Heart rate was 90."
  • Social Clues: "Lives alone," "No car," "Smokes a pack a day."
    It turns the sentence "Patient is a retired teacher who lives in a drafty apartment" into a structured list: Job: Retired, Housing: Unstable, Transport: None.

• Agent 2: The Translator (Risk Factor Normalizer)
  People write things differently. One doctor might write "No alcohol," another "Drinks socially," and a third "Sober." The computer gets confused.
  This agent acts like a translator. It takes all those different ways of saying the same thing and standardizes them into a clean, consistent category (e.g., converting all variations into "Current Moderate Use" or "Abstinent"). This allows the computer to do math on the data.

• Agent 3: The Summarizer (Note Summarizer)
  Patient notes can be 50 pages long. This agent reads the whole story and writes a short, "clinician-style" abstract. It keeps the important medical facts and risk factors but throws away the fluff. It's like turning a 50-page mystery novel into a one-page spoiler-free summary that still tells you who the villain is.

3. The Results: Does It Work?

The team tested this system on over 3,500 patient notes from a public database (MIMIC-III).

• The Detective was sharp: It successfully pulled out vital signs (like heart rate and blood pressure) and social details with very high accuracy (over 90% for many things). It even figured out that "living alone" and "housing instability" were significant risk factors.
• The Summarizer saved time: Even when they cut the text length by 60% to 90% (making the notes super short), the AI could still predict readmissions almost as well as if it had read the full, long notes.
• The Big Win: The system found that things like age, blood pressure, and housing stability were key predictors. It proved that you don't need a perfect, expensive database to predict heart failure risks; you can just use the notes doctors are already writing.

4. Why This Matters (The "So What?")

In many parts of the world (and even in some US hospitals), we don't have perfect, structured databases. We just have notes.

• For Developing Countries: Hospitals with limited technology can use this to analyze their handwritten or simple digital notes to spot high-risk patients.
• For Everyone: It reduces the need for humans to manually type data into computers. It turns "noise" (messy text) into "signal" (actionable data).

The Catch (Limitations)

The authors are honest about the flaws. Sometimes the AI might "hallucinate" (make up a fact) or miss a detail because the note was too messy. Also, they didn't have human doctors double-check every single output. So, this system is best used as a helper tool to flag potential risks for a human doctor to review, not as a robot that makes final medical decisions on its own.

In a Nutshell

ClinNoteAgents is like a super-efficient, tireless team of interns that reads thousands of messy doctor's notes, organizes the important facts, and creates a short "cheat sheet" that helps doctors predict which heart failure patients are most likely to come back to the hospital soon. It turns the chaos of human writing into clear, life-saving data.

Technical Summary

1. Problem Statement

Heart failure (HF) is a leading cause of hospital readmission, imposing significant clinical and financial burdens. While Electronic Health Records (EHRs) contain valuable data, they are often incomplete in resource-limited settings (e.g., developing nations) or rely heavily on unstructured free-text clinical notes (which constitute ~80% of US clinical data).

• The Challenge: Traditional models rely on structured fields or expert-crafted rules to interpret notes, failing to capture rich contextual information, social determinants of health (SDOH), and domain-specific jargon found in free text.
• The Gap: Existing NLP approaches often focus solely on social factors or rely on predefined taxonomies, missing the integration of clinical predictors with social factors in a unified, scalable framework that minimizes manual annotation.

2. Methodology: ClinNoteAgents

The authors propose ClinNoteAgents, a modular, LLM-based multi-agent system designed to transform unstructured discharge notes into structured, interpretable representations for two interdependent tasks: Risk Factor Mining and Readmission Prediction.

A. Data Source

• Dataset: MIMIC-III (Medical Information Mart for Intensive Care III).
• Cohort: 2,065 HF patients with 3,544 discharge notes (35.16% readmission rate within 30 days).
• Input: Discharge summaries containing demographics, chief complaints, comorbidities, vitals, and SDOH.

B. System Architecture

The system utilizes Qwen3-14B (with thinking mode enabled) across three specialized agents:

1. Risk Factor Extractor:

   • Extracts structured data into three categories:
     • Charted SDOH: Gender, age, language, marital status.
     • Uncharted SDOH: Alcohol/tobacco/drug use, transportation, housing, employment, parental status, social support.
     • Clinical Measurements: Vitals (temp, HR, RR, SpO2, BP, weight, height), chief complaints, and diagnoses.
   • Output: JSON-structured tuples (e.g., {"Condition": "Renal cell cancer"}).

2. Risk Factor Normalizer:

   • Addresses heterogeneity in free-text SDOH entries.
   • Two-Stage Process:
     1. Normalizer: Generates a concise set of standardized categories (e.g., converting "lives with mom" to "Living with Family Members").
     2. Labeler: Assigns extracted values to these generated categories.
   • Goal: Enable statistical correlation analysis by converting unstructured text into categorical variables.

3. Note Summarizer:

   • Generates clinician-style abstracts for downstream predictive modeling.
   • Two Modes:
     • Overall Summary: Condenses notes into a narrative including numerical values.
     • No-Number Summary: Replaces all numerical values with qualitative descriptors to mitigate LLM instability with raw numbers.

C. Evaluation Framework

• Extraction Fidelity: Compared against structured EHR tables (surrogate ground truth) using conditional accuracy and tolerance ranges (e.g., ±0.5°F for temperature). Diagnosis extraction was evaluated via an LLM-as-a-judge framework comparing extracted text to ICD-9 codes.
• Statistical Analysis: Conducted logistic regression (for vitals/age) and Chi-square tests (for SDOH) to identify significant risk factors.
• Prediction Performance: Tested three classifiers (Logistic Regression, ClinicalBERT, LoRA-finetuned Qwen3-8B) on:
  1. Raw discharge notes.
  2. Overall summaries.
  3. No-number summaries.
  4. Structural extraction summaries.

3. Key Results

A. Extraction and Normalization Performance

• Vitals: High conditional accuracy for cardiorespiratory measures (SpO2: 94.16%, RR: 93.77%, BP: >90%). Anthropometric variables (Height: 68.50%, Weight: 57.90%) showed lower accuracy due to unit conversion challenges (ft/in vs. cm/kg).
• SDOH: Gender extraction was nearly perfect (99.18% coverage, 99.94% accuracy). Language and Marital Status had lower coverage due to under-documentation in notes.
• Diagnoses: The LLM extracted fewer diagnoses than ICD-9 codes (avg 5.91 vs. 15.14) but achieved a 62.67% conditional accuracy and a mean similarity score of 3.04/5, indicating semantic alignment.
• Normalization: Successfully standardized heterogeneous SDOH entries into 5–10 coherent categories per variable, enabling statistical testing.

B. Risk Factor Correlation

• Clinical Factors: Logistic regression identified Age (positive association), Weight (negative association), and Blood Pressure (negative association) as statistically significant predictors.
• Social Factors: Housing was the only SDOH significantly correlated with readmission (p=0.012). Other factors (smoking, social support) were not significant, likely due to under-documentation in the notes.

C. Readmission Prediction

• Baseline: Raw discharge notes achieved the highest AUROC (LR: 0.6535).
• Summarization Impact:
  • No-Number Summary: Performed best among summaries (AUROC ~0.64 for LR), retaining ~61% of text while preserving predictive signal.
  • Overall Summary: Showed moderate performance drop (AUROC ~0.59) despite 83% text reduction.
  • Structural Extraction: Largest drop (AUROC ~0.57) with 91% reduction.
• Key Finding: LLM summarization preserves the majority of risk-relevant signals (comorbidities, stability, key vitals) even with 60–90% text compression, making it viable for scalable modeling.

4. Key Contributions

1. Unified Multi-Agent Framework: Introduces a novel pipeline that jointly extracts and harmonizes clinical and social risk factors from unstructured notes without requiring manual annotation or task-specific pretraining.
2. Scalability in Data-Limited Settings: Demonstrates a viable approach for HF risk modeling in environments where structured EHR data is scarce or unavailable (e.g., developing nations), relying solely on discharge notes.
3. Interpretability: The system produces structured, clinician-style abstractions and identified risk factors, bridging the gap between "black box" deep learning and interpretable clinical decision support.
4. Empirical Validation: Provides rigorous evaluation of LLM extraction fidelity against structured EHRs and validates the utility of LLM-generated summaries for downstream prediction tasks.

5. Significance and Limitations

• Significance: ClinNoteAgents offers a practical solution for leveraging the "dark data" in unstructured clinical notes. It reduces reliance on expensive structured data collection and manual feature engineering, enabling early detection of high-risk HF patients in diverse healthcare settings.
• Limitations:
  • Indirect Evaluation: Reliance on structured EHR fields as surrogate ground truth for SDOH and vitals.
  • Performance Plateau: Summarization did not improve prediction over raw notes, only maintained performance with reduced token counts.
  • Hallucination Risks: LLMs remain prone to omissions and hallucinations; the system is designed as a decision-support tool requiring clinician verification.
  • Documentation Bias: Low extraction coverage for certain variables (e.g., housing, marital status) reflects under-documentation in source notes rather than model failure.

In conclusion, ClinNoteAgents successfully operationalizes the use of LLMs to transform free-text clinical narratives into actionable, structured insights for heart failure readmission risk, balancing computational efficiency with clinical interpretability.