A Dataset and Resources for Identifying Patient Health Literacy Information from Clinical Notes

1. Problem Statement

Health literacy—the ability to obtain, understand, and use medical information to make informed decisions—is a critical determinant of patient outcomes. Low health literacy is associated with higher readmission rates, poorer health management, and increased healthcare utilization.

Despite its importance, current challenges include:

• Feasibility: Standardized screening tools (e.g., surveys) are often impractical for acutely ill inpatients and are not a priority for busy clinicians.
• Data Scarcity: Consequently, health literacy is rarely recorded as structured data in Electronic Health Records (EHRs). Instead, it exists as unstructured, narrative observations within clinical notes (e.g., social worker notes, discharge summaries).
• Extraction Difficulty: Manually reviewing these unstructured notes is labor-intensive. While Natural Language Processing (NLP) offers a solution, the field lacks a robust, annotated dataset derived from real clinical notes to train and evaluate automated detection models.

2. Methodology

The authors developed HEALIX, the first publicly available annotated dataset for health literacy in clinical notes, using a multi-stage curation and annotation pipeline.

A. Data Collection & Sampling

Data was sourced from the MIMIC-III database. The authors employed a three-pronged sampling strategy to maximize the retrieval of relevant notes while managing annotation costs:

1. Social Worker Note Sampling: A random sample of 300 social worker notes was selected, as these are most likely to contain patient behavioral and literacy assessments.
2. Keyword-Based Filtering: A list of 20+ keywords (e.g., "adherence," "insight," "compliance," "understands") was used to filter notes. For each keyword, up to 3 notes were sampled. Context windows (5 preceding and 5 following sentences) were extracted to capture relevant information.
3. LLM-Based Active Learning: To address class imbalance and increase diversity, an LLaMA 3-8B-Instruct model was used in an active learning loop. The model predicted labels for unannotated notes, and those with the highest predictive entropy (uncertainty) were selected for manual annotation.

B. Annotation Protocol

• Labeling Scheme: Notes were annotated at the note level into three categories: Low, Normal, and High health literacy. A fourth category, Not Related, was used for notes lacking literacy indicators.
  • Low: Patient cannot understand/interpret/use health info (e.g., "does not understand need to follow up").
  • Normal: Patient can verbalize understanding of diagnosis/treatment (e.g., "patient is aware of treatment plan").
  • High: Patient demonstrates understanding beyond verbal confirmation, showing critical thinking or informed decision-making (e.g., "demonstrated medication regimen," "weighing risks").
• Process: Two annotators independently labeled 589 notes. Disagreements were reconciled to create a gold-standard dataset.
• Context Handling: If a patient was cognitively impaired, the label was assigned to the decision-maker (e.g., family member) acting on their behalf.

C. Model Evaluation

The authors benchmarked zero-shot and few-shot prompting strategies across four open-source Large Language Models (LLMs) and a traditional SVM baseline:

• Models: LLaMA 3-8B, LLaMA 3.3-70B, Qwen 3-8B, Qwen 3-32B, and SVM.
• Task: Multiclass text classification.
• Evaluation Metrics: Precision, Recall, and F1-score were calculated under two schemes:
  • Strict: 4 classes (Low, Normal, High, Not Related).
  • Lenient: 3 classes (collapsing Normal and High into "Good").

3. Key Contributions

1. HEALIX Dataset: The creation of a novel, publicly available dataset containing 589 clinical notes across 9 note types (Social Work, Nursing, Discharge, Physician, etc.), annotated with fine-grained health literacy labels.
2. Novel Annotation Framework: The first study to annotate specific health literacy indicators in unstructured clinical narratives, distinguishing between "verbal confirmation" (Normal) and "demonstrated critical thinking" (High).
3. Active Learning Pipeline: Demonstration of a hybrid data curation workflow combining keyword filtering and LLM-based uncertainty sampling to efficiently build domain-specific datasets.
4. Baseline Benchmarks: Comprehensive evaluation of state-of-the-art LLMs on this task, providing a performance baseline for future research.

4. Results

Dataset Characteristics

• Distribution: 75 Low, 186 Normal, 45 High, and 283 Not Related notes.
• Demographics: Primarily Emergency Room patients (80%), White (68%), and English-speaking (67%). The 60-69 age group was most represented.
• Note Types: Social Work notes contained the highest proportion of Low literacy instances (53.3%), followed by Discharge Summaries.

Model Performance

• Best Performer: LLaMA 3.3-70B-Instruct achieved the highest performance, with an F1-score of 0.63 (Lenient) and 0.51 (Strict).
• Baseline: The SVM classifier underperformed all LLMs (Strict F1: 0.41).
• Zero-Shot vs. Few-Shot: Few-shot prompting did not consistently improve performance over zero-shot for these specific models; in some cases, it slightly degraded results.
• Error Analysis:
  • Models struggled most to distinguish between Normal and Not Related (confusing indirect cues with irrelevant text).
  • Confusion also existed between High and Normal, highlighting the difficulty in detecting "demonstrated" understanding versus "verbalized" understanding.
  • Inter-Annotator Agreement: Strict note-level F1 was 0.52, rising to 0.63 under the lenient scheme, indicating the inherent subjectivity of the task.

5. Significance and Future Directions

• Clinical Impact: HEALIX enables the automated identification of patients with low health literacy directly from routine clinical documentation. This allows healthcare systems to target educational interventions, potentially reducing readmissions and improving care equity.
• NLP Advancement: The study highlights that while LLMs show promise, they currently lack the deep contextual reasoning required to distinguish subtle nuances in health literacy (e.g., family inquiry vs. patient understanding).
• Limitations:
  • Source Bias: Data is limited to MIMIC-III (one hospital), which may not represent broader populations.
  • Context Truncation: Some notes were extracted as excerpts, potentially missing broader context.
  • Fine-Tuning: Models were evaluated in their pre-trained state; fine-tuning on HEALIX is expected to yield significant performance gains.

Conclusion: HEALIX fills a critical gap in health informatics by providing the necessary resource to transition health literacy detection from manual screening to scalable, automated NLP solutions, ultimately aiming to improve patient outcomes through better-targeted care.