💬 NLP

EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portal

This paper introduces EPPCMinerBen, a novel benchmark comprising three sub-tasks and 1,933 expert-annotated sentences from Yale New Haven Hospital's patient portal, to evaluate the performance of various large language models in analyzing electronic patient-provider communication, revealing that larger, instruction-tuned models generally outperform smaller ones, particularly in evidence extraction and fine-grained reasoning.

Original authors: Samah Fodeh, Yan Wang, Linhai Ma, Srivani Talakokkul, Jordan M. Alpert, Sarah Schellhorn

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samah Fodeh, Yan Wang, Linhai Ma, Srivani Talakokkul, Jordan M. Alpert, Sarah Schellhorn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling hospital where doctors and patients used to chat face-to-face, but now they mostly talk through a secure digital mailbox (the "Patient Portal"). These messages are goldmines of information: patients ask about side effects, doctors explain dosages, and together they make decisions. But there's a problem: there are too many messages for humans to read and understand one by one.

Enter EPPCMinerBen, a new "test drive" created by researchers to see if Artificial Intelligence (AI) can act as a super-efficient assistant to read, understand, and summarize these digital conversations.

Here is the story of the paper, broken down into simple concepts:

1. The Problem: The "Needle in a Haystack"

Think of the patient portal messages as a giant haystack. Hidden inside are needles—specific pieces of information like "The patient is worried about a side effect" or "The doctor is encouraging the patient."

Old Way: Humans had to read every single message and manually tag these needles. It was slow, expensive, and hard to scale.
New Hope: Can AI (Large Language Models or LLMs) do this automatically?
The Gap: We didn't have a good "ruler" to measure if the AI was actually doing a good job. Most AI tests were like asking a student to solve math problems, but we needed to see if they could understand a conversation.

2. The Solution: EPPCMinerBen (The "Driver's License Test" for AI)

The researchers built a special benchmark called EPPCMinerBen. Think of this as a driving test for AI cars, but instead of driving on a road, the AI is driving through a conversation.

To pass the test, the AI has to do three specific tasks for every sentence in a message:

The Big Picture (Code Classification): What is the main topic? (e.g., "Is this about a drug?" or "Is this about feelings?")
The Fine Details (Subcode Classification): What is the specific nuance? (e.g., "Is the patient asking for a drug, or complaining about it?")
The Proof (Evidence Extraction): Point to the exact words in the sentence that prove your answer. (e.g., Highlighting "I feel dizzy" to prove the patient is reporting a symptom).

3. The Contestants: The AI Race

The researchers gathered a "race track" of different AI models to see who would win.

The Heavyweights: Giant models with huge brains (70 billion parameters). Think of these as PhD students who read every book in the library.
The Middleweights: Medium-sized models.
The Lightweight: Tiny models (1–3 billion parameters). Think of these as smart high schoolers.
The Specialists: Models trained specifically on medical texts or social issues.

They tested these models in two ways:

Zero-Shot: "Here is the task. Go!" (No examples given).
Few-Shot: "Here is the task, and here are three examples of how to do it." (Like showing a student a practice test).

4. The Results: Who Won the Race?

The results were surprising and taught us a lot about how AI thinks:

Size Matters (But Not Always): The biggest, most powerful models generally won. They were the best at finding the "proof" (Evidence Extraction), getting about 83% accuracy. They could read the room and understand the context.
The "Distilled" Surprise: One medium-sized model (DeepSeek-R1-Distill) punched way above its weight class. It was like a small car with a turbo engine, performing almost as well as the giants. This suggests that teaching an AI how to think (reasoning) is just as important as how big its brain is.
The "Small Model" Struggle: The tiny models (1B or 3B) struggled, especially with the "Fine Details" (Subcodes). They often got confused, like a student who knows the vocabulary but can't understand the grammar.
The "Few-Shot" Boost: Giving the AI a few examples (Few-Shot) helped almost everyone, but it was a lifeline for the smaller models. It's like giving a student a cheat sheet; without it, they failed, but with it, they passed.
The Specialist Trap: Interestingly, models trained only on medical books didn't always beat the general-purpose models. Understanding human conversation (emotions, tone, back-and-forth) requires more than just medical facts; it requires "social intelligence."

5. Why This Matters

Imagine a future where an AI assistant sits in the background of a hospital's digital mailbox.

It reads thousands of messages instantly.
It flags to a doctor: "Hey, this patient is confused about their medication dosage."
It highlights the exact sentence: "I don't know if I should take this with food."
It helps doctors prioritize who needs help now.

This paper proves that while AI is getting very good at this, it's not perfect yet. It needs the right "training" (prompts) and the right "brain size" to handle the messy, emotional, and complex nature of human health conversations.

The Takeaway

EPPCMinerBen is the new standard ruler. It tells us that to build AI that truly helps doctors and patients, we need models that are not just smart, but also nuanced, context-aware, and capable of finding the "why" behind the words. It's a big step toward making healthcare communication more human, even when it's happening through a screen.

1. Problem Statement

Effective healthcare management, particularly for chronic conditions like cancer, relies heavily on consistent patient adherence, which is often hindered by communication breakdowns between patients and providers. While Electronic Patient-Provider Communication (EPPC) via secure messaging portals generates vast amounts of unstructured data containing valuable insights into patient needs, barriers, and intent, analyzing this data remains challenging.

Key Gaps Identified:

Scalability: Existing methods rely on manual coding or rule-based techniques, which are costly and lack scalability.
Contextual Limitations: Current Machine Learning (ML) approaches often focus solely on patient-authored messages, ignoring the interactive, bidirectional nature of the dialogue and the provider's response context.
Lack of Specialized Benchmarks: Existing biomedical NLP benchmarks (e.g., CBLUE, MedDG) focus on structured clinical dialogues or task-oriented interactions. They fail to capture the informal, socio-emotional, and relational dynamics specific to secure messaging.
Data Scarcity: There is a lack of de-identified, richly annotated datasets that reflect the complexity of real-world EPPC.

2. Methodology

2.1. Dataset Construction (EPPCMinerBen)

The authors constructed a novel dataset derived from 752 secure messages (449 patient, 303 provider) from the Yale New Haven Hospital patient portal.

Scale: 1,933 unique sentences containing 27,849 words and 33,388 tokens.
Annotation: Messages were annotated using a codebook inspired by the Roter Interaction Analysis System (RIAS).
Hierarchy: The annotation scheme is hierarchical:
- Codes (Coarse-grained): 9 high-level communication behaviors (e.g., "Information Giving," "Patient Partnership," "Shared Decision-Making").
- Subcodes (Fine-grained): Specific categories under each code (e.g., "Diagnostics," "Drugs," "Salutation").
Quality Control: Two annotators independently labeled the data with a Cohen's Kappa of 0.74. Discrepancies were resolved by senior adjudicators.

2.2. Task Formulation

The benchmark evaluates Large Language Models (LLMs) on three sequential sub-tasks designed to test discourse understanding and reasoning:

Code Classification: A sentence-level multi-label classification task to identify high-level communicative intents.
Subcode Classification: A conditional multi-class classification task that refines the high-level code into a specific subcode based on the predicted code.
Evidence Extraction: An information extraction task to identify the minimal text span within the sentence that justifies the predicted code-subcode pair.

2.3. Experimental Setup

Models Evaluated: A diverse range of LLMs was tested, categorized by size and domain:
- Large-scale (70B): Llama-3.1/3.3-70B-Instruct, DeepSeek-R1-Distill-Llama-70B.
- Medium-scale (14B–32B): QwQ-32B, Gemma-2-27B, DeepSeek-R1-Distill-Qwen-32B.
- Small-scale (1B–8B): Llama-3.1/3.2 series, Qwen2.5 series.
- Clinical-domain: Llama3-OpenBioLLM-70B, sdoh-llama-3-70B (focused on Social Determinants of Health).
Prompting Strategies: Models were evaluated under Zero-shot (no examples) and Few-shot (1-shot and 2-shot) settings to assess in-context learning capabilities.
Metrics: Micro-averaged Precision, Recall, and F1-score. Evidence extraction used a relaxed token-level matching strategy (Jaccard similarity $\ge$ 0.6).

3. Key Contributions

EPPCMinerBen Benchmark: The first benchmark specifically designed to evaluate LLMs on the interactive, relational, and socio-emotional dynamics of secure patient-provider messaging, addressing a critical gap in existing clinical dialogue datasets.
Richly Annotated Dataset: A de-identified dataset capturing bidirectional communication patterns, including information seeking, socio-emotional expressions, and shared decision-making, released via the NCI Cancer Data Service.
Hierarchical Evaluation Framework: A structured pipeline (Code $\to$ Subcode $\to$ Evidence) that tests models on discourse-level understanding, hierarchical reasoning, and grounding-based interpretability.
Comprehensive LLM Analysis: A systematic evaluation of how model scale, instruction tuning, and domain adaptation affect performance on nuanced clinical communication tasks.

4. Key Results

4.1. Overall Performance Trends

Model Scale Matters: Large, instruction-tuned models (70B parameters) generally outperformed smaller models, particularly in evidence extraction and classification.
Task Difficulty: Performance decreased as tasks became more granular:
- Evidence Extraction: Easiest task (Top F1: 82.84%).
- Code Classification: Moderate difficulty (Top F1: 67.03%).
- Subcode Classification: Hardest task (Top F1: 48.25%), highlighting the difficulty of fine-grained semantic reasoning.

4.2. Specific Model Findings

Top Performers:
- Llama-3.1-70B-Instruct: Led in Evidence Extraction (F1: 82.84%).
- Llama-3.3-70B-Instruct: Best in Code Classification (F1: 67.03%).
- DeepSeek-R1-Distill-Qwen-32B: Excelled in Subcode Classification (F1: 48.25%), outperforming larger 70B models in this specific task, suggesting distillation enhances hierarchical reasoning.
Few-Shot Impact: Few-shot prompting generally improved performance, especially for smaller models (e.g., Llama-3.1-8B failed at 0-shot but recovered to >50% with 1-shot). However, some models (e.g., Gemma-2-27B) showed performance degradation in few-shot settings, suggesting sensitivity to context overflow.
Domain Adaptation: The domain-specific model sdoh-llama-3-70B showed consistent performance but did not significantly outperform general-purpose instruction-tuned models. Conversely, Llama3-OpenBioLLM-70B underperformed, indicating that biomedical pre-training alone does not guarantee success in discourse analysis without strong instruction alignment.
Small Models: Models under 3B parameters struggled significantly, especially in subcode classification (F1 < 16%), failing to capture fine-grained reasoning.

4.3. Ablation Studies

Hierarchical vs. Flat: Separating code and subcode prediction (hierarchical approach) yielded significantly better results than a joint prediction task, confirming the value of coarse-to-fine reasoning.
Prompt Guidance: Removing explicit reasoning instructions from prompts caused a massive drop in Evidence Extraction performance (>35% F1 drop), proving that explicit task guidance is critical for alignment.

5. Significance and Implications

Clinical Utility: EPPCMinerBen enables the development of tools that can automatically identify communication breakdowns, unmet patient needs, and adherence barriers at scale, potentially improving care coordination and patient satisfaction.
NLP Advancement: The benchmark pushes the boundaries of clinical NLP by moving beyond entity extraction to discourse-level understanding and relational dynamics, requiring models to handle bidirectional, informal, and emotionally nuanced text.
Future Directions: The work highlights the need for multi-institutional, multilingual datasets and the exploration of fine-tuning strategies to further improve model generalization in diverse clinical settings.

In conclusion, EPPCMinerBen establishes a rigorous standard for evaluating LLMs in healthcare communication, demonstrating that while large instruction-tuned models show promise, significant challenges remain in fine-grained reasoning and the effective integration of socio-emotional context.