PVminerLLM: Structured Extraction of Patient Voice from Patient-Generated Text using Large Language Models

This paper introduces PVminer, a benchmark for structured extraction of patient voice from patient-generated text, and presents PVminerLLM, a supervised fine-tuned large language model that significantly outperforms prompt-based baselines in extracting codes, sub-codes, and evidence spans to enable scalable analysis of non-clinical health drivers.

The Gist

Imagine you are a doctor trying to understand your patients. You have their medical charts, which are like spreadsheets: they tell you exactly what medicine they take, their blood pressure numbers, and their test results. These are easy to read and analyze.

But patients also send you messages, fill out surveys, and tell stories about their lives. These are like journals written in a messy, emotional handwriting. They might say, "I'm scared to take this pill because I can't afford it," or "My landlord is evicting me, so I'm stressed."

This paper is about a new tool called PVminerLLM that helps doctors and researchers read those messy journals and turn them into neat, organized data, just like the spreadsheets.

Here is the story of how they built it, explained simply:

1. The Problem: The "Lost in Translation" Gap

For a long time, computers were great at reading medical charts (the spreadsheets) but terrible at reading patient stories (the journals).

• The Challenge: A patient's message is full of hidden clues. They might mention "housing instability" (they might lose their home) or "shared decision-making" (they want to help choose their treatment).
• The Old Way: Humans had to read every single message and manually write down these clues. This is like hiring a team of translators to read a million letters by hand. It's slow, expensive, and impossible to do for everyone.
• The New Hope: We have powerful AI (Large Language Models) that can read and write like humans. But when the researchers asked these AIs to read the patient messages, the AIs got confused. They would give vague answers, miss the point, or format the data messily. It was like asking a brilliant student to fill out a very strict, complicated government form without any practice—they knew the material, but they kept failing the specific test.

2. The Solution: Two Steps to Success

The researchers tried two approaches to teach the AI how to read these patient voices.

Step A: The "Strict Teacher" (Prompt Engineering)

First, they tried to be very specific with the AI. They wrote a super-detailed set of instructions (a "prompt") telling the AI exactly how to format the answer, what categories to look for, and how to quote the text.

• The Analogy: Imagine giving a robot a recipe and saying, "If you see a tomato, write 'Tomato' in column 3. If you see a sad face, write 'Sad' in column 4."
• The Result: It helped a little. The robot got better at following rules, but it still missed the subtle meanings. It was like a robot that could follow the recipe but didn't understand why the ingredients mattered. It was still making mistakes, especially with rare or tricky topics.

Step B: The "Apprentice" (Supervised Fine-Tuning)

This was the big breakthrough. Instead of just giving the AI instructions, they trained it. They showed the AI thousands of examples of patient messages and the correct way to label them. They let the AI practice, make mistakes, and learn from the corrections.

• The Analogy: Instead of just giving the robot a recipe, they hired a master chef (the human experts) to stand next to the robot for weeks, correcting its chopping, seasoning, and plating until the robot learned the art of cooking, not just the rules.
• The Result: This worked amazingly well. The AI (now called PVminerLLM) became an expert. It could read a messy patient message and instantly pull out the specific clues: "Ah, this person is worried about money (Social Determinant of Health)" or "This person wants to be part of the decision (Shared Decision Making)."

3. The Surprise: You Don't Need a Giant Brain

Usually, people think you need the biggest, most expensive AI supercomputer to do hard tasks.

• The Discovery: The researchers found that even smaller, cheaper AI models performed just as well as the giant ones after they were trained (fine-tuned).
• The Metaphor: It's like realizing you don't need a PhD in physics to fix a leaky faucet; you just need the right tool and a little practice. A small, specialized AI can do this job better than a giant, general AI that hasn't been trained for it. This means hospitals with smaller budgets can use this tool too.

4. Why This Matters (The "So What?")

Why do we care about turning patient stories into data?

• Seeing the Invisible: Right now, if a patient is struggling to pay for rent, that stress might not show up in their medical chart. But it affects their health! This tool makes those invisible struggles visible to doctors.
• Better Care: If a doctor knows a patient is stressed about housing, they can connect them with a social worker, not just prescribe more pills.
• Fairness: It helps ensure that the voices of people from different backgrounds are heard and counted, leading to fairer healthcare for everyone.

Summary

Think of PVminerLLM as a super-powered translator.

• Before: Patient stories were like a pile of unsorted letters in a language only a few humans could read.
• After: This tool reads those letters and instantly sorts them into neat folders: "Money Worries," "Emotional Support," "Treatment Choices."
• The Magic: It doesn't need a super-computer to do it; a trained, smaller AI can do the job perfectly, helping doctors understand their patients' real lives, not just their medical numbers.

Technical Summary

Here is a detailed technical summary of the paper "PVminerLLM: Structured Extraction of Patient Voice from Patient-Generated Text using Large Language Models."

1. Problem Statement

Patient-generated text (e.g., secure portal messages, survey responses, interview narratives) contains critical, unstructured information regarding patients' lived experiences, social determinants of health (SDOH), and engagement in care. However, this "patient voice" is rarely available in a structured format, limiting its utility in patient-centered outcomes research and clinical quality improvement.

Current challenges include:

• Unstructured Nature: The text is highly informal, overlapping, and context-dependent.
• Complexity: Signals often involve multiple simultaneous categories (e.g., emotional distress + financial insecurity + care preferences).
• Extraction Difficulty: Existing computational methods often focus on clinical notes (EHRs) rather than patient-generated text, or they reduce complex narratives to narrow, surface-level entities, missing the richness of social and relational contexts.
• Scalability: Manual abstraction is labor-intensive and expensive, while standard Large Language Model (LLM) prompting often fails to produce strictly structured, schema-compliant outputs required for large-scale analysis.

2. Methodology

The authors propose the PVminer Framework, which formalizes patient voice extraction as a schema-constrained structured prediction problem.

A. Task Formulation

The task involves extracting hierarchical labels and grounding them in specific text spans from a patient message ($s$).

• Inputs: A patient message and a direction indicator (Provider-to-Patient or Patient-to-Provider).
• Outputs: A set of tuples $(c, u, r)$, where:
  • $c$: A Code (high-level semantic category, e.g., "Partnership," "SDOH").
  • $u$: A Sub-code (granular distinction, e.g., "Build Trust," "Economic Stability").
  • $r$: An Evidence Span (the exact contiguous text in the input supporting the label).
• Schema: The schema includes 8 major Codes and 26 Sub-codes, allowing for multi-label, overlapping annotations.

B. Data and Annotation

• Dataset: A corpus of 1,137 messages (757 patient-authored, 380 provider-authored) from diverse sources (Yale New Haven Health, Texas charitable clinics, patient surveys).
• Annotation: Performed by domain experts using the eHOST platform. The dataset exhibits severe label imbalance, with a long tail of rare but semantically critical categories.
• Demographics: The dataset represents diverse racial, ethnic, and socioeconomic backgrounds.

C. Approaches Evaluated

1. Prompt Engineering (Zero-shot & Few-shot):
   • Baseline Prompt: Minimal task description.
   • Engineered Prompt: A sophisticated prompt featuring explicit output schemas, decision logic, disambiguation rules, self-validation constraints, and direction-aware control signals. It forces the model to decompose the task into interpretation, label selection, and span verification steps.
2. Supervised Fine-Tuning (PVminerLLM):
   • Model Architecture: Instruction-tuned LLMs (Llama-3.3-70B, Llama-3.1-8B, Llama-3.2-3B, Qwen2.5-1.5B) were fine-tuned using QLoRA (parameter-efficient adapters).
   • Training Objective: The model learns to map input messages to a serialized JSON string containing the structured annotations. A masked likelihood objective ensures the model focuses on generating valid Code/Sub-code combinations and exact Span boundaries, ignoring the instruction tokens during loss calculation.

3. Key Contributions

1. PVminer Framework: A novel formulation for extracting patient voice as a schema-constrained, multi-label, span-grounded structured prediction task.
2. Benchmark Creation: A comprehensive benchmark comparing prompt-based approaches (zero-shot/few-shot) against supervised fine-tuning across models ranging from 1.5B to 70B parameters.
3. PVminerLLM: A suite of supervised fine-tuned models specifically optimized for this task, demonstrating that high-fidelity extraction is achievable without relying on extreme model scale.
4. Systematic Analysis: Detailed evaluation showing that prompt engineering alone is insufficient for reliable extraction, while fine-tuning significantly bridges the gap between precision and recall.

4. Results

Performance Metrics

The models were evaluated on Code Prediction, Sub-code Prediction, and Span Extraction using Precision, Recall, and F1-scores.

• Prompt Engineering:

  • The "Engineered Prompt" significantly outperformed the baseline but still struggled with recall, particularly for rare Sub-codes and Span boundaries.
  • Zero-shot (70B Model): F1 scores were moderate (Code: ~62%, Sub-code: ~44%, Span: ~55%).
  • Few-shot (2-shot): Showed incremental improvements but failed to close the precision-recall gap.

• Supervised Fine-Tuning (PVminerLLM):

  • Significant Gains: Fine-tuning yielded substantial improvements across all metrics and model sizes.
  • Top Performance (70B Model):
    • Code F1: 83.82% (vs. 62.25% in zero-shot).
    • Sub-code F1: 80.74% (vs. 43.71% in zero-shot).
    • Span F1: 87.03% (vs. 55.04% in zero-shot).
  • Model Size Independence: Smaller models (e.g., 3B and 1.5B) achieved performance comparable to the 70B model after fine-tuning (e.g., 3B model achieved 80.33% Code F1). This indicates that task-specific supervision is more critical than model scale.

Domain-Specific Insights

• Prevalent Domains: "Partnership" and "SDOH" saw the most dramatic improvements with fine-tuning. For example, SDOH F1 jumped from 60.22% (2-shot) to 89.26% (SFT).
• Rare Domains: Fine-tuning significantly reduced the performance gap for rare and nuanced categories (e.g., "Shared Decision Making," "Active Participation") which were often missed by prompting.
• Error Reduction: Fine-tuning reduced common errors such as role confusion (Patient vs. Provider) and boundary hallucinations.

5. Significance and Implications

• Scalable Patient-Centered Care: PVminerLLM enables the systematic extraction of social and experiential signals from unstructured text at scale, moving beyond manual chart reviews.
• Health Equity: By reliably identifying SDOH (housing, financial insecurity) and psychosocial stressors, healthcare systems can better address non-clinical drivers of health outcomes and design targeted interventions.
• Feasibility of Deployment: The finding that smaller models (3B-8B) perform nearly as well as massive models after fine-tuning makes this technology accessible to resource-constrained healthcare settings (e.g., community clinics) without requiring massive computational infrastructure.
• Research Infrastructure: The framework provides a rigorous testbed for integrating patient voice into patient-centered outcomes research (PCOR), ensuring that social contexts are not lost in data-driven decision-making.

In conclusion, the paper demonstrates that while LLMs possess the semantic understanding to interpret patient text, supervised fine-tuning is essential to enforce the strict structural constraints required for reliable, large-scale clinical and social signal extraction.