PVminer: A Domain-Specific Tool to Detect the Patient Voice in Patient Generated Data

1. Problem Statement

Patient-generated data (e.g., secure messages, surveys, interviews) contains rich expressions of the Patient Voice (PV), encompassing both Patient-Provider Communication (PCC) behaviors (e.g., shared decision-making, partnership) and Social Determinants of Health (SDoH) (e.g., economic stability, access to care).

Current approaches face three critical limitations:

• Scalability: Traditional manual qualitative coding is labor-intensive and cannot scale to the massive volume of unstructured text generated in modern health systems.
• Fragmentation: Existing Machine Learning (ML) and Natural Language Processing (NLP) methods typically treat PCC and SDoH as separate tasks or rely on single-label classification, failing to capture the overlapping, multi-label nature of patient messages (where a single message may express gratitude, a financial struggle, and a medical query simultaneously).
• Domain Misalignment: Most pre-trained models (e.g., BioBERT, ClinicalBERT) are trained on provider-authored clinical notes or biomedical literature, which differ significantly in tone, structure, and vocabulary from patient-authored narratives.

2. Methodology: The PVminer Framework

The authors propose PVminer, a domain-adapted NLP framework designed to structure patient voice through a multi-label, multi-class prediction task. The framework consists of four integrated components:

A. Domain-Adapted Pre-training (PV-BERT)

To address the gap between clinical literature and patient language, the authors pre-trained two BERT variants on 6 million unlabeled patient-provider messages:

• PV-BERT-base and PV-BERT-large.
• Objective: Masked Language Modeling (MLM).
• Data: Segmented patient messages from Yale New Haven Health (YNHH), filtered to 510 tokens.
• Result: Models specifically tuned to the conversational, context-rich, and often informal nature of patient communication.

B. Unsupervised Topic Modeling (PV-Topic-BERT)

To capture latent thematic structures that might be missed by surface-level text:

• A BERTopic model was trained on 500k unlabeled messages using MiniLM-L6-v2 embeddings, UMAP for dimensionality reduction, and HDBSCAN for clustering.
• Output: A model that assigns topic distributions and extracts top keywords (e.g., "vaccine," "insurance," "bloodwork") for any given message.

C. Topic-Augmented Fine-Tuning

The core innovation of PVminer is the integration of topic information into the classification pipeline:

• Input Augmentation: For every input message, the most likely topic keywords (top 3) derived from the PV-Topic-BERT model are appended to the message text before tokenization.
• Metadata Injection: Author identity (Patient vs. Provider) is prepended as a special token (e.g., [PATIENT]) to help the model distinguish between requests and directives.
• Task Formulation: The system treats PV detection as a multi-label, multi-class classification problem. It predicts three hierarchical levels simultaneously:
  1. Code: High-level domains (e.g., Partnership, SDoH).
  2. Subcode: Fine-grained intents (e.g., Economic Stability, Gratitude).
  3. Combo: Unique pairings of Code + Subcode.

D. Evaluation Metrics

Performance was measured using Micro-averaged Precision, Recall, and F1-score across 5 independent runs, specifically designed for imbalanced multi-label datasets.

3. Key Contributions

1. Unified Multi-Label Task: Defined a comprehensive detection task that captures the layered, overlapping nature of patient voice (PCC + SDoH) rather than forcing mutually exclusive categories.
2. PVminer Framework: An integrated pipeline combining domain-specific pre-training, topic-informed representation learning, and hierarchical multi-label prediction.
3. Domain-Specific Models: Released PV-BERT-base, PV-BERT-large, and PV-Topic-BERT, demonstrating that pre-training on patient-generated text outperforms general and biomedical baselines.
4. Ablation Evidence: Quantified the specific value of author identity and topic augmentation, proving they are essential for high-fidelity detection.

4. Results

The study compared PVminer (using various encoders) against general-domain (BERT), biomedical (BioBERT, ClinicalBERT), and scientific (SciBERT) baselines.

• Code-Level Classification:
  • PV-BERT-large achieved the highest F1-score of 82.25%, outperforming the best baseline (BERT-base at 80.97%) and biomedical models (BioBERT at 78.64%).
• Subcode-Level Classification:
  • PV-BERT-base achieved the highest F1-score of 80.14%.
  • Domain-specific models consistently outperformed biomedical models, highlighting that patient language is distinct from clinical notes.
• Combo-Level Classification (Most Challenging):
  • PV-BERT-large achieved an F1-score of 77.87%, significantly higher than baselines (e.g., BioBERT at 74.70%).
• Ablation Study Findings:
  • Base PV-BERT-large: 73.94% (Combo F1).
  • + Author Identity: Improved to 75.33%.
  • + Topic Augmentation (Full PVminer): Achieved 77.87%.
  • Conclusion: Topic augmentation provided the most significant performance gain, helping the model disambiguate short or context-limited messages.

Class-Specific Performance:

• High Performance: Categories with explicit lexical cues (e.g., Partnership: Salutation/Signoff, SDOH: Economic Stability) achieved F1 > 85%.
• Low Performance: Categories requiring implicit reasoning or having sparse data (e.g., Shared Decision-Making: Provider Approval, Neighborhood & Built Environment) struggled (F1 < 50%), indicating a need for more data and context-aware modeling.

5. Significance and Impact

• Clinical Workflow Integration: PVminer enables the automated structuring of unstructured patient messages, allowing for:
  • Triage: Prioritizing high-risk messages (e.g., those indicating financial barriers or severe symptoms).
  • Quality Monitoring: Tracking the frequency of partnership behaviors and shared decision-making in digital care.
  • SDoH Integration: Systematically identifying social barriers (housing, transport, cost) that are often missed in structured EHR fields.
• Methodological Advancement: Demonstrates that domain adaptation (pre-training on patient text) combined with thematic augmentation (topic modeling) is superior to simply fine-tuning existing biomedical models for patient-facing tasks.
• Scalability: Provides a scalable solution to extract the "Patient Voice" at a scale impossible for human coders, supporting patient-centered outcomes research (PCOR).

Limitations & Future Work:
The authors note challenges with class imbalance (rare categories) and implicit meaning (shared decision-making nuances). Future work aims to expand annotated datasets, apply adversarial augmentation, and incorporate conversational context (multi-turn message threads) to better capture longitudinal concerns.