Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients

Technical Summary: Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Pemphigus

Problem Statement
Chronic dermatologic diseases, such as pemphigus, require lifelong follow-up, resulting in extensive, heterogeneous longitudinal clinical documentation. In routine practice, clinicians face significant challenges in synthesizing years of visit notes authored by different physicians to guide management decisions. The current reliance on partial chart reviews or copy-pasting previous documentation introduces variability and increases the risk of overlooking critical historical details (e.g., prior rituximab exposure, cumulative corticosteroid toxicity, or serious infections). While Artificial Intelligence (AI) and Large Language Models (LLMs) offer potential for clinical summarization, existing solutions often rely on cloud-based inference, raising concerns regarding patient privacy, data governance, and deployability in strict healthcare environments. Furthermore, most studies focus on short texts or synthetic tasks rather than real-world, multi-year longitudinal records.

Methodology
This retrospective case series evaluated a fully local, privacy-preserving Small Language Model (SLM) system designed to retrieve structured clinical features and generate longitudinal summaries from raw, unprocessed dermatology follow-up records.

• Dataset: The study utilized data from 30 pemphigus patients treated at Istanbul Research and Training Hospital. The dataset comprised 541 visit notes aggregated into 30 complete longitudinal records (totaling 89,336 words). No manual summarization or preprocessing was applied; notes were extracted directly from the hospital information system (DermaBase).
• Ground Truth: Two expert dermatologists annotated 56 clinically relevant features (categorical, numeric, and date-based) and generated clinician-facing final reports for each patient to serve as ground truth.
• Model and Deployment: The system employed the Qwen3 4B Thinking 2507 model, deployed locally on a workstation (RTX 4090, 24 GB VRAM, 96 GB RAM) via LM Studio. Internet access was disabled to ensure data privacy.
• Inference Procedure:
  • The model processed each patient's full longitudinal record as a single input (context length of 90,000 tokens).
  • Inference was conducted in two steps: first, retrieving 56 specific structured features; second, generating a free-text summary report (current_status).
  • To address the model's lack of medical domain training, minimal explanatory context (e.g., specific T-score thresholds for osteoporosis) was embedded directly into the prompts for features requiring standardized definitions.
  • The inference procedure was repeated 10 times per patient to assess stability, resulting in 16,800 total inferences.
• Evaluation:
  • Structured Features: Evaluated using exact-match accuracy against expert annotations.
  • Free-Text Summaries: Evaluated using automated metrics (BLEU, ROUGE) and a double-blinded human assessment by two expert dermatologists (>20 years experience) on a 10-point Likert scale for overall quality, clinical accuracy, and usefulness.

Key Results

• Feature Retrieval: The system achieved a mean accuracy of 82.25% across 1,680 feature retrieval tasks. Performance varied by data type:
  • Categorical features: 86.34%
  • Date-based features: 80.56%
  • Numerical features: 70.61% (errors primarily involved imprecise extraction rather than gross misclassification).
• Report Generation:
  • Automated text similarity scores were moderate (BLEU: 0.36; ROUGE-1: 0.52), with model-generated reports being approximately 107% longer than ground truth reports due to additional explanatory commentary.
  • Expert Evaluation: Dermatologists rated AI-generated summaries highly:
    • Overall Quality: 8.23 – 8.47
    • Clinical Accuracy: 7.93 – 8.20
    • Clinical Usefulness: 8.47 – 8.50
  • Preference: In 53.3% of evaluations (32/60), experts preferred the AI-generated report over the clinician-generated ground truth.
• Performance: The system demonstrated favorable inference speeds (~130 tokens/second), supporting near-real-time clinical use.

Key Contributions

1. Privacy-Preserving Implementation: The study demonstrates the feasibility of deploying a capable SLM entirely locally (offline) to process sensitive, multi-year patient records without data leaving the hospital infrastructure.
2. Longitudinal Synthesis: Unlike prior work focusing on single-visit notes, this research validates the ability of an SLM to synthesize information across years of heterogeneous clinical narratives to extract structured data and generate coherent summaries.
3. Real-World Evaluation: The evaluation used raw, unstructured clinical notes directly from hospital systems, capturing real-world challenges such as inconsistent documentation, hallucination risks, and the need for prompt refinement to handle clinical ambiguities (e.g., distinguishing "suspicion" from "confirmed" diagnosis).
4. Task-Specific Model Behavior: The findings highlight that the Qwen3 "Thinking" model is better suited for narrative synthesis and report generation than strict structured extraction, suggesting that model selection should be driven by the specific clinical workflow.

Significance and Claims
The paper claims that privacy-preserving, locally deployed SLMs can reliably generate clinically meaningful longitudinal summaries and, in certain evaluations, outperform or match human experts in terms of preference and utility. The authors position these systems not as autonomous decision-makers but as supportive tools to reduce clinician workload and cognitive load during routine follow-ups.

The study concludes that while local LMs show promise for longitudinal data retrieval, their real-world deployment requires careful prompt engineering to mitigate hallucinations and address documentation inconsistencies. Future work is proposed to focus on prospective evaluation of workload reduction, expansion to other chronic diseases, and the integration of Retrieval-Augmented Generation (RAG) to further improve factuality. The authors maintain a modest stance, acknowledging limitations such as the single-center design and the rarity of the disease, which necessitates larger multi-center datasets for broader validation.