Multimodal Fusion of Pathology Free-Text and Clinical Data Enhances Complication-Risk Discrimination After Implant-Based Breast Reconstruction

Technical Summary: Multimodal Fusion of Pathology Free-Text and Clinical Data for Complication-Risk Discrimination

Problem Statement
Implant-based breast reconstruction is the predominant surgical option following mastectomy, yet approximately one-third of patients develop complications within two years. Existing machine-learning models for predicting these complications rely exclusively on hand-curated, structured clinical variables (e.g., BMI, age, comorbidities), thereby ignoring prognostic information embedded in narrative pathology reports. While large language models (LLMs) have demonstrated the ability to extract categorical findings from clinical text, numeric morphometrics (such as mastectomy specimen weight) remain largely untapped. Furthermore, no prior study has successfully fused text-derived features with structured clinical data to improve risk stratification in this domain.

Methodology
The authors developed a fully on-premises, open-source multimodal pipeline to address this gap. The methodology consists of three primary components:

1. LLM-Based Information Extraction:

   • An open-source LLM (Gemma 3, 27B parameters) was deployed in a zero-shot setting to extract numeric morphometrics from free-text pathology reports.
   • The model was prompted to extract specific fields including left/right breast mastectomy specimen weights (in grams), dimensions, deep margins, and lymph node counts.
   • Inference was performed on-premises using 4× NVIDIA L40S GPUs with deterministic decoding to ensure reproducibility.
   • The extracted JSON outputs were parsed with strict schema validation, normalizing units (e.g., kg to g) and handling missing values.

2. Contrastive Multimodal Alignment (CLIP-Style):

   • To fuse unstructured text with structured data, the authors employed a dual-encoder architecture inspired by CLIP (Contrastive Language-Image Pre-training).
   • Encoders: Independent PubMedBERT-base transformers encoded the original pathology reports and templated clinical narratives (derived from structured variables).
   • Pooling: For documents exceeding the 512-token limit, overlapping windows were processed and aggregated via a learnable self-attention pooling layer to produce a single context-aware vector.
   • Training Objective: The model was trained using an InfoNCE contrastive loss to maximize cosine similarity between matched pathology-clinical pairs within a mini-batch.
   • Auxiliary Tasks: To ensure semantic retention, the training included auxiliary classifiers: a softmax classifier on the pathology [CLS] token to predict specimen laterality (left, right, bilateral) and a multi-label sigmoid classifier on the clinical [CLS] token to reconstruct attributes like smoking status and hypertension.

3. Predictive Modeling and Fusion:

   • Baseline: A LightGBM model trained on 62 structured clinical variables served as the primary baseline.
   • Morphometric Addition: Extracted numeric features (specifically total mastectomy weight) were added to the baseline.
   • Ensemble Fusion: The final model utilized a soft-voting ensemble. It combined:
     • The LightGBM classifier trained on morphometric features.
     • A Support Vector Classifier (SVC) trained on the 768-dimensional [CLS] embeddings from the dual encoder (truncated to 256 dimensions and L2 normalized).
   • The ensemble assigned equal weights (1:1) to the morphometric and embedding-based predictions.

Key Results

• Extraction Accuracy: The LLM achieved a combined extraction accuracy of 96.3% (78/81 cases) for mastectomy specimen weights. Specifically, it achieved 100% accuracy on left breast reports (37/37) and 93.2% on right breast reports (41/44). Errors were primarily due to reports omitting specimen weight, causing the model to erroneously extract implant weight.
• Cross-Modal Alignment: The CLIP-trained model successfully aligned clinical and pathology embeddings into a shared UMAP manifold. On a 215-pair validation set, the model achieved significant enrichment over random baselines in cross-modal retrieval (e.g., Clinical→Pathology Recall@50 = 0.781, a 3.36× improvement over chance).
• Risk Discrimination:
  • The baseline Clinical LightGBM achieved an AUROC of 0.740.
  • Adding only the extracted numeric weight (M2) yielded a marginal, non-significant increase to 0.744.
  • The full multimodal ensemble (M2 + both clinical and pathology [CLS] embeddings) achieved the highest performance with an AUROC of 0.764 (p=0.027 vs. baseline).
  • At a fixed 80% specificity, sensitivity improved from 0.573 (baseline) to 0.609 (best ensemble). At a 0.50 probability threshold, the Positive Predictive Value (PPV) increased from 0.668 to 0.828.
• Subgroup Performance: The model demonstrated robust performance across subgroups, with particularly strong discrimination in patients with shorter expander-to-implant intervals (<180 days), where the AUROC reached 0.796.
• Interpretability: SHAP analysis confirmed that the model relied on clinically coherent features. The top contributors were the expander-to-implant interval (TEtoImplant), BMI, and total mastectomy weight. Dependence plots revealed non-linear relationships, such as a threshold-like risk increase for intervals exceeding ~6–12 months.

Significance and Claims
The paper claims that this study provides the first evidence that open-source, on-premises LLMs can reliably extract numeric morphometrics and contextual embeddings from narrative pathology reports to enhance clinical risk prediction.

• Privacy and Accessibility: By operating entirely on-premises with open-source models, the framework ensures protected health information (PHI) never leaves institutional control and avoids per-call fees, making it accessible to resource-constrained centers.
• Unlocking Untapped Data: The study demonstrates that pathology text contains prognostic signals (e.g., fine-grained lexical cues, temporal relations, and uncertainty qualifiers) that are not captured by structured variables alone.
• Clinical Utility: While the statistical gain in AUROC is modest (+0.024), the authors argue that even small improvements yield clinically meaningful gains in identifying high-risk patients, potentially informing surgical planning, patient counseling, and postoperative follow-up.
• Generalizability: The authors position this interpretable, privacy-preserving multimodal framework as a generalizable approach for enhancing risk prediction across surgical oncology, moving beyond the limitations of tabular-only baselines.

The authors remain modest regarding limitations, acknowledging that the single-center nature of the study may limit generalizability due to domain shifts in report templates, and that the composite endpoint groups heterogeneous complications. They suggest future work should focus on multi-center validation and the integration of imaging data.