Scalable Patient-Level Prediction in a Trusted Research Environment: New Capabilities for the Veterans Precision Oncology Data Commons

Technical Summary: Scalable Patient-Level Prediction in a Trusted Research Environment

Problem Statement
The Veterans Precision Oncology Data Commons (VPODC) serves as a Trusted Research Environment (TRE) providing secure access to clinical, genomic, and imaging data from the Department of Veterans Affairs (VA). However, a significant barrier to scientific discovery is the requirement for researchers to have direct access to sensitive data to perform complex machine learning tasks. Traditional workflows often necessitate downloading data or using local environments, which compromises privacy and scalability. The challenge is to enable approved researchers to perform cohort discovery, train predictive models, and evaluate results on sensitive VA data without ever directly accessing the underlying raw datasets, while maintaining a cost-efficient and privacy-preserving infrastructure.

Methodology
The authors developed an extensible, app-centric architecture within the VPODC using the Gen3 open-source software stack. This infrastructure integrates authentication (Fence) and authorization (Arborist) services to manage user access and team-based collaboration. The core methodology involves three primary front-end applications integrated into a unified web UI:

1. ATLAS: An open-source tool from the OHDSI community used for cohort discovery and definition based on the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM).
2. Patient-Level Prediction (PLP) App: A customized interface integrating the OHDSI PLP R package. This app allows researchers to select cohorts, define observation and outcome windows, specify covariates, and train machine learning models. The backend utilizes Python scikit-learn for Support Vector Machines (SVM) and Random Forest (RF), and a modified cyclops package for Lasso Logistic Regression (LLR).
3. Results App: A React-based application for visualizing training job statuses, model performance metrics, diagnostic plots (e.g., ROC, calibration), and downloading results in ZIP format.

The infrastructure runs on Kubernetes, utilizing Karpenter for node autoscaling and custom web services (e.g., cohort-middleware, workflow-wrapper) to manage workflow execution and track costs via AWS tags.

To validate the system, the authors conducted two primary use cases:

• Germline Validation: Replicating a previously published study predicting all-cause mortality in patients with coronary artery disease and atrial fibrillation using synthetic Synpuf data.
• Cancer Research Application: Training models to predict the development of primary malignant neoplasm of the prostate in Veterans with an existing diagnosis of respiratory cancer. This involved training LLR, SVM, and RF models on a cohort of N=37,716 deceased Veterans (Cohort B) with 3-, 5-, and 10-year outcome windows.

Key Contributions
The paper presents three specific contributions:

1. Integrated Research Environment: A unified platform enabling hypothesis generation, cohort discovery, and PLP model development on sensitive VA data without direct data access.
2. Extensible Infrastructure: A scalable architecture that allows researchers to interact with data solely through front-end apps, ensuring data remains within the secure VPODC boundary while supporting team-based collaboration and granular access control.
3. Scientific Utility Demonstration: A proof-of-concept application predicting multiple malignancies (respiratory to prostate cancer), demonstrating the platform's ability to identify biologically meaningful covariates and replicate established research findings.

Results

• Infrastructure Validation: The PLP app successfully replicated the results of the germline study (Ahmadi et al.), producing consistent AUC and Brier metrics, confirming the accuracy of the infrastructure.
• Model Performance: In the respiratory-to-prostate cancer prediction task, the Lasso Logistic Regression (LLR) model outperformed SVM and RF models, which exhibited overfitting and poor calibration. The best-performing LLR model achieved test AUCs of 0.80 (3-year), 0.83 (5-year), and 0.85 (10-year) in internal validation.
• Covariate Identification: The trained models identified established predictors, including elevated Prostate-Specific Antigen (PSA) levels, abnormal metabolite levels (creatinine, neutrophils, leukocytes), and a significant association between Black or African American race and the subsequent development of prostate cancer in Veterans with respiratory cancer.
• Cost and Efficiency: On synthetic data comprising over one million individuals, training and testing classic ML models (e.g., Random Forest) completed in under four hours with a cost of less than one dollar. Real-data workflows on Cohort B demonstrated similar cost-efficiency.

Significance and Claims
The authors claim that the updated VPODC infrastructure enables rapid, cost-efficient, and privacy-preserving hypothesis testing and model selection. By removing the need for direct data access, the platform lowers the barrier to entry for researchers while maintaining strict security compliance (FedRAMP Low-Impact SaaS). The study demonstrates that the platform can support complex, multi-modal cancer research, such as investigating multiple primary malignancies, and successfully identifies clinically relevant risk factors. The authors note that while the current implementation supports classic machine learning, the modular design allows for future integration of deep learning and transformer architectures as cohorts expand. The paper emphasizes that the findings regarding multiple malignancies and racial disparities require further validation in larger, independent cohorts but serve as a proof-of-concept for the platform's utility in generating biologically meaningful insights.