PROACTIVE-AI: An Intelligent Dashboard for Assessing and Predicting of FDA-approved AI Software Performance in Real-World Clinical Scenarios

Technical Summary: PROACTIVE-AI Dashboard

Problem Statement
The rapid proliferation of FDA-approved Artificial Intelligence/Machine Learning (AI/ML) enabled medical devices (AIMDs) has outpaced the development of robust post-market surveillance mechanisms. While over 1,300 AI/ML devices were approved by the FDA as of July 2025, the regulatory framework remains largely passive, relying on voluntary manufacturer reporting of adverse events rather than proactive risk tracking. This creates a "blind spot" regarding device performance in real-world clinical settings. Key challenges include inconsistent pre-market evaluation practices, an over-reliance on retrospective single-site studies, and a lack of systematic post-market data. Furthermore, the fundamental nature of AI/ML products differs from traditional hardware devices; unlike static tools (e.g., stents), AI models can adapt and change behavior based on real-world data, rendering their performance at approval potentially distinct from their performance years later. Current regulatory frameworks lack standardized mechanisms to detect, report, or address these post-deployment shifts, leading to gaps in patient safety, transparency, and accountability.

Methodology
To address these gaps, the authors developed PROACTIVE-AI (Predictive Real-world Oversight, Assessment, Compliance, TrackIng, and surVEillance of AI), an interactive web-based dashboard. The system integrates and analyzes a dataset of approximately 1,303 unique FDA-cleared AI/ML devices spanning from 1995 to 2025. The methodology comprises three primary components:

1. Data Curation and Pre-processing:

   • The dataset combines FDA decision summary data with recall records sourced from Lee et al. [14].
   • To harmonize the data, the authors reconciled structural discrepancies between recalled (82 features) and non-recalled (21 features) datasets. One-hot encoded categorical variables were reconstructed, and missing features were introduced with placeholder values.
   • Missing values in the combined dataset were imputed using a large language model (GEMINI-3.1-PRO) via structured prompts to extract key attributes.
   • The final unified dataset included 1,243 non-recalled devices (Class 1) and 59 recalled devices (Class 0), partitioned into training, validation, and independent test sets using stratified sampling.

2. Interactive Dashboard Modules:

   • Knowledge Graph: A network visualization using D3.js that maps predicate relationships between devices. Nodes represent devices, color-coded by clinical panel and scaled by the number of descendants. It allows users to explore device lineage, manufacturer connections, and network density.
   • Time Analysis Panel: A longitudinal module visualizing trends across quarters and five-year periods. It tracks approval volumes, recall rates, clinical trial prevalence, and shifts in AI architecture (e.g., from unspecified to deep learning) and clinical specialties. It also annotates regulatory milestones to provide context for observed shifts.

3. AI-Driven Risk Assessment Tool:

   • Model Architecture: A CatBoost gradient boosting classifier was developed to distinguish between recalled and non-recalled devices. CatBoost was selected for its ability to handle categorical variables natively and its performance on structured tabular data.
   • Training Strategy: The model addressed class imbalance using class weights. It was trained with 287 boosting iterations, a learning rate of 0.03, and a tree depth of 6.
   • Validation: Robustness was assessed via 5-fold, 5-repeat stratified cross-validation. An independent test set (n=261) was withheld for final evaluation.
   • Interpretability: SHapley Additive exPlanations (SHAP) were used to quantify feature contributions, and Uniform Manifold Approximation and Projection (UMAP) was applied to visualize the separability of recalled vs. non-recalled clusters in a reduced-dimensional space.

Key Results

• Dashboard Visualization: The Knowledge Graph successfully mapped 1,780 nodes and 2,224 edges representing the predicate relationships of the 1,303 devices. The Time Analysis panel successfully identified longitudinal trends, including the rise of cardiovascular devices and the transition toward explicit deep learning architectures.
• Model Performance:
  • Cross-Validation: The CatBoost classifier achieved a mean ROC-AUC of 0.993 ± 0.014 and a mean PR-AUC of 0.967 ± 0.042 across repeated folds, demonstrating stable predictive performance.
  • Independent Testing: On the held-out test set, the model achieved a ROC-AUC of 0.961 and a PR-AUC of 0.886.
  • Classification Metrics: The model achieved an overall accuracy of 98.1%. For the minority class (recalled devices), it achieved a precision of 0.769, recall of 0.833, and an F1-score of 0.800, indicating effective identification of high-risk devices despite class imbalance.
• Feature Importance (SHAP Analysis):
  • The top three most influential features for predicting recall risk were: Number of Clinical Trials (Patients not images), Company, and Predicate Count.
  • Higher numbers of clinical trial patients were strongly associated with non-recall outcomes.
  • Lower predicate counts were strongly associated with recall predictions, suggesting that devices with fewer established predecessors (novel algorithms) carry higher risk.
  • UMAP visualization confirmed strong separation between recalled and non-recalled clusters, supporting the model's discriminatory capability.

Significance and Claims
The paper claims that PROACTIVE-AI provides a structured, quantitative framework for assessing the anticipated performance and recall risk of FDA-approved AI devices in real-world settings. By converting historical regulatory data into prospective risk scores, the tool aims to narrow the "trust gap" surrounding AI in healthcare.

The authors emphasize that their findings validate and extend previous descriptive work (specifically Lee et al. [14]) by moving from retrospective characterization to prospective prediction. The study confirms that a lack of clinical validation and the use of novel predicate chains are robust signals of post-market failure. The dashboard is presented as a resource for clinicians, regulators, and researchers to make informed decisions about device deployment and to prioritize post-market surveillance for high-risk devices, thereby enhancing patient safety and accountability without inventing new regulatory mandates.