Technical Summary: dashi – A Python Library for Dataset Shift Characterization

Problem Statement

The deployment of Artificial Intelligence (AI) in healthcare faces a critical challenge: the assumption of static data distributions between model development and real-world operation is frequently invalid. In dynamic clinical environments, dataset shifts—changes in the joint probability distribution $P(X, Y)$ between training and deployment—occur due to temporal evolution (e.g., changing clinical guidelines, epidemiological shifts) or multi-source heterogeneity (e.g., differences between hospitals).

These shifts manifest as:

• Covariate Shift: Changes in input feature distribution $P(X)$.
• Prior (Label) Shift: Changes in outcome prevalence $P(Y)$.
• Concept Shift: Changes in the conditional relationship $P(Y|X)$.

Current approaches to data quality often rely on static, univariate checks (completeness, conformance) or retrospective performance metrics. These methods fail to detect complex, multivariate "unknown unknowns" and silent failures where models degrade without triggering standard alerts. Furthermore, while theoretical foundations for shift detection exist, there is a lack of accessible, model-agnostic software tools to operationalize these concepts for daily MLOps in health AI.

Methodology

The paper introduces dashi, an open-source Python library designed to bridge the gap between theoretical drift detection and practical health AI monitoring. It employs a dual-approach architecture:

1. Unsupervised (Data-Centric) Module

This module is model-agnostic, analyzing the information properties of data batches without relying on predictive algorithms.

• Distribution Estimation: It constructs data temporal maps (evolution over time) and data source maps (heterogeneity across sites) by estimating empirical probability distributions.
• Dimensionality Reduction: To handle high-dimensional clinical data, it integrates techniques like PCA, SVD, MCA, and FAMD to project data into lower-dimensional subspaces.
• Density Projection: It uses Kernel Density Estimation (KDE) to generate continuous joint probability density functions, visualized as 2D/3D manifolds.
• Distance Metrics: It quantifies divergence between batches using the Jensen-Shannon (JS) distance, a symmetric metric bounded between 0 (identical) and 1 (disjoint).
• Visual Analytics:
  • Information Geometric Temporal (IGT) Plots: Projects temporal batch dissimilarities onto a non-parametric statistical manifold to visualize trajectories, seasonality, and abrupt shifts.
  • Multi-Source Variability (MSV) Plots: Visualizes spatial heterogeneity using two novel metrics:
    • Global Probabilistic Deviation (GPD): A scalar measuring total dispersion across all sources.
    • Source Probabilistic Outlyingness (SPO): Identifies specific sources that deviate significantly from the central tendency.
• Conditional Analysis: Allows for the isolation of $P(X|Y)$ to specifically detect concept shifts.

2. Supervised (Model-Centric) Module

This module directly measures the impact of distributional shifts on predictive performance.

• Automated Pipeline: It automates the training and cross-batch evaluation of baseline predictive models (defaulting to Random Forests and Histogram Gradient Boosting).
• Training Strategies: Supports cumulative (training on accumulated historical data) and from-scratch (training on single batches) strategies.
• Cross-Batch Evaluation: Models trained on specific temporal or source batches are evaluated against all other batches to generate performance matrices.
• Metrics: Tracks standard classification (AUC, F1, Precision/Recall) and regression (MAE, RMSE, $R^2$) metrics to quantify performance degradation.

Key Contributions

1. Dual-Approach Framework: dashi uniquely combines unsupervised distributional analysis with supervised performance tracking, allowing users to correlate data shifts directly with model degradation.
2. Novel Metrics and Visualizations: It introduces IGT plots for temporal tracking and MSV metrics (GPD/SPO) for multi-source characterization, moving beyond simple pairwise drift detection to global topological analysis.
3. Operationalization of Theory: The library translates complex statistical concepts (statistical manifolds, JS divergence) into an accessible, programmatic pipeline compatible with standard EHR formats (pandas DataFrames) and multimodal data (via embeddings for text/images).
4. Comprehensive Taxonomy: The authors provide a structured table linking specific shift typologies (abrupt, gradual, seasonal, subgroup) to detection methods within dashi, their effects on AI, and recommended mitigation strategies.

Results

The utility of dashi was validated across three health AI case studies:

• Simulated Gestational Diabetes (GDM) Cohort (Temporal Concept Shift):

  • dashi successfully detected a simulated period of inconsistent coding (2021–2022) where the conditional distribution $P(X|Y)$ was entangled.
  • IGT plots clearly delineated two distinct temporal subgroups separated by a topological shift.
  • Supervised analysis confirmed that models trained on the "noisy" early batches suffered severe generalization failures on later, corrected data, validating the library's ability to predict silent failures.

• Mexico COVID-19 Cohort (Multi-Source Domain Adaptation):

  • The library identified severe structural heterogeneities across National Health System (NHS) institutions.
  • MSV plots highlighted specific institutions (e.g., NHS types 1 and 15) as outliers with high SPO values and distinct class imbalances.
  • Cross-institution performance matrices revealed that models trained on certain domains failed when transferred to others, while biclustering identified groups of hospitals with compatible data distributions.

• Valencian Community 112 Emergency Dispatch (Multimodal Temporal Shift):

  • Applied to 2.5 million free-text emergency call records, dashi utilized contextual embeddings (BioLord-2023) to track unstructured data dynamics.
  • IGT plots identified an abrupt shift in late 2014 (due to system migration) and a secondary shift in March 2020 (linked to pandemic lockdowns).
  • The library captured strong seasonality in call patterns and demonstrated how training on the pre-2014 batch resulted in the lowest generalization scores across the timeline.

Significance and Claims

The paper positions dashi not merely as an analytical tool, but as a foundational mechanism for continuous algorithmic auditing essential for Trustworthy AI.

• From Reactive to Proactive: The authors argue that dashi enables a paradigm shift from reactive analysis of "silent failures" to proactive, real-time data governance. By detecting underlying data instabilities before they manifest as clinical errors, institutions can trigger recalibration or restrict deployment.
• Bridging the Gap: The library addresses the bottleneck between theoretical shift detection and practical MLOps, providing an end-to-end toolkit that translates complex mathematical theory into actionable insights.
• Regulatory Compliance: The authors claim that dashi provides the necessary infrastructure to comply with emerging regulatory standards for Trustworthy AI by ensuring that a model's statistical assumptions remain aligned with evolving clinical realities.
• Cost-Effectiveness: By utilizing efficient tree-based models and focusing on interpretability over absolute predictive maximization, the library offers a cost-effective solution for robustness and safety in dynamic healthcare environments.

The paper concludes that managing dataset shift is not an isolated post-deployment task but a continuous requirement across the entire AI lifecycle, and dashi facilitates this rigorous auditing at every stage.