Generalizable deep learning for photoplethysmography-based blood pressure estimation -- A Benchmarking Study

This benchmarking study evaluates the generalizability of five deep learning models for cuffless blood pressure estimation from PPG signals, revealing significant performance degradation on external datasets due to distribution shifts and demonstrating that sample-based domain adaptation can effectively improve out-of-distribution robustness.

The Gist

Imagine you want to teach a robot to guess your blood pressure just by looking at a video of your finger pulse (a signal called PPG). This is a huge goal because it could replace the old, tight arm-cuff method with a simple, painless sensor.

This paper is like a stress test for the smartest robots (Deep Learning models) currently available to do this job. The researchers wanted to see if these robots are truly smart or if they are just "cheating" by memorizing the specific people they trained on.

Here is the breakdown of their findings using some everyday analogies:

1. The Setup: The "School" vs. The "Real World"

The researchers used a massive library of data called PulseDB (think of it as a giant school) to train five different types of AI models.

• In-Distribution (ID) Testing: This is like giving the students a final exam using the exact same questions they practiced in class. The models did great! They got high scores.
• Out-of-Distribution (OOD) Testing: This is the real test. The researchers took the models out of the "school" and threw them into completely different environments (external datasets) with different people, different sensors, and different health conditions.

The Result: When the models left the classroom, they stumbled. Their performance dropped significantly. It turns out, many models were just memorizing the "classroom" patterns rather than learning the actual rules of blood pressure.

2. The "Diet" Problem: Why Some Models Failed

The researchers discovered that the biggest reason the models failed wasn't the model's brain, but the data diet it was fed.

• The MIMIC Dataset: Imagine a training diet made mostly of heavy, salty foods (older patients in hospitals). When the model tried to guess the blood pressure of a healthy young person (a light diet), it got confused. The "flavors" were too different.
• The VitalDB Dataset: This was a more balanced diet. Models trained here were better at guessing blood pressure for new types of people.
• The Lesson: If you train a model only on sick, older hospital patients, it won't know how to handle healthy people. The "distribution" (the mix of people) matters more than the complexity of the AI.

3. The "Calibration" Crutch

There are two ways to train these models:

• Calibration (The Crutch): You let the model look at a specific patient's data before testing them. It's like giving a student the answer key for the specific test they are about to take. The model does very well here because it's just memorizing that one person.
• Calibration-Free (The Real World): The model has to guess a stranger's blood pressure without seeing them first. This is much harder. The study showed that while models are great with the crutch, they often fail when the crutch is removed.

4. The Magic Trick: "Re-weighting" (Domain Adaptation)

The researchers tried a clever trick to fix the "diet" problem. They realized the models were failing because the blood pressure numbers in the training data didn't match the test data.

• The Analogy: Imagine you are teaching a chef to cook soup. You only gave them recipes for spicy soup. When you ask them to make a mild soup, they over-salt it.
• The Fix: Instead of giving them new recipes, you told them: "When you see a mild ingredient in your training, pay extra attention to it. When you see a spicy one, pay less attention."
• The Result: By mathematically "re-weighting" the training data to match the target population, the models got better at guessing. It wasn't a miracle cure, but it was a significant improvement, like giving the chef a better spice guide.

5. The Big Takeaway

The most important message of this paper is a warning to the scientific community:

"Just because a model gets an 'A' in the classroom doesn't mean it will pass the real-world test."

Currently, many AI blood pressure tools look amazing in research papers because they are tested on the same data they were trained on. But when you take them to a real hospital or a different country, they often fail.

The Verdict:

• Don't trust the "In-Class" scores. They are too optimistic.
• Choose your data carefully. Training on diverse, healthy, and varied data (like the VitalDB subset) works better than just using the biggest hospital dataset (MIMIC).
• We need better tools. To make this technology safe for doctors to use, we need to stop testing models in a vacuum and start testing them on strangers from different backgrounds.

In short: The AI is smart, but it's currently a bit of a "spoiled student" that only knows how to perform in the specific room it was taught in. We need to teach it to handle the chaos of the real world.

Technical Summary

1. Problem Statement

Photoplethysmography (PPG) offers a non-invasive, cost-effective method for estimating blood pressure (BP), serving as a potential alternative to traditional cuff-based measurements. While deep learning (DL) models have shown promise in inferring BP from raw PPG waveforms, most existing research relies on in-distribution (ID) testing, where training and test data come from the same distribution.

The core problem addressed is the lack of Out-of-Distribution (OOD) generalization. Real-world applications involve data from diverse sources with different sensor hardware, signal qualities, and patient physiologies. Models trained on specific datasets often fail when applied to external datasets due to distribution shifts (e.g., differences in BP ranges, demographics, or recording conditions). There is a critical need to benchmark DL models on external datasets and identify training strategies that yield robust generalization.

2. Methodology

Datasets

• Training Data: The study utilized PulseDB, a large-scale dataset containing over 5 million 10-second segments from MIMIC-III and VitalDB.
  • Subsets: The authors generated nine subsets based on source (Combined, VitalDB, MIMIC) and scenario:
    • Calib: Subjects overlap between train/test (patient-specific calibration).
    • CalibFree: No subject overlap (generalization to unseen patients).
    • AAMI: A stricter calibration-free scenario focusing on the tails of the BP distribution (adhering to AAMI standards).
• External Test Data (OOD): Four external datasets were used to evaluate generalization: Sensors (MIMIC-III), UCI (MIMIC-II), BCG (healthy subjects), and PPGBP (cardiovascular patients). These differ significantly in size, signal quality, and population.

Model Architectures

Five state-of-the-art deep learning architectures were evaluated:

1. LeNet1D: A simple feed-forward CNN.
2. XResNet1d50 & XResNet1d101: Residual networks with skip connections.
3. Inception1D: Uses multi-scale convolutional filters.
4. S4 (Structured State Space Sequence): A model designed for long-range dependencies in time series.

Training & Evaluation Procedures

• Metrics: Primary metric was Mean Absolute Error (MAE) for Systolic (SBP) and Diastolic (DBP) BP. Secondary analysis included Bland-Altman plots to assess bias and limits of agreement.
• Domain Adaptation Strategy: The authors tested a sample-based domain adaptation approach using importance weighting.
  • Instead of using individual target labels, they inferred sample weights based on the label distribution (BP histogram) of the target domain compared to the source domain.
  • Training samples were reweighted to mimic the target distribution, aiming to minimize the predictive error on the target dataset without requiring individual target labels.

3. Key Contributions

1. Comprehensive Benchmarking: The first extensive comparative study of DL models for PPG-based BP estimation across nine PulseDB subsets and four external datasets, evaluating both ID and OOD performance.
2. Identification of Generalization Gaps: Demonstrated that ID performance is overly optimistic and that model performance degrades significantly when tested on external datasets due to distribution mismatches.
3. Analysis of Dataset/Scenario Impact: Identified that the VitalDB subset (specifically in CalibFree and AAMI scenarios) yields better generalization than MIMIC-based models, challenging the common reliance on MIMIC for training.
4. Domain Adaptation Validation: Validated a simple, label-distribution-based reweighting technique that improves OOD performance, particularly for models trained on diverse datasets.

4. Key Results

Model Performance

• Best Architecture: XResNet1d101 consistently achieved the best or near-best performance across scenarios.
• ID Performance (PulseDB):
  • Calib (with calibration): MAE of ~9.0 mmHg (SBP) and 5.8 mmHg (DBP).
  • CalibFree/AAMI (without calibration): MAE increased to ~13.9 mmHg (SBP) and 8.5 mmHg (DBP).
• OOD Performance (External Datasets):
  • Without calibration, MAE on external datasets ranged from 10.0 to 18.6 mmHg (SBP) and 5.9 to 10.3 mmHg (DBP).
  • Models trained on MIMIC showed poor generalization to external datasets compared to those trained on VitalDB or Combined subsets.
  • AAMI scenarios generally resulted in higher errors due to the broader BP distribution and stricter testing protocols.

Impact of Domain Adaptation

• Importance Weighting: Applying sample weights based on target label distributions improved OOD performance in 55% of external test cases.
• Magnitude of Improvement:
  • Mean improvement for SBP: 2.66 mmHg.
  • Mean improvement for DBP: 0.86 mmHg.
  • Notable gains were observed for MIMIC-trained models when adapted to external datasets, reducing MAE by up to 13 mmHg in specific cases.
• Correlation: A strong correlation was found between the Earth Mover's Distance (EMD) of BP distributions (dissimilarity) and OOD error. Higher distribution mismatch led to higher prediction errors.

Bland-Altman Analysis

• Models trained on standard subsets showed significant negative bias (underestimation) when tested on AAMI datasets, often exceeding 10–15 mmHg, indicating systematic deviations due to distribution shifts.

5. Significance and Conclusion

• Re-evaluating Training Data: The study challenges the prevailing use of MIMIC as the primary training source for generalizable BP estimation, suggesting VitalDB (or Combined subsets) provides better robustness for unseen patients.
• Beyond ID Benchmarks: The authors emphasize that ID benchmarks are insufficient for clinical deployment. Models must be validated on external, diverse datasets to ensure safety and reliability.
• Practical Recommendations:
  • Use XResNet1d101 or Inception1D architectures.
  • Prioritize training on VitalDB or Combined subsets, specifically using CalibFree or AAMI scenarios to force generalization to unseen patients.
  • Employ importance weighting based on target label distributions to mitigate distribution shifts.
• Clinical Reality Check: Despite improvements, the best MAE scores (approx. 7–10 mmHg) still fall short of the IEEE standard for clinical use (Grade A/B requires <5–7 mmHg). The authors suggest future work should incorporate clinical metadata, stricter quality control, and foundation models to bridge this gap.

In summary, this paper provides a critical roadmap for developing robust, generalizable PPG-based BP estimators, highlighting that dataset selection and distribution-aware training strategies are more critical than model architecture complexity alone.