SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG

The paper introduces SignalMC-MED, a comprehensive benchmark utilizing 22,256 synchronized single-lead ECG and PPG visits to evaluate biosignal foundation models across 20 clinical tasks, demonstrating that domain-specific models with multimodal fusion and full-duration signals outperform general time-series approaches while revealing that larger model sizes do not guarantee superior performance.

The Gist

Imagine you are a doctor trying to understand a patient's health just by listening to their heartbeat (ECG) and feeling their pulse (PPG). For years, we've had tools to do this, but they were like listening to a single second of a song and trying to guess the whole melody.

Recently, a new generation of "super-smart" AI models (called Foundation Models) has emerged. These are like musical prodigies who have listened to billions of hours of music and can instantly recognize patterns. But here's the problem: nobody knew if these prodigies could actually handle a real, messy, 10-minute live performance in a busy emergency room, especially when you have two different instruments playing at once (the heart's electrical signal and the blood flow pulse).

Enter SignalMC-MED. Think of this paper as the creation of the ultimate "Taste Test" or "Olympics" for these AI models.

Here is the breakdown of what the researchers did, using simple analogies:

1. The Arena: A Busy Emergency Room

The researchers didn't use clean, perfect lab data. They went to a real hospital emergency department and grabbed 22,000 patient visits.

• The Data: For every patient, they recorded 10 minutes of two signals simultaneously:
  • ECG: The electrical spark of the heart (like the conductor's baton).
  • PPG: The pulse of blood flowing through the skin (like the sound of the drums).
• The Challenge: They created 20 different tasks to test the AI. Some were easy (guessing the patient's age or gender), some were hard (predicting if they had diabetes or heart failure), and some were like solving a puzzle (estimating their blood sugar or kidney function just from the heartbeat).

2. The Contestants: The AI Models

The researchers invited eight different "contestants" to compete:

• The Generalists: Models trained on any kind of time-series data (like stock prices or weather). They are smart but not specialized in medicine.
• The Specialists: Models trained only on heartbeats (ECG) or only on pulses (PPG).
• The Bilinguals: A model trained on both heartbeats and pulses together.
• The Old Guard: Traditional doctors' methods using hand-crafted math formulas (the "hand-crafted features").

3. The Rules of the Game

To make it fair, the AI models weren't allowed to "study" the specific patients in the test group. They had to act like a frozen encyclopedia:

1. They looked at the 10-minute signal.
2. They broke it into tiny 10-second chunks.
3. They turned each chunk into a "summary note" (a feature vector).
4. They averaged those notes to get a "visit summary."
5. A simple, linear model (like a basic calculator) tried to make predictions based only on that summary.

This tested how good the AI's understanding of the raw data was, without relying on the AI to "learn" the specific task during the test.

4. The Big Discoveries (The Plot Twist)

Here is what the "Olympics" revealed:

• Specialists Beat Generalists: The models trained specifically on heart data (the specialists) crushed the general time-series models. It's like a cardiologist beating a generalist in a heart exam. If you want to predict heart issues, you need a heart-trained AI.
• Two Heads Are Better Than One: When the AI looked at both the ECG and the PPG together, it performed significantly better than looking at just one. It's like trying to understand a movie by only watching the video (ECG) vs. watching the video and listening to the audio (PPG). The combination gives a much clearer picture.
• Time is the Secret Ingredient: The longer the signal, the better the AI performed. A full 10-minute recording was far superior to a short 10-second clip. It's the difference between judging a chef by one spoonful of soup versus tasting the whole bowl. The AI needed the full context to spot subtle patterns.
• Bigger Isn't Always Better: Surprisingly, the massive, huge AI models didn't always beat the smaller, medium-sized ones. Sometimes, a smaller model was just as good. It suggests that for this specific job, you don't need a supercomputer; you need the right kind of training.
• The "Old Guard" Still Has Value: The traditional, hand-crafted math features (the "Old Guard") were incredibly strong. In fact, they often beat the fancy new AI models. The best strategy? Combine them. When you mix the AI's "intuition" with the doctor's "math," you get the strongest result of all.

5. Why This Matters

Before this paper, we didn't know if these fancy new AI models were actually ready for the real world. This study says: "Yes, but with conditions."

• Don't just use any AI: Use one trained on heart data.
• Don't just use one signal: Use both the electrical and pulse signals if you can.
• Don't rush: Give the AI a longer look at the patient (10 minutes is better than seconds).
• Respect the classics: Don't throw away the old, proven math methods; combine them with the new AI for the best results.

In a nutshell: SignalMC-MED is the rulebook and the scoreboard that tells us how to build the best AI doctors for the future, ensuring they are accurate, reliable, and ready to help in a real emergency room.

Technical Summary

Here is a detailed technical summary of the paper "SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG."

1. Problem Statement

While biosignal foundation models (FMs) have shown promise in clinical prediction tasks, their evaluation has been limited by a lack of standardized benchmarks that reflect real-world clinical conditions. Specifically:

• Data Limitations: Most existing benchmarks focus on short signal segments (typically 10 seconds) and unimodal data (ECG or PPG), whereas clinical monitoring often involves long-duration (minutes to hours), synchronized multimodal data.
• Evaluation Gaps: There is no systematic comparison of domain-specific biosignal FMs against general time-series FMs, nor is there a clear understanding of how multimodal fusion, signal duration, and model scale impact performance on realistic emergency department (ED) data.
• Baseline Ambiguity: The relative value of learned FM representations compared to traditional, hand-crafted physiological features remains unrigorously assessed in a multimodal context.

2. Methodology

Dataset: SignalMC-MED

The authors introduce SignalMC-MED, a benchmark derived from the MC-MED dataset, comprising 22,256 adult emergency department visits.

• Data Structure: Each visit includes synchronized, single-lead Electrocardiogram (ECG) and Photoplethysmogram (PPG) signals with a duration of 10 minutes.
• Preprocessing: Signals are resampled to a common frequency (default 250 Hz), cleaned using NeuroKit2, and normalized.
• Splitting: Chronological train/validation/test splits are used (15,832 / 2,030 / 1,995 visits) with no patient overlap to simulate real-world deployment and prevent temporal leakage.

Evaluation Framework

The study employs a unified linear probing framework to isolate representation quality:

1. Segmentation: 10-minute signals are divided into non-overlapping 10-second segments.
2. Feature Extraction: Frozen FMs extract feature vectors for each segment.
3. Aggregation: Segment-level features are aggregated via mean pooling to create a visit-level representation.
4. Fusion: For multimodal settings, ECG and PPG visit-level vectors are averaged (late fusion).
5. Downstream Tasks: Linear models (Ridge regression for regression, Logistic regression for classification) are trained on top of the frozen features.
6. Robustness: Experiments are repeated with 10%, 25%, 50%, and 100% of training data to assess data efficiency.

Models Evaluated

The benchmark evaluates 8 representative approaches across three categories:

• General Time-Series FMs: MOMENT (small/base/large) and Chronos-Bolt (tiny/mini/small/base).
• Unimodal Biosignal FMs:
  • ECG-specific: D-BETA, ECGFounder, xECG.
  • PPG-specific: PaPaGei.
• Multimodal Biosignal FM: CSFM (trained on paired ECG+PPG).
• Baselines: Hand-crafted domain features (ECG and PPG) extracted via NeuroKit2 and pyPPG.

Tasks

The benchmark includes 20 clinically relevant tasks:

• Demographics: Age regression, Sex classification.
• Clinical Disposition: ED disposition classification (Discharge vs. Admit/Observation/ICU).
• Laboratory Values: Regression of 8 lab tests (e.g., Potassium, Glucose, eGFR).
• Diagnosis Detection: Classification of 9 prior ICD-10 diagnosis groups (e.g., Atrial Fibrillation, Heart Failure, Diabetes, Anemia).

3. Key Contributions

1. SignalMC-MED Benchmark: A large-scale, clinically grounded benchmark featuring synchronized 10-minute ECG and PPG data with 20 diverse downstream tasks, designed to reflect routine emergency care.
2. Systematic Head-to-Head Evaluation: A comprehensive comparison of general time-series models, unimodal biosignal FMs, multimodal FMs, and hand-crafted features across ECG-only, PPG-only, and ECG+PPG settings.
3. Empirical Insights: Detailed analysis of how modality specialization, fusion strategies, signal duration, and model scale influence performance, providing practical guidance for deploying biosignal FMs.

4. Key Results

Model Performance

• Domain-Specific Superiority: Domain-specific biosignal FMs consistently outperform general time-series FMs (MOMENT, Chronos-Bolt) across all modalities.
• Top Performers: CSFM-base (multimodal) achieved the best overall performance and mean rank. xECG-10min (ECG-specific) consistently ranked second, demonstrating strong cross-modality generalization even on PPG-only inputs.
• Weak Performers: PaPaGei (PPG-specific) ranked last or second-to-last across all settings, suggesting that modality-specific pretraining alone is insufficient without diverse subject-level data (PaPaGei was trained on only 13.5k subjects vs. >1.5M for top models).
• Hand-Crafted Features: Domain features provided a strong baseline, outperforming general time-series FMs and ranking 4th overall.

Multimodal Fusion

• Complementarity: ECG + PPG fusion yielded robust improvements over unimodal inputs for nearly all models.
• Fusion Strategy: Simple late feature-level fusion (averaging unimodal vectors) slightly outperformed CSFM's internal multimodal fusion mechanism, suggesting that preserving unimodal structure before combining may be more effective than complex internal alignment for this data.

Signal Duration vs. Model Scale

• Duration Matters: Using the full 10-minute signal consistently outperformed shorter segments (10s to 5m), indicating that longer temporal context provides critical predictive information beyond short-window morphology.
• Scale Saturation: Larger model variants (e.g., CSFM-large vs. CSFM-base) did not reliably outperform smaller ones under frozen linear evaluation. This suggests diminishing returns from parameter scaling when the downstream task is linear and key features are already accessible.

Feature Engineering

• Complementarity: Concatenating hand-crafted ECG features with learned FM embeddings consistently improved performance, even for top-performing models. This implies current FMs do not fully internalize all established morphological abstractions.

5. Significance and Implications

• Standardization: SignalMC-MED establishes a standardized protocol for evaluating biosignal FMs on realistic, long-duration, multimodal data, moving beyond short-segment, unimodal benchmarks.
• Deployment Guidance:
  • Prioritize Data: Longer signal segments are more impactful than increasing model size.
  • Leverage Multimodality: Even simple fusion of ECG and PPG yields significant gains; multimodal data should be utilized whenever available.
  • Model Selection: Domain-specific models trained on diverse, large-scale datasets (e.g., CSFM, xECG) are superior to general time-series models or models trained on limited subject diversity.
  • Hybrid Approaches: Combining learned representations with hand-crafted physiological features remains a highly effective strategy for maximizing performance.
• Future Directions: The findings highlight the need for pretraining on diverse subject populations rather than just massive segment counts, and suggest that future research should focus on advanced fusion strategies and long-term temporal modeling.

In summary, the paper demonstrates that for cardiovascular prediction, specialized, multimodal foundation models trained on diverse, long-duration data represent the state-of-the-art, but they must be evaluated against strong physiological baselines and optimized for signal duration rather than just parameter count.