ECG-InterpBench: Benchmarking the Interpretability of ECG Foundation Models with Matched-Scale Sparse Autoencoders

Technical Summary: ECG-InterpBench

Problem Statement
Existing benchmarks for Electrocardiogram (ECG) foundation models (FMs) predominantly evaluate downstream predictive performance, such as accuracy in rhythm classification or risk prediction. These evaluations fail to reveal how the internal hidden representations are organized. Specifically, current methods do not determine whether these representations can be faithfully decomposed, whether clinically relevant measurements are localized to identifiable features, or whether the same structural properties are recovered across independent analyses. This gap is critical in clinical settings because shared representations may propagate acquisition artifacts, demographic confounding, or cohort-specific shortcuts to every downstream model built upon them. Furthermore, existing interpretability tools (e.g., output attribution or linear probes) address different questions than the audit of task-general hidden representations and often lack the rigor to distinguish between predictive accuracy and true representation-level interpretability.

Methodology
The paper introduces ECG-InterpBench, a benchmark designed to systematically evaluate the interpretability of ECG foundation model representations using Sparse Autoencoders (SAEs) as standardized measurement instruments. The core design principle is capacity matching: to enable controlled comparisons, the benchmark evaluates models using SAEs with matched dictionary widths and sparsity settings, ensuring that differences in results stem from the foundation models themselves rather than variations in the measurement capacity of the SAEs.

Key methodological components include:

• Models and Data: The benchmark evaluates six frozen ECG foundation models (CSFM, CARDIAC-FM, ECG-FM, ECG-JEPA, HuBERT-ECG, and ST-MEM) across the PTB-XL dataset. An external validation is performed on a credentialed MIMIC-IV-ECG cohort.
• Experimental Grid: The evaluation spans a 450-cell atlas constructed from:
  • 6 Models
  • 5 standardized relative encoder depths ($q \in \{0, 0.25, 0.5, 0.75, 1\}$)
  • 5 matched dictionary expansion factors ($E \in \{1, 4, 8, 16, 32\}$)
  • 3 random seeds
    This results in 75 "exact match" blocks where all six models share the same target depth, dictionary capacity, sparsity protocol, and seed.
• SAE Configuration: The SAEs use a BatchTopK activation function with a fixed active feature budget relative to the dictionary size ($k/d = 1/8$). All source layer atlases are standardized to dimension $d=768$.
• Metrics: The benchmark evaluates three complementary axes:
  1. Sparse Reconstruction Fidelity: Measured by normalized reconstruction $R^2$, dead-feature fraction, and realized mean sparsity ($L_0$).
  2. Single-Feature Clinical Accessibility: The correlation between individual SAE code coordinates and 49 clinically meaningful ECG measurements (e.g., intervals, amplitudes, ST-T changes). Features are selected on a training subset and evaluated on held-out test data to prevent leakage.
  3. Cross-Seed Reproducibility: The stability of decoder directions and feature subspaces across different random seeds, measured via matched cosine similarity and subspace overlap.
• Uncertainty Quantification: The framework incorporates patient-sampling uncertainty via bootstrapping and design uncertainty via seed and depth variations, using paired statistical comparisons.

Key Contributions

1. Capacity-Controlled Benchmark: Introduction of a framework for comparing representation-level interpretability across ECG FMs under a matched SAE protocol, separating model differences from measurement capacity differences.
2. 450-Cell Interpretability Atlas: Construction of a comprehensive atlas mapping how representation interpretability changes across encoder depth and SAE scale, providing a structured profile for each model.
3. Leakage-Controlled Metrics: Definition of metrics spanning sparse fidelity, single-feature clinical accessibility, and cross-seed reproducibility, ensuring that feature selection occurs strictly on training data before held-out evaluation.
4. Robust Comparisons: Provision of comparisons supported by patient and design uncertainty, sparsity sensitivity analysis (comparing fixed $k/d$ vs. fixed $k/N$), and external-cohort validation.

Results
The benchmark reveals that ECG foundation models exhibit distinct, metric-specific interpretability profiles; no single model dominates across all dimensions.

• Reconstruction Fidelity: HuBERT-ECG achieves the highest reconstruction fidelity (AUC 0.985 on PTB-XL, 0.990 on MIMIC-IV), followed by ECG-JEPA and CSFM.
• Clinical Accessibility: ECG-JEPA demonstrates the highest single-feature clinical accessibility and concept coverage (AUC 0.324 and 0.781, respectively), outperforming other models in localizing clinical concepts to sparse features.
• Stability and Utilization: ECG-FM exhibits the lowest dead-feature fraction (0.099) and highest feature stability, while CSFM shows the highest subspace stability.
• External Validation: The MIMIC-IV-ECG replication confirms that the separation between reconstruction leaders (HuBERT-ECG) and accessibility leaders (ECG-JEPA) is preserved across datasets.
• Rank Stability: Reconstruction rankings are highly stable across SAE capacities (Kendall $\tau = 1.00$), whereas accessibility and coverage rankings show moderate stability, indicating that some interpretability properties are more sensitive to dictionary scale than others.

Significance and Claims
The paper claims that ECG-InterpBench provides a capacity-controlled and reproducible framework for comparing ECG foundation models across distinct dimensions of representation interpretability. It argues that representation interpretability is not equivalent to predictive accuracy or linear decodability but requires explicit, complementary measurements.

The authors emphasize that the benchmark characterizes the structure of frozen model representations under a matched protocol. They explicitly state that the reported model orderings are specific to the evaluated representation properties (fidelity, accessibility, stability) and should not be interpreted as rankings of diagnostic performance or overall clinical value. The work does not establish physiological mechanisms or prospective patient outcomes. Instead, it establishes a foundation for the principled development and selection of interpretable ECG foundation models by connecting representation geometry with clinical concepts across models and scales. The framework is presented as extensible to other clinical modalities, including medical imaging, EEG, and electronic health records.