When Measurement Conventions Masquerade as Calibration Gains in Cardiac Digital Twins

Technical Summary: When Measurement Conventions Masquerade as Calibration Gains in Cardiac Digital Twins

Problem Statement
Cardiac digital twins rely on an "observation operator" ($H_{op}$) to map clinical images (specifically echocardiograms) into physiological measurements like Left Ventricular Ejection Fraction (EF). While recent deep learning pipelines have advanced segmentation accuracy (often measured by Dice scores), systematic errors in $H_{op}$ can bias downstream state updates regardless of the underlying model's quality.

The core problem identified is that calibration studies often assume a fixed reference convention, leading to "phantom" gains. Specifically, models evaluated against clinical EF labels may appear to improve calibration simply because they inadvertently align with a measurement convention mismatch (e.g., single-plane vs. biplane) rather than because they genuinely reduce segmentation error. The authors argue that without disentangling measurement conventions from model performance, digital twin pipelines risk incorporating structured biases that distort physiological state estimation.

Methodology
The authors conducted an audit of EF observation operators using four front-end architectures sharing a U-Net backbone, applied to two major datasets: CAMUS and EchoNet-Dynamic.

1. Models Evaluated:
   • B1: Frame-wise segmentation.
   • B2: Adjacent-frame-smoothed segmentation.
   • P1: Combines FiLM (Feature-wise Linear Modulation) with a phase head.
   • P2: Adds cross-patient cyclic consistency and Mean-Teacher-style EMA averaging.
2. Observation Operator: A single-plane method-of-disks approximation was used to extract volumes and EF from apical four-chamber clips for all models.
3. Evaluation Metrics:
   • $EF_{MAE_{gt}}$: Error against ground-truth masks processed by the same extractor (isolating segmentation error).
   • $EF_{MAE_{clin}}$: Error against clinical EF labels (including convention mismatch).
   • Bias Analysis: Quantification of the gap between single-plane extractor outputs and biplane clinical references.
4. Datasets:
   • CAMUS: Evaluated on a held-out subset ($n=50$). Clinical EF is biplane (4- and 2-chamber views), while the extractor uses only the 4-chamber view, creating a known plane mismatch.
   • EchoNet-Dynamic: Evaluated on the official test split ($n=1,287$). Labels and extractor are both single-plane (4-chamber), serving as a plane-aligned replication to test if "gains" persist when the dominant mismatch is removed.

Key Contributions

1. EF Observation-Operator Audit: A framework to distinguish between genuine model improvements and artifacts caused by measurement-reference convention mismatches.
2. Evidence of Convention Confounding: Demonstration that the apparent calibration gains of phase-conditioned models (P1/P2) on CAMUS are largely explained by the gap between single-plane and biplane EF measurements, not by improved segmentation.
3. Negative Replication: A plane-aligned replication on EchoNet-Dynamic showing that the "phase conditioning advantage" disappears when the plane mismatch is removed, and in some cases, reverses.
4. Convention-Aware EF Audit (CAEA) Protocol: A five-step protocol to prevent models from receiving credit for calibration gains caused by the measurement-reference pair.

Results

• CAMUS Findings: Phase-conditioned models (P1/P2) appeared to reduce bias toward zero when compared to biplane clinical EF, suggesting a calibration gain. However, when evaluated against ground-truth masks processed by the same single-plane extractor ($MAE_{gt}$), the error was statistically indistinguishable across all models. In fact, P1 showed the largest $MAE_{gt}$ and most negative bias.
• The Convention Gap: The study quantified the "single-plane–biplane" gap. The single-plane ground-truth EF exceeded biplane clinical EF by +6.30 points. This offset explained nearly all the baseline bias observed in the B1 model. Once this convention component was subtracted, the residual segmentation bias was statistically indistinguishable from zero.
• EchoNet-Dynamic Findings: In the plane-aligned setting, the CAMUS results did not replicate. The baseline models (B1/B2) showed negative bias, and the phase-conditioned models (P1/P2) did not improve bias over the baselines. Furthermore, P1 and P2 produced invalid EF estimates (where $\hat{ESV} \ge \hat{EDV}$) in 10–20 cases, a deployment risk not present in B2.
• Downstream Implications:
  • Hemodynamics: Under CAMUS conditions, EF bias propagated to significant errors in Stroke Volume (SV) and Cardiac Output (CO) (up to ~16-18% error). Under plane-aligned EchoNet conditions, these errors dropped to ~3–4%.
  • Stratum Assignment: The convention mismatch caused frequent disagreement in EF stratum assignment (e.g., classifying a patient as HFrEF vs. HFpEF), with rates around 44–54% for the models tested.
  • Conformal Prediction: Residual widths were significantly larger under convention-confounded evaluation (CAMUS) compared to plane-aligned evaluation (EchoNet), indicating that calibration residuals contain structured reference mismatch, not just segmentation noise.

Significance and Claims
The paper claims that the apparent superiority of phase-conditioned models in previous literature may be an artifact of measurement conventions rather than genuine algorithmic progress. The authors assert that:

• Genuine gains must survive plane alignment: A model should only be credited with calibration improvements if the gain persists after aligning the measurement plane and auditing residual differences in frame selection, contours, and preprocessing.
• Dice is insufficient: High geometric overlap (Dice) does not guarantee calibrated physiological measurements; small contour shifts at End-Diastole (ED) and End-Systole (ES) can be amplified by the EF ratio.
• Protocol Necessity: The proposed Convention-Aware EF Audit (CAEA) is essential to separate genuine observation operator calibration from measurement artifacts. It mandates checking seed consistency, characterizing convention offsets, and replicating under aligned references before claiming calibration improvements.

The authors conclude that among the evaluated front-ends, B2 (adjacent-frame-smoothed) is the most deployable option, as it supports streaming inference, produces no invalid EF estimates, and achieves the best performance in plane-aligned settings. The study emphasizes that calibration should be assessed for the full "model–extractor–reference" pair, not just the model architecture.