Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification

Technical Summary: Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification

Problem Statement
Automated classification of Cardiac Magnetic Resonance (CMR) videos remains challenging due to the reliance of recent deep learning models solely on raw image intensities. While these models capture temporal dependencies, they often overlook underlying myocardial deformations. Pathological conditions such as cardiomyopathy and myocarditis manifest as regional wall motion abnormalities and altered strain patterns, which are geometrically encoded in deformation fields. Although deformable shape representations have proven robust complements to texture features in neurodegenerative disease classification, extending this to cardiac video analysis is non-trivial. Existing methods typically combine shape and texture features via simple concatenation or element-wise addition. These approaches fail to model cross-modal feature dependencies, ignore the varying diagnostic importance of different cardiac phases, and apply uniform weighting across timepoints, thereby missing the dynamic interplay between myocardial motion and image intensity.

Methodology: ShapeFuse
The authors propose ShapeFuse, a framework that unifies deformable shape and image texture representations in a shared latent space. The architecture consists of three primary stages:

1. Feature Extraction:

   • Shape Representation: A shape encoder-decoder network ($\theta^E_S, \theta^D_S$) estimates frame-wise velocity fields ($v$) by aligning a source frame ($I_0$) to target frames ($I_t$) using diffeomorphic registration principles (Stationary Velocity Fields). This yields latent geometric deformation features ($z_{vt}$).
   • Texture Representation: An image encoder ($\theta^E_I$) extracts latent texture features ($z_{ft}$) from the raw CMR frames.

2. Latent Shape-Aware Feature Fusion:

   • Bidirectional Cross-Modal Temporal Attention (BCTA): Instead of static fusion, ShapeFuse employs a bidirectional attention mechanism in the latent space. Each modality (shape and texture) learns its own query, key, and value projections. This allows the shape sequence to attend to relevant texture frames across time and vice versa. The mechanism explicitly parameterizes cross-phase dependencies, enabling each modality to adaptively weight the other based on spatio-temporal correspondence.
   • Adaptive Gating: A per-timepoint adaptive gate ($g_t$) dynamically controls the relative contribution of shape and texture features. This gate, computed via a sigmoid function over concatenated latent features, allows the model to learn which modality is more informative at specific cardiac phases, rather than treating them symmetrically.
   • Diagnostic Importance Pooling: Following the gating mechanism, a learned temporal pooling strategy (based on Bahdanau Attention) aggregates the fused sequence. This assigns higher weights to diagnostically critical timepoints (e.g., peak systolic contraction) rather than applying uniform temporal pooling.

3. Training Strategy:

   • The model is trained in two stages. First, the shape network is optimized to minimize a registration loss (combining image dissimilarity and velocity field smoothness). Second, with the shape encoder frozen, the image encoder, attention modules, gating mechanism, pooling strategy, and classifier are jointly optimized using binary cross-entropy loss.

Key Contributions
The paper claims three primary contributions:

1. Bidirectional Cross-Modal Temporal Attention: A mechanism that learns mutual dependencies between latent deformable shape and texture representations across the cardiac cycle, moving beyond simple concatenation.
2. Adaptive Fusion and Pooling: A gating strategy that dynamically weights shape and texture contributions at each cardiac phase, coupled with a learned temporal pooling that prioritizes diagnostically relevant timepoints.
3. State-of-the-Art Performance and Interpretability: Demonstration of superior classification performance on real-world cine CMR videos, supported by attention maps that localize diagnostically critical phases and quantify modality contributions.

Experimental Results
The framework was evaluated on a dataset of 510 cine CMR video sequences from 125 subjects, featuring a class imbalance of myocardial scar-induced wall motion abnormalities.

• Classification Performance: ShapeFuse consistently outperformed competing fusion strategies (including addition, concatenation, weighted, bilinear, and standard attention fusion) across multiple image encoder backbones (ResNet, EfficientNet, DenseNet, ViT) and registration networks (VoxelMorph and TLRN). For instance, with a ResNet backbone and VoxelMorph registration, ShapeFuse achieved an accuracy of 0.888 and an F1-score of 0.902, surpassing the best baseline (Weighted fusion) which scored 0.832/0.857.
• Interpretability: Grad-CAM visualizations showed that ShapeFuse consistently highlighted the myocardial region across timepoints, whereas baseline methods produced diffuse or inconsistent activations.
• Attention Analysis: The learned attention maps revealed that deformation features consistently attended to mid-systolic texture frames, while texture representations sought phase-aligned geometric context. The adaptive gating distribution indicated that texture representations benefit significantly from geometric grounding.

Significance
The paper asserts that ShapeFuse represents the first framework to jointly learn the fusion of deformable shape and texture representations in a shared latent space for cardiac video classification. By replacing naive fusion with dynamic, cross-modal attention and adaptive gating, the method captures complementary physiological information that static approaches miss. The authors claim this leads to state-of-the-art classification performance and improved interpretability, as the attention mechanisms successfully identify the specific cardiac phases and modality contributions that are most critical for diagnosis. Future work is proposed to extend this to multi-label pathology classification and explore self-supervised pretraining to reduce reliance on labeled data.