Deep Shape Regression for Planar Curves with Multimodal Covariates

Technical Summary: Deep Shape Regression for Planar Curves with Multimodal Covariates

Problem Statement
The paper addresses the challenge of estimating how the shape of planar curves varies with high-dimensional, multimodal covariates (e.g., combining tabular demographics, disease status, genomics, and medical images). In statistical shape analysis, a curve's shape is defined as its equivalence class under translation, rotation, scaling, and reparametrisation. Existing methods often struggle to handle multimodal data or fail to account for reparametrisation invariance (elasticity), leading to "washed out" mean estimates where sharp features are blurred due to misalignment across observations. The authors aim to develop a regression model that is invariant to translation, rotation, and scaling, while also handling sparsely and irregularly sampled curves and complex covariate structures.

Methodology
The proposed approach integrates statistical shape analysis, elastic functional data analysis, and deep learning.

1. Mathematical Formulation:

   • Curves are represented as complex-valued functions $y: [0, 1] \to \mathbb{C}$.
   • The target is the conditional full Procrustes mean, defined as the minimizer of the expected squared full Procrustes distance to the observed curves, conditional on covariates $x$.
   • A key theoretical extension established in the paper is that this conditional mean is the leading eigenfunction of the conditional Hermitian covariance surface $C(s, t | x) = E[\tilde{y}(s) \tilde{y}(t) | X = x]$, where $\tilde{y}$ represents the pre-shape (centered and unit-norm) curves.

2. Deep Conditional Covariance Smoothing:

   • To estimate the conditional covariance surface, the authors propose a novel smoother using a cubic B-spline basis $b(t)$. The surface is modeled as $\hat{C}(s, t | x) = b(s)^T \Xi(x) b(t)$.
   • The coefficient matrix $\Xi(x)$ is constrained to be Hermitian and positive semi-definite via an outer product structure $\Xi(x) = \theta(x)\theta(x)^H$.
   • Multimodal Encoding: The vector $\theta(x)$ is constructed additively over covariates. Each covariate $x_j$ is mapped through a modality-specific encoder $\phi_j(x_j)$:
     • Scalar covariates: Penalized B-splines.
     • Tabular vectors: Multilayer perceptrons (MLPs).
     • Images (e.g., sMRI): Convolutional neural networks (CNNs).
   • The model is trained via penalized least squares using stochastic gradient descent (SGD), minimizing the difference between observed off-diagonal products of curve points and the estimated covariance surface.

3. Elastic Mean Estimation:

   • To achieve invariance to reparametrisation (warping), the authors propose an iterative algorithm.
   • The process alternates between:
     1. Covariance Smoothing: Fitting the conditional covariance and extracting the leading eigenfunction (the current mean estimate).
     2. Rotational Alignment: Aligning observed curves to the current mean.
     3. Parametrisation Alignment: Warping the curves to align with the mean using a square-root-velocity (SRV) framework.
   • This loop continues until the training loss stabilizes, effectively recovering crisp mean curves even with misaligned features.

4. Size and Shape:

   • Since the Procrustes mean is scale-invariant, the authors model size separately via a log-size regression on the same covariates. The final "size-and-shape" mean is the product of the predicted size and the unit-norm shape mean.

Key Contributions

• Theoretical Extension: Proves that the conditional full Procrustes mean is the leading eigenfunction of the conditional covariance, extending previous unconditional results to the regression setting.
• Novel Architecture: Introduces a deep conditional covariance smoother that accommodates multimodal covariates (images, sequences, tabular data) through modality-specific encoders, overcoming the limitations of classical tensor-product spline smoothers.
• Elastic Estimation Algorithm: Proposes an iterative scheme that jointly estimates the conditional mean and performs rotational and parametrisation alignment, addressing the "washing out" of features common in non-elastic methods.
• Application to Neuroimaging: Demonstrates the method on hippocampal outlines from the Alzheimer's Disease Neuroimaging Initiative (ADNI), successfully recovering covariate effects consistent with existing literature.

Results

• Simulation Study: Using synthetic data with known conditional means and multimodal covariates (scalar and image), the full method (with elastic alignment) outperformed variants without elastic alignment and variants using a rank-1 covariance approximation. The full method achieved the lowest mean squared prediction error (MSPE) and preserved sharp features (cusps) that were blurred in the non-elastic variant.
• ADNI Application: Applied to 239 hippocampal outlines, the model recovered known biological effects:
  • Increasing age and Alzheimer's disease (AD) diagnosis were associated with hippocampal shrinkage.
  • Sex showed no significant effect (likely due to pre-alignment volume matching).
  • An interaction between AD and APOE-$\varepsilon4$ allele dosage was detected, showing greater shrinkage in AD patients who are $\varepsilon4$ carriers, consistent with literature.
  • A sensitivity analysis suggested the model could incorporate whole-brain sMRI scans as covariates.

Significance and Claims
The authors claim that their method provides a general framework for conditional covariance estimation in functional data analysis, bridging the gap between neural functional models and conditional covariance smoothing. They emphasize that the model is "by construction" invariant to translation, rotation, and scaling, and capable of handling sparse, irregular data.

The paper modestly frames its application to hippocampal outlines as a "first application" to recover covariate effects consistent with the literature, rather than a definitive clinical breakthrough. Methodologically, the authors acknowledge open questions, such as the selection of penalty parameters via cross-validation rather than restricted maximum likelihood (which is difficult with neural encoders) and the need for future work to extend the approach to closed curves and three-dimensional shapes. They note that while the current implementation uses an established R package for the alignment step, integrating this directly into a deep learning framework could improve computational speed.