HyperCT: Low-Rank Hypernet for Unified Chest CT Analysis

1. Problem Statement

Non-contrast chest CT scans are a standard modality for pulmonary screening (e.g., detecting nodules or emphysema) but contain rich anatomical data regarding the heart and great vessels. This has led to the paradigm of opportunistic screening, where these scans are repurposed to detect extra-pulmonary conditions like cardiac structural and functional abnormalities.

However, current approaches suffer from two main limitations:

1. Isolation: Conventional pulmonary tasks and opportunistic cardiac tasks are typically addressed by separate, isolated models. There is no unified framework to perform both simultaneously.
2. Suboptimal Multi-Task Learning (MTL): Standard MTL approaches (e.g., hard parameter sharing or loss weighting) often assume tasks are competing and focus on mitigating negative transfer. The authors argue this is misaligned with medical screening, where findings are often synergistic and comorbid (e.g., cardiac enlargement often co-occurs with pulmonary congestion). Furthermore, existing MTL methods struggle to scale to high-capacity architectures like Vision Transformers (ViTs) without prohibitive computational costs.

2. Methodology: HyperCT

The authors propose HyperCT, a framework that unifies conventional and opportunistic screening by dynamically adapting a pre-trained Vision Transformer backbone via a Hypernetwork integrated with Low-Rank Adaptation (LoRA).

Core Architecture

• Base Model: A frozen, pre-trained ViT backbone (specifically DINOv3, ViT-base variant) serves as the feature extractor.
• Hypernetwork ($h_\phi$): A neural network that takes a task embedding ($e_k$) as input and generates task-specific parameters for the base model.
• Dynamic Parameter Generation: Instead of sharing static weights across all tasks, the hypernetwork generates a unique set of weights $\theta_k$ for each specific task $k$ based on its embedding.
  • $\theta_k = h_\phi(e_k)$
  • The prediction is $\hat{y}_k = f_{\theta_k}(x)$.

Low-Rank Adaptation (LoRA) Integration

To make the hypernetwork computationally feasible for large ViTs, the authors do not generate full-rank weight matrices. Instead, they use LoRA:

• Mechanism: For each target module $m$ in the ViT (e.g., attention projections, MLP layers), the hypernetwork generates two low-rank matrices, $A_m$ and $B_m$, with rank $r \ll \min(d_{in}, d_{out})$.
• Weight Update: The effective weight for a task is the frozen pre-trained weight plus the low-rank update:
  $$W_{task} = W_{base} + \frac{\alpha}{r} (B_m A_m)$$
• Efficiency: This reduces the complexity of the hypernetwork from quadratic ($O(D^2)$) to linear ($O(D)$) relative to the ViT hidden dimension, allowing the generation of task-specific adaptations without exploding the parameter count.
• Module Conditioning: The hypernetwork also receives a module indicator ($\phi_{pos}$) to ensure it generates distinct weights for different layers within the ViT, rather than identical weights for all layers.

Training Strategy

• Data: 36,286 non-contrast chest CTs from Columbia University (CU) and Weill Cornell Medicine (WCM).
• Tasks: 25 binary classification tasks (18 conventional pulmonary, 7 opportunistic cardiac).
• Labeling: Pulmonary labels were extracted from free-text radiology reports using Llama3.1. Cardiac labels were derived by matching CT scans to echocardiography reports within a $\pm 180$-day window.
• Optimization: The model is trained using Binary Cross-Entropy (BCE) loss. During training, one task is randomly sampled per sample to compute the loss.

3. Key Contributions

1. First Unified Framework: The first model to jointly perform 18 conventional pulmonary and 7 opportunistic cardiovascular screening tasks on non-contrast chest CTs, bridging a critical gap in holistic patient assessment.
2. Scalable Hypernetwork Design: Successfully integrates LoRA into a hypernetwork architecture, overcoming the scalability limitations that previously restricted hypernetworks to small models. This enables efficient, task-specific adaptation of large foundation models (ViTs).
3. Comprehensive Validation: Validated across retrospective, prospective, and multi-institutional cohorts, demonstrating robustness to distributional shifts common in real-world deployment.
4. Synergistic Learning: Demonstrates that explicitly modeling task relationships via dynamic parameter generation outperforms standard MTL approaches that assume task conflict.

4. Experimental Results

Performance Metrics (AUC)

• Retrospective Evaluation: HyperCT achieved an overall average AUC of 78.1% (CU) and 76.5% (WCM), outperforming strong MTL baselines (e.g., MGDA, UW, RLW) and matching the performance of dedicated Single-Task Learning (STL) models.
• Prospective Evaluation: In prospective cohorts (real-world data), HyperCT significantly outperformed STL models (77.8% vs 75.4% on CU), suggesting that the multi-task dynamic architecture learns more robust representations that generalize better to distributional shifts.
• Ablation Studies:
  • LoRA Rank: Performance improved with higher ranks, with $r=16$ yielding optimal results.
  • Backbone: The 2D-pretrained DINOv3 backbone significantly outperformed the 3D-pretrained CTViT, highlighting the transferability of large-scale 2D natural image pretraining to medical 3D tasks.
  • Module Selection: Applying LoRA to both Attention and MLP modules (HyperCT) outperformed variants applying it to only one.

Clinical Utility (Decision Curve Analysis)

• Net Benefit: Decision Curve Analysis (DCA) showed that HyperCT provides a positive net benefit over "treat all" and "treat none" strategies across a wide range of thresholds (5–80%) for all 7 cardiac tasks.
• Generalization: This clinical utility was maintained in the external WCM prospective cohort, proving the model's value for selective referral to echocardiography.

Interpretability

• PCA of Weights: Principal Component Analysis of the generated LoRA weights revealed distinct clustering between Opportunistic (Cardiac) and Conventional (Pulmonary) tasks, indicating the hypernetwork successfully learns domain-specific parameter subspaces.
• Saliency Maps: Grad-CAM visualizations confirmed that the model focuses on relevant anatomical regions (cardiac silhouette for cardiac tasks, lung parenchyma for pulmonary tasks).

5. Significance

HyperCT represents a paradigm shift in medical imaging AI by moving from isolated, single-task models to a unified, parameter-efficient, and holistic assessment tool.

• Clinical Impact: It maximizes the value of existing non-contrast chest CT data, enabling the detection of life-threatening cardiac conditions without additional radiation or cost.
• Technical Impact: It solves the scalability bottleneck of hypernetworks in large-scale medical imaging, providing a blueprint for applying dynamic parameter generation to foundation models in other domains.
• Efficiency: By maintaining a constant parameter count regardless of the number of tasks, it offers a cost-effective solution for hospitals needing to deploy multiple screening capabilities simultaneously.