Nodule-Aligned Latent Space Learning with LLM-Driven Multimodal Diffusion for Lung Nodule Progression Prediction

1. Problem Statement

Early diagnosis of lung cancer is critical but challenging due to biological uncertainty and a limited understanding of the mechanisms driving nodule progression. Current clinical practice relies on Low-Dose Computed Tomography (LDCT) screening, where indeterminate nodules require follow-up imaging (6–12 months) to monitor progression. This delay often postpones definitive diagnosis and treatment for malignant cases.

Existing AI approaches typically treat nodule diagnosis as a static, single-time-point classification task. While recent generative models can synthesize images, they often lack the fine-grained control necessary for clinical prognosis. Specifically, they struggle to align generated image variations with specific patient Electronic Health Record (EHR) data and precise nodule attributes (e.g., diameter, margin), which are crucial for predicting malignancy.

2. Methodology: NAMD Framework

The authors propose Nodule-Aligned Multimodal (Latent) Diffusion (NAMD), a framework designed to generate "virtual" 1-year follow-up LDCT scans based on baseline scans and patient/nodule metadata. The architecture consists of three core components:

A. Nodule-Aligned Latent Space

Instead of using a standard Variational Autoencoder (VAE), NAMD introduces a Nodule-Aligned VAE to compress high-dimensional LDCT images into a latent space where distances directly correspond to changes in nodule attributes.

• Latent Alignment Loss ($L_{align}$): This loss function enforces that the distance between latent codes ($z_i, z_j$) matches the distance between their corresponding metadata feature vectors ($f_i, f_j$). It uses a Kullback-Leibler (KL) divergence to align the distribution of latent similarities with the distribution of feature similarities.
• Predictive Representation Loss ($L_{pred}$): A lightweight binary classifier (benign vs. malignant) is trained on the latent space to ensure the embeddings capture clinically relevant diagnostic features.
• Total VAE Loss: Combines reconstruction loss ($L_{rec}$), KL divergence ($L_{KL}$), perceptual loss ($L_{LPIPS}$), and the proposed alignment and prediction losses.

B. LLM-Driven Diffusion Control

To incorporate patient-specific context, the model utilizes a Large Language Model (LLM) as an auxiliary context encoder.

• Data Processing: Patient EHR data (nodule attributes and personal history) is converted into a radiology report format.
• Soft-Prompt Adaptation: Using MedGemma 1.5 (a medical-focused LLM), the system employs Parameter-Efficient Fine-Tuning (PEFT) via soft-prompt tuning. Learnable prompt vectors are prepended to the input to adapt the frozen LLM weights to the specific task.
• Cross-Attention Injection: The LLM generates an embedding ($C$) from the EHR text, which is injected into the U-Net backbone of the diffusion model via cross-attention layers. This allows the diffusion process to be conditioned on fine-grained patient metadata.

C. Two-Stage Training Strategy

1. Unconditional Stage: Pre-trains the denoising U-Net on unpaired latent representations to learn the general distribution of lung nodule appearances.
2. Conditional Stage: Fine-tunes the model to predict the follow-up latent ($z^{(2)}$) given the baseline latent ($z^{(1)}$) and the LLM-derived context ($C$). The baseline latent is channel-concatenated with the noisy target latent, while the context is injected via cross-attention.

3. Key Contributions

1. Nodule-Aligned Latent Space: A novel latent space construction where latent distances are explicitly constrained to reflect changes in nodule attributes (e.g., diameter), enabling the model to capture the trajectory of nodule evolution.
2. LLM-Driven Control Mechanism: An innovative integration of medical LLMs with diffusion models using soft-prompt tuning to condition image generation on structured EHR data, ensuring generated outcomes adhere to clinical attributes rather than loose semantics.
3. Virtual Follow-Up Synthesis: A framework that successfully synthesizes 1-year follow-up images that preserve clinically relevant features, effectively bridging the gap between baseline data and future clinical reality.

4. Experimental Results

The model was evaluated on the National Lung Screening Trial (NLST) dataset (1,121 training pairs, 450 test pairs).

• Diagnostic Performance:
  • Using NAMD-generated follow-up images, the malignancy prediction model achieved an AUROC of 0.805 and AUPRC of 0.346.
  • This significantly outperforms models using only real baseline scans (AUROC: 0.742) and approaches the performance of models using real follow-up scans (AUROC: 0.819).
  • NAMD surpassed all state-of-the-art baselines (including McWGAN, CorrFlowNet, and standard SD1.5/Flux) in both AUROC and AUPRC.
• Image Quality:
  • The generated images achieved competitive FID (83.06) and LPIPS (0.220) scores, comparable to the best baselines.
• Ablation Studies:
  • Removing the alignment loss ($L_{align}$) caused AUROC to drop to 0.765 and FID to worsen significantly (101.3), proving that structuring the latent space is critical for capturing clinically meaningful progression signals.
  • t-SNE visualizations confirmed that the aligned latent space organizes nodules by diameter (a key clinical metric), whereas unaligned spaces resulted in entangled representations.

5. Significance and Conclusion

The paper demonstrates that NAMD can effectively predict lung nodule progression one year in advance by synthesizing virtual follow-up scans.

• Clinical Impact: By generating high-fidelity follow-up images that reflect likely disease trajectories, NAMD enables earlier and more accurate malignancy diagnosis, potentially reducing the critical delays associated with current surveillance protocols.
• Technical Insight: The study highlights that for clinical generative tasks, pixel-perfect reconstruction is not the ultimate goal. Instead, the model's ability to encode semantically meaningful progression signals (via the aligned latent space) is what drives downstream diagnostic accuracy.
• Future Direction: This work establishes a new paradigm for integrating multimodal data (images + EHR) via LLMs into diffusion models for precise, patient-specific medical prognosis.