🤖 machine learning

LUNG-KGMM: Knowledge-Guided Multimodal Learning for Lung Cancer Incidence Prediction

This paper proposes LUNG-KGMM, a knowledge-guided multimodal framework that integrates electronic health records, radiology reports, chest radiographs, and clinical guidelines to achieve superior and portable lung cancer incidence prediction while addressing data heterogeneity and leakage through specialized processing pipelines.

Original authors: Chunlei Yang, Shuyan Li, Zhong Cao

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chunlei Yang, Shuyan Li, Zhong Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Lung cancer remains one of the most formidable health challenges worldwide, responsible for a significant portion of cancer-related deaths. For decades, doctors have relied on screening tools like low-dose computed tomography scans to catch the disease early in high-risk individuals, but these tests are expensive and not suitable for everyone. In the meantime, hospitals generate a vast, continuous stream of patient data during routine care: electronic health records that track a person's history, radiology reports written by specialists after looking at X-rays, and the X-ray images themselves. The challenge for modern medicine is not a lack of data, but rather the difficulty of weaving these different threads together into a single, coherent picture that can predict who might develop lung cancer in the coming years. Researchers are increasingly turning to artificial intelligence to help, hoping to build systems that can learn from these diverse sources to spot subtle warning signs that a human might miss or that are too complex to calculate manually.

In this context, a team of researchers has developed a new framework called LUNG-KGMM, designed to predict the risk of developing lung cancer over a period of one to six years. The system does not rely on a single type of information. Instead, it acts as a synthesizer, bringing together four distinct types of data: a patient's longitudinal electronic health records, the text of their radiology reports, the visual features of their chest X-rays, and structured knowledge derived from official medical guidelines. The researchers built this tool to address a common problem in medical AI: the tendency for models to inadvertently incorporate future information in the data they are trained on. To prevent this, they created a rigorous process to clean the radiology reports, stripping out any language that explicitly mentions a cancer diagnosis or past treatment, ensuring the model only learns from the clues available at the time of the check-up. They also introduced a way to handle patients who were not followed up for the full six years, allowing the system to learn from partial information without making false assumptions.

The core of this new approach is a unique way of handling medical guidelines. Rather than just reading a rulebook, the system translates the relationships between findings and recommended actions into a structured map. For instance, if a report mentions a "nodule," the system understands that this finding is linked to specific attributes like size and density, and that these attributes trigger certain follow-up actions according to medical standards. This creates an auditable stream of knowledge that helps the model understand not just what was found, but what that finding implies in a clinical context. The researchers tested this framework using a large public database of medical records and images, and then validated their findings on a separate, real-world dataset from a hospital platform in Xiamen, China. This second step was crucial to see if the model could work outside the controlled environment where it was built, even though the data in the second location was subject to different privacy rules and language.

The results showed that the new system outperformed existing methods when tested on the public database. It achieved a higher level of accuracy in distinguishing between patients who would develop lung cancer and those who would not, compared to models that used only electronic records, only images, or only the text of the reports. The study found that the radiology reports themselves carried the strongest signal, likely because they are a professional summary of the visual findings. However, adding the structured guideline knowledge and the image data provided a measurable, incremental improvement, suggesting that combining these sources offers a more complete view of risk than any single source alone. The researchers also discovered that while the model could be applied to the Chinese dataset without retraining, its performance dropped significantly when it tried to use synthetic images generated from text descriptions instead of real X-rays. This highlighted that the system relies heavily on the specific language and the actual visual data it was trained on, and that it cannot simply be copied and pasted into a different hospital system without local adjustments.

Ultimately, this work demonstrates that it is possible to build a transparent, reproducible system that learns from the complex, messy reality of routine hospital care to predict future cancer risk. The researchers did not claim to have solved the problem of lung cancer prediction or to have created a tool ready for immediate clinical use. Instead, they showed that a framework integrating text, images, and structured medical knowledge can learn effectively from historical data. They emphasized that for such a system to be useful in a real hospital, it would need to be adapted to local languages, reporting styles, and data privacy rules, and it would require prospective testing to prove it works in the future, not just the past. The study serves as a proof of concept, showing that when artificial intelligence is carefully guided by clinical knowledge and rigorous data cleaning, it can begin to make sense of the vast amounts of information doctors generate every day.

Technical Summary: LUNG-KGMM

Problem Statement
Early identification of lung cancer risk is essential for timely intervention, yet existing prediction models face two primary limitations: a reliance on single data modalities and an inability to effectively leverage structured clinical knowledge. Furthermore, routine-care risk identification differs from protocolized screening (e.g., LDCT), and existing datasets often suffer from issues such as modality heterogeneity, potential data leakage in radiology reports (e.g., inclusion of prior cancer history or treatment details), and incomplete follow-up data. There is a need for a reproducible framework that can align multimodal data, mask unobserved follow-up horizons, and validate performance in real-world hospital settings distinct from public datasets.

Methodology
The authors propose LUNG-KGMM, a knowledge-guided multimodal longitudinal learning framework designed for 1-to-6-year incident lung cancer prediction. The methodology addresses the identified gaps through the following components:

Data Construction & Sanitization:
- Cohorts: A development cohort was built from the public MIMIC-IV, MIMIC-CXR, and MIMIC-IV-Note databases. A real-world validation cohort was constructed from the Xiamen Medical Big Data Platform.
- Leakage-Sanitized Reports: A specific pipeline processes radiology reports to remove "leakage" terms (direct cancer diagnoses, prior history, treatment history) while retaining index-time risk findings (e.g., nodules, ground-glass opacities, suspicious morphology).
- Index-Time Samples: Samples are constructed with pre-index EHR data, sanitized report tokens, optional image embeddings, and guideline-derived knowledge tokens.
Multimodal Architecture:
- EHR & Text: Longitudinal EHR variables and radiology reports are encoded using a one-layer Transformer encoder with two attention heads.
- Image Representation: For MIMIC, multiview chest radiograph (CXR) features are extracted using pre-trained TorchXRayVision (DenseNet121) and aggregated. For the Xiamen cohort (where real images were restricted), report-conditioned synthetic images generated by RoentGen-v2 were used for sensitivity analysis only, not as primary evidence.
- Knowledge-Guided Stream (KG): Clinical guidelines (e.g., pulmonary nodule management) are not used as executable rules but as a structured knowledge stream. Report triggers are expanded into "finding-attribute-action" relation tokens (e.g., linking a nodule mention to the concept that size and density are relevant attributes). This creates an auditable, structured re-encoding of report evidence.
Fusion and Prediction:
- Fusion Strategy: The model concatenates vectors from EHR, sanitized reports, KG terms, and image features. While attention and gated fusion were tested, concatenation was retained as it yielded the best performance.
- Training Objective: A horizon-masked cumulative binary cross-entropy loss is used. This handles incomplete follow-up by applying the loss only to observable time horizons (1–6 years) and treating unobserved horizons as missing rather than negative outcomes. The output predicts monotonic cumulative risk logits for each year.

Key Contributions

Real-World Cohort Construction: The authors constructed a large-scale, real-world Xiamen lung cancer risk cohort containing EHR, radiology reports, and 1–6-year incident labels, alongside a public MIMIC development cohort.
Knowledge-Guided Multimodal Model: The proposal of LUNG-KGMM, which integrates EHR, leakage-sanitized reports, CXR features, and report-triggered guideline relations. The model achieves the strongest point estimate among compared models on the MIMIC test set.
Cross-Cohort Evaluation: The study conducts within-cohort evaluation and cross-cohort "no-retraining" analyses to characterize performance portability and the limitations of transferring models between different healthcare systems and languages.

Results

MIMIC Performance: On the MIMIC test set, LUNG-KGMM achieved a macro 1/3/6-year AUROC of 0.885, outperforming strong baselines including MLP (0.870), DrFuse-style fusion (0.876), and MedFuse-style fusion (0.825). It also achieved a macro AUPRC of 0.507 and a Brier score of 0.034.
Ablation Insights:
- Modality Contribution: The radiology report (sanitized) was the strongest single modality (AUROC 0.873), significantly outperforming EHR-only (0.743) and image-only (0.787 for multiview XRV).
- Knowledge Value: The Knowledge Graph (KG) representation provided the best guideline encoding (0.885) compared to structured terms, retrieved text, or executable paths, adding a modest but complementary gain over the no-guideline model (0.860).
- Fusion: Concatenation fusion outperformed attention and gated fusion strategies.
Cross-Cohort Portability: When applied to the Xiamen cohort without retraining:
- The "no-image" transfer (EHR + Report + KG) achieved a macro AUROC of 0.750 (MIMIC $\to$ Xiamen).
- Adding the image stream (using synthetic images for Xiamen) degraded performance to 0.592, highlighting a significant domain mismatch between real MIMIC CXR embeddings and report-conditioned synthetic images.
- The results indicate that while the task schema is portable, fixed model weights are not directly transferable across different languages, reporting practices, or data governance environments without local adaptation.

Significance and Claims
The paper claims that LUNG-KGMM provides a reproducible research framework for longitudinal lung cancer incidence prediction that explicitly addresses modality alignment, data leakage, and follow-up masking. The study demonstrates that a transparent multimodal model can be trained for this task and that structured guideline relations (KG) can augment risk prediction when derived from report evidence.

However, the authors maintain a modest stance regarding clinical utility:

The performance gains (e.g., 0.015 AUROC over MLP) are described as "modest point-estimate improvements" that support incremental model value but do not establish clinical benefit without prospective evaluation.
The image and KG contributions are characterized as "incremental" improvements over the strong semantic baseline provided by sanitized reports.
The cross-cohort results emphasize that local terminology harmonization, feature alignment, and likely site-specific retraining are required before deployment in real-world settings.
The use of synthetic images is explicitly framed as a "negative missing-image sensitivity analysis" and not a substitute for real imaging evidence.

In conclusion, the work validates the feasibility of a knowledge-guided multimodal approach for risk prediction while highlighting the practical challenges of data heterogeneity and the necessity of local adaptation for real-world application.

Technical Summary: LUNG-KGMM

More like this