📄 medicine

Interpretable Multi-modal Deep Learning Radiomics for Preoperative Mapping of Tertiary Lymphoid Structure Associated B-Cell Tumor Immune Microenvironment and Immunotherapy Response in Clear Cell Renal Cell Carcinoma

This study developed and validated a non-invasive, interpretable multi-modal deep learning radiomics model based on arterial-phase CT images that accurately predicts tertiary lymphoid structures in clear cell renal cell carcinoma, thereby enabling preoperative risk stratification and guiding immunotherapy decisions.

Original authors: Yingjie Xv, Hongjian Liu, Zongjie Wei, Bangxin Xiao, Zhanpeng Yuan, Qing Jiang, Xuan Zhang, Feng Li, Yong Chen, Ming Qiu, Xiang Peng, Jiaxin Su, Weiyang He, Mingzhao Xiao

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingjie Xv, Hongjian Liu, Zongjie Wei, Bangxin Xiao, Zhanpeng Yuan, Qing Jiang, Xuan Zhang, Feng Li, Yong Chen, Ming Qiu, Xiang Peng, Jiaxin Su, Weiyang He, Mingzhao Xiao

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your body as a bustling city under siege by a sneaky criminal gang called cancer. Usually, the city's police force (your immune system) tries to fight back, but sometimes the criminals build invisible walls that hide them from the police. In some cases, however, the city manages to build its own mini-stations right inside the enemy territory. Scientists call these "Tertiary Lymphoid Structures" (TLS). Think of TLS as tiny, makeshift police academies built right inside the tumor, where immune cells gather, train, and launch a coordinated attack. The more of these academies a tumor has, the better the patient's chances of surviving and responding to powerful new drugs called immunotherapies. The problem? To see these academies, doctors currently have to perform surgery or a biopsy to take a physical piece of the tumor out and look at it under a microscope. It's invasive, risky, and you can't do it before the surgery to plan the best attack.

This is where the story gets exciting. What if you could look at a standard medical scan, like a CT scan (which is just a super-detailed 3D X-ray), and use a super-smart computer brain to "see" these invisible police academies without ever cutting the patient open? That is exactly what a team of researchers from China set out to do. They wanted to build an artificial intelligence (AI) detective that could look at a CT scan of a kidney tumor and predict whether these helpful immune structures were hiding inside. If they could do this, doctors could know a patient's prognosis and whether immunotherapy would work before the surgery even begins, turning a blind guess into a clear, data-driven map.

The AI Detective and the Invisible Map

The researchers, led by a team from Chongqing Medical University and several other hospitals, decided to build a "multi-modal" AI detective. In plain English, "multi-modal" means they didn't just rely on one way of looking at the data. Instead, they combined three different ways of analyzing the CT scans:

2D Analysis: Looking at the tumor like a single flat photograph (a slice).
2.5D Analysis: Looking at the tumor from three different angles (front, side, and top) at the same time, like holding a cube and looking at it from three sides.
3D Analysis: Looking at the tumor as a full, solid 3D block.

They also added a fourth tool: "Radiomics." Think of this as a digital magnifying glass that measures thousands of tiny, invisible textures in the image—things like how rough the surface is, how the colors blend, and how the pixels are arranged. These textures are too small for a human eye to notice but might hold the secret to whether the tumor is hiding an immune academy.

The team trained their AI on a massive dataset of 1,361 patients with a specific type of kidney cancer called clear cell renal cell carcinoma (ccRCC). They split these patients into different groups: some to teach the AI (the training set), some to test it (the test sets), and even a group of patients they would treat in the future to see if the AI's predictions held up (the prospective set).

The Big Findings: The "Late Fusion" Winner

After training their AI, the researchers had to decide which detective was the best. They found that the 2.5D model (looking at the tumor from three angles) was the sharpest single detective, followed closely by the traditional radiomics model. However, the real magic happened when they combined them.

They tried two ways to combine the detectives:

Early Fusion (Pre-fusion): Mixing all the raw data together before making a decision.
Late Fusion (Late-fusion): Letting each detective make its own guess first, and then having a "boss" AI (using a method called Support Vector Machine, or SVM) listen to all the guesses and make the final call.

The Late Fusion strategy won the race. This combined model, which the authors call the LFM, achieved an impressive accuracy score (AUC) of 0.938 in the first test group. To put that in perspective, an AUC of 1.0 is a perfect score, and 0.5 is a coin flip. This means the AI was incredibly good at spotting the invisible immune academies just by looking at the CT scan.

But the team didn't stop there. They wanted to know: Is this AI actually seeing the biology, or is it just guessing? To prove it, they dug deeper. They looked at the actual tissue samples from the patients the AI had flagged as "TLS-positive." Using a high-tech microscope and gene sequencing, they found that the AI was right. The tumors it predicted had:

Higher densities of inflammatory cells (the police).
More B-cells (a specific type of immune soldier).
Increased activity in genes known to build these immune structures (specifically the CXCL13-CXCR5 axis).

This confirmed that the AI wasn't just seeing random patterns; it was detecting the biological footprint of a real immune response.

Predicting the Future and Saving Lives

The ultimate test for any medical tool is whether it helps patients. The researchers used their LFM model to predict two critical things:

Survival: Patients whose tumors were predicted to have TLS (the immune academies) lived longer without the cancer coming back. In the test groups, this difference was statistically significant, meaning it was a real effect, not a fluke.
Immunotherapy Response: For patients with advanced cancer who received immunotherapy (drugs that wake up the immune system), the AI could predict who would respond well. The model correctly identified "responders" (those who got better) versus "non-responders" with an accuracy of 0.807.

The team also tested their model on a "prospective" group—patients treated after the model was built. The AI maintained its high accuracy (AUC of 0.881), showing that it wasn't just memorizing old data but could actually predict outcomes for new patients.

The Limits and the Future

While the results are promising, the authors are careful not to call this a "solved" problem. They note that their study was retrospective (looking back at past data) and mostly focused on patients in China, so the model needs to be tested on more diverse populations to ensure it works everywhere. They also admit that their biological analysis was based on a relatively small group of patients, so the exact molecular mechanisms need further study.

However, the core message is clear: This study suggests that we can build a non-invasive "map" of the tumor's immune environment using standard CT scans and AI. By combining different ways of looking at the image and validating it with real biological data, the team has created a tool that could help doctors decide who needs surgery, who needs immunotherapy, and who is likely to do well—all before making a single cut. It's a step toward a future where AI acts as a translator, turning the static gray images of a CT scan into a dynamic story of the body's fight against cancer.

Technical Summary: Interpretable Multi-modal Deep Learning Radiomics for Preoperative Mapping of TLS in ccRCC

Problem Statement
Clear cell renal cell carcinoma (ccRCC) is the predominant form of renal malignancy. While immune checkpoint inhibitors (ICIs) have revolutionized treatment for advanced cases, clinical responses vary significantly. Tertiary lymphoid structures (TLS)—ectopic lymphoid aggregates within the tumor microenvironment—are strongly associated with improved prognosis and ICI responsiveness. However, current assessment of TLS relies on invasive histopathological examination of surgical or biopsy specimens, which is limited by spatial heterogeneity, procedural risks, and subjectivity. There is a critical need for a robust, non-invasive preoperative surrogate to assess TLS status and guide therapeutic decision-making.

Methodology
This multicenter, retrospective-prospective study enrolled 1,361 patients with ccRCC from eight tertiary hospitals in China. The cohort was partitioned into a primary training set ( $n=686$ ), two external test sets ( $n=244$ and $n=147$ ), a prospective validation set ( $n=67$ ), an ICI therapy cohort ( $n=152$ ), and a multi-omics cohort ( $n=65$ ).

The study employed a three-phase workflow:

Single-Modality Model Development: Preoperative arterial-phase contrast-enhanced CT (CECT) images were utilized to train three categories of models:
- 2D Deep Learning (DL): Utilized axial slices with maximum tumor cross-sections.
- 2.5D DL: Incorporated three orthogonal views (axial, sagittal, coronal).
- 3D DL: Analyzed cubic volumes centered on the tumor.
- Radiomics: Extracted 1,315 features (first-order, shape, texture) processed via dimensionality reduction (mRMR, Elastic Net) and fed into machine learning (ML) algorithms.
- Architectures: Various CNNs (e.g., ResNet, DenseNet, VGG) pre-trained on ImageNet or Med3D were benchmarked.
Fusion Strategy: The optimal single-modality models (identified via Test Set 1 performance) were integrated using two strategies:
- Pre-fusion Model (PFM): Feature-level integration (concatenating DL features with radiomic features) followed by ML classification.
- Late-fusion Model (LFM): Decision-level integration (stacking risk scores from individual models) followed by ML classification.
Biological Interpretability: The best-performing model was validated against digital pathology (HoVer-Net cell segmentation on Whole Slide Images) and bulk RNA sequencing (differential expression analysis, immune deconvolution via CIBERSORTx) to uncover the biopathological basis of predictions.

Key Contributions

Multi-modal Framework: The study systematically benchmarks and fuses 2D, 2.5D, and 3D DL architectures with traditional radiomics, addressing a gap in existing literature which often relies on single-modality approaches.
Rigorous Validation: The model was validated across external multi-center cohorts, a prospective cohort, and an ICI-treated cohort, ensuring robustness beyond single-center retrospective data.
Biological Interpretability: Unlike "black box" AI models, this study links radiomic predictions to specific pathological features (inflammatory cell density) and transcriptomic signatures (CXCL13-CXCR5 axis, B-cell markers), providing a mechanistic rationale for the model's performance.
Clinical Utility: The framework serves as a non-invasive tool for preoperative TLS mapping, prognostic stratification, and predicting ICI response.

Results

Single-Modality Performance: In Test Set 1, the 2.5D-DenseNet161 model achieved the highest AUC among DL models (0.899), followed by 2D-ResNet50 (0.888) and Radiomics-XGBoost (0.878). The 3D-ResNet101 performed lower (0.756).
Fusion Performance: The Late-fusion Model (LFM), specifically an SVM-based ensemble, outperformed the Pre-fusion Model (PFM). The LFM achieved an AUC of 0.938 (95% CI: 0.911–0.961) in Test Set 1.
External and Prospective Stability: The LFM maintained high performance across Test Set 2 (AUC 0.918), the prospective set (AUC 0.881), and the multi-omics set (AUC 0.877).
Prognostic and Therapeutic Value:
- Patients predicted as TLS-positive by the LFM demonstrated significantly prolonged Progression-Free Survival (PFS) in the primary and external test sets ( $P<0.05$ ).
- In the ICI cohort, the LFM effectively discriminated responders from non-responders with an AUC of 0.807.
Biological Correlation: LFM-predicted TLS-positive cases showed significantly higher densities of inflammatory and necrotic cells on pathology. Transcriptomic analysis confirmed upregulation of TLS-associated genes (CXCL13, CXCR5, MS4A1, PDCD1) and enrichment of humoral immune response pathways, alongside increased infiltration of naive and memory B cells.

Significance and Claims
The authors claim that this multi-modal LFM provides a reliable, non-invasive surrogate for assessing TLS status in ccRCC, overcoming the limitations of invasive tissue sampling. The study asserts that the model's ability to capture imaging phenotypes related to local immune-inflammatory activity offers a solid mechanistic basis for its predictions. Clinically, the tool is positioned to support preoperative risk stratification and personalized immunotherapy decision-making.

The paper acknowledges limitations, including the retrospective nature of the primary cohorts, the exclusive recruitment from Chinese centers (limiting generalizability to diverse populations), and the reliance on H&E staining for TLS ground truth. The authors note that the biological mechanisms identified are preliminary and require further validation through functional experiments. They conclude that while the model shows promise, it is a supportive tool for clinical decision-making rather than a standalone diagnostic standard.

The AI Detective and the Invisible Map

The Big Findings: The "Late Fusion" Winner

Predicting the Future and Saving Lives

The Limits and the Future

Technical Summary: Interpretable Multi-modal Deep Learning Radiomics for Preoperative Mapping of TLS in ccRCC

More like this