🤖 AI

A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification

The paper introduces MFM-Geom, a geometric multimodal foundation model that integrates bi-parametric MRI and clinical reports using Riemannian deep learning to achieve superior and robust prostate cancer classification with significantly reduced training data requirements.

Original authors: Juan A. Olmos, Antoine Manzanera, Fabio Martínez

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juan A. Olmos, Antoine Manzanera, Fabio Martínez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you only have half the clues. In the world of medicine, specifically when looking for prostate cancer, doctors often rely on two main types of evidence: a special kind of 3D picture of the prostate (called bp-MRI) and a written report containing the patient's history and blood test numbers. For a long time, computer programs designed to help doctors have been like detectives who only look at the pictures, ignoring the written notes. They try to guess the answer just by staring at the image, which can be tricky because medical images are complex and there aren't always thousands of examples to study. This is where "Foundation Models" come in—these are giant, pre-trained AI brains that have already learned a lot about medical images and text, kind of like a student who has read every textbook in the library before starting a specific exam. The big question researchers are asking is: Can we build a smarter detective that looks at both the pictures and the notes at the same time, using a special kind of math that understands how these different clues fit together geometrically?

This paper introduces a new detective named MFM-Geom. Instead of just gluing a picture and a text report together at the very end, this model uses a clever trick called "geometric learning." Think of the computer's understanding of the picture and the text as two different languages. Usually, AI tries to translate them into a single, flat list of numbers. But MFM-Geom treats these clues like a shape in a curved space (a "Riemannian manifold"), where the distance and angle between the picture clues and the text clues matter just as much as the clues themselves. By using this curved, geometric approach, the model can find hidden connections between what the MRI shows and what the clinical report says, even when it hasn't seen many examples to learn from.

The researchers tested this new detective on a dataset of prostate cancer cases. They found that MFM-Geom was incredibly good at spotting significant cancer, even when they only gave it 10% of the usual training data to learn from. In this "small data" scenario, MFM-Geom scored an AUC-PR of 90.67, which was significantly better than the old methods that only looked at images or used standard translation techniques. For instance, it beat the best image-only model by 8.3% in this specific metric. The paper also tested the model on a completely different set of data (an external dataset called PROSTATE158) to see if it could generalize, and it maintained a strong AUC-PR of 90.6, proving that this geometric approach is robust.

The paper explicitly argues against the idea that simply combining image and text predictions at the very end (a "decision-level fusion") is the best way to go. They suggest that this old method misses the complex, early relationships between the image details and the clinical variables. Instead, they propose that these clues should be woven together deep inside the model's brain using their geometric structure. They also show that using a model pre-trained on general images (like ImageNet) isn't as effective as using one pre-trained on biomedical data (BiomedCLIP), suggesting that the "medical vocabulary" learned beforehand is crucial.

In their experiments, the team compared MFM-Geom against several other setups: a standard image-only model, a model that just averages image patches, and a model using the same image structure but pre-trained on non-medical data. The results showed that the geometric head (the part that handles the curved math) consistently outperformed the others, especially in difficult cases involving intermediate levels of cancer. The authors also looked at "attention maps," which are like heatmaps showing what the AI is focusing on. They found that the image part of the model focused on the actual lesions, while the text part focused on key variables like prostate volume and specific zones, confirming that the model was learning the right things. While the model is highly effective, the authors note that it does take a bit longer to run (about 14.3 ms per sample compared to 7.6 ms for a simpler model) and that the model's internal "thought process" can still be hard to fully interpret, though the attention maps offer a glimpse. Ultimately, this work suggests that treating medical data as a geometric shape rather than a flat list of numbers could be a powerful way to improve cancer detection, especially when data is scarce.

Technical Summary: MFM-Geom for Prostate Cancer Classification

Problem Statement
Prostate cancer (PCa) is a prevalent malignancy where the timely and accurate classification of clinically significant prostate cancer (csPCa) is critical for treatment outcomes. Current diagnostic workflows rely heavily on bi-parametric MRI (bp-MRI) and clinical variables (e.g., PSA, age). However, expert evaluation is time-consuming and subject to inter-reader variability. Existing computer-aided diagnosis (CAD) methods predominantly utilize unimodal, imaging-based deep learning models, often neglecting complementary clinical context. Furthermore, approaches that do combine imaging and clinical data frequently rely on decision-level fusion, which overlooks complex, early-stage correlations between imaging features and clinical variables. Additionally, these models often struggle with data scarcity, limiting their ability to learn robust representations.

Methodology
The authors propose MFM-Geom, a geometric multimodal foundation model designed to jointly learn representations from bp-MRI volumes and clinical reports. The pipeline consists of three main components:

Multimodal Representation Learning:
- Imaging Branch: The model utilizes independent volumetric branches for each bp-MRI sequence (T2W, ADC, DWI). It adapts the BiomedCLIP foundation model's image encoder (a Vision Transformer, ViT) for volumetric inputs using a weight-inflation strategy. Input Volumes of Interest (VoIs) are processed into 3D patches via an Adapter 3D (A3D) layer, generating patch embeddings and a class embedding.
- Text Branch: Clinical variables (Age, PSA, PSAD, Prostate Volume, and zone-specific extensions) are converted into a structured "fill-in-the-blank" text report. This report is tokenized and processed by the BiomedCLIP text encoder (PubMedBERT), yielding text token embeddings and a class embedding.
- Contrastive Alignment: The model is fine-tuned using a contrastive loss (InfoNCE) to align the imaging class embeddings and text class embeddings into a shared latent space, maximizing similarity for corresponding pairs and minimizing it for non-corresponding pairs.
Geometric Classification Head:
- Instead of relying solely on class token embeddings, the method unifies all patch and text token embeddings into a single multimodal sequence.
- A Symmetric Positive Definite (SPD) descriptor ( $S_0$ ) is computed from this sequence to capture pairwise relationships and correlations between imaging and clinical variables.
- To preserve the intrinsic geometric structure of the SPD manifold, the model employs Riemannian deep learning. This involves a BiRe block (comprising a Bilinear Mapping layer and a Rectification layer) that processes the SPD matrices while maintaining their geometric properties.
- The representations are projected into the Euclidean domain via a LogEig layer and passed through a Multi-Layer Perceptron (MLP) for final classification.
Training Strategy:
- The model leverages pretraining on millions of biomedical image-text pairs.
- Fine-tuning utilizes a combined loss function: binary cross-entropy for classification and contrastive loss for alignment.
- Optimization uses an AdamW optimizer for standard layers and a geometry-aware algorithm for the SPD layers.

Key Contributions
The paper outlines four primary contributions:

Adaptation of Biomedical FM: Adapting a multimodal biomedical foundation model for PCa diagnosis, utilizing independent encoders for bp-MRI volumes and clinical reports derived from variables like prostate zone location and extra-prostatic extension.
SPD Descriptor: Introducing an SPD descriptor to capture multimodal correlations from encoder embeddings, offering an alternative to standard class token-based classification.
Geometric Head: Developing a deep Riemannian learning module (Geometric Head) to process the SPD descriptor while preserving the geometric structure of the SPD manifold.
Contrastive Learning: Implementing a contrastive learning loss to align PCa-related imaging and text embeddings within a multimodal latent space.

Results
The model was evaluated on the PI-CAI dataset (internal) and the PROSTATE158 dataset (external).

Data Efficiency: Using only 10% of the training data, MFM-Geom achieved an AUC-PR of 90.67 in the task of distinguishing non-csPCa from intermediate/high-malignancy cases. This outperformed unimodal baselines (UFM-C, UFM-GAP) and models without geometric heads by significant margins (e.g., +8.3% over UFM-C).
Generalization: On the external PROSTATE158 dataset, the geometric head demonstrated robustness, with MFM-Geom and the unimodal geometric variant (UFM-Geom) achieving AUC-PR scores of 90.6 and 90.61, respectively, outperforming ImageNet-pretrained and standard classification baselines.
Performance Metrics: The geometric models consistently showed lower False Positive Rates at 95% True Positive Rate (FPR95) compared to baselines.
Interpretability: Attention maps indicated that the image encoder focused on lesions, while the text encoder highlighted relevant clinical variables (e.g., PSAD, lesion extension), confirming the model learned coherent multimodal representations.

Significance and Claims
The authors claim that MFM-Geom addresses the limitations of existing unimodal and decision-level fusion approaches by creating a unified latent space that respects the geometric structure of multimodal data. The paper asserts that the proposed geometric head outperforms traditional classification heads, particularly under conditions of limited data availability. The results suggest that integrating clinical context with imaging via a geometric foundation model enhances the detection of intermediate-malignancy lesions, which is clinically relevant for decisions regarding active surveillance versus aggressive treatment. The work demonstrates that fine-tuning biomedical foundation models with geometric learning modules can achieve competitive performance with state-of-the-art CNN-based approaches while utilizing significantly less training data.

Technical Summary: MFM-Geom for Prostate Cancer Classification

More like this