CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

This paper proposes CFCML, a novel coarse-to-fine crossmodal learning framework that bridges the modality gap between medical images and tabular data through multi-granularity feature exploration and a hierarchical anchor-based relationship mining strategy, achieving state-of-the-art disease diagnosis performance on MEN and Derm7pt datasets.

The Gist

Imagine you are a detective trying to solve a complex medical mystery. To catch the culprit (the disease), you have two types of clues:

1. The Visual Clues: High-tech medical images (like MRI scans) that show where the problem is and what it looks like.
2. The Written Clues: A patient's chart (tabular data) with facts like age, symptoms, and medical history.

The problem is, these two types of clues speak completely different languages. The images are huge, detailed, and visual, while the charts are short, text-based, and factual. Trying to combine them is like trying to mix oil and water; they don't naturally blend, and often, the computer gets confused, missing important details.

This paper introduces a new detective tool called CFCML (Coarse-to-Fine Crossmodal Learning). Think of it as a super-smart translator and organizer that helps the computer understand both clues perfectly. Here is how it works, broken down into simple steps:

1. The "Coarse" Stage: The Rough Draft

The Problem: If you try to compare a whole MRI scan (which has thousands of tiny pixels) with a short sentence about a patient's age, it's an unfair fight. The image is too loud, and the text is too quiet. Also, early parts of an image show general shapes, while deep parts show specific details.

The Solution (MG-CIE Module):
Imagine you are organizing a library.

• The "Coarse" step: Instead of trying to read every single book (pixel) at once, the system creates a "summary" of the image at different levels of detail. It looks at the big picture first, then zooms in.
• The Translation: It takes the short text clues and expands them slightly, and it condenses the massive image data. It forces them to be the same "size" so they can talk to each other.
• The Result: It creates a rough draft where the image and the text start to understand each other. It's like having a translator who says, "Okay, this picture of a tumor matches with this sentence about 'pain in the head'."

2. The "Fine" Stage: The Expert Review

The Problem: Even after the rough draft, the computer might still be distracted by irrelevant details. It might focus on the background of the photo instead of the tumor, or get confused by a patient's name instead of their age. It needs to focus only on what matters for the specific disease.

The Solution (CCRM Strategy):
This is where the system gets really smart. It uses a technique called "Class-Aware Relationship Mining."

• The "Class" Concept: Think of "classes" as different types of suspects (e.g., "Benign Tumor" vs. "Cancer").
• The "Anchors": The system creates three types of "anchors" (like magnets) to pull the right clues together:
  1. Sample Anchors: It looks at individual patients and asks, "Does this patient's image and chart look like other patients with the same disease?"
  2. Unimodal Anchors: It creates a "perfect average" image for a specific disease and a "perfect average" chart for that same disease.
  3. Crossmodal Anchors: It creates a "perfect average" combination of both image and chart for that disease.

The Magic: The system uses these anchors to pull all the "Cancer" clues together into one tight group and push all the "Benign" clues far away. It's like a bouncer at a club who only lets people with the same VIP pass (disease type) into the same room, regardless of whether they arrived by car (image) or bus (text). This forces the computer to ignore the noise and focus only on the features that actually define the disease.

Why is this better than before?

Previous methods were like trying to glue the image and text together with duct tape. They often missed the small, important details in the images or ignored the specific context of the text.

• Old Way: "Here is a picture, here is a list of numbers. Let's just smash them together."
• New Way (CFCML): "Let's first translate them so they speak the same language (Coarse), and then let's organize them by who they belong to, grouping all the 'Cancer' cases together and separating them from the 'Healthy' cases (Fine)."

The Results

The researchers tested this on two real-world medical datasets:

1. Brain Tumors (MEN): Diagnosing different grades of meningiomas.
2. Skin Lesions (Derm7pt): Distinguishing between moles and melanoma.

The new method beat all the previous "State-of-the-Art" (SOTA) methods. It was more accurate, better at spotting the tricky cases, and could clearly see the disease areas in the images (proven by heatmaps showing exactly where the computer was looking).

In a Nutshell

CFCML is a two-step process that teaches a computer to stop treating medical images and patient charts as strangers. First, it translates them into a common language. Second, it organizes them into tight-knit groups based on the specific disease, ensuring the computer learns exactly what to look for to make a life-saving diagnosis.

Technical Summary

1. Problem Statement

In clinical practice, accurate disease diagnosis relies on integrating multimodal medical images (e.g., MRI, dermatoscopy) and tabular clinical data (e.g., age, medical history, lesion details). However, existing Crossmodal Learning (CML) methods face two critical challenges:

• The Modality Gap: There is a significant semantic and structural gap between image data (high-dimensional, spatial) and tabular data (low-dimensional, attribute-based). Most existing methods focus only on high-level encoder outputs, neglecting local information and failing to fully align features across different granularities.
• Lack of Class-Awareness: Many methods overlook the extraction of task-relevant, class-specific information during the fusion process. This leads to the retention of redundant, class-irrelevant features, which hinders diagnostic accuracy.
• Token Disparity: Images typically contain thousands of tokens (spatial patches), while tabular data has very few attributes. Direct fusion often leads to information overload or imbalance.

2. Methodology: CFCML Framework

The authors propose CFCML, a Coarse-to-Fine Crossmodal Learning framework designed to progressively reduce the modality gap through two main stages:

A. Coarse Stage: Multi-Granularity Crossmodal Information Enhancement (MG-CIE)

This stage aims to preliminarily reduce the modality gap by exploring relationships between image features at various encoder depths and tabular data.

• Feature Extraction:
  • Images: Multi-granularity features are extracted from four stages of an image encoder (using nnMamba for 3D MRI or Swin Transformer for 2D images).
  • Tabular Data: Instead of simple one-hot encoding, the authors use a pretrained CLIP text encoder to convert clinical attributes into semantic text embeddings (e.g., "The age of the patient is 45").
• Token Mapping: To address the token count disparity (e.g., 49k image tokens vs. 5 tabular tokens), distinct adapters map image and tabular tokens to predefined, comparable numbers ($n_x$ and $n_t$) using Conv1d operations.
• Cross-Modal Interaction: A Crossmodal Information Enhancement (CIE) module utilizes multi-head cross-attention. For each granularity level, one modality acts as the "Query" (primary) and the other as "Key/Value" (auxiliary) to generate supplementary features.
• Fusion: Enhanced features from all four granularity levels are concatenated and fused to create robust, enhanced unimodal representations for both images and tabular data.

B. Fine Stage: Class-Aware Crossmodal Relationship Mining (CCRM)

This stage further narrows the modality gap and extracts discriminative features by leveraging class labels as a bridge.

• Prototype Construction: The framework calculates two types of prototypes based on class labels:
  • Unimodal Prototypes: Average features for each class within a specific modality.
  • Crossmodal Prototypes: Average features for each class across all modalities.
• Hierarchical Anchor-Based Contrastive Learning (CL): Three specific contrastive strategies are employed to cluster same-class samples and separate different classes:
  1. Sample-Anchor: Treats individual samples as anchors, pulling them closer to same-class prototypes and pushing them away from different-class samples.
  2. Unimodal-Prototype Anchor: Aligns unimodal prototypes with crossmodal prototypes of the same class.
  3. Crossmodal-Prototype Anchor: Uses crossmodal prototypes as anchors to align all unimodal prototypes of the same class.
• Final Prediction: The class-aware features are concatenated and passed through an MLP classifier. The total loss combines classification loss (Cross-Entropy) with the three contrastive losses.

3. Key Contributions

1. Coarse-to-Fine Framework: A novel architecture that progressively reduces the modality gap by first aligning multi-granularity features (Coarse) and then refining them with class-aware contrastive learning (Fine).
2. MG-CIE Module: Introduces a mechanism to explore inter-modal relationships across multiple encoder stages, effectively handling the token disparity between images and tabular data via adaptive token mapping.
3. CCRM Strategy: Proposes a hierarchical anchor-based contrastive learning approach using unimodal and crossmodal prototypes to explicitly extract class-aware information and dissolve modality boundaries.
4. Superior Performance: Demonstrates state-of-the-art results on two distinct medical datasets, outperforming existing uncertainty-based, feature-disentanglement, and attention-based methods.

4. Experimental Results

The method was evaluated on two datasets:

• MEN Dataset: A private dataset of 796 patients with meningiomas (3 grades) using 3 MRI sequences and 6 clinical attributes.
• Derm7pt Dataset: A public dataset of 827 skin lesion cases (Melanoma vs. Nevus) using clinical/dermatoscopy images and 5 clinical attributes.

Key Findings:

• MEN Dataset: CFCML achieved an AUC of 98.57%, outperforming the best SOTA method (GLoMo) by 1.53%. It also showed significant gains in Accuracy (95.61%) and Macro-F1 (91.13%).
• Derm7pt Dataset: CFCML achieved an AUC of 90.52%, improving upon the best SOTA by 0.91%. It also led in Sensitivity, Specificity, and Balanced Accuracy.
• Ablation Studies:
  • Removing MG-CIE caused a significant drop in performance, confirming the value of multi-granularity exploration.
  • Removing CCRM reduced class separation capabilities, proving the necessity of prototype-based contrastive learning.
  • Using CLIP for tabular embedding outperformed traditional MLP embeddings by ~1.5%.
  • Conv1d for token mapping was more efficient and effective than the Perceiver approach.

5. Significance

• Clinical Impact: The framework provides a more accurate tool for disease diagnosis by effectively fusing visual and non-visual clinical data, which is crucial for complex conditions like brain tumors and skin cancer.
• Methodological Advancement: It addresses the specific challenge of "Image-Tabular" fusion, which is harder than "Image-Image" fusion due to the vast structural differences. The proposed "Coarse-to-Fine" approach offers a new paradigm for handling heterogeneous medical data.
• Interpretability: Visualization via Grad-CAM and MDA (Manifold Discovery) confirmed that the model focuses on relevant lesion areas and creates tighter, more separable feature clusters for different disease classes compared to existing methods.

Limitations & Future Work:
The current method requires manual tuning of token mapping numbers ($n_x, n_t$) and incurs slightly higher computational complexity due to multi-granularity processing. Future work aims to automate token mapping and reduce parameter counts.