💻 computer science

Label Adaptive Gating for Learning Per Label Modality Importance in Multimodal Multi Label Clinical Diagnosis

This paper proposes Label Adaptive Gating (LAG), a novel fusion mechanism that learns distinct modality importance for each clinical label via cross-attention, outperforming existing single-output fusion baselines across multiple multi-label clinical datasets while providing clinically sensible modality attributions.

Original authors: Van-Thien Tran, Thanh-Nghi Doan, Song-Toan Tran

Published 2026-09-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Van-Thien Tran, Thanh-Nghi Doan, Song-Toan Tran

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In a modern hospital, a patient's story is told through many different voices. There are the structured numbers from blood tests and vital signs, the unstructured paragraphs of a doctor's handwritten notes, and the visual snapshots of X-rays or heart rhythms. For decades, computer scientists have tried to build artificial intelligence that can listen to all these voices at once to make a diagnosis. The goal is simple: combine these different types of information so the computer understands the full picture better than it could from any single source. However, real-world medicine is rarely a single question with a single answer. A patient in an intensive care unit often suffers from several conditions at the same time—perhaps heart failure, an infection, and kidney trouble all occurring together. These conditions are related, but they are not identical. The information needed to spot heart failure might be hidden in a blood test, while the signs of pneumonia are clearly visible on a chest X-ray.

For a long time, the tools used to combine these medical data sources were designed for simpler problems. They would look at a patient's entire record and decide, once and for all, which type of information was most important. It was as if the computer decided that blood tests were the most important thing for every single patient, or that X-rays were the key to every diagnosis. This approach forces a compromise. If a specific disease relies heavily on a lab result, but the computer has decided that X-rays are generally more important, it might miss the subtle signs of that disease. The challenge for researchers has been to build a system that can be flexible enough to say, "For this specific condition, I need to listen closely to the blood work," while simultaneously saying, "For that other condition, I need to focus entirely on the image."

A team of researchers has proposed a new way to solve this problem, called label-adaptive gating. Instead of making one global decision about what information matters, their system learns a separate strategy for every single disease it is trying to predict. Imagine a room full of experts, each waiting to diagnose a different condition. When the patient with heart failure arrives, the heart expert steps forward and asks the team to focus on the heart monitor and blood pressure. When the patient with a lung infection arrives, the lung expert steps forward and asks the team to look at the X-ray. In this new system, the computer does exactly that. It maintains a unique "question" for every disease it is tracking. When it tries to predict heart failure, it asks the data sources, "Which of you has the best answer for heart failure?" and listens to the response. When it switches to predicting kidney trouble, it asks a different question and listens to a different set of answers. This allows the system to weigh the importance of an X-ray versus a blood test differently for every single diagnosis, rather than forcing one rule to fit all.

The researchers tested this idea against five other common ways of combining medical data. They used three large, real-world datasets containing records from thousands of patients, covering everything from chest X-rays and electrocardiograms to detailed clinical notes and lab results. The datasets were different in size and the types of diseases they covered, ranging from six specific lung conditions to eight broad categories of organ-system failures. To ensure a fair test, every method was given the same amount of computing power and trained using the exact same rules. They ran the experiments multiple times to make sure the results were consistent and not just a lucky accident.

The results showed that the new, flexible approach was the most reliable overall. It performed best on two of the three datasets and came in a very close second on the third. In the third dataset, a simpler method that adjusted its focus based on the patient rather than the disease performed slightly better, but the difference was so small that it could be considered a tie. Crucially, the new method was the only one that could explain its own reasoning. Because it learns a specific strategy for each disease, researchers could look at the system and see exactly which data source it was using for each condition. For example, when predicting kidney issues, the system correctly learned to pay the most attention to laboratory blood tests. When predicting heart problems, it focused on clinical notes and heart rhythms. This ability to show its work is vital in medicine, where doctors need to trust why a computer made a suggestion.

However, the study also revealed that this flexibility comes with a requirement: the system needs enough data to learn these specific strategies. On one of the smaller datasets, where there were many diseases to predict but fewer patients to learn from, the system's ability to distinguish between data sources faded. In that case, it defaulted to a simple average, treating all information sources as equally important. This suggests that while the method is powerful, it is not magic; it needs sufficient evidence to learn the nuances of each disease. The researchers also found that a more complex method, which tried to let the different data sources talk to each other in a deep, intricate way, performed the worst of all. This indicates that simply adding more complexity does not always lead to better results, especially when the goal is to predict multiple conditions at once.

The study concludes that for the complex task of diagnosing multiple conditions simultaneously, the best approach is to let the computer decide what matters for each specific disease, rather than forcing it to make a single decision for the whole patient. The new method offers a balance of high performance and clear reasoning, making it a strong candidate for future medical AI tools. While it is not a perfect solution for every situation, it represents a significant step forward in teaching computers to understand the messy, multi-faceted reality of human illness. By treating each diagnosis as a unique question with its own answer key, the system moves closer to the way human doctors actually think, shifting their focus and attention depending on the specific problem at hand.

Problem Statement

Real-world clinical diagnosis is inherently a multi-label problem, where a single patient may present with multiple, partially correlated findings (e.g., pulmonary edema and cardiomegaly) simultaneously. Existing multimodal fusion mechanisms in medical machine learning are predominantly designed for single-output tasks (e.g., predicting a single binary outcome like in-hospital mortality).

Current fusion strategies determine modality importance either:

Globally: Once per dataset.
Locally: Once per sample.

In both cases, a single weighting policy must serve all labels at once. This creates a failure mode where the model is forced to compromise; for instance, if a chest radiograph is critical for one finding but laboratory values are critical for another, a sample-level gate cannot optimize for both simultaneously. The authors argue that modality usefulness is a property of the individual label, not the dataset or the patient, yet current architectures lack the granularity to represent this.

Methodology: Label-Adaptive Gating (LAG)

The authors propose Label-Adaptive Gating (LAG), a fusion mechanism that resolves modality importance at the per-label level rather than the per-sample or per-dataset level.

Architecture:
- The model takes $M$ modality embeddings (e.g., ECG, CXR, Labs) of width $d$ .
- Instead of a single query vector, LAG introduces $L$ learned query vectors, where $L$ is the number of labels.
- For each label $\ell$ , a specific query vector $q_\ell$ performs scaled dot-product attention over the stacked modality embeddings (acting as keys and values).
- This produces a label-specific context vector $c_\ell$ and an attention distribution $a_\ell$ over the modalities.
- Each context vector feeds into a classifier head to produce the logit for that specific label.
Key Design Features:
- No Anchor Modality: Unlike cross-attention baselines that require one modality to be nominated as an "anchor" (query source), LAG derives queries directly from the label set, removing a dataset-specific hyperparameter.
- Parameter Efficiency: The mechanism adds only $L \times d$ parameters (the query vectors) on top of a standard cross-attention block. It is significantly cheaper than transformer encoders or hypernetworks.
- End-to-End Learning: The modality importance for each label is learned directly during training without post-hoc attribution methods.

Experimental Setup

The study validates LAG against five representative baselines and unimodal models across three independent, genuinely multi-label clinical datasets:

Symile-MIMIC: 6 co-occurring CheXpert findings (CXR, ECG, Labs).
HAIM-MIMIC-MM: 14 CheXpert findings (CXR, Text, Chart/Lab events).
eICU-CRD: 8 organ-system diagnosis categories (Chart, Lab, Clinical notes).

Protocol:

Controlled Variables: All models used identical encoder capacities, hyperparameters, and training protocols.
Robustness: Experiments were run with four different random seeds per mechanism per dataset, with patient-level re-splitting to prevent data leakage.
Metrics: Evaluation focused on macro-AUROC (threshold-free) with per-label decision thresholds fitted only on the validation set.

Baselines Compared:

Feature Concatenation (Concat)
Learned Scalar Gating (Gate)
Confidence-Gated Cross-Attention (Attention)
Confidence-Gated Transformer Encoder (Transformer)
Hypernetwork-Generated Classifier Weights (Hyperfusion)

Key Results

Overall Performance: LAG achieved the best average rank (1.33 out of 6) across the three datasets and the smallest average gap to the best mechanism on each dataset ($-0.0005$ macro-AUROC).
Dataset-Specific Outcomes:
- Symile-MIMIC & HAIM: LAG tied or won first place.
- eICU: LAG placed a close second to the Scalar Gating baseline. The margin was small but consistent across all four seeds.
Transformer Failure: The Confidence-Gated Transformer Encoder was the worst-performing mechanism on all three datasets, ranking last at every seed on the two larger datasets (Symile-MIMIC and eICU). This suggests that increased capacity does not guarantee better performance in multi-label settings with limited data.
Fusion vs. Unimodal:
- On HAIM and eICU, fusion provided significant gains over the best single modality (approx. 0.044 and 0.021 AUROC, respectively).
- On Symile-MIMIC, the best unimodal model (CXR) nearly matched the best fusion model, indicating that for this specific dataset, the added complexity of fusion was not justified by performance gains.

Interpretability Analysis

The authors inspected the learned attention matrices to verify if LAG learned clinically sensible modality dependencies:

eICU: The mechanism showed clinically sensible differentiation. For example, the "Renal" and "Hematology" labels correctly assigned higher attention weights to the "Lab" modality, while other labels relied more on clinical notes or charts.
Symile-MIMIC: The model correctly prioritized CXR for all findings, with attention weights scaling sensibly with the visual distinctiveness of the finding (e.g., higher weight for "Edema" than "No Finding").
HAIM: The attention weights collapsed to a uniform average across all 14 labels. The authors attribute this to data scarcity; with only ~1,034 patients spread across 14 labels, there was insufficient signal for the 14 query vectors to learn distinct distributions. This highlights that interpretability is data-hungry and not guaranteed by architecture alone.

Significance and Claims

The paper makes four primary contributions:

Architectural Innovation: It formulates LAG, the first mechanism to explicitly learn modality importance per label rather than per sample or dataset, without requiring an anchor modality.
Rigorous Comparison: It provides a matched experimental comparison of six fusion families under identical conditions, isolating the effect of the fusion mechanism from encoder capacity or training protocols.
Reliability: It establishes LAG as the most reliable default mechanism for multi-label clinical diagnosis among those tested, offering the best average rank and smallest performance gap to the optimum.
Honest Interpretability: It demonstrates that LAG can provide direct, clinically sensible per-label attributions when data is sufficient, while explicitly reporting when these attributions collapse due to data scarcity.

Recommendations:

LAG is recommended as the default fusion mechanism for multi-label clinical tasks due to its robustness and lack of anchor-modality hyperparameters.
Scalar Gating is a legitimate alternative, particularly on large datasets where it may slightly outperform LAG.
Transformers should be avoided as a default for multi-label clinical fusion, as they consistently underperformed in this study.
Fusion Validation: Before applying complex fusion, practitioners should verify that fusion actually outperforms the strongest single-modality baseline on their specific dataset, as this was not true for all datasets in the study.

The authors conclude that while LAG resolves the label-level granularity issue, future work should explore mechanisms that condition on both the label and the sample simultaneously.