💻 computer science

LARGO: Low-Rank Hypernetwork for Handling Missing Modalities

LARGO is a novel low-rank hypernetwork that addresses the challenge of missing modalities in multimodal image analysis by compressing multiple dedicated models into a single network via Canonical Polyadic tensor decomposition of convolutional weights, achieving state-of-the-art performance across diverse medical and non-medical datasets.

Original authors: Niels Vyncke, Pooya Ashtari, Aleksandra Pižurica

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Niels Vyncke, Pooya Ashtari, Aleksandra Pižurica

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to diagnose a brain tumor. Usually, you have a full toolkit: four different types of MRI scans (like taking photos from four different angles or with four different filters). When you have all four, the diagnosis is easy and accurate.

But in the real world, things go wrong. Maybe the machine for one specific scan is broken, the patient can't stay still long enough for a second scan, or the hospital simply doesn't have the equipment. Suddenly, you only have one or two scans. Most computer programs (AI models) trained to diagnose tumors get confused or fail completely when they don't have their full set of tools. They are like a chef who can only cook a perfect meal if they have every ingredient in the kitchen; if they are missing salt, they can't make the soup.

The Old Way: One Chef Per Recipe

Traditionally, to solve this, researchers tried two main approaches:

The "Jack-of-All-Trades" Chef: Train one giant AI to handle any combination of missing scans. The problem is, this chef gets confused. They try to be good at everything but end up being mediocre at specific tasks.
The "Specialist" Team: Train a separate, perfect AI for every single possible combination (e.g., one AI for "T1 only," another for "T2 only," another for "T1 and T2," etc.). With 4 types of scans, there are 15 different combinations. This means you need 15 different AI models. This is like hiring 15 different chefs just to handle different missing ingredients. It takes up a massive amount of storage space and computing power, making it impractical for hospitals.

The New Solution: LARGO (The "Magic Blueprint")

The authors of this paper propose a clever new system called LARGO. Instead of hiring 15 chefs or training one confused chef, they built a "Magic Blueprint Generator."

Here is how it works, using an analogy:

1. The Family of Recipes
Imagine that all 15 different "Specialist" AIs are actually very similar. They all know how to look at a brain, they all know how to spot a tumor, and they all use similar "mathematical recipes" (weights) to do it. The only difference is which scan they are looking at.

2. The Compression Trick (CP Decomposition)
The authors realized that because these 15 models are so similar, they are full of redundancy. It's like having 15 books where 90% of the text is identical, and only the chapter titles change.
LARGO uses a mathematical trick called Canonical Polyadic (CP) decomposition. Think of this as taking those 15 books and compressing them into a single, tiny "Master Blueprint."

This Master Blueprint doesn't store 15 separate models.
Instead, it stores a few "factor matrices" (like a set of Lego bricks and a set of instructions).
One specific set of instructions tells the blueprint: "If the patient has only the T2 scan, assemble the model using these specific bricks."

3. The "On-Demand" Construction
When a patient arrives with missing scans, the LARGO system doesn't need to load a new model. It simply looks at the Master Blueprint, reads the instruction for that specific missing-scenario, and instantly reconstructs the perfect, specialized AI model for that exact situation.

It's like having a 3D printer that can print a specific tool for a specific job in seconds, rather than keeping a warehouse full of every possible tool.

What Did They Prove?

The researchers tested this "Magic Blueprint" on real medical data:

Brain Tumors (BraTS 2018): They tested it on 15 different scenarios of missing MRI scans. LARGO was the best at diagnosing tumors in 14 out of 15 scenarios compared to the best existing methods.
Stroke Detection (ISLES 2022): They tested it on 7 different scenarios of missing stroke scans. LARGO was the best in all 7 scenarios.
Non-Medical Test (avMNIST): To prove it wasn't just a medical trick, they tried it on a mix of images and audio (like recognizing a number from a picture and a sound). It worked well there too, showing the method is flexible.

The Bottom Line

LARGO is a way to have the specialized performance of 15 different AI models without the storage cost of keeping 15 different models. It does this by realizing that these models are cousins, not strangers, and compressing them into a single, efficient "weight space" blueprint that can rebuild the right model instantly whenever it's needed.

The paper claims this results in higher accuracy (better diagnoses) and better efficiency (less computer memory needed) compared to current state-of-the-art methods, without needing complex multi-step training or "hallucinating" fake data to fill in the gaps.

Technical Summary: LARGO (Low-Rank Hypernetwork for Handling Missing Modalities)

Problem Statement

In multimodal image analysis, particularly in medical imaging (e.g., brain tumor segmentation, stroke lesion detection), acquiring all available modalities is often infeasible due to equipment constraints, scan time limits, or patient conditions. This results in incomplete modality sets during inference. Standard deep learning models struggle with such missing inputs without explicit architectural design.

Existing approaches face a fundamental trade-off between specialization and scalability:

Single-model methods (e.g., zero-imputation, shared encoders) force one network to handle all $2^N - 1$ modality combinations, often compromising performance on specific configurations.
Dedicated models (training a separate model for each combination) maximize specialization but scale exponentially in storage and computational cost.
Generative and distillation methods risk hallucinating clinically dangerous features or require complex multi-stage training.

Most current solutions operate in feature space, engineering robust representations to handle missing inputs. This paper proposes operating in weight space to address the redundancy inherent in training separate models for different modality subsets.

Methodology: LARGO

The authors propose LARGO (Low-rAnk hypeRnetwork for missinG mOdalities), a hypernetwork framework that compresses the family of $2^N - 1$ dedicated missing-modality models into a single network.

Core Architecture

Hypernetwork Formulation: Instead of learning $M = 2^N - 1$ separate sets of weights, LARGO constructs a single hypernetwork $H_\phi$ that implicitly contains all dedicated models $\{F_{\theta(m)}\}_{m=1}^M$ .
Tensor Decomposition (CPD): The method exploits the observation that models sharing access to the same modalities learn similar patterns, implying parameter redundancy. LARGO stacks the convolutional weights of all $M$ $M$ dedicated models along a new "model dimension" and applies Canonical Polyadic (CP) tensor decomposition.
- The weight tensor $W \in \mathbb{R}^{M \times C_{in} \times C_{out} \times K}$ is decomposed into factor matrices $A, B, C, D$ .
- Factor matrix $A$ indexes the modality combinations, while $B, C, D$ represent input channels, output channels, and kernel dimensions, respectively.
Low-Rank Reparameterization:
- Inner Layers: Convolutional layers are reparameterized using the CP decomposition. The rank $R$ is selected to maintain a parameter budget equivalent to a single dedicated model (achieving a $1/M$ compression ratio).
- Stem and Head Layers: These layers remain uncompressed to preserve input/output flexibility, though their parameter contribution is negligible compared to inner layers.
Inference: During inference, for a specific modality combination $m$ , the hypernetwork reconstructs the dedicated convolutional weights $W^{(m)}$ by contracting the factor matrices (specifically selecting the $m$ -th row of factor matrix $A$ ). This avoids computing unused parameters for other modality subsets.

Training Procedure

Loss Function: The model is trained using a standard segmentation loss ( $L_{Dice} + L_{CE}$ ).
Strategy: The training employs random modality dropout combined with full-modality guidance. At each iteration, a subset $m$ is sampled to minimize the loss for that specific missing-modality scenario, while a term using the full-modality input ensures the network is guided by complete data throughout training.
Initialization: Factor matrices $B, C, D$ are initialized via Kaiming normal; matrix $A$ is initialized with ones. Weights are trained from scratch rather than derived from pre-trained weights.

Key Contributions

Novel Hypernetwork Formulation: LARGO is the first method to jointly reparameterize the kernel weights of an entire family of $2^N - 1$ dedicated missing-modality models into a single, parameter-efficient representation using CP decomposition.
Weight-Space Operation: Unlike feature-space methods, LARGO operates directly in weight space, compressing the exponential family of models into a single compressed weight manifold without auxiliary losses, teacher networks, or multi-stage training.
State-of-the-Art Performance: The method achieves superior results on medical imaging benchmarks while maintaining a parameter count comparable to a single dedicated model.

Experimental Results

Datasets and Scenarios

BraTS 2018: 4 modalities (FLAIR, T1, T1c, T2), 15 missing-modality scenarios. Task: Brain tumor segmentation (WT, TC, ET).
ISLES 2022: 3 modalities (DWI, ADC, FLAIR), 7 missing-modality scenarios. Task: Stroke lesion segmentation.
avMNIST: 2 modalities (Image, Audio), 3 scenarios. Task: Classification (Proof-of-concept for non-medical, heterogeneous modalities).

Quantitative Performance

BraTS 2018: LARGO ranked first in 15/15 scenarios for Whole Tumor (WT), 11/15 for Tumor Core (TC), and 14/15 for Enhancing Tumor (ET).
- Average Dice improvements over baselines (mmFormer, M³AE, ShaSpec, SimMLM): +0.68% (WT), +2.53% (TC), and significant gains in ET.
- LARGO achieved an average Dice of 87.05% (WT), 79.35% (TC), and 62.83% (ET).
ISLES 2022: LARGO ranked first in all 7 scenarios.
- Average Dice: 62.67%, outperforming the next best (M³AE at 60.14%).
- Notable gains in difficult single-modality cases (e.g., ADC-only: 42.49% vs. 40.03%).
avMNIST: LARGO achieved the highest average accuracy (97.00%), a +0.44% improvement over the closest competitor, demonstrating applicability to heterogeneous non-medical modalities.

Efficiency and Complexity

Parameter Efficiency: LARGO maintains a parameter count similar to a single dedicated model (e.g., 22.6M on BraTS 2018) while implicitly representing all 15 dedicated models. This is significantly more efficient than ShaSpec (187.7M parameters) and comparable to other baselines.
Computational Overhead: Dynamic weight reconstruction introduces a 24.6% increase in GFLOPs and 25.1% increase in peak VRAM compared to a single nnU-Net backbone. The authors note this is acceptable as missing-modality models typically introduce additional operations anyway.
Scalability: The method is practical for $N \lesssim 10$ modalities, as the uncompressed stem/head layers (which grow exponentially) remain a negligible fraction of the total parameters.

Significance and Claims

The paper claims that LARGO bridges the gap between the specialization of dedicated models and the scalability of single-model approaches. By exploiting structural similarities in convolutional kernels across modality-specific models via tensor decomposition, LARGO eliminates the need for exponential storage and compute scaling.

The authors position LARGO as a principled hypernetwork framework that:

Avoids the "hallucination" risks of generative imputation.
Avoids the performance compromises of shared-encoder architectures.
Achieves state-of-the-art results without complex training pipelines (e.g., knowledge distillation).

The proof-of-concept on avMNIST suggests the method's potential extends beyond medical imaging to any task involving heterogeneous modalities with missing data. The authors conclude that future work will explore more expressive, nonlinear hypernetwork formulations and extensions to transformer-based architectures.