MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalities

In the world of modern medicine, the most accurate diagnoses often come from combining different types of medical images. A doctor might look at a standard X-ray, a magnetic resonance scan, and a specialized tissue scan all at once, letting the strengths of one image compensate for the weaknesses of another. This practice, known as multimodal fusion, allows for a much clearer picture of what is happening inside the body, particularly when detecting complex conditions like brain tumors. However, a significant barrier stands in the way of using this powerful combination on a global scale: patient privacy. Hospitals cannot simply send their patient scans to a central server to be combined, as strict laws protect individual medical data from leaving the local institution. Furthermore, even when hospitals agree to collaborate, they rarely have the exact same set of images for every patient. One hospital might have a full suite of scans, while another, perhaps in a different country or with older equipment, might only have a few. This inconsistency creates a puzzle where the tools designed to combine the images often fail because they expect every piece of the puzzle to be present.

Researchers have long sought a way to train artificial intelligence to learn from these scattered, incomplete datasets without ever moving the raw data from the hospital where it was collected. This approach, called federated learning, allows hospitals to share only the lessons their computers learn, not the patients themselves. Yet, a new problem has emerged: how do you teach a computer to recognize a tumor when it is only seeing a fraction of the available image types, and when the only labels it has are as simple as a "yes" or "no" on a report, rather than a detailed drawing of the tumor's shape? The challenge is to build a system that can learn from these sparse, private, and incomplete clues without losing accuracy or compromising privacy.

A team of researchers at the Indian Institute of Science has developed a new framework called MOSAIC to solve this exact problem. Their work focuses on training an artificial intelligence to segment, or outline, brain tumors using only image-level labels—simple indicators of whether a tumor is present or absent—while navigating a landscape where different hospitals possess different, incomplete sets of MRI scans. The researchers found that by creating a system that does not care about the specific identity of the missing scans, but instead focuses on the underlying patterns of the data, they could achieve high accuracy without needing the detailed, time-consuming annotations that usually make such training possible.

The core of their solution lies in how the system handles the missing pieces. In a typical scenario, if a hospital lacks a specific type of MRI scan, the computer might try to guess what that missing scan would look like or simply ignore it, often leading to errors. The MOSAIC system takes a different approach. It uses a specialized module that acts as a translator, converting whatever scans a hospital has available into a shared, common language. This translation happens locally at each hospital, so no raw images ever leave the site. The system only needs to know how many types of scans are present, not which specific ones they are, allowing a hospital with a unique combination of scans to join the network without any special setup or retraining.

To ensure that all the hospitals are learning from the same "truth" despite their different data, the researchers introduced a method of aligning the data using frequency patterns. Imagine that every medical image has a unique musical signature made up of different tones and rhythms. While the specific notes might change depending on which scans are available, the overall rhythm and structure of the music remain consistent. The MOSAIC system listens to these rhythmic patterns, or spectral signatures, rather than the raw visual details. By comparing these compact, non-reversible frequency patterns across all hospitals, the system can align their understanding of what a tumor looks like without ever exposing the actual patient images. This method, which the authors call spectral prototype alignment, effectively bridges the gap between hospitals that have different sets of tools, ensuring that the collective intelligence grows stronger even when individual pieces are missing.

The training process is divided into two distinct stages to handle the inherent noise of learning from simple "yes" or "no" labels. In the first stage, the system generates rough guesses, or pseudo-labels, about where the tumor might be located based on the available scans. Because the data is incomplete, these initial guesses can be messy or inaccurate. The second stage acts as a dedicated refinement network, a second layer of intelligence that takes these rough guesses and cleans them up. It uses a combination of techniques, including a "teacher" model that provides stable guidance and a voting system that weighs different clues to decide the final shape of the tumor. This two-step process allows the system to break through the usual accuracy limits of weak supervision, turning coarse, noisy hints into precise, detailed outlines of the tumor.

The researchers tested their framework on three different datasets involving brain tumors from multiple institutions, including the FeTS2022 challenge, the BraTS-MEN dataset for meningiomas, and the BraTS-SSA dataset from Sub-Saharan Africa. These tests involved a variety of scenarios where hospitals had different combinations of MRI scans, ranging from full sets to single scans. The results were striking. Using only image-level labels and dealing with missing scans, the MOSAIC system achieved a Dice score of 0.84 on the FeTS2022 dataset, a metric that measures how well the predicted tumor outline matches the actual tumor. This performance surpassed all other methods that relied on more detailed supervision, such as bounding boxes or point annotations, and even outperformed a centralized model that had access to all the data combined. In fact, the system performed so well that it approached the accuracy of fully supervised methods, which require experts to manually draw every tumor boundary, despite using only the simplest form of label.

One of the most significant findings was the system's ability to adapt to new, unseen hospitals. Because the translation module depends only on the number of available scans rather than their specific type, a new hospital could join the network and start contributing immediately. The researchers demonstrated that a new institution could join an already-trained network and achieve performance within a very small margin of the existing members without needing to retrain the entire system from scratch. This capability is crucial for real-world deployment, where medical protocols and equipment availability vary widely and change over time. The system also showed a remarkable ability to quantify its own uncertainty, correctly identifying when it was less confident about the tumor boundaries due to missing data, a feature that is essential for building trust in clinical settings.

The study also explored what happens when different alignment strategies are used. The researchers compared their frequency-based approach against other methods that tried to align data by matching simple averages or standard deviations of the images. They found that these simpler methods failed to capture the complex, texture-based signatures that distinguish different types of medical scans. Only the frequency-based approach, which looked at the underlying rhythmic patterns of the data, was able to successfully align the diverse datasets. This suggests that for tasks involving incomplete and heterogeneous data, looking at the "music" of the data rather than just its "volume" is a far more effective strategy.

The implications of this work extend beyond just brain tumors. The ability to train powerful medical AI models without moving patient data, while accommodating the reality that different hospitals have different tools and levels of detail, addresses a fundamental bottleneck in global healthcare. By proving that high accuracy is possible with simple labels and incomplete data, the researchers have opened a path for more hospitals to participate in collaborative AI research. The system does not require expensive, time-consuming manual annotations, nor does it demand that every hospital have the same expensive equipment. Instead, it creates a flexible, privacy-preserving network where the collective knowledge of many imperfect sources can be synthesized into a highly accurate, trustworthy model.

In the end, the MOSAIC framework demonstrates that the limitations of privacy and incomplete data do not have to be barriers to progress. By designing a system that respects the constraints of the real world—where data is siloed, labels are scarce, and equipment varies—the researchers have shown that it is possible to build a collaborative intelligence that is not only secure but also superior to traditional, centralized approaches. The work suggests that the future of medical AI lies not in gathering all the data in one place, but in creating smart, adaptable systems that can learn from the diverse and fragmented reality of global healthcare.