Challenges and proposed solutions in modeling multimodal medical data: A systematic review

Imagine you are trying to solve a massive, complex puzzle, but instead of just one box of pieces, you have several boxes. One box contains colorful pictures (like medical scans), another has long lists of numbers (like genetic codes), a third holds daily diary entries (like patient records), and a fourth has data from a smartwatch (like heart rate). In the world of medical science, this is called multimodal data. "Multi" means many, and "modal" refers to the different types or "modes" of information. Doctors have always known that looking at just one type of information—like only the X-ray or only the blood test—is like trying to understand a whole movie by looking at a single frame. To get the full story of a patient's health, you need to combine all these different pieces.

However, mixing these different puzzle boxes together is incredibly tricky. The pieces don't fit together naturally; they are different shapes, sizes, and colors. Some boxes are missing pieces entirely, some have thousands of tiny pieces while others have only a few, and sometimes the pieces are so small or the picture is so blurry that it's hard to tell what's going on. This is the problem that scientists in the field of Artificial Intelligence (AI) are trying to solve. They are building computer programs that can learn to put these different puzzle boxes together to help doctors make better decisions. But before they can build the perfect robot puzzle-solver, they have to figure out how to handle the messiness of the real world.

This paper is a giant "report card" for the last decade of trying to solve this puzzle. The authors, a team of researchers from universities in Germany, didn't build a new robot themselves. Instead, they acted like detectives, hunting down 69 different studies published between 2011 and late 2023. They read every single one to find out what problems these scientists ran into when trying to mix medical data and, more importantly, what tricks they used to fix those problems. They wanted to know: What are the biggest headaches in this field, and are we getting any better at solving them?

The Five Big Headaches

After sorting through hundreds of papers, the team found that almost everyone was fighting the same five battles.

1. The Missing Piece Problem
Imagine you are trying to solve the puzzle, but for some patients, the "X-ray" box is completely empty. Maybe the machine broke, the patient couldn't afford the test, or they dropped out of the study. In the real world, this happens all the time. The paper found that 15 out of the 69 studies (about 22%) had to deal with this.

• The Fix: Instead of just guessing or leaving the spot blank, smart computer programs are learning to "hallucinate" the missing piece. They use techniques like Generative Adversarial Networks (GANs), which are like two AI artists playing a game: one tries to draw a fake X-ray that looks real, and the other tries to spot the fake. Through this practice, the AI learns to create a realistic guess for the missing data. Other methods use knowledge distillation, where a "teacher" AI that knows everything about one type of data teaches a "student" AI how to guess the rest, even if it hasn't seen all the pieces.

2. The Tiny Crowd Problem
Sometimes, you have a puzzle, but you only have a handful of people to help you solve it. In medical research, this is called a small dataset. The paper found this was the most common challenge, appearing in 17 studies (25%). If you try to teach a computer to recognize a rare disease with only five examples, it will likely just memorize those five pictures and fail when it sees a new one.

• The Fix: To get around this, researchers are using Transfer Learning. This is like taking a student who has already studied for a million hours on general pictures (like cats and cars) and giving them a quick crash course on medical images. They also use Data Augmentation, where the computer takes the few pictures it has and artificially spins, flips, or brightens them to create "new" practice pictures, making the tiny crowd feel like a huge stadium.

3. The Size Mismatch Problem
Imagine trying to mix a bucket of sand with a single grain of rice. The bucket of sand represents Imaging data (which can have millions of pixels), and the grain of rice represents Genomic data (which might only have a few thousand numbers). If you just dump them together, the computer will ignore the rice and only look at the sand. This is called imbalance in dimensionality, and 7 studies tackled it.

• The Fix: The solution is to shrink the bucket of sand down to the size of the rice, or to give the rice a special "weight" so the computer pays attention to it. Researchers use weighted loss functions, which are like a referee shouting, "Hey, don't ignore the small data! It's important!" This forces the computer to listen to both the giant images and the tiny genetic lists equally.

4. The "Black Box" Problem
Even when the computer solves the puzzle perfectly, it often refuses to explain how it did it. In medicine, this is dangerous. If a computer says, "This patient has cancer," but can't tell the doctor why, the doctor can't trust it. This lack of interpretability was a challenge in 14 studies.

• The Fix: Scientists are building "flashlights" for the AI. They use tools like Grad-CAM and SHAP, which highlight exactly which parts of an X-ray or which specific genes the computer looked at to make its decision. It's like the computer drawing a circle around the suspicious spot on the X-ray and saying, "I saw this, so I made my choice."

5. The Mixing Strategy Problem
Finally, there is the question of how to mix the data. Should you mix the raw pieces together first (Early Fusion), mix them after they are partially solved (Intermediate Fusion), or solve them separately and combine the answers at the end (Late Fusion)? The paper found that 39 out of 69 studies (more than half) used Intermediate Fusion.

• The Fix: This approach is like having a team of specialists. The X-ray expert and the blood test expert each study their own clues first, then they meet in the middle to compare notes before making a final decision. This seems to work better than mixing everything at the start or waiting until the very end.

What the Paper Says (and Doesn't Say)

The authors are careful not to declare a total victory. They found that while there are many clever tricks, no single method is the magic bullet that works for every disease or every type of data. The best solution depends entirely on the specific puzzle you are trying to solve.

They also explicitly ruled out some ideas. For instance, they noted that simply pasting all the data together without thinking about the differences (a method called "Early Fusion") often fails when the data types are too different, like mixing a video with a spreadsheet. They also pointed out that many studies rely on public datasets (like the Alzheimer's Disease Neuroimaging Initiative, used in 18 studies, or the Cancer Genome Atlas, used in 8). While these are great for testing, the paper suggests that relying too much on them might mean we aren't testing our ideas on the messy, real-world data found in actual hospitals.

Furthermore, the paper highlights a worrying gap: only 19 out of the 69 studies (27.5%) shared both their data and their computer code. This means that for most of these clever solutions, other scientists can't easily check the work or build upon it. The authors suggest that the field needs to be more open, like sharing a recipe rather than just the final cake.

The Bottom Line

This review suggests that we are making progress, but the road is still bumpy. The most frequent problems are missing data and not having enough examples to learn from. The most promising tools right now involve teaching computers to guess missing pieces, borrowing knowledge from other tasks, and using "flashlights" to see how they think.

The paper concludes that the future isn't about finding one perfect algorithm. Instead, it's about building flexible systems that can handle missing pieces, work with small crowds, and explain their reasoning. As the authors put it, the goal is to move from just "making models" to making models that are robust, trustworthy, and ready to help real doctors in real hospitals. Until then, the puzzle remains a work in progress, but we are definitely getting better at finding the pieces.