Neuro-JEPA: Building and Evaluating Health-System Scale Neuroimaging Foundation Models with Mixture of Experts
Neuro-JEPA is a scalable, multimodal foundation model that leverages a sparse Mixture-of-Experts architecture and latent predictive learning on 1.55 million clinical MRI scans to achieve state-of-the-art, robust generalization across diverse neurological imaging tasks and health systems, often outperforming larger models with significantly less pretraining data.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The human brain is a complex organ, and understanding its health often relies on looking inside it without making a single cut. Doctors use magnetic resonance imaging, or MRI, to create detailed pictures of brain tissue. These pictures are not all the same; they are taken using different settings that highlight different features, much like adjusting the focus on a camera to see either the texture of a leaf or the water inside it. One setting shows the brain's structure clearly, another highlights fluid, and a third suppresses that fluid to make swelling or disease stand out. For a human expert, reading a patient's health requires looking at all these different views together, combining the clues from each to form a complete diagnosis. However, teaching computers to do this has been difficult. Most artificial intelligence systems have been trained to look at only one type of picture at a time, missing the rich connections between the different views. This leaves a gap between how human doctors think and how current computer programs work.
A team of researchers at New York University and Stanford University has built a new kind of artificial intelligence designed to bridge this gap. They created a system called Neuro-JEPA, a computer model trained to understand the brain by looking at all three types of MRI pictures simultaneously. Instead of learning from a small collection of textbook examples, this model studied a massive archive of 1.55 million brain scans from nearly 430,000 patients. These scans came from a large hospital system and covered a wide range of conditions, from strokes and tumors to neurodegenerative diseases like Alzheimer's. The researchers did not just feed the data into a standard computer program; they built a specialized architecture that allows the model to learn from the relationships between the different image types. By doing this, the model learned to create a unified understanding of the brain that works across different hospitals, different scanners, and different diseases.
The results of this work show that the new model is significantly better at understanding brain images than previous systems. When tested on dozens of tasks, such as identifying a stroke, predicting how a patient will recover, or determining if a person has early signs of dementia, the new model consistently outperformed existing tools. It succeeded even when it had to work with very little labeled data, a common problem in medicine where expert doctors are needed to mark up images for training. In some cases, the model improved AUC by up to 0.15 compared to the best previous systems, particularly for conditions like multiple sclerosis and brain inflammation. Crucially, the model achieved these results while being trained on less than one-third of the data used by other leading models, suggesting that the way the model was designed was more important than simply having more data.
The researchers also tested whether the model could handle real-world challenges, such as looking at patients from a different hospital or using a type of scan it had never seen before. The model maintained its high performance across these different settings, proving that it learned general principles about brain anatomy and disease rather than just memorizing specific images. It could even predict outcomes like how long a patient would stay in the hospital or whether a mild memory problem would progress to full dementia, using only the standard MRI scans that are already part of routine care. This capability is vital because it could allow doctors to get more information from scans they already take, potentially catching diseases earlier or planning treatments more effectively without needing expensive or invasive additional tests.
To build this system, the researchers had to solve several specific problems. They found that simply copying existing methods used for other types of data did not work well for brain scans. They discovered that the model needed a special way of hiding parts of the image during training to force it to learn the most important details, rather than just guessing based on simple patterns. They also introduced a mechanism that allowed the model to focus its attention on the brain tissue itself, ignoring the empty space around the skull, which made the learning process much more efficient. Furthermore, they used a design that allowed the model to activate different "experts" for different parts of the image, ensuring that the system could handle the vast variety of brain shapes and pathologies it encountered. This approach allowed the model to scale up effectively, becoming more powerful as it was given more data and more computing power.
The study explicitly challenges the idea that the only way to build a better medical AI is to collect more data. While data is important, the researchers demonstrated that the design of the model and the quality of the training process are decisive factors. They showed that a carefully designed model trained on a curated set of 1.55 million scans could outperform models trained on over five million scans. They also ruled out the idea that these models only work in the specific hospital where they were trained, proving that the knowledge gained is transferable to other institutions and even to different types of medical imaging. The work suggests that the future of medical imaging lies not in creating a separate computer program for every single disease, but in building a single, versatile system that can understand the brain as a whole, adapting to new questions and new patients as they arise.
This research represents a significant step toward making artificial intelligence a reliable partner in neurological care. By showing that a single model can learn from diverse data and apply that knowledge to a wide range of clinical problems, the team has provided a framework that could eventually support doctors in making faster and more accurate decisions. The model's ability to predict disease progression and patient outcomes from routine scans opens the door to earlier interventions and better management of chronic conditions. As the researchers continue to refine these tools and expand their training to include even more diverse populations and imaging types, the potential for these systems to improve patient care grows, moving the field from experimental prototypes toward practical, general-purpose tools for understanding the human brain.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.