Genome-scale molecular profiling from routine histopathology with a multimodal pathology foundation model
The paper introduces Fuji, a multimodal pathology foundation model trained on over 60,000 whole-slide images paired with genomic data that successfully infers genome-scale molecular states, mutational processes, and clinical outcomes directly from routine histology, thereby bridging the gap between tissue morphology and underlying molecular mechanisms.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine you are looking at a city from a high tower. From this height, you can see the shape of the streets, the density of the buildings, and how the neighborhoods are organized. You can tell a bustling downtown from a quiet suburb just by looking at the layout. Now, imagine that every building in this city is actually a living cell, and the streets are the tissues of the human body. For over a century, doctors have been climbing this tower, using microscopes to look at these "cities" (tissues) to diagnose diseases like cancer. They call this histopathology. It's like reading a map of the city to understand what's happening inside.
But here is the twist: the city's layout isn't just random. It is built by the invisible rules of the "blueprints" inside every building—the DNA. When the DNA gets damaged or mutated, it changes how the buildings are constructed, how they talk to each other, and how the neighborhood grows. For a long time, scientists thought you had to go inside every single building to read those blueprints (using expensive DNA sequencing) to know what was wrong. But what if the shape of the city itself told you everything about the blueprints? What if you could look at the map and know exactly which genes were broken, without ever stepping inside a single building? This is the big question this paper tackles: Can the visual "shape" of a tumor tell us its genetic secrets?
The Paper: "Fuji" – The Translator Between Shape and Code
In this study, a team of researchers introduces a new artificial intelligence model they call Fuji. Think of Fuji not just as a picture-recognizer, but as a super-smart translator that learns to speak two languages at once: the language of pictures (what a tumor looks like under a microscope) and the language of code (the DNA mutations and RNA messages inside the cells).
Usually, AI models trained on medical images are like tourists who only learn to recognize landmarks. They can say, "That looks like a lung," or "That looks like a tumor," but they don't understand why the tumor looks that way. They treat the picture as just a pretty image. Fuji is different. The researchers taught it to look at the picture and immediately ask, "What genetic code caused this specific shape?"
To do this, they didn't just show Fuji pictures. They fed it a massive library of 60,546 whole-slide images (huge, high-resolution digital photos of tissue samples) and paired every single one with its corresponding genetic data: 45,675 RNA profiles (which tell us which genes are active) and 22,683 whole-genome sequences (the full DNA code). It's like giving the AI a million textbooks where every page has a photo of a city on the left and the exact blueprint of that city's construction on the right.
How Fuji Learned to "See" Genes
The researchers used a clever two-step training method, like a master chef teaching an apprentice.
- Step One (The Taste Test): First, they taught Fuji to recognize the basic "flavor" of the tissue. They used two other powerful AI models (GigaPath and CHIEF) as "teachers." Fuji learned to look at the tissue and match the visual patterns, even if the tissue was prepared in two different ways: some were preserved in chemicals (FFPE, the standard for hospitals) and some were frozen fresh (fresh-frozen, used for research). Fuji learned to ignore the differences in how the tissue was preserved and focus only on the true shape of the cells. This is crucial because it means Fuji can work with the slides doctors actually use every day.
- Step Two (The Deep Dive): Next, they challenged Fuji to connect the shape to the code. They gave it three tasks at once:
- Reconstruct the image: If you hide a part of the picture, can Fuji guess what it looks like? This keeps it good at seeing details.
- Reconstruct the RNA: Can Fuji look at the shape and guess what the RNA messages are saying? This links the visual to the molecular activity.
- Decode the mutations: Can Fuji look at the shape and guess which specific genetic "typos" (mutations) are present? This is the big magic trick.
What Fuji Discovered
Once trained, Fuji proved that the "shape" of a tumor is indeed a direct reflection of its genetic code. The researchers found that Fuji could predict complex genetic states with incredible accuracy, often outperforming all previous AI models.
- The "Instability" Detectives: The team tested if Fuji could spot when a tumor's DNA was falling apart. They looked for four specific types of genetic chaos: Microsatellite Instability (MSI), Chromosomal Instability (CIN), Homologous Recombination Deficiency (HRD), and Whole-Genome Duplication (WGD). Fuji got these right with an accuracy (AUC) as high as 0.98. To put that in perspective, if a human expert was 80% sure, Fuji was 98% sure. It could tell the difference between a stable genome and a chaotic one just by looking at the tissue slide.
- The "Extra" DNA: One of the most exciting finds was about extrachromosomal DNA (ecDNA). This is a weird, circular piece of DNA that floats outside the normal chromosomes and makes cancer grow super fast. Usually, you need expensive, complex tests to find it. Fuji, however, learned to spot the unique "footprint" ecDNA leaves on the tissue structure. In patients with lower-grade glioma and endometrial carcinoma, the presence of these invisible DNA circles, predicted by Fuji from the slide alone, was a strong sign that the patient would have a harder time surviving.
- The "Smoking" and "Repair" Signatures: The researchers also asked: Can Fuji tell if a tumor was caused by smoking or by a broken DNA repair system? They found that tobacco-related mutations and mismatch repair defects leave distinct, localized marks on the tissue architecture. It's as if smoking leaves a specific kind of soot on the city walls, and Fuji can point exactly to where that soot is. These patterns were so clear that Fuji could find them in completely different groups of patients, proving the pattern is real and not just a fluke.
- Predicting the Future: Fuji didn't just look at the past; it looked at the future. The model could predict Tumor Mutational Burden (TMB) (how many mutations a tumor has) and neoantigen load (how visible the tumor is to the immune system). More importantly, in a small group of lung cancer patients, Fuji's predictions about who would respond to immunotherapy were better than other models. It suggested that the shape of the tumor holds clues about whether the immune system can fight it.
Why This Matters
The most important thing about Fuji is that it works on the slides doctors already have. Most of the world's cancer patients get a standard slide (FFPE) that is preserved in chemicals. Previous AI models struggled to connect these standard slides to the genetic data because that data usually comes from fresh-frozen samples. Fuji solved this by learning to ignore the "chemical smell" of the slide and focus on the "city layout."
The authors suggest that this could change how cancer is treated. Instead of waiting days or weeks for expensive genetic tests, doctors could potentially use Fuji to get a "genomic report" from a routine microscope slide in minutes. It could help decide which patients need targeted drugs, which ones might respond to immunotherapy, and which ones are at higher risk, all without needing to sequence the DNA first.
Of course, the paper is careful to note that this is a "proof of concept." While the results are strong and consistent across different datasets, the immunotherapy predictions were based on a small group of patients and need to be tested in larger, real-world clinical trials before doctors can rely on them. But the core idea—that the visual shape of a tumor is a readable map of its genetic code—seems to be a solid discovery. Fuji has shown us that if you look closely enough at the city, you can read the blueprints.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.