💻 computer science

Cell-Type Prototype-Informed Neural Network for Gene Expression Estimation from Pathology Images

This paper proposes the Cell-type Prototype-informed Neural Network (CPNN), a framework that leverages single-cell RNA-sequencing data to derive cell-type prototypes and learn compositional weights from pathology images, thereby achieving state-of-the-art, interpretable gene expression estimation at both slide and patch levels.

Original authors: Kazuya Nishimura, Ryoma Bise, Shinnosuke Matsuo, Haruka Hirose, Yasuhiro Kojima

Published 2026-03-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kazuya Nishimura, Ryoma Bise, Shinnosuke Matsuo, Haruka Hirose, Yasuhiro Kojima

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Reading a City from a Satellite Photo

Imagine you have a satellite photo of a bustling city (this is the Pathology Image). You can see the buildings, the parks, and the roads. But you want to know something invisible: what the people inside the buildings are thinking and doing (this is the Gene Expression).

Usually, to know what people are thinking, you have to knock on every single door and ask them (this is RNA Sequencing). But knocking on every door is expensive, slow, and sometimes impossible.

Scientists have tried to guess what people are thinking just by looking at the buildings from space. They've gotten pretty good at it, but they often treat the city as one big, blurry blob. They miss the fact that the city is actually made of thousands of individual people (cells) with different jobs.

This paper introduces a new method called CPNN. It's like giving the satellite observer a secret cheat sheet based on interviews with the actual people, even though the observer never saw them directly.

The Problem: The "Blurry Soup" vs. The "Individual People"

The Slide (The City): When doctors look at a tissue sample under a microscope, they see a "slide." It's a mix of millions of cells. The gene expression they measure is like a big pot of soup. They can taste the soup and know it's "salty" or "spicy" (high or low gene activity), but they can't tell which specific ingredient (cell) added the salt.
The Single-Cell Data (The Interviews): Scientists have databases where they have interviewed individual cells (Single-Cell RNA sequencing). They know exactly what a "Muscle Cell" says versus what a "Skin Cell" says.
- The Catch: These interviews are noisy (people stutter or lie), and they don't have photos of the city to match them up with.

The Old Way: Previous AI models tried to guess the soup's flavor just by looking at the pot. They were okay, but they didn't understand the recipe.

The New Way (CPNN): This model says, "Let's use the interviews to build a Prototype Menu."

How CPNN Works: The "Recipe Card" Analogy

The authors created a system called Cell-type Prototype-informed Neural Network (CPNN). Here is how it works, step-by-step:

1. Creating the "Prototype Menu" (Cell-Type Prototypes)

Imagine you want to guess the flavor of a soup. You know that if the soup has a lot of "Tomato Cells," it will be red and acidic. If it has "Carrot Cells," it will be orange and sweet.

The researchers take the noisy interview data (Single-Cell data) and clean it up to create a perfect "Recipe Card" for each cell type.
Recipe Card for "Immune Cell": "If you see this cell type, expect these 50 genes to be active."
Recipe Card for "Cancer Cell": "If you see this cell type, expect these 200 genes to be active."
These cards are called Prototypes. They represent the "ideal" voice of each cell type.

2. Looking at the City (The Image)

Now, the AI looks at the pathology image (the satellite photo). It doesn't just guess the soup flavor; it tries to figure out what kind of people are in the buildings.

It looks at a patch of the image and asks: "Is this area full of Immune Cells? Or Cancer Cells?"
It calculates a Weight (a percentage). For example: "This patch is 60% Immune Cells and 40% Cancer Cells."

3. Mixing the Soup (The Prediction)

Finally, the AI mixes the Recipe Cards based on the Weights it found in the image.

Math Magic: (60% of Immune Recipe) + (40% of Cancer Recipe) = The Predicted Gene Expression.

Because the AI is forced to use these "Recipe Cards," it can't just make up random numbers. It has to follow the biological rules of how cells actually behave. This makes the prediction much more accurate and trustworthy.

Why is this a Big Deal?

1. It's Cheaper and Faster

Instead of doing expensive genetic testing on every patient, doctors can just take a picture of the tissue slide, and this AI can predict the genetic profile with high accuracy.

2. It's "Explainable" (The "Why" Factor)

Old AI models are "black boxes." You ask, "Why did you predict this cancer is aggressive?" and the AI says, "I don't know, the pixels looked right."

CPNN is transparent. Because it uses the "Recipe Cards," it can tell you: "I predicted this is aggressive because I see a lot of 'Cycling' cells (cells that are dividing rapidly) in this area."
It tells the doctor which cell types are driving the result.

3. It Handles the "Noise"

Single-cell data is messy (like a noisy phone call). The researchers built a special filter (called Modality Correction) that adjusts for the differences between the "interview" (single-cell data) and the "photo" (tissue image). It's like a translator who knows exactly how to clean up a bad phone connection so you understand the message.

The Results: The "Taste Test"

The researchers tested this on three types of cancer (Breast, Kidney, Lung) and compared it to 13 other top AI methods.

The Score: CPNN won almost every time. It was the best at predicting the genetic "flavor" of the tissue.
The Proof: When they looked at the "Weights" the AI calculated, they matched real biological facts. For example, the AI correctly identified that "Basal" breast cancers have more rapidly dividing cells than "Luminal" cancers, just like real doctors know they should.

Summary

Think of CPNN as a detective who solves a mystery by combining a crime scene photo with a database of suspect profiles.

Old Detectives: Look at the photo and guess.
CPNN: Looks at the photo, identifies the suspects (cell types) present, checks their known profiles (prototypes), and reconstructs the crime (gene expression) based on who was there.

This makes the prediction not only more accurate but also gives doctors a clear explanation of why the prediction was made, bridging the gap between what a tissue looks like and what it is doing at a molecular level.

1. Problem Statement

Estimating gene expression profiles directly from Whole Slide Images (WSIs) of pathology slides is a promising, low-cost alternative to RNA sequencing, enabling large-scale molecular profiling. However, existing methods face a fundamental limitation:

The Aggregation Gap: Current approaches treat gene expression as a direct signal from the slide or patch level. They fail to model the biological reality that observed "bulk" or "spatial" expression is actually an aggregation of underlying cell-level expressions.
Data Modality Mismatch: While single-cell RNA sequencing (scRNA-seq) provides cell-level resolution, these datasets are noisy, suffer from batch effects, and lack corresponding histology images. Conversely, WSI-based datasets have images but lack cell-level resolution.
Lack of Biological Priors: Existing models rely solely on image features to predict high-dimensional gene expression without incorporating the stable gene-gene co-variation patterns inherent to specific cell types.

2. Methodology: Cell-Type Prototype-Informed Neural Network (CPNN)

The authors propose CPNN, a framework that bridges the gap between histology images and gene expression by explicitly modeling the relationship between cell-type prototypes and observed bulk/spatial expression.

Core Concept

The model assumes that the gene expression of a tissue patch is a linear mixture of cell-type-specific expression profiles (prototypes).
$\mu_g^b(X) = \alpha_g \sum_i \sum_c w(x_i)_c \cdot \bar{T}_{c,g} + \beta_g$
Where:

$\bar{T}_{c,g}$ : Cell-type Prototypes. Mean expression profiles for cell type $c$ and gene $g$ , derived from scRNA-seq data.
$w(x_i)_c$ : Compositional Weights. The proportion of cell type $c$ in image patch $x_i$ , predicted by the neural network.
$\alpha_g, \beta_g$ : Modality Correction parameters (scaling and bias) to align the single-cell derived prototypes with the bulk/spatial measurement scale.

Key Components

Prototype Generation (Batch-Agnostic):
- Raw scRNA-seq data is noisy and batch-affected. CPNN uses Negative Binomial Regression to estimate normalized, batch-agnostic mean expression profiles ( $\bar{T}$ ) for each cell type. These serve as informative priors.
Image Encoder & Weight Prediction:
- The WSI is cropped into patches. An image encoder (e.g., CONCH) extracts features.
- A Multi-Layer Perceptron (MLP) predicts the compositional weights ( $w$ ) for each cell type based on the patch features.
Expression Reconstruction:
- Predicted gene expression is calculated as the dot product of the predicted weights and the prototypes, adjusted by modality correction factors ( $\alpha, \beta$ ).
Optimization Strategy:
- Loss Function: The model minimizes a Negative Log-Likelihood assuming a Negative Binomial distribution (for slide-level) or Pearson Correlation Loss (for patch-level spatial data).
- Regularization: A soft consistency constraint ( $L_R$ ) is added to ensure the predicted weights and updated prototypes remain consistent with initial deconvolution results, preventing the model from ignoring the biological priors.
- Prototype Refinement: Unlike static priors, the prototypes ( $\bar{T}$ ) are updated during training to mitigate noise and the modality gap between scRNA-seq and histology.

Adaptation for Patch-Level (Spatial Transcriptomics)

For spatial transcriptomics (ST), the framework is adapted by replacing the absolute value-based loss (Negative Binomial) with a Pearson Correlation loss. This focuses on the relative relationships between patches rather than absolute counts, which are often noisy in ST data.

3. Key Contributions

Novel Problem Formulation: A new approach to WSI-based gene expression estimation that explicitly leverages external cell-level datasets (scRNA-seq) to guide the prediction, rather than treating expression as a black-box regression task.
CPNN Architecture: A neural network that integrates cell-type prototypes and compositional weights, acting as a biologically grounded regularizer. It effectively handles the modality gap between single-cell data and histology images.
Interpretability: By outputting compositional weights, the model provides interpretable insights into which cell types drive specific gene expression predictions, a feature missing in standard deep learning approaches.
Comprehensive Evaluation: Extensive validation across three slide-level datasets (BRCA, KIRC, LUAD) and three patch-level spatial datasets (CSCC, Her2st, STNet).

4. Experimental Results

The method was evaluated against 13 state-of-the-art baselines, including Multiple Instance Learning (MIL) and Multiple Instance Regression (MIR) methods (e.g., HE2RNA, tRNAformer, MOSBY, S4MIL).

Slide-Level Estimation:
- CPNN achieved State-of-the-Art (SOTA) performance in Spearman Correlation Coefficient (SCC) across all three datasets (BRCA, KIRC, LUAD).
- It outperformed advanced attention-based and state-space models (e.g., MambaMIL, ILRA), demonstrating that incorporating cell-level priors is more effective than purely architectural improvements.
Patch-Level Estimation:
- When integrated as a plug-in with existing architectures (STNet, TRIPLEX), CPNN consistently improved SCC performance.
- The largest gains were seen on the CSCC dataset, where matched single-cell and spatial data allowed for better modality alignment.
Ablation Studies:
- Removing prototype initialization or modality correction significantly degraded performance, confirming the necessity of both the priors and the alignment mechanisms.
- Granularity Analysis: The model remained robust across different cell-type label granularities (coarse to fine), though fine-grained labels (49 types) yielded the best results.
Biological Validation:
- Visualizing the inferred weights revealed biologically plausible patterns. For example, "Basal-like" BRCA tumors showed high weights for proliferative (Cycling) prototypes, and "LumA" tumors showed distinct activation patterns consistent with known biology.

5. Significance

Biological Grounding: CPNN moves beyond "black-box" deep learning by embedding biological knowledge (cell-type composition) directly into the network's architecture. This acts as a powerful regularizer, especially when training data is limited or noisy.
Clinical Utility: The ability to estimate gene expression from standard pathology slides (WSIs) at a fraction of the cost of sequencing enables large-scale retrospective studies and personalized treatment stratification.
Interpretability: The model does not just predict expression; it explains why by revealing the underlying cell-type composition, offering a new tool for pathologists to understand the cellular drivers of molecular phenotypes.
Generalizability: The framework is flexible, applicable to both bulk (slide-level) and spatial (patch-level) transcriptomics, and can be integrated into existing pipelines as a plug-in component.

In summary, CPNN successfully bridges the gap between histology and genomics by treating gene expression estimation as a cell-type deconvolution problem guided by deep learning, achieving superior accuracy and interpretability compared to existing methods.