🤖 machine learning

HEXST: Hexagonal Shifted-Window Transformer for Spatial Transcriptomics Gene Expression Prediction

HEXST is a novel hexagonal-shifted-window Transformer that improves spatial gene expression prediction from histology images by aligning with hexagonal spot-array geometry and employing contrast-sensitive objectives to preserve spatial heterogeneity, outperforming existing state-of-the-art models across multiple datasets.

Original authors: Keunho Byeon, Jin Tae Kwak

Published 2026-05-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Keunho Byeon, Jin Tae Kwak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Reading the City Map from a Satellite Photo

Imagine you have a massive, high-resolution satellite photo of a bustling city (this is a histology slide, or a picture of tissue under a microscope). You can see the buildings, the roads, and the parks. However, you don't know what's happening inside the buildings. Are the factories running? Are the schools full? Are the hospitals busy?

In the medical world, finding out what's happening inside the "buildings" (cells) usually requires a very expensive, slow, and complicated process called Spatial Transcriptomics. It's like sending a drone into every single building to take a census. While this gives you perfect data, it's too costly to do for every patient in a hospital.

The goal of this paper is to build a smart AI that can look at the satellite photo (the cheap, common tissue slide) and accurately guess what's happening inside the buildings (the gene expression) without ever needing to send the drones.

The Problem: The "Square Peg in a Round Hole" Issue

The researchers found two main problems with how other AI models tried to solve this:

The Wrong Grid: The expensive "drone" technology (like the 10x Genomics Visium platform) doesn't sample the city in a perfect square grid (like a chessboard). Instead, it samples in a honeycomb pattern (hexagons).
- The Analogy: Imagine trying to organize a crowd of people standing in a honeycomb pattern, but you force them into square boxes. Some people who are standing right next to each other get put in different boxes, while people far away get stuck in the same box. The AI gets confused about who is actually a neighbor.
The "Blurry" Effect: Existing AI models are too nice. When they try to guess the activity in a specific spot, they tend to average out the answers.
- The Analogy: If one building is a loud rock concert and the one next to it is a quiet library, a "too nice" AI might guess they are both "moderately noisy." This over-smoothing blurs the sharp boundaries between different types of tissue, which is exactly where the important medical clues are hiding.

The Solution: HEXST (The Honeycomb-Smart AI)

The authors created a new model called HEXST (Hexagonal Shifted-Window Transformer). Here is how it fixes the problems:

1. Speaking the Language of Honeycombs

Instead of forcing the data into square boxes, HEXST builds its own hexagonal windows.

The Analogy: Instead of using a square ruler to measure a honeycomb, HEXST uses a hexagonal ruler. It groups the "spots" (the data points) exactly the way nature arranged them.
The Shift: To make sure the AI sees the whole picture, it doesn't just look at one static honeycomb grid. It shifts the grid slightly, like sliding a window pane back and forth. This ensures that every spot gets to be in the center of a window at some point, allowing the AI to understand the neighborhood from every angle.

2. The "HexRoPE" Compass

Standard AI models use a compass that only knows North, South, East, and West (Cartesian coordinates). But in a honeycomb, "North" isn't just straight up; there are six directions.

The Analogy: HEXST invents a new compass called HexRoPE. This compass understands the six directions of the honeycomb lattice. It tells the AI, "Hey, that spot is actually your neighbor to the top-right, not just 'up'." This helps the AI understand the true shape of the tissue.

3. Keeping the Edges Sharp (The "Deviation" Trick)

To stop the AI from blurring the boundaries between the rock concert and the library, HEXST uses a special training trick.

The Analogy: Instead of just asking the AI, "How loud is this building?" (which leads to average guesses), the AI is also asked, "How much different is this building compared to its neighbors?"
By focusing on the differences (deviations) rather than just the absolute numbers, the AI learns to preserve the sharp edges and the unique "personality" of each gene's location. It learns that a sharp boundary is a feature, not a mistake to be smoothed out.

4. Learning from the Experts (The "Foundation Model")

HEXST doesn't learn from scratch. It uses a "pre-trained" brain (a foundation model called scFoundation) that has already read millions of genetic books.

The Analogy: Imagine teaching a student to read a map. Instead of starting with the alphabet, you give them a student who has already memorized the entire encyclopedia of how genes work. HEXST aligns its guesses with this expert's knowledge, ensuring its predictions make biological sense.

The Results: Does It Work?

The researchers tested HEXST on seven different datasets (different types of tissue and species, including humans and mice).

The Scoreboard: HEXST beat every other top model (like PEKA, NH2ST, and others) in almost every category.
The Proof: When they looked at the heatmaps (the visual guesses of gene activity), HEXST's maps looked much sharper and more accurate than the others. It didn't blur the lines between different tissue types.
Real-World Test: They also tested if this "smart guess" could help with real medical tasks. They used HEXST to predict prostate cancer grades and patient survival times for lung cancer. The model, which only looked at the tissue pictures, performed better than models that just looked at the pictures without the gene-prediction help. It got closer to the accuracy of having the actual expensive genetic tests.

Summary

HEXST is a new AI tool that finally respects the honeycomb shape of modern tissue scanners. By using hexagonal windows, a special six-direction compass, and a trick to keep edges sharp, it can accurately predict complex genetic activity from simple microscope slides. This could eventually allow doctors to get deep genetic insights from the cheap, standard slides they already use every day, without needing expensive new equipment.

Technical Summary: HEXST - Hexagonal Shifted-Window Transformer for Spatial Transcriptomics

Problem Statement
Spatial transcriptomics (ST) enables the profiling of gene expression within tissue sections while preserving spatial context, which is crucial for understanding tissue architecture and disease microenvironments. However, the high cost and low throughput of ST technologies hinder their routine deployment in large clinical cohorts. Consequently, computational methods have emerged to infer spatial gene expression directly from ubiquitous hematoxylin and eosin (H&E) stained histology slides.

Despite recent advances, existing models face two fundamental limitations:

Geometric Mismatch: Widely used spot-array ST platforms (e.g., 10x Genomics Visium) sample tissue on approximately hexagonal lattices. However, most existing models impose Cartesian (square grid) or geometry-agnostic neighborhood structures. This mismatch leads to anisotropic receptive fields, grouping non-neighboring spots while splitting immediate neighbors, thereby failing to encode the true local geometry as an architectural inductive bias.
Over-smoothing of Spatial Heterogeneity: Standard training objectives rely on point-wise regression (e.g., minimizing per-spot error). This often favors predictions that regress toward local means, resulting in over-smoothed gene expression profiles. Such smoothing obscures gene-specific spatial heterogeneity and sharp biological boundaries (e.g., tumor-stroma interfaces), which are critical for identifying localized cellular interactions.

Methodology
The authors propose HEXST (Hexagonal Shifted-Window Transformer), a geometry-aligned framework designed to predict spatial gene expression from histology images while respecting the hexagonal sampling of ST platforms. The architecture comprises four key components:

Hexagonal Shifted-Window Multi-Head Self-Attention (HexMSA):
- HEXST adapts the shifted-window Transformer architecture to hexagonal spot arrays.
- It maps Cartesian spot coordinates to a hexagonal lattice representation using axial and cube coordinates.
- Spots are partitioned into multi-scale hexagonal windows based on their lattice positions.
- A shifting mechanism offsets window centers across successive blocks, facilitating cross-window information exchange and expanding the receptive field from local to tissue-wide contexts without boundary artifacts.
- Within each window, self-attention is computed over a fixed set of discrete hexagonal "slots," ensuring computational efficiency.
Hexagonal Rotary Positional Encoding (HexRoPE):
- To encode relative spatial relationships inherent to hexagonal lattices, HEXST introduces HexRoPE.
- Unlike standard Cartesian Rotary Positional Embeddings (RoPE), HexRoPE rotates query and key features along the three principal directions of the hexagonal lattice (u, v, w axes) based on relative cube-coordinate offsets.
- This preserves translation equivariance within each window and provides geometrically consistent positional information.
Transcriptomic Feature Alignment:
- To inject transcriptomic priors, the model aligns its output embeddings with representations from a pretrained single-cell foundation model (scFoundation).
- A learnable linear projection maps HEXST's visual embeddings to the transcriptomic space, optimized via a cosine alignment loss. This leverages knowledge from large-scale single-cell datasets to guide the prediction of gene expression patterns.
Deviation-Matching Objective:
- To counteract over-smoothing and preserve gene-specific spatial contrast, HEXST introduces a deviation-matching loss.
- Instead of solely minimizing absolute error, this objective aligns the deviations (relative variations from the batch mean) between predictions and ground truth across spots.
- This encourages the model to capture fine-scale spatial variations and sharp boundaries rather than collapsing toward local averages.

Key Contributions

Geometry-Aware Architecture: HEXST is the first model to explicitly align the Transformer architecture with the hexagonal geometry of spot-array ST platforms through hexagonal window partitioning and HexRoPE.
Enhanced Spatial Contrast: By combining point-wise regression with a deviation-matching objective and transcriptomic priors, the method effectively preserves gene-specific spatial heterogeneity and sharp boundaries that are often lost in standard regression approaches.
Robust Performance: The framework demonstrates consistent improvements over state-of-the-art methods across diverse datasets, tissues, and species.

Results
The authors evaluated HEXST on seven spatial transcriptomics datasets aggregated from the SpaRED benchmark, covering diverse tissues (human and mouse) and experimental protocols.

Quantitative Performance: HEXST consistently outperformed seven state-of-the-art baselines (including STNet, Hist2ST, EGNv1/v2, TCGN, NH2ST, and PEKA) across five evaluation metrics: Gene-wise Pearson Correlation (PCCF), Spot-wise PCC (PCCS), Mutual Information (MIF), and two Area Under the Curve (AUC) metrics for distinguishing expression levels.
- Compared to the best-performing baseline (PEKA), HEXST achieved improvements of +0.0364 in PCCF, +0.0558 in PCCS, and +0.0431 in MIF.
Qualitative Analysis: Visual comparisons of gene expression heatmaps showed that HEXST retained sharp local variations and boundaries, whereas baseline models tended to blur these features or misidentify high-expression regions.
Downstream Clinical Tasks: The transcriptomic representations learned by HEXST were successfully transferred to two downstream clinical tasks on TCGA cohorts lacking ST measurements:
- Prostate Cancer Grading (TCGA-PRAD): HEXST-guided representations improved Gleason grading accuracy by 2.82% and Cohen's $\kappa$ by 0.0338 compared to image-only baselines, outperforming the PEKA baseline.
- Survival Prediction (TCGA-LUAD): HEXST improved the concordance index (C-index) for overall survival prediction by 0.0115 over image-only baselines, approaching the performance of bulk RNA-seq baselines more closely than competing methods.

Significance and Claims
The paper claims that HEXST addresses the critical gap between the geometric reality of ST platforms and the assumptions of existing computational models. By explicitly modeling hexagonal geometry and preserving spatial heterogeneity, HEXST enables more accurate inference of spatial gene expression from routine H&E slides. The authors posit that this capability facilitates large-scale exploratory studies of tissue organization and supports transcriptomics-guided clinical research, such as prognosis prediction, without the need for costly experimental spatial transcriptomics assays. The work suggests that geometry-aware modeling is essential for unlocking the full potential of spatial transcriptomics in clinical practice. Future work is noted to involve comparisons with other foundation models and extensions to continuous-coordinate spatial assays.