HistAgent: an evidence-grounded agent for spatial molecular reasoning from routine histology
HistAgent is a unified framework that combines a visual-omics foundation model with a spatial AI agentic module to infer, interpret, and retrieve spatial molecular information from routine H&E histology images, offering a cost-effective and generalizable alternative to expensive spatial transcriptomics while enabling traceable, question-driven biological analysis.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the world of medicine, a pathologist's most trusted tool is a simple glass slide stained with purple and pink dyes. This routine test, known as hematoxylin and eosin staining, reveals the shape and arrangement of cells within a tissue sample, allowing doctors to diagnose diseases like cancer by looking at the tissue's architecture. However, this view is limited to what the eye can see; it cannot reveal the specific chemical messages or genetic instructions that cells are using to communicate and function. To see those invisible molecular details, scientists must turn to a more advanced and expensive technology called spatial transcriptomics. This method maps the activity of thousands of genes while keeping them in their original physical locations within the tissue, offering a deep dive into the biological machinery at work. Yet, because this advanced test is costly, technically difficult, and not available in every hospital, it remains out of reach for many routine diagnostic needs. The challenge has been to bridge the gap between the simple, widely available stained slides and the complex, hidden molecular world they contain.
A team of researchers has now built a new digital system that attempts to solve this problem by teaching artificial intelligence to read the molecular story hidden inside a standard stained slide. They call their creation HistAgent. Instead of simply guessing what genes might be present, the system works like a careful investigator that gathers evidence before drawing a conclusion. It starts by analyzing a small patch of the stained tissue, looking at both the specific cells in focus and the surrounding neighborhood to understand the context. Based on this visual information, the system generates a ranked list of the most likely active genes in that specific spot, effectively translating the picture of the tissue into a list of molecular suspects.
What makes this approach different from previous attempts is how it handles the results. Rather than just outputting a long, confusing list of numbers, the system organizes its findings into structured "evidence cards." These cards summarize the top genes, the types of cells present, and the functional programs the cells are running. A second part of the system, acting as a reasoning agent, uses these cards to answer specific questions about the tissue. If a researcher asks why a certain area looks like an immune hotspot, the system can point to the specific genes and cell types it found in that region to support its answer. This allows for a back-and-forth conversation where the user can dig deeper, asking follow-up questions about the local environment or the biological processes at play, with every answer tied directly to the visual and molecular evidence the system found.
The researchers trained this system on a massive collection of tissue samples from humans and mice, covering dozens of different organs. They used a development corpus comprising 2.23 million paired spots from 936 tissue slides. When they asked the system to identify known biological features, such as specific immune structures or tumor types, it performed with high accuracy, often matching the results of the expensive, direct molecular tests. In one specific test involving kidney cancer, the system successfully located complex immune structures within the tissue and explained their presence by citing the specific immune cells and chemical signals it detected, all within minutes and at a fraction of the cost of the traditional method.
Beyond just identifying features, the system proved capable of predicting patient outcomes. By analyzing the entire slide, it could estimate the likelihood of a tumor being aggressive or a patient having a poor prognosis, performing as well as or better than existing advanced models. The researchers also built a searchable library of millions of tissue spots, allowing users to type in a description or upload a new image to find similar molecular patterns across a vast database of known tissues. This turns a simple microscope slide into a window for exploring the deep molecular landscape of the body, making high-level biological insights accessible without the need for expensive, specialized equipment. The work suggests that routine histology, long considered a static image, can be transformed into a dynamic source of molecular information, provided the right digital tools are used to interpret the evidence.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.