VINTER: a generative vision-language model for annotating and interrogating spatial tissues

Tissues are not just flat maps of cells; they are complex, three-dimensional neighborhoods where the identity of a single cell depends heavily on who lives next door. In the human body, a cell's function is often dictated by its immediate surroundings, much like how a person's role in a community changes based on the people they interact with daily. For decades, scientists have struggled to read these neighborhoods in a single, unified way. They could look at the tissue under a microscope to see its shape and structure, or they could measure the chemical signals inside the cells to understand their activity, but doing both simultaneously has been difficult. Traditional methods often force a choice: either describe the tissue by its visual appearance or by its molecular signature, rarely allowing the two to speak to each other in real time. This separation leaves a gap in understanding how the physical layout of a tumor or an organ influences the behavior of the cells within it.

A new approach called VINTER aims to close this gap by treating the tissue as a single, readable story. The researchers developed a computer model that can look at a picture of a tissue sample, read the list of active genes inside a specific spot, and simultaneously consider the gene activity of the six spots surrounding it. Instead of forcing the computer to choose between the image and the genes, the model reads all three sources—the visual structure, the local genes, and the neighborhood context—as one continuous sequence of information. This allows the system to make a decision about what kind of tissue it is looking at, while also keeping a record of exactly which pieces of evidence led to that conclusion. Because the genetic information is presented as a list of names rather than abstract numbers, the researchers can edit the input, removing or rearranging specific genes to see how the model's decision changes. This turns the process of identifying tissue from a static label into an interactive interrogation, where scientists can ask "what if" questions about the tissue's composition.

The team tested this system on a variety of biological challenges, starting with the human brain. They asked the model to identify the different layers of the cerebral cortex, a task where it outperformed existing methods that rely solely on gene data or image data alone. The model achieved a high level of accuracy, correctly identifying the cortical layers in the vast majority of cases. More importantly, the system demonstrated a remarkable ability to transfer its knowledge to new situations without needing to be retrained. When the researchers applied a model trained on breast cancer data to a completely different group of patients with triple-negative breast cancer, it successfully identified tissue types it had never seen before. This suggests that the model learned a fundamental language of tissue organization that applies across different diseases and patients, rather than just memorizing specific examples.

One of the most striking discoveries came from studying tertiary lymphoid structures, which are small, organized clusters of immune cells that form inside tumors and can influence how the body fights cancer. The researchers used the model's ability to edit inputs to figure out exactly what signals these structures send to their surroundings. They found that the model's judgment of whether a structure was "mature" or "immature" depended on two opposing programs of gene activity in the neighborhood. One program, driven by follicular B cells, made the structure look more mature, while another, driven by plasma cells that produce antibodies, made it look less mature. By mapping these signals back onto the tissue, they discovered a hidden spatial pattern: the mature, organizing signals were concentrated in the center of the structure, while the antibody-producing signals were pushed to the very edges. This revealed that the "maturity" of these immune hubs is not a single state but a dynamic balance between a core and a periphery, a nuance that standard methods would have missed.

The power of this approach also extended to finding hidden details in triple-negative breast cancer, a disease known for its complexity. In many cases, pathologists label a region of tissue as "stroma," or connective tissue, which is a broad category that can hide significant differences. The model, however, looked at the probability of every possible tissue type for each spot and found that within these broadly labeled stromal regions, there were distinct zones of immune activity. Some areas were rich in immune cells and showed signs of an active, inflamed response, while others were quiet and silent. This ability to see the "hot" and "cold" zones within a single label helped the researchers identify a specific type of immune environment that is known to respond well to certain cancer treatments. Furthermore, by looking at the secondary probabilities of the model, they were able to flag spots that were labeled as benign but actually contained hidden cancer cells. When they reviewed these flagged spots, they recovered nearly half of the cancer that had been missed by the standard single-label approach, particularly in the most aggressive subtypes of the disease.

Finally, the researchers showed that this method works across different scales of observation, from large tissue samples down to individual cells. They applied the same model to high-resolution data from colorectal cancer, where it successfully identified immune structures and distinguished between different types of immune cells without any additional training. The model correctly identified that certain immune cells, specifically those producing a specific type of antibody, gathered at the edges of these structures, while others remained in the center. This consistency across different technologies and resolutions suggests that the model has captured a robust understanding of how tissue is organized. By turning spatial annotation into an interrogable reading of tissue, VINTER offers a new way to explore the body's neighborhoods, allowing scientists to ask questions about the evidence behind every biological decision and to see the hidden patterns that define health and disease.