🧬 biology

A leakage-controlled evaluation framework for ontology-grounded semantic representations of single-cell clusters

This paper introduces a leakage-controlled evaluation framework demonstrating that frozen, ontology-grounded semantic representations of single-cell clusters consistently outperform gene-name embeddings across external datasets, whereas unconstrained LLM-generated interpretations and post hoc tuning fail to provide robust value once methodological biases are eliminated.

Original authors: Mohamed KONE, Gaoussou HAIDARA, Chao HOU, Shulin Wang

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed KONE, Gaoussou HAIDARA, Chao HOU, Shulin Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to solve a mystery inside a bustling city, but instead of interviewing people, you are looking at tiny, glowing clues left behind by millions of microscopic citizens. This is the world of single-cell RNA sequencing, a technology that lets scientists peek inside individual cells to see which genes are "switched on." Think of these genes as the cell's to-do list or its ID card. When scientists find a group of similar cells, they usually look at the top few genes on their lists to figure out what kind of cell it is. These top genes are called marker genes.

However, just like a list of ingredients doesn't always tell you the flavor of the dish, a list of gene names can be confusing. Two different groups of cells might have different gene names but actually be doing the same job, or they might have similar names but be doing totally different things. To fix this, scientists often use Gene Ontology, which is like a giant, organized library of biological concepts. Instead of just listing "Gene A" and "Gene B," you can say, "These cells are busy with 'cell division' and 'energy production.'" This makes the data easier to understand.

Recently, everyone got excited about Large Language Models (LLMs)—the super-smart AI chatbots that can write stories and explain complex ideas. People wondered: Could we just ask an AI to read these gene lists and write a perfect, easy-to-understand story about what the cells are doing? It sounds like a magic shortcut. But as this new study shows, taking shortcuts in science can sometimes lead you to the wrong answer, especially if the AI accidentally "guesses" the answer before you even ask.

The Great AI Guess Hunt

This paper is essentially a rigorous "lie detector test" for using AI to understand cells. The researchers, led by Mohamed Kone and his team at Hunan University, set out to see if an AI (specifically a model called DeepSeek) could turn messy gene lists into better, smarter descriptions of cell groups. They didn't just ask the AI to write a story; they built a fortress of rules to make sure the AI wasn't guessing.

The Setup: The Trap of the "Smart" AI
The team started with a hopeful idea: maybe the AI could look at a list of genes and say, "Ah, these cells are immune cells fighting a virus!" The problem is, these AI models are so good at reading that they might have seen the answer in their training data. If you ask, "What do these genes mean?" and the AI says, "They are immune cells," it might just be repeating something it memorized, not actually figuring it out from the genes. This is called leakage. It's like asking a student a math problem, and they get the right answer not because they solved it, but because they saw the answer key on the teacher's desk.

To stop this, the researchers built a series of "gates" or checkpoints:

The Free-Text Gate: They let the AI write a free-form story. But the AI kept slipping up, using words that gave away the cell type (like saying "T-cells" when it wasn't supposed to). The AI was guessing by using its own memory instead of the clues.
The Constrained Gate: They tried to stop the guessing by only letting the AI pick from a pre-approved list of biological terms. But even then, the AI didn't do any better than a simple, boring computer program that just counts how many genes support each term.
The Recovery Gate: They tried to trick the AI by hiding some of the gene clues. The AI couldn't recover the missing information any better than random guessing once the "guessing" list of possible answers was removed.

The Verdict: The AI Didn't Win
After running all these tests, the researchers found that the AI did not provide any special magic. Once they blocked the guessing routes, the AI's "smart" descriptions were no better than a simple, rule-based system. In fact, the AI often just repeated what it already knew, rather than learning from the specific genes in front of it. The study explicitly rules out the idea that using a fancy AI to generate free-text descriptions is a reliable way to improve cell analysis when you need to be sure the results are real and not just a lucky guess.

The Real Hero: The "Boring" Rule-Based System
So, if the AI failed, what worked? The researchers turned to a method that sounds much less exciting but turned out to be the real champion: Deterministic Ontology Grounding.

Imagine you have a bag of mixed-up Lego bricks (the genes). Instead of asking a creative friend to build a castle (the AI), you use a strict, step-by-step instruction manual (the deterministic rules) to sort the bricks into specific categories based on their shape and color.

The team took the gene lists and mapped them to the "library" of biological concepts (Gene Ontology) using a strict, unchangeable set of rules.
They didn't let the rules change based on the results. They locked the rules in place before they looked at the final answer.
They tested these locked rules on three completely new sets of data (immune cells, pancreas cells, and brain cells) that the AI had never seen before.

The Results: Rules Beat Hype
The results were clear and consistent. The "boring" rule-based system consistently did a better job at grouping the cells correctly than just using the raw gene names.

On the immune cell dataset, the rule-based system improved the accuracy score by +0.0260.
On the pancreas dataset, it improved the score by a huge +0.2702.
On the brain cell dataset, it improved the score by +0.2839.

The best "rule" wasn't the same for every tissue. For the brain cells, a focused list of 12 concepts worked best. For the pancreas, a broader list of 30 concepts worked better. This suggests that there is no single "magic formula" for all cells; the right amount of detail depends on the specific biological puzzle you are solving.

Why This Matters
The most important takeaway isn't that the AI is useless. It's that we need to be careful how we test it. This paper proves that if you don't build strict walls against guessing, an AI might look like a genius when it's actually just a parrot.

The study concludes that when you strip away the guessing and the luck, explicit, structured biological knowledge (the rule-based system) is more reliable than unconstrained AI generation for understanding cell groups. It's a reminder that in science, sometimes the most reliable tool isn't the flashiest one, but the one that follows the rules and doesn't try to guess the answer. The researchers didn't find a new AI superpower; they found a better way to make sure we aren't fooling ourselves when we use them.

Technical Summary: A Leakage-Controlled Evaluation Framework for Ontology-Grounded Semantic Representations of Single-Cell Clusters

Problem Statement

Single-cell RNA sequencing (scRNA-seq) studies rely on marker genes to summarize cellular identity. While these lists are interpretable, they are sparse symbolic summaries that often fail to capture functional relationships between cell populations that are evident to domain experts. Large Language Models (LLMs) have been proposed to synthesize biological knowledge from these sparse inputs to generate richer semantic representations. However, evaluating LLM-derived representations is vulnerable to benchmark leakage, privileged candidate information, and post hoc method selection. Specifically, LLMs may inadvertently reconstruct evaluation labels through lineage or population descriptions, or gain unfair advantages from prompt structures containing hidden full-marker information. This creates a risk that apparent performance gains are artifacts of data leakage rather than genuine biological inference.

Methodology

The study employs a staged, leakage-controlled evaluation framework designed to isolate genuine semantic value from confounding factors. The workflow proceeds through three distinct phases:

LLM Branch Evaluation and Closure:
- Free-Text Reasoning: DeepSeek was initially tested to generate free-text biological descriptions from marker genes. Audits revealed that even without explicit cell-type fields, the model leaked identity information through lineage and population language.
- Constrained Selection: The model was restricted to selecting from a supplied list of Gene Ontology (GO) identifiers. This did not outperform deterministic ranking.
- Recovery Controls: Experiments tested the model's ability to recover concepts from perturbed (incomplete) marker sets. "Oracle-assisted" recovery (where the model sees missing context) showed gains, but these were attributed to privileged information. In the independent-universe gate, where candidates were built only from observed perturbed markers without privileged context, DeepSeek recovered zero concepts and matched the deterministic baseline.
- Outcome: All LLM branches were closed as they failed to demonstrate model-specific value after controlling for leakage and candidate-universe biases.
Deterministic Ontology-Grounded Development:
- The study shifted to deterministic ontology grounding, mapping marker genes to GO biological-process concepts using a frozen MyGene.info cache.
- Scoring: Marker contribution ( $w_i$ ) was calculated based on gene rank and expression fraction differences inside/outside the cluster. Concept scores ( $S_k(c)$ ) were the sum of weighted markers.
- Aggregation: Concepts were ranked by supporting gene count, best marker rank, static score, and GO ID. Four fixed representation rules were defined:
  - MSOA Top-12: Uniform aggregation of the top 12 concepts.
  - Weighted MSOA Top-12: Weighted aggregation based on normalized static scores.
  - Evidence Top-20: Uniform aggregation of top 20 concepts.
  - GO Unranked Top-30: Concepts sorted by ID, selecting top 30.
- Embedding: Selected concepts were encoded using the frozen sentence-transformers/all-MiniLM-L6-v2 model.
- Development Decision: No universal adaptive rule (e.g., support-rescue mechanisms) was selected because none consistently outperformed the fixed baselines across all development datasets (PBMC3k, HumanPancreas, MouseRetina) while maintaining stability.
Frozen External Validation:
- Protocol Lock: Before any performance metrics were inspected, three independent external datasets were registered: Wilk 2020 (Immune), Pancreas Multiomics 2022, and a Motor-Cortex Inhibitory-Neuron dataset.
- Eligibility: Datasets were audited for label validity and marker cache completeness.
- Evaluation: The four frozen ontology representations were tested against a baseline of raw gene-name embeddings. The primary metric was the Adjusted Rand Index (ARI) between representation-derived partitions and biological labels. Stability was assessed via seven hierarchical clustering settings and 50-seed K-means analyses.

Key Results

LLM Failure: After applying leakage and candidate-universe controls, DeepSeek-based reasoning, constrained selection, and semantic recovery demonstrated no model-specific value. The apparent gains in earlier stages were explained by privileged information or label leakage.
Ontology Gains: All four deterministic ontology-grounded representations improved the Adjusted Rand Index (ARI) over raw gene-name embeddings across all three independent external datasets.
- MSOA Top-12 achieved the largest mean Delta ARI (+0.1542) across the three datasets.
- GO Unranked Top-30 showed the strongest performance on the Pancreas dataset (Delta ARI +0.2702).
- MSOA Top-12 was strongest on the Inhibitory-Neuron dataset (Delta ARI +0.2839).
Stability: The performance gains persisted across seven hierarchical clustering settings and 50-seed K-means analyses, indicating the signal was not an artifact of a specific clustering configuration.
Encoder Sensitivity: A supplementary analysis using SapBERT showed that three unweighted rules remained non-negative across datasets, though effect sizes and weighted aggregation results were encoder-dependent.

Key Contributions

Leakage-Controlled Benchmark: The paper establishes a framework for evaluating biological semantic representations that rigorously separates label leakage, privileged candidate information, and explicit biological knowledge. It demonstrates how unconstrained LLM reasoning can be confounded by these factors.
Deterministic Representation Family: It formalizes a family of deterministic, marker-supported ontology-grounded representations (MSOA, Evidence, GO-ID variants) that map sparse marker lists to structured biological concepts without relying on generative text.
Prospectively Frozen Validation: The study implements a "no-final-method" development phase followed by a strictly frozen external validation protocol. This prevents post hoc tuning and ensures that reported gains are reproducible across independent biological settings.

Significance and Claims

The paper claims that the signal persisting under frozen external validation comes from explicit structured biological knowledge, not from unconstrained generated interpretation or post hoc tuning.

Modest Claim on LLMs: The study does not claim LLMs are generally unsuitable for single-cell biology. Rather, it concludes that in the specific setting of marker-to-semantic-representation, the evaluated DeepSeek mechanisms did not add reliable value once leakage and privileged candidate controls were applied.
Modest Claim on Methodology: The authors do not claim to have discovered a single "optimal" ontology aggregation rule. Instead, they highlight that useful ontology breadth differs across biological settings (e.g., broader GO coverage benefits pancreas data, while focused Top-12 benefits inhibitory neurons).
Primary Contribution: The core contribution is the evaluation framework itself—a protocol that preserves negative results, closes development branches based on prespecified gates, and validates frozen rules on independent data. This framework allows for the distinction between apparent LLM-mediated gains and reproducible benefits of explicit ontology grounding.