Application of large language models to the annotation of cell lines and mouse strains in genomics data

This study demonstrates that Large Language Models, specifically GPT-4o enhanced with retrieval-augmented generation, can effectively support human curators by significantly outperforming traditional string-matching methods in annotating mouse strains and cell lines in genomics data, thereby improving the efficiency and quality of metadata curation through a human-in-the-loop workflow.

The Gist

Imagine a massive, chaotic library called GEO (the Gene Expression Omnibus). This library holds hundreds of thousands of scientific experiments, like recipes for how cells behave. But here's the problem: the "recipe cards" (metadata) are written in messy, inconsistent handwriting. One scientist might write "Mouse Type A," another "C57BL/6," and a third might just say "The black mouse."

To make these recipes useful for other scientists, librarians (curators) have to manually read every card, figure out what the messy handwriting actually means, and file it under a strict, official label (like "C57BL/6J Mouse Strain"). This is the job of a database called Gemma. It's a vital job, but it's slow, expensive, and even the best human librarians make mistakes when they get tired or confused by typos.

The Big Idea
The authors of this paper asked a simple question: Can a super-smart AI (specifically, a Large Language Model called GPT-4o) act as a "super-librarian" to help speed this up?

They didn't ask the AI to replace the humans entirely. Instead, they wanted to see if the AI could act as a high-speed assistant that does the heavy lifting, leaving the humans to just double-check the work.

The Experiment: Two Specific Tasks

The researchers tested the AI on two very specific, tricky tasks:

1. Identifying Mouse Strains: Figuring out exactly which breed of mouse was used in an experiment.
2. Identifying Cell Lines: Figuring out exactly which type of human or animal cells were grown in a petri dish.

They gave the AI over 9,000 real-world examples (the "messy recipe cards") and asked it to match them to the official labels.

How They Did It (The "Magic" Tricks)

To help the AI, the researchers used two main tricks:

• The "Zero-Shot" Prompt: They didn't train the AI on thousands of examples first. They just gave it the instructions and the messy text, like asking a smart friend, "Read this and tell me what kind of mouse this is."
• RAG (Retrieval-Augmented Generation): This is like giving the AI a searchable encyclopedia right next to it.
  • For Mouse Strains, the AI had a short list of 156 official names to choose from.
  • For Cell Lines, the list was huge (46,000 names!). The AI couldn't hold all of them in its "brain" at once. So, the researchers used a "vector search" (a fancy way of saying "smart search") to find the top 50 most likely matches from the encyclopedia and handed those to the AI to make the final decision.

The Results: A Tale of Two Tasks

1. The Mouse Strain Mission (The Success Story)

• The AI's Performance: The AI got it right 77% of the time.
• The Comparison: They also tried a simple computer method that just looks for matching letters (like a "Find and Replace" tool). That method only got 6% right because it got confused by typos and weird spellings.
• The Takeaway: The AI is much better at understanding context. If a scientist wrote "C57/Bl6" (a typo), the AI knew it meant "C57BL/6." The simple computer tool just saw a mismatch and gave up.

2. The Cell Line Mission (The Harder Challenge)

• The AI's Performance: The AI got it right 59% of the time.
• Why it was harder: There are way more cell lines (46,000 vs. 156), and their names are often confusing codes (like "HEK293" or "HeLa"). The "smart search" sometimes picked the wrong 50 candidates, so the AI couldn't find the right answer even if it wanted to.

The Best Part: Catching Human Errors

Here is the most surprising discovery. The researchers used the "official" human annotations as the "correct" answer. But when the AI disagreed with the human, the researchers checked the original scientific papers.

• Result: The AI was right 200+ times.
• What happened? The human librarians had made mistakes! Sometimes the scientist who submitted the data wrote one thing in the title and something else in the methods section. The AI read the whole story and spotted the inconsistency, while the human, looking at just one part, missed it.

The Verdict: AI as a Co-Pilot, Not the Pilot

The paper concludes that AI is not ready to replace human curators yet. It still makes mistakes, often due to typos in the source text or "hallucinations" (making up facts).

However, the AI is an incredible "Co-Pilot."

• The Workflow: Imagine a human librarian sitting at a desk. The AI zooms through 1,000 cards in a second, says, "I think this is a C57 mouse, and here is the quote from the paper that proves it."
• The Human's Job: The human just reads the quote, nods, and clicks "Approve." If the AI is wrong, the human fixes it.

Why This Matters

This approach turns a slow, painful process into a fast, efficient one. It allows scientists to organize the world's biological data much faster, ensuring that future researchers can find the exact mouse strain or cell line they need without getting lost in a sea of messy handwriting.

In short: The AI is the super-fast scanner that finds the needles in the haystack, and the human is the expert who makes sure the needle is actually a needle and not a piece of straw. Together, they are unstoppable.

Technical Summary

1. Problem Statement

The reuse of functional genomics data in repositories like the Gene Expression Omnibus (GEO) relies heavily on accurate, consistent, and comprehensive metadata. However, current metadata often suffers from missing information, ambiguity, typos, and inconsistencies between the GEO records and associated publications.

• Current Bottleneck: Resources like Gemma rely on manual curation to map free-text descriptors (e.g., mouse strains, cell lines) to controlled ontology terms. This process is time-consuming, costly, and prone to human error.
• The Gap: While Large Language Models (LLMs) show promise in natural language understanding, their efficacy in large-scale, domain-specific biomedical metadata curation—specifically for entity-to-ontology mapping—has not been systematically evaluated against human-curated ground truth at scale.

2. Methodology

The authors evaluated OpenAI's GPT-4o as an assistive tool for annotating two specific entity types: mouse strains and cell lines.

Data Sources

• Reference Dataset: 9,390 manually curated transcriptomic experiments from the Gemma database.
• Input Data: GEO metadata (titles, summaries, designs, sample characteristics) and, where available, the full text of associated journal articles (Methods sections).
• Ontologies:
  • Mouse Strains: 156 terms derived from the Experimental Factor Ontology (EFO) and Gemma Ontology (TGEMO).
  • Cell Lines: 46,032 unique terms from the Cell Line Ontology (CLO) and EFO.

Experimental Design

The study employed a Zero-Shot prompting strategy combined with Retrieval-Augmented Generation (RAG) to ground the model in domain-specific knowledge.

1. Mouse Strain Annotation (Direct Prompting):

   • The prompt included GEO metadata, publication text, and a JSON list of all 156 mouse strain ontology terms (labels, URIs, descriptions).
   • GPT-4o was instructed to output detected strains, their URIs, and verbatim supporting quotes in a structured JSON format.

2. Cell Line Annotation (Two-Stage RAG):

   • Due to the context window limit (128k tokens) preventing the inclusion of all 46,032 cell lines in a single prompt, a two-stage approach was used:
     • Stage 1: GPT-4o identified free-text mentions of cell lines in the input.
     • Stage 2: These mentions were embedded using text-embedding-3-large and compared against a vector database of ontology terms. The top 50 most similar ontology terms were retrieved and fed back to GPT-4o to map the free-text mentions to specific ontology terms.

3. Baseline Comparison:

   • A Regular Expression (Regex) string-matching method was used for mouse strains to compare LLM performance against a simpler computational approach.

4. Evaluation Metrics:

   • Ground Truth: Existing Gemma annotations (manually reviewed and corrected for errors found by the model).
   • Metrics: Precision, Recall, and F1 Score were calculated based on True Positives (TP), False Positives (FP), and False Negatives (FN).
   • Error Analysis: Discrepancies were manually reviewed to distinguish between model errors, human curation errors, and source data inconsistencies.

3. Key Results

Mouse Strain Annotation

• GPT-4o Performance: Correctly annotated 77% of experiments.
  • Average Recall: 0.82
  • Average Precision: 0.82
  • Average F1 Score: 0.82
• Regex Baseline Performance: Correctly annotated only 6% of experiments.
  • While recall was high (0.70), precision was extremely low (0.32) due to false positives from common English words or partial matches.
• Error Correction: GPT-4o identified and helped correct 230 existing errors in Gemma's manual curation, primarily caused by inconsistent reporting between GEO metadata and publications.

Cell Line Annotation

• GPT-4o Performance: Correctly annotated 59% of experiments.
  • Average Recall: 0.72
  • Average Precision: 0.72
  • Average F1 Score: 0.72
• Analysis: Lower performance compared to mouse strains is attributed to the massive size of the Cell Line Ontology (46k terms) and the high lexical complexity of cell line names (short alphanumeric codes). The two-stage RAG process introduced retrieval errors where the correct term was not ranked in the top 50 candidates.

Error Characterization

• Nature of Errors: Model errors closely resembled human errors, often stemming from:
  • Typos in source text (e.g., "C57/Bl6" instead of "C57BL/6").
  • Inconsistent naming conventions.
  • Discrepancies between GEO records and publications.
• Hallucinations: Occasional hallucinations occurred (e.g., inferring a specific strain not explicitly mentioned), but the model consistently provided accurate supporting quotes. This allowed human curators to quickly verify and correct the error.

4. Key Contributions

1. Systematic Evaluation: Provided the first large-scale assessment (9,000+ experiments) of GPT-4o for ontology-based curation in genomics, moving beyond small-scale sentence-level evaluations.
2. Demonstration of Assistive Value: Showed that while LLMs cannot yet fully replace human curators, they significantly outperform traditional string-matching methods and can effectively support humans.
3. Human-in-the-Loop Workflow: Proposed a workflow where LLMs perform initial screening and entity normalization, providing supporting evidence (quotes) for rapid human validation.
4. Discovery of Curation Errors: Demonstrated that LLMs can identify inconsistencies in high-quality, manually curated databases (Gemma) that human curators missed, highlighting the value of cross-referencing metadata with full-text publications.

5. Significance and Future Directions

• Scalability: The findings suggest that integrating LLMs into curation pipelines can drastically improve the efficiency and quality of metadata annotation for rapidly growing biomedical repositories.
• Limitations: The study notes that performance is currently limited by context window sizes (requiring RAG for large ontologies) and the quality of source data (typos/inconsistencies).
• Future Work: The authors suggest exploring fine-tuning, improved retrieval strategies for large ontologies, and extending this approach to other metadata categories (e.g., tissues, diseases, drugs).
• Conclusion: LLMs are ready to serve as powerful assistive tools in a human-in-the-loop workflow, bridging the gap between the volume of genomic data and the capacity for manual curation.