Longevity Bench: Are SotA LLMs ready for aging research?

This paper introduces LongevityBench, a comprehensive benchmark designed to evaluate state-of-the-art large language models on their ability to interpret diverse aging-related biodata and grasp fundamental biological principles of aging, ultimately revealing their current limitations and outlining strategies to enhance their utility in longevity research.

The Gist

Imagine you have a group of very smart, super-advanced robots (the AI models) that can read almost everything ever written on the internet. They can write poetry, solve math problems, and chat like humans. But here's the big question: Do they actually understand how living things work, or are they just really good at guessing based on patterns they've seen before?

To find out, the scientists at Insilico Medicine created a special "final exam" called LongevityBench.

Think of this exam like a driver's test for aging. Just as a driving test checks if you can actually handle a car in rain, snow, and traffic—not just recite the rules of the road—LongevityBench checks if these AI robots truly understand the biology of getting old.

The Test Subjects: The "Super-Students"

The researchers picked 15 of the smartest AI models currently available (like the latest versions of GPT, Gemini, Claude, and others). They threw a massive pile of biology homework at them. This wasn't just reading a textbook; it was real-world data.

The homework covered five different "subjects":

1. The Crystal Ball (Survival): Looking at a person's medical records and blood tests, can the AI guess how long they will live?
2. The Time Machine (Age Prediction): Looking at a sample of DNA or blood proteins, can the AI tell you exactly how old the person is?
3. The Genetic Puzzle (Mutations): If you change a specific gene in a fly or a mouse, will they live longer or shorter? Can the AI predict the outcome?
4. The Cancer Detective: Looking at tumor data, can the AI guess which patient will stay healthy longer?
5. The Gene Writer: Can the AI look at a partial list of active genes and "fill in the blanks" with the rest of the list correctly?

The Results: Who Passed?

The results were a mix of "brilliant" and "confused."

• The Top Performers: The Google Gemini 3 Pro and OpenAI's GPT-5/o3 models came out on top. They were like the valedictorians of the class, getting the highest average scores. They were particularly good at looking at medical records and guessing survival times.
• The "One-Trick Ponies": Some models were amazing at one thing but terrible at another. For example, Claude Sonnet was the best at predicting cancer survival but struggled with other tasks. It's like a student who is a genius at math but fails history.
• The "Random Guessers": When the test got really hard—like trying to predict age just by looking at a list of proteins (proteomics)—most of the AIs dropped to near-random guessing. It was as if they were just flipping a coin.

The Big Surprise: The "Question Trick"

The most interesting part of the paper wasn't just who got the best grades, but how they took the test.

The researchers noticed that the AI's performance changed wildly depending on how the question was asked.

• The Analogy: Imagine asking a student, "Who is taller: Alice or Bob?" They might get it right. But if you ask, "Who is shorter: Alice or Bob?" they might get it wrong, even though the answer is the same.
• The Reality: The AI models were very sensitive to the wording. If you asked them to pick the older person between two people (a "pairwise" choice), they often failed. But if you asked them to guess the age group (e.g., "Is this person in their 60s or 70s?"), they did much better.

This suggests the AI isn't building a deep, internal "map" of how aging works. Instead, they are often just recognizing specific patterns in the question format. If you change the format, their "knowledge" seems to vanish.

The "Compression" Problem

There was another weird glitch. When the AI tried to guess exactly how many months a person had left to live, they almost always underestimated it.

• The Metaphor: It's like a weather forecaster who, no matter the data, always predicts it will rain tomorrow. Even if the sky is blue, they say "rain."
• The Cause: The AI seems to be trained on so much text about "disease" and "death" that whenever it sees a medical record, it panics and assumes the worst, ignoring the fact that many people live long lives despite having health issues.

The Verdict: Are They Ready for Science?

Not quite yet.

The paper concludes that while these AI models are incredibly powerful tools for writing code, summarizing papers, or brainstorming ideas, they are not yet ready to be trusted as independent scientists in the field of aging.

• They are great at: Reading medical records and giving a "best guess" on survival.
• They are bad at: Understanding the deep, complex mechanics of how genes and proteins interact to cause aging, especially when the data is messy or the question is phrased differently.

The Takeaway

The scientists aren't saying "stop using AI." They are saying, "Use AI, but check its work."

Think of LongevityBench as a report card. It tells researchers, "Hey, this AI is good at math, but don't let it drive the car yet." The goal now is to use these test results to train the next generation of AI so they don't just memorize facts, but actually understand the biology of life and death.

In short: The robots are smart, but they still need a human teacher to make sure they aren't just making things up.

Technical Summary

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into biomedical research workflows has raised concerns regarding their actual utility versus superficial pattern matching. While LLMs excel at text generation and code assistance, there is a lack of empirical evidence demonstrating whether they possess a coherent internal understanding of complex biological processes, specifically aging.

• The Gap: Aging biology involves multi-scale data (from molecular omics to clinical phenotypes) and requires reasoning about low-level biodata to derive high-level conclusions (e.g., predicting time-to-death from blood markers). No systematic benchmark existed to evaluate if foundation models can reliably interpret biodata in the context of aging.
• The Risk: Overreliance on models that fail to grasp biological ground truth could lead to erroneous scientific conclusions.

2. Methodology: LongevityBench Framework

The authors introduced LongevityBench, a comprehensive evaluation framework designed to test 15 State-of-the-Art (SotA) LLMs across 17 distinct tasks. The benchmark is built on the premise that a model capable of "understanding" aging should perform consistently across different data modalities and question formats.

A. Dataset Composition

The benchmark aggregates 30,193 prompts (approx. 50 million tokens) derived from seven diverse, publicly available data sources:

1. General Population Survival: NHANES-IV (1999, 2001) data including demographics, biometrics, and blood biomarkers.
2. Aging Trajectories: Open Genes database for gene expression changes in human tissues (mRNA level).
3. Multi-mutant Lifespan: Synergy Age database (Drosophila and Mouse) for genetic intervention effects.
4. Cancer Survival: TCGA data (KICH, LUAD, SKCM) with RNAseq and clinical diagnosis.
5. Methylation Aging: GEO studies (Illumina Human Methylation arrays) for epigenetic age prediction.
6. Transcriptomic Aging: GTEx portal data for multi-tissue RNA expression.
7. Proteomic Aging: Olink Explore 3072 plasma proteomics dataset.

B. Task Formats

To distinguish between rote memorization and genuine reasoning, tasks were presented in multiple formats:

• Binary/Multi-class Classification: (e.g., "Did the patient survive >10 years?" or "Select the correct age bin").
• Pairwise Comparison: (e.g., "Which of these two patients will live longer?" or "Which sample is older?").
• Regression: Predicting exact numerical values (e.g., Time-to-Death in months, % lifespan increase).
• Generative: Predicting sets of genes/proteins given partial profiles (e.g., "Generate the remaining top 50 expressed genes").

C. Evaluation Metrics

• Classification/Pairwise: Unadjusted Accuracy.
• Regression: Mean Absolute Error (MAE).
• Generative: Jaccard Index (overlap between predicted and ground-truth gene sets).

D. Models Evaluated

15 models from major developers were tested:

• OpenAI: GPT-4.1, GPT-4.1-mini, GPT-5, GPT-5-mini, GPT-5.2, o3, o4-mini.
• Google: Gemini 2.5 Pro/Flash, Gemini 3 Pro/Flash Preview.
• Anthropic: Claude Sonnet 4.5.
• xAI: Grok-3, Grok-4.
• DeepSeek: DeepSeek-R1-0528.

3. Key Results

A. Overall Performance

• No Universal Dominance: No single model achieved top-3 performance across all 17 tasks.
• Top Performers: Google's Gemini 3 Pro achieved the best average rank (5.0), followed closely by OpenAI's GPT-5 and o3 (both 5.06).
• Modality Variance: Performance was highly heterogeneous. A model excelling in clinical survival prediction might fail completely in proteomic generation.

B. Specific Domain Findings

• Clinical Survival (NHANES): Models performed well on binary survival tasks (accuracy >0.85 for top models like Gemini 3 Pro). However, pairwise comparisons (who lives longer?) dropped significantly (0.60–0.67), and regression tasks showed systematic underestimation of long-term survival (compressing predictions to 50–100 months).
• Omics (Transcriptomics/Proteomics):
  • Transcriptomics: Pairwise age prediction was near random chance (0.48–0.55), suggesting models struggle to extract aging signals from raw gene expression profiles in this format. However, multiclass binning improved performance slightly.
  • Proteomics: This was the most challenging domain. Even top models achieved near-random accuracy in binary classification and extremely low Jaccard indices (<0.03) in generative tasks, likely due to the scarcity of proteomic data in training corpora.
• Genetics: In multi-mutant lifespan tasks, models improved when provided with extended biological context (Gene Ontology annotations), though some models (e.g., o3) remained stable regardless of context, suggesting internalized knowledge.

C. Format Dependence (Critical Finding)

The study revealed that model rankings are unstable across question formats.

• A model might rank #1 in a multiclass age prediction task but #10 in a pairwise comparison of the same data.
• This inconsistency suggests that current LLMs lack a coherent internal representation of aging biology and instead rely on superficial correlations specific to the prompt structure.

4. Key Contributions

1. First Specialized Benchmark: Introduction of LongevityBench, the first systematic framework to evaluate LLMs specifically on aging biology across clinical, genomic, epigenomic, and proteomic domains.
2. Empirical Evidence of Limitations: Demonstrated that SotA LLMs struggle with:
   • Extracting aging signals from raw omics data (especially proteomics).
   • Generalizing across question formats (format bias).
   • Accurate regression of long-term survival times.
3. Framework for Improvement: Proposed the MMAI Gym for Science, an iterative pipeline using LongevityBench to guide Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) to align models with biological ground truth.
4. Leaderboard & Resources: Established a public leaderboard (insilicobench.insilico.com) to track progress as models evolve.

5. Significance and Implications

• For Researchers: The paper provides a critical "reality check." Researchers should not rely on general-purpose LLMs for complex aging analysis without rigorous validation. Specific models should be selected based on the specific data modality (e.g., Gemini 3 Pro for clinical survival, but caution advised for proteomics).
• For AI Development: The findings highlight that "fluency" does not equal "biological understanding." Future model training must move beyond text prediction to include structured reasoning over low-level biodata to achieve true scientific utility.
• Future Directions: The authors emphasize that LongevityBench is a starting point. Future iterations will include chemical screening, cross-species translation, and cost/latency metrics to create a holistic tool for AI-driven life science research.

Conclusion: While current LLMs show promise in specific niches (like clinical record interpretation), they are not yet ready for autonomous, high-stakes aging research due to significant gaps in omics interpretation, format robustness, and mechanistic reasoning. LongevityBench serves as a necessary diagnostic tool to bridge this gap.