🧬 biology

Pre-Transformer Models for Longevity Science and Deep Ageing Clocks: An Empirical Analysis

This empirical study demonstrates that for biological-age prediction across diverse ageing cohorts, pre-transformer models (such as penalized regression and gradient boosting) generally outperform or match attention-based transformers in accuracy, data efficiency, cross-cohort transfer, and mortality risk stratification, establishing them as the rational default for longevity science while positioning transformers as a specialized tool only viable with exceptionally large datasets.

Original authors: John Feng

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: John Feng

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to guess how old a person really is, not by looking at their driver's license, but by reading the tiny chemical notes written on their cells, blood, or even their brain scans. Scientists call this "biological age." While your birthday tells you how long you've been on Earth, your biological age tells you how fast your body is actually wearing out. Some people are 50 but feel like 30; others are 30 but feel like 50. To figure this out, researchers build "ageing clocks"—computer programs that take a bunch of messy data (like DNA tags or blood sugar levels) and spit out an age estimate. The difference between this estimate and the person's real birthday is called "age acceleration," and it's a superpower: it can predict who might get sick or pass away sooner.

For years, the most popular way to build these clocks was using older, simpler math tools. But recently, a new, super-complex type of AI called a "Transformer" has taken over the tech world, becoming the star of everything from writing stories to recognizing images. Naturally, scientists asked: "Should we swap our old, reliable clocks for these flashy new Transformers?" It's like asking if we should replace a trusty, fuel-efficient bicycle with a massive, high-tech rocket ship just to get to the grocery store. The big question is: does the rocket actually get us there faster, or does it just burn a ton of fuel and crash because the road is too bumpy?

This paper is the ultimate race track test to find the answer. The author, John Feng, set up a giant, controlled tournament. He took six different groups of people and five different types of data (from DNA to brain images) and ran eight different types of computer models against each other. He didn't just see which one guessed the birthday closest; he also checked which one was cheapest to run, which one was easiest to understand, and—most importantly—which one actually predicted who would die sooner.

The results were a bit of a shock to the tech hype machine. The old-school, simpler models (like "penalized regression" and "gradient boosting") won the race in five out of the six categories. They were accurate, cheap, and their predictions actually matched up with real-life health risks. The new Transformers? They mostly stumbled. On the small, messy datasets that scientists usually have, the Transformers often got confused and made worse guesses than the simple models. They only managed to win in one specific case: a huge dataset with 34,000 people, and even then, they only beat the old models by a tiny, almost invisible margin of 0.10 years.

Here is the twist that makes the story really interesting: the model that was the best at guessing the birthday (the pretrained Transformer) was actually the worst at predicting who was at risk of dying. It was like a student who memorized the textbook perfectly but failed the real-world test. The simpler models, while not always the absolute fastest at guessing the number, were much better at spotting the real biological danger signs.

The paper concludes that for the world of ageing research, the "pre-Transformer" toolkit (the older, simpler models) is still the smartest choice. It's the bicycle that gets you there reliably. The Transformers are powerful, but they are like that rocket ship: they only make sense if you have a massive amount of data and unlimited fuel, and even then, they might not get you to the right destination. The authors suggest that we shouldn't throw away our old tools just because the new ones look cooler; sometimes, the simple, boring math is exactly what saves the day.

Technical Summary: Pre-Transformer Models for Longevity Science and Deep Ageing Clocks

Problem Statement

The field of computational biogerontology relies on "ageing clocks"—models that map high-dimensional molecular or physiological data to an estimate of biological age. The deviation from chronological age (age acceleration) serves as a predictor for morbidity and mortality. While the field has historically utilized pre-transformer methods (penalized regression, kernel machines, tree ensembles, and pre-2017 deep networks), the dominance of attention-based transformers in general artificial intelligence raises a critical question: do transformers outperform these established methods on the specific data conditions of ageing research?

Current literature often argues a priori that ageing cohorts (small sample sizes, high dimensionality, noisy labels, and regulatory requirements) favor parsimonious models, but this has lacked empirical validation through head-to-head comparisons. This paper addresses the gap by conducting a controlled benchmark to determine if the migration to transformers is justified or if pre-transformer toolkits remain the rational default.

Methodology

The study establishes a unified benchmark across six public cohorts spanning five data modalities:

DNA Methylation: GSE40279 (blood, 450K array, $n=656$ , $p \approx 20,000$ ).
Blood Biochemistry: NHANES (clinical markers, $n \approx 34,000$ , 34 features).
Transcriptome: GTEx v8 (whole blood, $n \approx 755$ , 5,000 genes).
Gut Microbiome: American Gut (16S profiles, $n \approx 9,500$ , $\sim 1,500$ taxa).
Accelerometry: NHANES (7-day minute-level series, $n \approx 7,000$ ).
Brain MRI: Cam-CAN (T1 structural scans, $n \approx 650$ ).

Model Classes: Eight distinct model classes were evaluated:

Classical Pre-Transformers: Elastic Net, Kernel SVR, Random Forest, Gradient Boosted Decision Trees (GBDT).
Pre-Transformer Deep Nets: Multilayer Perceptron (MLP), and modality-native deep networks (Autoencoder for tabular, ConvLSTM for accelerometry, 3D-CNN for MRI).
Transformers: From-scratch (Tr-scr) and Pretrained-then-Fine-tuned (Tr-pre) variants (FT-Transformer/TabTransformer for tabular, Sequence Transformer for accelerometry, Vision Transformer for MRI).

Protocol:

Evaluation: A single nested cross-validation protocol (5-fold outer loop repeated 10 times) with identical data splits, features, and random seeds for all models.
Hyperparameter Tuning: Optuna (100 trials) used within the inner loop.
Metrics:
- Accuracy: Mean Absolute Error (MAE) and Pearson correlation against chronological age.
- Efficiency: Training compute cost relative to Elastic Net.
- Interpretability: Fraction of top-attributed features that are literature-validated ageing markers.
- Biological Validity: Hazard Ratio (HR) of the age-acceleration residual against all-cause mortality (tested on NHANES).
- Transferability: Performance degradation ( $\Delta$ MAE) when training on GSE40279 and testing on an independent cohort (GSE87571).

Key Results

1. Accuracy Across Modalities

Tabular Omics: On methylation, transcriptome, and microbiome data, penalized regression (Elastic Net) and gradient boosting were the best or tied-best performers.
- Methylation: Elastic Net achieved the lowest MAE (3.42 yr); the from-scratch transformer performed worst (5.22 yr) due to overfitting.
- Transcriptome: GBDT led (8.07 yr); from-scratch transformer trailed (9.90 yr).
Structured Data: On accelerometry and MRI, modality-native pre-transformer deep nets (ConvLSTM and 3D-CNN) outperformed all other classes.
- Accelerometry: ConvLSTM (12.66 yr) vs. Transformers (>13 yr).
- MRI: 3D-CNN (4.74 yr) vs. Vision Transformer (7.48 yr).
Large- $n$ Cohort: The only cohort where a transformer won was NHANES blood chemistry ( $n \approx 34,000$ ). The pretrained transformer achieved a marginally lower MAE (6.40 yr) than GBDT (6.50 yr), a difference of only 0.10 yr.
Aggregate: While the pretrained transformer had the lowest mean gap-to-best across all cohorts (0.18 yr), this aggregate advantage is driven entirely by the single large- $n$ cohort.

2. Data Efficiency and Compute

Break-even Point: On the NHANES cohort, the pretrained transformer only began to outperform gradient boosting at a training sample size of $n \approx 23,000$ . Below this threshold, pre-transformer models were strictly superior.
Compute Cost: The pretrained transformer required approximately 21,000× the compute of an Elastic Net and 23× the compute of a native deep net to achieve a marginal 0.05 yr improvement in the mean gap-to-best. From-scratch transformers were the least efficient.

3. Interpretability and Marker Recovery

Sparse linear models (Elastic Net) and tree ensembles (GBDT) recovered literature-validated ageing markers at significantly higher rates (0.74 and 0.73, respectively) compared to deep and transformer models (0.29–0.46).
Deep and transformer attributions were diffuse across many weakly-weighted inputs, whereas sparse models identified specific, known biomarkers (e.g., serum albumin, C-reactive protein).

4. Biological Validity (Mortality Prediction)

The Paradox: The model with the lowest chronological-age error (the pretrained transformer) carried the weakest mortality signal (Hazard Ratio $\approx 1.13$ per SD).
Risk Stratification: Both transformer variants ranked at the bottom for risk stratification. Conversely, GBDT, which had nearly identical age accuracy to the transformer (6.50 vs. 6.40 yr), demonstrated a much stronger mortality signal (HR $\approx 1.27$ ).
Conclusion: Minimizing chronological-age error does not maximize biological signal; high-capacity attention models appear to learn residuals that are less informative for mortality risk.

5. Cross-Cohort Transfer

When transferring a methylation clock from GSE40279 to GSE87571, the Elastic Net showed the least degradation ( $\Delta$ MAE +1.2 yr).
The from-scratch transformer degraded the most (+4.0 yr), followed by native deep nets (+2.8 yr). The pretrained transformer showed intermediate robustness (+1.9 yr), suggesting self-supervised pretraining aids transfer, but not to the level of sparse linear models.

Significance and Claims

The paper concludes that for the data conditions characterizing human ageing research (small $n$ , high $p$ , tabular omics, and structured imaging), the pre-transformer toolkit is the rational default.

Structural Superiority: Pre-transformer models (penalized regression, GBDT, ConvLSTM, 3D-CNN) are not merely "legacy" tools but are structurally better suited for the $p \gg n$ regime and the specific inductive biases required for structured data.
Transformer Limitations: Transformers do not win on accuracy in typical ageing scenarios. Their only advantage appears in the specific conjunction of large sample sizes ( $n > 23,000$ ) or large unlabelled corpora for pretraining, and even then, the accuracy gain is negligible (0.10 yr) and comes at a prohibitive compute cost.
Biological Validity: The pursuit of minimal chronological-age error via high-capacity models may be counterproductive, as these models produce age-acceleration residuals with weaker associations to mortality.
Recommendation: Attention mechanisms should be viewed as a targeted addition rather than a replacement. They are justified only when specific cohorts provide genuine sequence structure and unlabelled scale that align with the transformer's inductive biases. For the vast majority of ageing research, the pre-transformer toolkit offers the best balance of accuracy, interpretability, transferability, and biological validity.