📊 statistics

Mortality Forecasting as a Flow Field in Tucker Decomposition Space

This paper proposes a novel mortality forecasting method that reframes the problem as integrating a flow field through the low-dimensional score space of a Tucker tensor decomposition, achieving significantly lower bias and error across ages, horizons, and sexes compared to traditional Lee-Carter, Hyndman-Ullah, and UN World Population Prospects models.

Original authors: Samuel J. Clark

Published 2026-03-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Samuel J. Clark

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Predicting the Future of Human Lifespans

Imagine you are trying to predict how fast a car will be driving 50 years from now.

The Old Way (The "Time Machine" Approach):
Most current methods (like Lee-Carter or the UN's standard model) look at the car's speed over time. They draw a straight line through the past speeds and say, "If the car was speeding up by 1 mph every year, it will keep speeding up by 1 mph forever."

The Problem: This assumes the rules of the road never change. If the car hits a traffic jam (a pandemic) or runs out of gas (biological limits), the straight line prediction fails. Also, these methods often predict the average speed (life expectancy) first, and then try to guess how that speed is distributed across different parts of the car (different ages), which often leads to messy, inconsistent results.

The New Way (The "Flow Field" Approach):
Samuel Clark's paper proposes a different way to think about it. Instead of looking at time, imagine the car is driving through a landscape or a flow field.

The landscape represents different levels of health. The "bottom" of the valley is high mortality (short lives), and the "top" of the mountain is low mortality (long lives).
Every country is a car driving up this mountain.
The paper argues that no matter where the car starts, it follows a very specific, narrow river (a flow field) as it climbs. The path is predictable.
The only question is: How fast is the car moving up the river right now?

How the Method Works (The "River" Analogy)

The author uses a mathematical tool called Tucker Decomposition. Think of this as a high-tech map that compresses millions of data points (ages, sexes, countries, years) into a simple, 5-dimensional "scorecard."

The River is One-Dimensional:
The paper discovered something amazing: even though the map has 5 dimensions, the cars (countries) are essentially moving in a straight line along a single river. The speed of the car and the shape of the river are tightly linked. If you know how far up the river a country is (its current life expectancy), you can predict exactly what its age-specific death rates look like.
The Speed Limit (The Speed Function):
The model learns a "speed limit" for the river.
- The Insight: Countries don't improve at a constant rate. They improve fastest when they are in the "middle" of the journey (moving from poor health to good health).
- The Catch: As countries get to the very top of the mountain (the healthiest nations like Japan or Sweden), they naturally slow down. It's harder to get from 85 years to 86 than it is to get from 60 to 61.
- The model uses a "speed function" that gets slower as you get closer to the top, preventing the unrealistic prediction that humans will live to 150 years old.
The "Era" Weighting:
The model is smart about when it looks at history. It knows that the speed of improvement in 1950 (when antibiotics were new) was different from the speed in 2020. It gives more weight to recent history and less weight to the distant past, so the prediction feels "current."

Why This is Better Than the Competition

The paper compares this new "Flow Field" method against three other major methods:

Lee-Carter: The classic straight-line method.
Hyndman-Ullah: A slightly more complex version of the classic.
pyBayesLife: A re-implementation of the UN's official method (which predicts life expectancy first, then tries to guess the age details).

The Results:

Accuracy: All methods are roughly equally good at predicting the average life expectancy for the next 10 years.
The Long Game (50+ years): This is where the Flow Field wins.
- The old methods (Lee-Carter) keep drawing that straight line and predict life expectancy will shoot up to 100+ years, which is likely too optimistic (or pessimistic, depending on the method). They have a huge bias (systematic error).
- The Flow Field stays on the "river." It knows that as countries get healthier, progress slows down. It predicts a more realistic future.
The "Age" Problem:
- The UN method (pyBayesLife) predicts the average life first, then tries to "reverse engineer" the details for every single age (babies, teenagers, seniors). This is like trying to guess the shape of a puzzle by looking at the box cover. It often gets the details wrong.
- The Flow Field predicts the entire puzzle at once. It predicts the death rates for a 5-year-old and a 90-year-old simultaneously. This means the predictions are coherent: it never predicts that men will suddenly live longer than women at age 20 if that contradicts history.

The "PyBayesLife" Side Quest

The author also rewrote the UN's official software (BayesLife) from scratch in Python because the original R software was "locked" with data from countries that weren't part of the study. This new version (pyBayesLife) was used as a fair referee. Even with this improved referee, the Flow Field method still won on accuracy for age-specific details.

The Takeaway for Regular People

Imagine you are planning a pension fund or a hospital system for the next 50 years.

Old Methods might tell you: "People will live to 95, so build massive hospitals." (Risk: You might overbuild if they only live to 90).
The Flow Field tells you: "People will live longer, but the rate of improvement is slowing down. They will likely reach 92, not 95."

Why it matters:
The Flow Field method is more honest. It doesn't assume the future will be a perfect straight line from the past. It respects the "physics" of human biology—that getting healthier gets harder the healthier you already are. By navigating this "flow field," it provides a more stable, less biased, and more realistic map for the future of human longevity.

In short: Instead of guessing the future by drawing a straight line, this method finds the winding river that all countries are actually swimming in, and predicts where the current will take them next.

1. Problem Statement

Mortality forecasting has historically followed two distinct paradigms:

Time-Series Extrapolation (e.g., Lee-Carter, Hyndman-Ullah): These methods decompose mortality rates (often via SVD) and extrapolate temporal components using stochastic models (e.g., Random Walk with Drift). While effective for short horizons, they often suffer from systematic bias at long horizons (50+ years) and can produce structurally implausible outcomes (e.g., negative mortality, sex crossovers).
Level-Dependent Scalar Forecasting (e.g., UN WPP/bayesLife): This approach forecasts life expectancy ( $e_0$ $e_{0}$ ) as a scalar using a double-logistic function dependent on the current mortality level. The age-specific rates are then reconstructed using a separate model life table system.
- The Core Bottleneck: The scalar approach creates an information bottleneck. It compresses the entire $Sex \times Age$ mortality surface into a single number ( $e_0$ ) and then attempts to reconstruct the high-dimensional surface. This discards information about age patterns, sex differentials, and country-specific structural deviations, leading to significant errors in age-specific forecasts even if the $e_0$ forecast is accurate.

The paper proposes a unified framework that treats mortality transition not as a time-series extrapolation or a scalar recovery problem, but as a flow through a structured, low-dimensional space.

2. Methodology: The Flow-Field Approach

The proposed method integrates tensor decomposition with dynamical systems theory.

A. Data Representation: Tucker Decomposition

The Human Mortality Database (HMD) data is organized as a 4-way tensor $M \in \mathbb{R}^{S \times A \times C \times T}$ (Sex, Age, Country, Year).

Decomposition: A rank- $(2, 42, 46, 100)$ Tucker decomposition is applied to the logit-transformed mortality rates ( $\logit(q_x)$ ).
Effective Core: For any country-year, the mortality schedule is determined by an effective core matrix $G_{ct}$ .
Dimensionality Reduction: PCA on the vectorized effective core matrices reveals that 5 principal components (PCs) capture 97.1% of the variance.
- PC1 ( $s_1$ ): Represents the mortality level (highly correlated with $e_0$ ).
- PCs 2–5 ( $s_2, \dots, s_5$ ): Represent the structural shape of the mortality schedule (age patterns and sex differentials).

B. The Flow-Field Dynamics

The authors demonstrate that the derivatives of these 5 PCs are tightly correlated ( $r \approx -0.92$ between $\Delta s_1$ and $\Delta s_2$ ), implying that the mortality transition is essentially a one-dimensional flow through the 5D score space.

Navigation Coordinate: The system navigates in $s_1$ space (the level score) rather than calendar time or $e_0$ .
Speed Function ( $g^*(s_1)$ ): A nonparametric, era-weighted function estimates the rate of change $ds_1/dt$ . It uses a truncated exponential kernel to weight recent data more heavily, avoiding bias from historical eras with different improvement rates.
Trajectory Functions ( $f^*_k(s_1)$ ): For $k=2 \dots 5$ , these functions map the level score $s_1$ to the structural scores. They act as a continuous model life table system in Tucker coordinates.
Score Relaxation: Structural scores ( $s_k$ ) for a specific country do not immediately snap to the canonical trajectory. Instead, they relax toward the canonical path with empirically calibrated half-lives (12–32 years), preserving country-specific structural deviations (e.g., Russia's excess male working-age mortality) for decades.

C. Forecasting Architecture

Advance Level: Update $s_1$ using the era-weighted speed function.
Map Structure: Compute structural scores $s_k$ using trajectory functions and apply score relaxation.
Reconstruct Surface: Map the score vector back to the full $Sex \times Age$ mortality surface using the shared Tucker basis matrices.
Compute $e_0$ : Calculate life expectancy from the reconstructed surface only for reporting. Crucially, $e_0$ is not used as a feedback loop for navigation, eliminating the divergence between "navigation $e_0$ " and "surface $e_0$ ."

D. Tail Extension

To forecast beyond observed mortality frontiers (e.g., $e_0 > 85$ ), the authors implement a joint tangent extension. At a transition point ( $s_1 \approx -12$ , $e_0 \approx 78$ ), the LOWESS fits are extended linearly using the empirical slope of frontier countries (Japan, Sweden, etc.), ensuring monotonic improvement without saturation.

3. Key Contributions

Unified Framework: The first method to unify tensor decomposition (multi-population structure) and level-dependent forecasting (dynamics) into a single coherent system where the forecast is the reconstruction.
Direct Surface Prediction: By forecasting the full $Sex \times Age$ surface directly in score space, the method avoids the information loss inherent in the "scalar $e_0$ $\to$ age schedule" pipeline.
Structural Coherence: The shared Tucker basis matrices guarantee that all forecasts are structurally coherent (smooth age profiles, no negative rates, consistent sex differentials) by construction.
pyBayesLife Reimplementation: The authors created pyBayesLife, a de novo Python reimplementation of the UN's R-based bayesLife and MortCast pipelines. This was necessary because the R packages had hard-coded dependencies on World Population Prospects (WPP) data and internal datasets that prevented a fair, HMD-only comparison. This reimplementation also improved numerical stability and MCMC efficiency.
Bias Correction: The system achieves near-zero systematic bias in long-horizon forecasts by anchoring the speed function to cross-sectional data rather than extrapolating temporal trends.

4. Results

The system was evaluated using leave-country-out cross-validation (48 HMD countries, 50-year horizon, 9,507 test points) and compared against Lee-Carter (LC), Hyndman-Ullah (HU), and the re-implemented pyBayesLife.

A. Life Expectancy ( $e_0$ ) Accuracy

MAE: Flow-field: 4.100 years. (Comparable to LC: 3.818, HU: 4.094, pyBayesLife: 4.764).
Bias: The flow-field has the lowest aggregate bias (+1.058 years).
- LC and HU show large negative bias (-3.2 to -3.5 years), underpredicting longevity.
- pyBayesLife shows large positive bias (+3.3 years), overpredicting longevity.
Horizon Performance: At short horizons (1–5 years), time-series methods (LC/HU) are more accurate due to jump-off adjustments. However, at long horizons (26–50 years), the flow-field significantly outperforms all others (MAE 5.01 vs. 6.0–7.1), as it does not drift into implausible territory.

B. Age-Specific Mortality Accuracy

This is the most significant finding. The comparison was made on 1.66 million age-specific test points.

MAE ( $l_x$ -weighted log( $m_x$ )):
- Flow-field: 0.424
- pyBayesLife: 1.143
- Improvement: The flow-field is 2.7 times more accurate.
Bias Structure:
- Flow-field bias is near-zero across all ages and horizons.
- pyBayesLife exhibits strong, structured bias (underpredicting mortality at young ages, overpredicting at old ages) due to the reconstruction bottleneck.
Sex Differential: The flow-field reproduces the male-female mortality gap far more accurately because both sexes emerge from the same surface, whereas pyBayesLife reconstructs them separately, introducing errors in the differential.

5. Significance and Implications

Policy Impact: Systematic bias in mortality forecasting leads to structurally underfunded pension systems (if life expectancy is underpredicted) or overestimated solvency (if overpredicted). The flow-field's low bias makes it superior for long-range planning (50–75 years).
Architectural Insight: The results prove that the "scalar $e_0$ bottleneck" is the primary source of error in current UN-style pipelines. Collapsing a high-dimensional surface to a scalar and attempting to recover it discards irretrievable information.
Generalizability: The method can be applied to non-HMD populations (Tier 1: $e_0$ only; Tier 2: age-specific rates) by mapping them into the flow field, providing structurally coherent forecasts even for data-sparse regions.
Scientific Debate: The flow-field's behavior at the mortality frontier (decelerating improvement rates) provides empirical support for the existence of a biological limit or a plateau in life expectancy gains, contributing to the debate on human lifespan limits.

In summary, the paper presents a paradigm shift from "extrapolating time" to "navigating a structured space," demonstrating that direct surface prediction in a low-dimensional tensor space yields superior accuracy, coherence, and bias properties compared to traditional scalar-mediated pipelines.