📊 statistics

Forecasting sub-population mortality using credibility theory

This paper extends classical credibility theory to forecast mortality rates for small sub-populations by deriving explicit predictors that optimally balance reliable super-population trends with limited sub-population data, thereby minimizing forecast error without relying on a specific super-population model structure.

Original authors: Mathias Lindholm, Gabriele Pittarello

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mathias Lindholm, Gabriele Pittarello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how long people will live in a specific, small town. You have two sources of information:

The National Report: A massive, highly reliable study covering the entire country. It has millions of data points, so its predictions are very stable and trustworthy.
The Local Diary: A small notebook kept by the town mayor. It only has data for 500 people. Because the sample is so small, the numbers jump around wildly. If three people in the town happen to get sick in one year, the local diary might predict a terrible future, even if that was just bad luck.

The Problem:
If you rely only on the National Report, you might miss the unique quirks of your small town (maybe they have cleaner air or a different diet). If you rely only on the Local Diary, your prediction will be a mess of noise and random fluctuations.

The Solution (Credibility Theory):
This paper introduces a "smart blending" technique called Credibility Theory. Think of it as a dimmer switch or a volume knob that automatically adjusts how much you trust the National Report versus the Local Diary.

How the "Dimmer Switch" Works

The authors created a mathematical formula that acts like a wise referee. It looks at the Local Diary and asks: "Is this data reliable enough to stand on its own?"

Scenario A: The Tiny Village (Low Credibility)
Imagine a village with only 50 people. The data is too shaky. The "dimmer switch" turns the Local Diary down to almost zero and turns the National Report up to 100%.
- Result: The prediction looks exactly like the National Report. Why? Because with so few people, the local noise is too dangerous to trust.
Scenario B: The Medium Town (Medium Credibility)
Imagine a town with 5,000 people. The data is getting better, but still has some wobbles. The "dimmer switch" splits the difference. Maybe it trusts the National Report 60% and the Local Diary 40%.
- Result: The prediction is a smooth blend. It follows the national trend but gently bends to account for the town's specific habits.
Scenario C: The Large City (High Credibility)
Imagine a city with 100,000 people. The local data is rock solid. The "dimmer switch" turns the National Report down and the Local Diary up to 100%.
- Result: The prediction relies almost entirely on the local data because it's now reliable enough to override the general national average.

The "Stochastic Process" Twist

Most previous methods assumed that mortality rates were static (like a fixed number). However, the authors realized that mortality is more like a weather pattern. It changes over time, driven by hidden forces (like pandemics, medical breakthroughs, or climate change).

They extended their "dimmer switch" to handle this moving target. They treat the future not as a fixed number, but as a cloud of possibilities (a stochastic process).

The Analogy: Instead of just predicting "It will be 70 degrees tomorrow," they predict "It will likely be between 65 and 75 degrees, but here is how much we trust our local thermometer vs. the national weather station."

Why This Matters

The paper proves that this "smart blending" method is mathematically superior to trying to guess the future using only the small local data.

It prevents overreaction: If a small town has a weird spike in deaths one year, the model won't panic and predict a disaster. It says, "That's probably just random noise; let's stick closer to the national trend."
It captures uniqueness: If a town consistently lives longer than the national average, the model slowly learns to trust the local data more, eventually giving that town its own unique forecast.

The Bottom Line

This paper gives us a new tool for forecasting the future of small groups. It's like having a GPS that knows when to trust the satellite map (the big data) and when to trust the local driver's knowledge (the small data).

By mathematically balancing these two sources, we can make much safer, more accurate predictions for small communities, insurance companies, and governments, without getting fooled by random chance.

1. Problem Statement

The paper addresses the challenge of forecasting mortality rates for small sub-populations (e.g., specific demographic groups, regions, or cohorts) that are part of a larger super-population.

The Dilemma: Small sub-populations often lack sufficient historical data or exposure (number of individuals at risk) to produce reliable mortality forecasts when modeled independently. Conversely, relying solely on the super-population model ignores sub-population-specific effects (biological markers, behavioral factors, or demographic variables) that cause deviations from the general trend.
The Gap: Existing multi-population models (e.g., Li-Lee) often rely on complex parameter sharing or strong Bayesian priors. Classical credibility theory, typically used in non-life insurance for static claim frequencies, has not been fully adapted to handle mortality rates driven by latent stochastic processes (like the time-varying period effects in Lee-Carter models).

2. Methodology

The authors extend classical credibility theory to the context of stochastic mortality modeling. The core approach involves modeling the sub-population mortality rate as a weighted average of a global model and a sub-population-specific relative survival model.

A. Statistical Framework

Data Generation: The model assumes a Poisson death count structure:
$D_{x,t}^i \mid E_{x,t}^i, \Theta_x^i, \mu_{x,t} \sim \text{Poisson}(E_{x,t}^i \Theta_x^i \mu_{x,t})$
Where:
- $D_{x,t}^i$ and $E_{x,t}^i$ are deaths and exposure for sub-population $i$ , age $x$ , time $t$ .
- $\mu_{x,t}$ is the global mortality rate (estimated from the super-population), modeled as a stochastic process (e.g., via Lee-Carter or APC models).
- $\Theta_x^i$ is a random effect representing the sub-population's relative risk, assumed to be independent of the global trend and stable over time.
Credibility Estimator: The authors derive a linear credibility estimator for the future mortality rate $\hat{\mu}_{x,t'+h}(\Theta_x^i)$ . The estimator is a weighted average:
$\hat{\mu}_{x,t'+h}(\Theta_x^i) = (1 - z_x^i)\mu_{x,t'+h} + z_x^i (\mu_{x,t'+h} R_x^i)$
Where:
- $\mu_{x,t'+h}$ is the forecasted global mortality rate.
- $R_x^i$ is a relative survival factor estimated from the sub-population's historical data.
- $z_x^i$ is the credibility weight, determined by the exposure size and the variance of the random effect $\Theta_x^i$ .

B. Key Theoretical Derivations

Handling Stochasticity: Unlike classical credibility which assumes static means, this framework accounts for the stochastic nature of the global mortality trend ( $\mu_{x,t}$ ) and the Poisson variation in death counts.
Explicit Error Formulas: The authors derive an explicit expression for the expected quadratic forecast error, allowing for the quantification of uncertainty in the predictions.
Plug-in Estimation: Since the theoretical estimator relies on unknown conditional expectations and variances, the paper proposes a computable "plug-in" version using method-of-moments estimators for the variance of the random effects ( $\text{Var}(\Theta_x^i)$ ).

3. Key Contributions

Extension of Credibility Theory: The paper successfully adapts credibility theory from static insurance pricing to dynamic, stochastic mortality forecasting. It bridges the gap between multi-population models and credibility theory.
Model Agnosticism: The credibility predictor is independent of the specific form of the super-population model. Whether the global trend is modeled via Lee-Carter, APC, or Renshaw-Haberman, the credibility adjustment mechanism remains valid.
Automatic Calibration: The credibility weight ( $z_x^i$ $z_{x}^{i}$ ) automatically adjusts based on data availability.
- If exposure is low or variance is high, the model leans heavily on the global model (borrowing strength).
- If exposure is high and the sub-population deviates significantly, the model leans toward the sub-population specific model.
Uncertainty Quantification: The derivation of the expected quadratic forecast error provides a rigorous way to calculate prediction intervals that account for both parameter uncertainty and process variation.

4. Results

The methodology was validated through a simulation study using Italian mortality data (Human Mortality Database) split into three sub-populations of varying sizes (Small, Medium, Large).

Single Dataset Illustration:
- Weight Dynamics: As more historical data (calendar years) were added, the credibility weights shifted smoothly from the global model toward the sub-population specific model.
- Age Dependence: For older ages (where death counts are higher), the credibility weights increased, pulling the forecast closer to the relative survival model. For younger ages (low counts), the forecast remained close to the global model.
- Small Populations: For the smallest sub-population (500 individuals), the model effectively prevented overfitting by relying heavily on the global trend, whereas larger sub-populations could support more specific modeling.
Out-of-Sample Performance (30 Simulations):
- The credibility approach (Model A) consistently outperformed or matched the separate sub-population model (Model C), which often suffered from convergence issues or high variance due to small sample sizes.
- The credibility approach acted as an optimal compromise: it performed similarly to the global model (Model D) for low-exposure groups but captured specific trends better for high-exposure groups, often outperforming the global model in terms of Poisson deviance and Mean Squared Error (MSE).
- Benchmarking: The credibility model generally outperformed the "separate model" approach, particularly for the smallest sub-population where independent modeling failed.

5. Significance and Implications

Practical Utility: The method provides a robust, data-driven solution for actuaries and demographers dealing with small groups (e.g., specific occupational groups, small regions, or niche demographic segments) where traditional modeling fails due to data scarcity.
Theoretical Advancement: It demonstrates that credibility theory can be successfully applied to complex, time-series-driven mortality models, offering a simpler alternative to fully Bayesian multi-population models which require strong prior assumptions.
Risk Management: By providing explicit formulas for forecast error, the approach enhances risk management capabilities for pension funds and insurers, allowing for more accurate capital requirements and pricing for small sub-portfolios.

In summary, the paper proposes a flexible, mathematically rigorous framework that balances the stability of a global mortality trend with the specificity of local data, automatically calibrating the balance based on the reliability of the available data.