A Dirichlet-Multinomial-Poisson framework for the coherent analysis and forecast of cause-specific mortality

1. Problem Statement

Traditional approaches to forecasting mortality often model cause-specific death rates independently (e.g., applying the Lee-Carter model to each cause separately). This approach suffers from a fundamental incoherence problem: the sum of the projected deaths by specific causes often does not match the projected total number of deaths in the population.

• Consequences: This leads to implausible long-term projections, violates basic demographic constraints (mutually exclusive and collectively exhaustive causes), and complicates policy planning and resource allocation.
• Statistical Challenge: Modeling mortality by cause requires handling compositional data (proportions summing to 1) while simultaneously modeling the total count of deaths, which is subject to its own stochastic variation. Standard transformations (like log-ratio) used in Compositional Data Analysis (CoDa) can distort rate-based interpretations or fail to propagate uncertainty correctly between totals and components.

2. Methodology: The Dirichlet-Multinomial-Poisson (DMP) Framework

The authors propose a hierarchical probabilistic framework that treats both the total number of deaths and the allocation of deaths across causes as random variables. This ensures coherence by construction.

A. Hierarchical Structure

The model decomposes the mortality process into two linked layers:

1. Total Mortality (Poisson Layer): The total number of deaths ($Y_{a,t}$) for age $a$ and period $t$ follows a Poisson distribution:
   $$Y_{a,t} \sim \text{Poisson}(E_{a,t} m_{a,t})$$
   Where $E_{a,t}$ is exposure and $m_{a,t}$ is the total mortality rate.
2. Cause Allocation (Dirichlet-Multinomial Layer): Conditional on the total deaths $Y_{a,t}$, the vector of deaths by cause ($Y_{a,t,1}, \dots, Y_{a,t,C}$) follows a Dirichlet-Multinomial (DM) distribution.
   • The probability of death by cause $c$ ($\gamma_{a,t,c}$) follows a Dirichlet distribution parameterized by a vector $\alpha_{a,t} = \phi \gamma_{a,t}$.
   • The probabilities are modeled via a softmax function of a latent linear predictor $\eta_{a,t,c}$:
     $$\gamma_{a,t,c} = \frac{\exp(\eta_{a,t,c})}{\sum_{c'} \exp(\eta_{a,t,c'})}$$
   • The DM distribution accounts for overdispersion (extra variation beyond the multinomial) via the concentration parameter $\phi$.

B. Covariate Modeling (Age-Period and Lee-Carter)

The latent predictors ($\eta_{a,t,c}$) and the log-total rate ($\log m_{a,t}$) are modeled using two distinct structures to capture temporal dynamics:

1. Age-Period (AP) Model:
   • Total rate: $\log m_{a,t} = \nu_0 + \delta_a + \pi_t$
   • Cause probabilities: $\eta_{a,t,c} = \phi_{0,c} + \zeta_{a,c} + \lambda_{t,c}$
   • Parameters are smoothed using Second-Order Random Walks (RW2) to prevent overfitting and capture smooth demographic transitions.
2. Lee-Carter (LC) Model:
   • Total rate: $\log m_{a,t} = \nu_0 + \delta_a + \beta_a \kappa_t$
   • Cause probabilities: $\eta_{a,t,c} = \phi_{0,c} + \zeta_{a,c} + \vartheta_c \kappa_t$
   • The period effect $\kappa_t$ is shared between the total mortality level and the cause composition, driven by a First-Order Random Walk (RW1). This allows the model to capture systematic reallocations of deaths across causes driven by the same secular trends affecting total mortality.

C. Inference and Forecasting

• Bayesian Estimation: Parameters are estimated using Hamiltonian Monte Carlo (HMC) via the Stan software. Non-informative priors are used to let the data drive the posterior.
• Coherent Projections: Future forecasts are generated by simulating the stochastic evolution of the period effects ($\pi_t$ or $\kappa_t$) and the cause-specific loadings. Because the model is generative, the sum of projected cause-specific rates is guaranteed to equal the projected total rate, and uncertainty bands naturally propagate from both the total count and the compositional allocation.

3. Key Contributions

1. Unified Coherent Framework: The paper introduces a single probabilistic model that simultaneously models total mortality counts and cause-specific proportions, eliminating the incoherence found in separate modeling approaches.
2. Double Source of Dispersion: By combining Poisson (for totals) and Dirichlet-Multinomial (for composition), the model captures variation in both the volume of deaths and the mix of causes, including overdispersion in the latter.
3. Integrated Uncertainty Quantification: Unlike methods that add uncertainty bands post-hoc, this framework derives predictive uncertainty directly from the joint posterior distribution, providing coherent uncertainty statements for both aggregates and subgroups.
4. Flexibility: The framework is compatible with standard demographic models (AP and LC), allowing researchers to choose the temporal structure that best fits the data while maintaining coherence.

4. Empirical Results

The framework was tested on data from the Human Mortality Database (1979–2023) for the United States and France, separated by sex. The study used a rolling 15-year out-of-sample (OOS) evaluation, comparing the DMP models (DMP-AP and DMP-LC) against:

• Standard Lee-Carter (LC) fitted independently per cause.
• Compositional Data Analysis (CoDa) methods (Kjærgaard et al., 2019).

Key Findings:

• Total Mortility Accuracy: The DMP-LC specification consistently achieved the lowest Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) for total mortality in both countries and sexes. It outperformed the standard LC and CoDa approaches significantly.
• Cause-Specific Performance:
  • DMP-LC and DMP-AP were systematically among the top performers for major causes (e.g., Cardiovascular Diseases, Neoplasms).
  • While CoDa sometimes showed lower errors for specific individual causes (particularly in France for females), it performed poorly on total mortality and lacked the system-level coherence.
  • Standard LC models, when fitted independently, often produced incoherent totals even if individual cause fits were reasonable.
• Stability: The LC-based coherent model (DMP-LC) provided more stable extrapolations over long horizons compared to the AP-based model, which showed higher volatility in out-of-sample forecasts for France.
• Visual Fit: The models successfully tracked observed trends and produced widening, plausible uncertainty intervals for the forecast period, particularly for causes with high variability.

5. Significance and Implications

• Policy Relevance: The framework addresses a critical gap in public health and actuarial science. Policymakers can now rely on forecasts where the sum of specific disease burdens matches the total mortality burden, ensuring resource allocation strategies are based on consistent data.
• Risk Management: For longevity risk management and pension planning, the ability to quantify joint uncertainty across causes and totals is vital. The DMP framework provides a unified distribution for these risks.
• Methodological Advancement: The paper demonstrates that integrating compositional constraints directly into the likelihood function (via Dirichlet-Multinomial) is superior to post-hoc corrections or separate modeling. It validates the use of Bayesian hierarchical modeling for complex demographic forecasting tasks.

In conclusion, the paper establishes the Dirichlet-Multinomial-Poisson (DMP) framework, particularly the DMP-LC variant, as a robust, coherent, and accurate method for forecasting cause-specific mortality, offering a significant improvement over traditional independent modeling approaches.