📊 statistics

A Beta-GAM Hidden Markov Model for Proportion Time Series

This paper proposes a Beta-GAM Hidden Markov Model for proportion time series that combines regime-switching dynamics with flexible spline-based covariate effects, validated through simulation and applied to Russian mortality data to identify latent structural regimes underlying historical mortality shocks.

Original authors: Andrea Nigri, Han Lin Shang, Marco Bonetti

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrea Nigri, Han Lin Shang, Marco Bonetti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a story that is being told by a single, fluctuating number—like the percentage of female deaths in a specific age group over many years. This number doesn't just wiggle randomly; it follows a pattern, but that pattern changes depending on the "mood" of the era. Sometimes the story is calm and predictable; other times, it's chaotic and volatile.

The authors of this paper have built a new mathematical tool, a Beta-GAM Hidden Markov Model, to decode these stories. Here is how it works, explained through simple analogies:

1. The "Chameleon" Storyteller (The Hidden Regimes)

Think of the data (the proportion of deaths) as a chameleon. The chameleon changes its color based on its environment, but you can't see the environment directly; you only see the color.

The Hidden States: In the paper's model, there are invisible "regimes" or "moods" (like a calm mood vs. a chaotic mood). The data switches between these moods over time.
The Switching: The model figures out when the story switches from a "calm era" to a "chaotic era" and back again, even though the switch isn't explicitly marked in the data.

2. The "Flexible Ruler" (The GAM Part)

Usually, when statisticians try to describe how a number changes over time (like age), they use a straight ruler or a simple curve. But real life is rarely a straight line.

The Solution: This model uses a "flexible ruler" made of splines (think of a bendable plastic strip). This allows the model to draw smooth, wiggly curves that perfectly fit the data's shape without forcing it into a straight line.
The Benefit: It can capture complex patterns, like how the gap between male and female mortality widens in young adulthood and narrows later, all while the "mood" of the era stays the same.

3. The "Precision Dial" (The Beta Part)

The data being studied is a proportion (a number between 0 and 1, like 0.45 or 0.80).

The Dial: Imagine a dial that controls how "tight" or "loose" the data points are around the average.
- High Precision: The data points are huddled tightly together like a flock of birds (very predictable).
- Low Precision: The data points are scattered like a flock of birds in a storm (very noisy).
The Innovation: This model lets the "Precision Dial" change depending on which "mood" (regime) the system is in. One era might be very stable (tight flock), while another is chaotic (scattered flock).

4. The "Detective's Filter" (Avoiding False Alarms)

When you try to find hidden moods in data, it's easy to get greedy and invent too many moods (e.g., "Maybe there are 5 moods instead of 2?"). This often leads to nonsense results where the math breaks down.

The Filter: The authors created a special "degeneracy filter." It's like a quality control inspector. If the model tries to invent a mood that doesn't make sense (for example, a mood that is so chaotic the math explodes), the filter kicks it out. This ensures the final answer is stable and real, not a mathematical glitch.

5. The "Confidence Net" (Bootstrap)

How do we know the model is right?

The Method: The authors used a technique called bootstrapping. Imagine they took the story, scrambled the pages slightly, and re-told the story 200 times. If the model gives the same answer every time, they know they can trust it. If the answers jump around wildly, they know to be careful.

The Real-World Test: Russian Mortality

The authors tested this tool on real data: the ratio of female to total deaths in Russia from 1960 to 2014.

What they found: The model successfully identified two distinct eras:
1. The Chaotic Era: A time with high volatility and a deep drop in the ratio (meaning many more men were dying from accidents, violence, or alcohol). This matched the turbulent times of the late Soviet health crisis and the 1990s transition.
2. The Stable Era: A time with a smoother, more predictable pattern where the ratio was higher and less volatile.
The Result: The model didn't just say "it changed"; it showed how the shape of the age-related mortality changed during these different eras and gave a clear timeline of when the shifts happened.

Summary

In short, this paper introduces a smart, flexible way to analyze percentages that change over time. It combines three powerful ideas:

Hidden Moods: It finds invisible shifts in the data's behavior.
Flexible Shapes: It draws smooth, complex curves to fit the data perfectly.
Smart Filtering: It prevents the math from breaking down by rejecting impossible solutions.

The result is a tool that can tell the story of how a population's health patterns evolve, distinguishing between times of stability and times of crisis, even when the data is messy.

Technical Summary: A Beta-GAM Hidden Markov Model for Proportion Time Series

Problem Statement
The paper addresses the modeling of univariate proportion time series taking values in the open interval $(0, 1)$ . Such data are prevalent in economics, finance, and demography (e.g., mortality ratios, bid distributions, income shares). Existing approaches face limitations: standard Hidden Markov Models (HMMs) often assume constant emission parameters within latent states, failing to capture covariate-dependent structures. Conversely, Markov-switching Generalized Additive Models (MS-GAMs) allow for smooth covariate effects but are not specifically designed for bounded proportion data, which requires a distribution that respects the $(0, 1)$ support and handles heteroscedasticity appropriately. The authors propose a framework that integrates the flexibility of GAMs with the regime-switching capabilities of HMMs, specifically tailored for Beta-distributed emissions.

Methodology
The proposed model is a Hidden Markov Model where the observed variable $y_t$ follows a Beta distribution conditional on a latent state $z_t = k$ . The Beta distribution is parameterized by a state-specific mean $\mu_{kt}$ and a state-specific precision $\phi_k$ .

Emission Distribution:
The conditional distribution is $y_t | z_t = k \sim \text{Beta}(\mu_{kt}\phi_k, (1-\mu_{kt})\phi_k)$ .
The mean $\mu_{kt}$ is linked to covariates $x_t$ via a logistic transformation of an additive predictor:
$\text{logit}(\mu_{kt}) = \eta_{kt} = \sum_{j=1}^p f_{kj}(x_{tj})$
where each $f_{kj}(\cdot)$ is a smooth function estimated using cubic B-spline basis expansions. The precision $\phi_k$ is constant within a state but varies across states, allowing for regime-dependent variability.
Estimation (Penalized EM Algorithm):
Parameters $\Theta = (\pi, A, \{\beta_k\}, \{\phi_k\})$ are estimated via a penalized Expectation-Maximization (EM) algorithm.
- E-step: Computes forward-backward recursions to obtain smoothed posterior probabilities of the latent states ( $\gamma_{tk}$ ) and state transitions ( $\xi_{tij}$ ). Calculations are performed on the log-scale to ensure numerical stability.
- M-step: Maximizes the penalized expected complete-data log-likelihood. A quadratic penalty (P-splines) is applied to the B-spline coefficients $\beta_k$ to control smoothness and prevent overfitting. Unlike standard EM, the M-step does not yield closed-form updates for the emission parameters; instead, it employs numerical optimization (L-BFGS-B) to maximize the penalized Q-function for each state independently. This constitutes a Generalized EM algorithm.
Model Selection and Diagnostics:
The authors introduce a two-stage grid search to select the number of latent states $K$ and the smoothing penalty $\lambda$ .
- Diagnostic Filter: To address the issue of degenerate solutions (where overspecifying $K$ leads to states with negligible occupancy and explosive precision estimates), a diagnostic criterion is applied. Models are flagged and discarded if precision estimates hit an upper bound or if the differences between ordered precision estimates show abrupt "explosions."
- Selection Criteria: Among non-degenerate models, the optimal $K$ and $\lambda$ are selected using information criteria (AIC, BIC, ICL) that utilize effective degrees of freedom (EDF) rather than raw parameter counts to account for the penalty-induced shrinkage. The procedure selects $\lambda$ within each $K$ using AIC (less conservative) and selects $K$ using BIC (more conservative), with a preference for richer structures if models fall within two BIC units.
Uncertainty Quantification:
Confidence intervals for transition probabilities and state-dependent parameters are derived via a parametric bootstrap procedure, simulating new datasets from the fitted model and re-estimating parameters.

Key Contributions
The paper makes four primary methodological contributions:

Integration of GAMs and Beta-HMMs: It embeds B-spline based GAMs within a Beta emission HMM, allowing the conditional mean to vary nonlinearly with covariates within each latent regime while maintaining state-specific precision.
Frequentist Penalized Estimation: It adopts a penalized EM approach rather than a fully Bayesian one, avoiding the specification of priors for high-dimensional spline coefficients and offering computational efficiency for large samples.
Diagnostic Filtering for Degeneracy: It introduces a specific diagnostic criterion to detect and exclude degenerate solutions arising from overspecified latent states, a common issue in flexible HMMs.
Analytical Derivations: It provides explicit analytical derivations for the complete-data likelihood and the derivatives of the penalized log-likelihood, establishing a formal mathematical foundation for the estimation routine.

Results

Simulation Study: A Monte Carlo study with $T=2,500$ and $K=4$ demonstrates that the model accurately recovers transition dynamics, state precisions, and latent-state decoding (Viterbi accuracy $\approx 93.2\%$ ). The two-stage selection procedure successfully identifies the correct number of states and appropriate smoothing levels. The method remains stable even under reduced sample sizes ( $T=1,500$ ) and lower state persistence, though performance metrics degrade as expected.
Empirical Application: The model is applied to Russian age-specific mortality data (1960–2014, ages 0–40), modeling the female-to-total mortality ratio. The selected model identifies two persistent latent regimes:
- State 1: A more dispersed regime with a deeper trough in the young-adult age range (15–35), associated with periods of elevated and volatile male excess mortality.
- State 2: A highly concentrated regime with a shallower trough, representing a more stable mortality pattern.
  The Viterbi-decoded sequence suggests a transition from the dispersed to the concentrated regime in the late 1960s, with the concentrated regime prevailing thereafter, aligning with historical demographic shifts in Russia (e.g., the late Soviet health crisis and subsequent recovery).

Significance and Claims
The authors claim that the proposed Beta-GAM HMM offers a practical balance between flexibility, interpretability, and computational tractability for analyzing proportion time series. The framework successfully separates latent regime dynamics from smoothly varying covariate effects. The inclusion of a diagnostic filter is highlighted as a crucial step for ensuring model stability and interpretability, particularly when the number of states is unknown. The application to Russian mortality data illustrates the model's ability to summarize smooth age patterns while identifying substantively meaningful structural changes driven by historical events. The paper concludes that the methodology provides a robust tool for describing and monitoring structural changes in demographic proportions, with narrow bootstrap confidence intervals indicating reliable uncertainty quantification.