A continuous-time Markov chain framework for population size estimation from multi-list data: accounting for absorbing lists and asymmetric interactions

Technical Summary: A Continuous-Time Markov Chain Framework for Population Size Estimation from Multi-List Data

Problem Statement
Estimating the size of hidden or hard-to-reach populations is critical for public policy, typically achieved through Multiple Systems Estimation (MSE) or capture-recapture methods using multi-list data. Standard approaches, primarily log-linear models, rely on the assumption that the target population is closed (no entries or exits) and that individuals are homogeneous in their probability of being recorded. However, these assumptions are frequently violated in social and epidemiological contexts where data sources include "absorbing lists"—records that capture individuals as they leave the population, such as death registers or emigration records.

The authors argue that failing to account for the absorbing nature of such lists violates the closure and homogeneity assumptions. Specifically, individuals recorded on an absorbing list have a reduced window of opportunity to be captured by other lists, leading to a violation of the independence assumptions inherent in standard log-linear modeling. The paper posits that applying standard log-linear models to data containing absorbing lists results in biased (specifically overestimated) population size estimates. Furthermore, while ecological capture-recapture methods have long addressed open populations, such models have not been effectively adapted to the MSE context due to a lack of temporal information in administrative datasets.

Methodology
The paper proposes a Continuous-Time Markov Chain (CTMC) framework to model the data collection process for multi-list data.

• State Space and Transitions: The framework defines a state space $L$ as the power set of $k$ lists. Individuals transition from an initial state of being unobserved ($\emptyset$) to states representing combinations of lists they have been recorded on. The process is modeled as a continuous-time Markov chain with a generator matrix $Q$.
• Parameterization: Transition rates are defined by list-specific capture parameters ($\lambda_i$) and interaction parameters ($\mu_K$) representing dependencies between lists. The model assumes that capture times on an operational scale are exponentially distributed.
• Absorbing Lists: To handle absorbing lists (e.g., death records), the framework modifies the generator matrix $Q$ by setting transition rates to zero for any state containing the absorbing list. This mathematically enforces the constraint that once an individual enters an absorbing state, they cannot transition to any other state, effectively modeling the "exit" from the observable population.
• Ordered Lists and Asymmetry: The framework is extended to handle ordered lists (where recording on list $L_j$ requires prior recording on $L_i$) by reducing the state space and removing redundant parameters. It also allows for general asymmetric interactions (e.g., referrals) by estimating directional interaction parameters ($\mu_{ij}$ vs. $\mu_{ji}$).
• Estimation: The model estimates the total population size $N$ by maximizing a likelihood function. The authors utilize a Poisson approximation for the observed counts, which yields log-likelihood and AIC values directly comparable to standard log-linear models. Model selection is performed using AIC-based forward, backward, and bidirectional selection strategies.

Key Contributions

1. Framework Development: The authors introduce a CTMC framework for MSE that is algebraically equivalent to the standard log-linear model when lists are independent but offers greater flexibility for modeling dependencies and specific data collection dynamics.
2. Absorbing List Modeling: The paper provides a specific mechanism to model absorbing lists within the MSE context, addressing a known limitation of standard log-linear models which assume a closed population.
3. Simulation Evidence: Through simulation studies, the authors demonstrate that:
   • The CTMC and log-linear models yield similar results when lists are independent or when dependencies are standard.
   • In the presence of an absorbing list, standard log-linear models (even with model selection) tend to overestimate population size and exhibit higher variance.
   • The CTMC model with an absorbing list constraint provides more accurate and less variable estimates in these scenarios.
4. Empirical Application: The framework is applied to two real-world datasets:
   • Stroke Data (Northwest England, 1994-1995): A dataset including a death register (absorbing list). The CTMC model produced slightly lower population estimates (777–815) compared to log-linear models (997–1015), consistent with simulation findings that log-linear models overestimate in this context.
   • Drug Use Data (City of London, 2018-2019): A dataset involving ordered lists (community treatment entry vs. exit). The CTMC framework successfully modeled the ordered structure, yielding estimates (404–421) that were comparable to, though slightly higher than, log-linear estimates, with confidence intervals that encompassed alternative estimates.

Results and Claims of Significance
The paper claims that the primary significance of this work lies in its ability to explicitly account for absorbing lists and asymmetric interactions in MSE, which standard log-linear models cannot do without ad-hoc modifications.

• Bias Reduction: The authors conclude that ignoring the absorbing nature of lists (like death records) leads to significant overestimation of population size. The CTMC framework corrects this by modeling the directional flow of individuals out of the observable population.
• Flexibility: The framework is presented as a versatile tool that can adapt to various data collection processes, including ordered lists and varying observation periods, without requiring the strict closure assumptions of traditional MSE.
• Practical Utility: While the CTMC model has a higher computational cost (requiring matrix inversion of size $2^k \times 2^k$), the authors suggest that for datasets with absorbing lists, the trade-off is necessary to avoid bias. They recommend using accelerated forward or bidirectional selection for the CTMC model, or alternatively, using a log-linear model with "forced" absorbing interactions as a computationally cheaper approximation.

The authors modestly note that while the CTMC model offers new prospects for analyzing multi-list data, diagnostic testing for model assumption violations remains an area for future research. They also highlight that the framework's ability to identify absorbing lists when the nature of the lists is undisclosed is currently under investigation.