📊 statistics

Two-Stage Estimation of Population Abundance with Robust Inference under Interacting Survey Protocols

This paper proposes a two-stage estimation framework with a robust variance estimator to accurately quantify population abundance under interacting survey protocols, effectively addressing sample loss and detection interference while providing more reliable uncertainty estimates than traditional Bayesian hierarchical models.

Original authors: Yusaku Ohkubo, Tatsuki Shimamoto, Yuya Eguchi, Hirotaka Katahira

Published 2026-07-23

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Yusaku Ohkubo, Tatsuki Shimamoto, Yuya Eguchi, Hirotaka Katahira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to count a secret population of invisible creatures living in a forest. You can't just see them all at once, so you have to use special tools to find them. Sometimes, you use a super-expensive, super-accurate tool, like a high-tech drone that sees everything. Other times, you use a cheaper, simpler tool, like a flashlight that might miss a few critters. The big problem in ecology (the study of how living things interact with their world) is that these tools aren't perfect. They often miss animals, or they might even scare them away or hurt them while you are looking. If you use a cheap tool first and then a fancy tool, the cheap tool might have already damaged the evidence, making the fancy tool's results look weird. Scientists need a way to combine these messy, imperfect clues to get a true count of the population, without getting tricked by the tools themselves.

This paper tackles a tricky puzzle: what happens when your survey methods interfere with each other? The authors, researchers from universities in Japan, looked at a specific scenario where a "low-cost" method (like combing a squirrel's fur) is followed by a "high-cost" method (like shaving the fur). They found that the first method doesn't just miss some parasites; it actually damages or removes some of them before the second method even gets a chance to look. This creates a double whammy of errors: missed detections and lost samples. To solve this, they invented a new "two-stage" math recipe. Instead of trying to guess everything at once, they split the problem into two steps. First, they calculate how many parasites were likely destroyed by the combing. Second, they use that number to fix the detection rate of the combing method. They tested this idea with computer simulations and real data from invasive squirrels, finding that their new method gives a much more honest picture of the uncertainty than the old, popular "all-in-one" models, which tend to be overly confident and wrong.

The Detective's Dilemma: When Tools Fight Each Other

Counting wildlife is hard. Imagine trying to count every single fish in a pond without scaring them away or missing the ones hiding under lily pads. Ecologists often use N-mixture models, which are fancy statistical tools that try to guess the true number of animals (the "N") based on how many they actually saw. These models assume that if you miss an animal, it's just bad luck, not because the animal was hiding or because your tool broke.

But what if your tools are fighting each other?

The authors of this paper noticed a weird problem in the real world. They were studying Pallas's squirrels (a type of squirrel that has invaded parts of Japan) and the tiny parasites living on them. To count the parasites, they used two methods:

Combing: Running a comb through the squirrel's fur for a few minutes. This is cheap and easy, but it's not perfect.
Shaving: Shaving the squirrel's fur completely and then looking through the hair. This is expensive and takes forever, but it's supposed to catch everything.

The plan was simple: Use the comb first to get a quick count, then shave the squirrel to see what the comb missed. By comparing the two, they could figure out how good the comb was. But here's the twist: The comb was destroying the evidence.

When they combed the squirrels, they didn't just miss some parasites; they actually knocked some off or damaged them so badly that the shaving method couldn't find them later. It was like trying to count broken glass by sweeping it up with a broom, only to realize the broom broke half the shards before you could count them. If you just compare the "comb count" to the "shave count," you get the wrong answer because the "shave count" is now too low. The math gets confused, and the final population estimate becomes a mess.

The Two-Stage Solution: Fixing the Broken Chain

The authors realized that the old way of doing math (called a "joint hierarchical model") was trying to solve the whole puzzle at once. It was like trying to fix the broom, sweep the floor, and count the glass all in one giant, tangled knot. The result? The math got confused, the numbers were biased (skewed), and the scientists were way too confident in their wrong answers.

So, the authors proposed a Two-Stage Estimation framework. Think of it as a relay race where the baton is passed carefully, rather than everyone running in a circle.

Stage 1: The "What If" Prediction
First, they looked at the squirrels that were only shaved (the control group) and the ones that were shaved after being combed (the treatment group). They used this data to answer a specific question: "How many parasites did the combing process destroy?"
They created a "counterfactual" prediction. This is a fancy way of saying, "If the combing hadn't damaged anything, how many parasites should have been there?" This step isolates the damage caused by the survey method itself.

Stage 2: The Correction
Once they knew how many parasites were likely lost to damage, they used that number to fix the math for the second stage. They calculated the detection probability of the combing method based on this "undamaged" number, rather than the broken number they actually saw.

To make sure they weren't fooling themselves, they invented a special kind of error bar (called a robust variance estimator). Usually, when you use a number from Step 1 to do Step 2, you forget that Step 1 had its own mistakes. This new math tool makes sure those mistakes are carried over, so the final answer isn't too confident. It's like adding a safety margin to your calculation to say, "We think the answer is X, but it could be a bit higher or lower because our first guess wasn't perfect."

What the Numbers Say: Simulations and Real Squirrels

The team didn't just guess; they tested their idea in two ways.

1. The Computer Simulation
They created fake worlds in a computer where they knew the exact number of parasites. They ran the simulation 1,000 times with different settings.

The Old Way (Bayesian Joint Model): Even though the computer knew the true answer, the old model kept guessing wrong. It was biased, meaning it consistently overestimated or underestimated the numbers. Worse, its "confidence intervals" (the range where it thought the answer lay) were too narrow. It was like a detective saying, "I'm 95% sure the thief is in this tiny closet," when the thief was actually in the whole house. In many cases, the true answer wasn't even in the range the model claimed to be 95% sure of.
The New Way (Two-Stage): The authors' method got the numbers right on average. More importantly, its confidence intervals were honest. When they said they were 95% sure, the true answer was actually in that range 95% of the time. Even when they messed up the math on purpose (by making the data "messy" or "over-dispersed"), the new method stayed reliable, while the old one fell apart.

2. The Real Squirrel Data
They applied their method to real data from 180 squirrels in Japan (60 for the control group, 120 for the treatment group).

The Damage: They found that the combing method destroyed about 30% of the parasites. The "survival probability" of a parasite after combing was only about 0.7 (meaning 70% survived, 30% were lost).
The Comparison: When they compared their new method to the old "all-in-one" model, the old model was slightly too optimistic about how well the combing worked. But the biggest difference was in the uncertainty. The new method's error bars were about 50% wider than the old model's. This sounds bad, but it's actually good! It means the new method is admitting, "We aren't 100% sure, and here is the honest range of possibilities," whereas the old model was pretending to be more certain than it really was.

Why This Matters

This paper suggests that when different survey methods interact and mess with each other, we need to stop trying to solve everything at once. By separating the steps—first figuring out the damage, then fixing the detection rate—we get a much clearer, more honest picture of nature.

The authors admit that their method isn't a magic wand for every single problem in the world. For instance, if the data has complex patterns over time or space (like squirrels moving in groups), the math might need more tweaking. But for now, this two-stage approach offers a solid, reliable way to count populations without getting tricked by our own tools. It teaches us that sometimes, to get the right answer, you have to stop and fix the broken pieces before you try to count the whole.

Technical Summary: Two-Stage Estimation of Population Abundance with Robust Inference Under Interacting Survey Protocols

Problem Statement
Estimating population abundance in ecology is frequently complicated by the interaction between multiple survey protocols. While combining high-cost, high-accuracy methods with low-cost, low-accuracy methods offers a practical strategy for resource optimization, such combinations can introduce systematic bias when the protocols interfere with one another. Specifically, a low-accuracy method (e.g., combing) may physically damage or remove individuals (sample loss) before a subsequent high-accuracy method (e.g., shaving) is applied. This interaction violates the standard assumption of independent sampling, leading to two distinct sources of bias: (1) missed detections (parasites remaining undetected) and (2) sample destruction (parasites lost during the initial survey). Naïve calibration strategies that fail to account for this sample loss result in the overestimation of detection probabilities and the underestimation of true abundance. Furthermore, fully joint hierarchical models (such as Bayesian N-mixture models) applied to these interacting processes often suffer from weak identifiability and feedback loops, where the observation model of the low-accuracy counts influences the estimation of the sample loss parameters, leading to biased estimates and underestimated uncertainty.

Methodology
The authors propose a two-stage estimation framework designed to decouple the calibration of sample loss from the estimation of detection probability. The data structure involves three groups: a control group (high-accuracy method only), a treatment group (low-accuracy method followed by high-accuracy method), and a test group (low-accuracy method only).

Model Formulation: The authors extend the N-mixture model to explicitly represent heterogeneity in the detection process. The latent true abundance ( $y_{TRUE}$ ) follows a Poisson distribution. The observation process is decomposed into:
- Stage 1 (Sample Loss Calibration): The treatment group's high-accuracy counts ( $y_{high}$ ) are modeled as a function of the true abundance and a survival probability ( $p_{intact}$ ), representing the proportion of individuals surviving the low-accuracy procedure. This stage estimates the counterfactual abundance that would have been observed had no sample loss occurred.
- Stage 2 (Detection Probability): Conditional on the predicted abundance from Stage 1, the low-accuracy counts ( $y_{low}$ ) are modeled to estimate the detection probability ( $p_{low}$ ) of the initial method.
Estimation Strategy:
- Stage 1: Parameters ( $\beta$ for abundance covariates and $p_{intact}$ ) are estimated via Maximum Likelihood Estimation (MLE) using a standard Poisson Generalized Linear Model (GLM) formulation.
- Stage 2: The detection probability ( $p_{low}$ ) is estimated via MLE using the predicted intensities from Stage 1 as offsets.
- Variance Estimation: Crucially, the authors derive a sandwich-type robust variance estimator for the Stage 2 estimator. This estimator formally propagates the uncertainty from the first-stage predictions into the second stage, correcting for the fact that the regressors in the second stage are generated (predicted) rather than fixed.

Key Contributions

Decoupling of Processes: The framework separates the estimation of sample loss from detection probability, avoiding the feedback mechanisms and weak identifiability inherent in fully joint hierarchical models.
Robust Inference: The derivation of a robust variance estimator ensures that uncertainty quantification remains valid even under certain forms of model misspecification (e.g., over-dispersion or negative binomial outcomes when a Poisson model is assumed).
Two-Stage Efficiency: The method allows for the use of standard Poisson GLM machinery while rigorously accounting for the generated regressors in the second stage.

Results
The performance of the proposed method was evaluated through simulation studies and applied to real-world data on ectoparasites (lice, ticks, mites, and fleas) collected from invasive Pallas's squirrels (Callosciurus erythraeus) in Japan.

Simulation Studies:
- Bias: The proposed two-stage estimator produced unbiased point estimates for both $p_{intact}$ and $p_{low}$ across various scenarios. In contrast, the Bayesian hierarchical joint model exhibited systematic bias, overestimating the impact of treatment variables and failing to converge to true values even with correct model specification.
- Uncertainty Quantification: The proposed method's robust standard errors yielded empirical coverage rates close to the nominal 95% level. Conversely, the Bayesian joint model and naïve standard errors consistently underestimated uncertainty, resulting in coverage rates as low as 50% in some scenarios.
- Robustness to Misspecification: Under model misspecification (e.g., data generated from a negative binomial distribution but analyzed with a Poisson model), the proposed method maintained coverage at or above the nominal level (albeit with some conservativeness), whereas the Bayesian joint model showed significant under-coverage.
Empirical Application:
- Analysis of the squirrel ectoparasite data revealed that the combing method resulted in a sample loss rate of approximately 30% ( $p_{intact} \approx 0.70$ ), confirming that the combing process physically removes parasites before the shaving method can detect them.
- The proposed method yielded a wider 95% confidence interval for the detection probability ( $p_{low}$ ) compared to the Bayesian joint model, reflecting a more honest assessment of uncertainty. The Bayesian model's narrower intervals were deemed unreliable given the simulation results showing poor coverage.

Significance and Claims
The paper claims that the proposed two-stage framework provides a more reliable approach for estimating population abundance when survey protocols interact, particularly in scenarios involving sample loss. The authors emphasize that their method offers superior uncertainty quantification compared to standard Bayesian joint models, which tend to be overly optimistic and prone to bias due to feedback loops. By separating the calibration of sample loss from detection estimation and utilizing a robust variance estimator, the method addresses the limitations of existing N-mixture models in the context of heterogeneous and interfering survey protocols. The authors note that while the method assumes independence (i.i.d. setting) and fixed covariates, it establishes a flexible foundation for ecological monitoring programs where combining high- and low-cost survey methods is necessary. They acknowledge that future work is needed to address spatio-temporal dependence and more complex detection heterogeneity.

The Detective's Dilemma: When Tools Fight Each Other

The Two-Stage Solution: Fixing the Broken Chain

What the Numbers Say: Simulations and Real Squirrels

Why This Matters

Technical Summary: Two-Stage Estimation of Population Abundance with Robust Inference Under Interacting Survey Protocols

More like this