📊 statistics

A Total Statistical Error Framework for Comparing Census Data Collection Methods

This paper introduces a Total Statistical Error framework for comparing census imputation methods based on unit-level enumeration and address correctness, demonstrating through Australian 2021 census data that augmenting post-enumeration survey models with census nonresponse indicators effectively corrects severe biases caused by dependent nonresponse.

Original authors: Siu-MIng Tam, Anders Holmberg

Published 2026-08-04

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Siu-MIng Tam, Anders Holmberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect snapshot of a bustling city. You want to know exactly who lives where, how many people are in each house, and where they usually sleep. This is the job of a census. But life is messy. People move, they forget to fill out forms, or they are sleeping in a temporary spot like a student dorm or a construction camp. When the census takers can't find a person or doesn't know their "usual" address, they have to guess. This guessing game is called imputation.

Now, imagine you have two different ways to make these guesses. Maybe one method uses a "nearest neighbor" approach (finding the most similar person who did answer and copying their details), and another uses a complex computer brain called a "random forest" to predict the answer. How do you know which guesser is better? Usually, statisticians check two things separately: Did they find the person? And did they guess the right address? But what if one method finds more people but guesses their addresses wrong, while the other finds fewer people but gets the addresses right? It's like comparing a chef who cooks a huge meal but burns half of it, versus one who cooks a small meal perfectly. You need a way to measure the whole meal, not just the parts. This paper builds a new "scorecard" to solve exactly that problem, helping officials decide which guessing method gives the most accurate picture of the population.

The Great Census Guess-Off

In the world of counting people, getting the address right is just as important as getting the count right. If a student is counted at their university dorm but their "usual" address is actually their parents' house, that's a mistake for local planning. The authors of this paper, Siu-Ming Tam and Anders Holmberg, realized that statisticians didn't have a single, fair way to compare different guessing methods. They wanted a score that combined "Did we find them?" and "Did we get their address right?" into one simple number.

They created a new framework called the Total Statistical Error (TSE) framework. Think of it like a video game where every person in the country is a character. For each character, you get a "Correct" point (1) only if two things happen: the census actually found them, and it assigned them to the correct home address. If the census missed them entirely, or if it found them but put them in the wrong house, you get a "Wrong" point (0). By adding up all these points, you get a Correctness Rate. This rate is the ultimate tie-breaker. If Method A has a higher correctness rate than Method B, Method A wins. It doesn't matter if Method A found slightly fewer people; if it got the right people in the right places, it's the better tool.

The Two Ways to Check the Score

The paper explains that there are two ways to calculate this score, depending on what information you have.

Paradigm I: The Mystery Box (Validation Survey)
Sometimes, you don't know the "true" address for everyone. You only have a sample. Imagine you have a giant box of census data, but you don't know the real answers. To check your work, you send out a special team (a Post-Enumeration Survey, or PES) to a random sample of people to ask, "Hey, where do you really live?" You compare the census guesses to the PES answers. This is like a teacher grading a test by only checking a few random questions. The paper shows that if the people who don't answer the PES are different from those who do (for example, if people with wrong addresses are harder to find), your score will be biased. It's like if the teacher only graded the easy questions because the hard ones were too messy to read. The authors found that this "dependent nonresponse" can make a bad method look great.

Paradigm II: The Master Key (Direct Benchmark)
Sometimes, you have the "truth" for everyone. Maybe you have a previous census or a perfect government register. In this case, you don't need to guess or sample; you just compare the new method directly against the master list. It's like having the answer key for the whole test. You can calculate the exact correctness rate without any guessing. The paper uses this method for their main experiment because they had access to a massive dataset from the 2021 Australian Census where they knew the true answers for everyone.

The Big Experiment: Who Guesses Best?

To test their new framework, the authors ran a massive simulation using real data from the 2021 Australian Census. They took a group of 30,000 people and pretended that 2%, 5%, or 10% of them had "missing" data (like income or job type). Then, they let two different guessing machines try to fill in the blanks:

k-Nearest Neighbors (kNN): This method looks for the person in the database who is most similar to the missing one and copies their info.
Random Forest (RF): This is a more complex computer model that builds many decision trees to predict the missing info.

They tested these methods under a tricky condition called NMAR (Not Missing At Random). This means the people who were "missing" weren't missing by chance; they were missing because they had high incomes, and high-income people were less likely to answer the census questions. This is a realistic scenario that makes guessing very hard.

The Shocking Result:
When they looked at the "Overall Correctness Rate" (the total score including everyone), all three methods looked almost identical. They all scored around 95% to 98%. Why? Because most people didn't have missing data, so they were counted correctly automatically. The overall score was hiding the real problem.

But when they zoomed in on just the people who needed guessing (the "Conditional Correctness Rate"), the story changed completely.

The Random Forest model was the clear winner at guessing individual details. It outperformed kNN on every single variable tested. For example, it guessed the correct job industry 34% of the time (compared to 29% for kNN) and the correct income 17% of the time (compared to 12.5% for kNN).
However, when looking at the whole picture (getting all four missing details correct at once for the same person), the k-Nearest Neighbor method (specifically using just 1 neighbor) actually pulled ahead slightly. It got all four details right for 3.7% of the missing people, while Random Forest only got 3.4%.

Why the difference? Because Random Forest guessed each detail separately. It might get the job right for one person but the income wrong. The kNN method grabbed all the details from the same "neighbor," so the answers stayed consistent with each other. It was like getting a full outfit from one friend versus getting a shirt from one friend and pants from another; the full outfit just fits better together. So, while Random Forest is the better "specialist" for individual facts, the simpler kNN method was slightly better at keeping the whole story coherent.

The Trap of the Biased Survey

The most critical finding of the paper comes from the "Paradigm I" simulation. The authors simulated a scenario where the "check-up team" (the PES) had trouble finding people who had been guessed incorrectly.

The Standard Approach: If you just look at the people who answered the PES, you might think your guessing method is 97% accurate.
The Reality: The true accuracy was only about 95%.
The Bias: The standard approach was overestimating the quality by about 2.5 percentage points.

But here is the kicker: when they looked at specific groups (like employed people), the error exploded. Because the group of missing people was small (only about 1.5% to 6% of the group), the small bias got magnified massively. The standard approach claimed the accuracy for employed people was around 50-60%, when the true accuracy was actually near 0%! It was a catastrophic overestimation.

The Solution:
The authors tested a fix called CBB-NR. This involves adding a simple "flag" to the survey model: "Was this person's data missing and imputed in the first place?"

When they added this simple flag, the bias dropped by 90%.
Surprisingly, adding the actual guessed values (the income or job guesses) didn't help much. The simple "Yes/No" flag that said "This person was a guess" was the magic ingredient.

Why This Matters

This paper gives statisticians a new, fair ruler to measure census quality. It proves that looking at the "big picture" (overall accuracy) can hide serious flaws in how we guess missing data. It also warns us that if our check-up surveys miss the people who are hardest to find (the ones with wrong guesses), we will think our methods are much better than they really are.

The authors show that by using their new "Correctness Rate" and fixing the survey bias with a simple flag, we can finally compare methods like kNN and Random Forest fairly. They found that while complex computer models are great at guessing single facts, simpler methods that keep all the facts together might actually be better for getting the whole story right. Most importantly, they showed that without fixing the bias in our check-up surveys, we are flying blind, thinking our maps are perfect when they might be full of holes.

Technical Summary: A Total Statistical Error Framework for Comparing Census Data Collection Methods

Problem Statement
Population censuses increasingly rely on imputation to assign usual-residence (UR) addresses for non-responding dwellings. However, national statistical agencies lack a formal statistical framework to compare competing imputation strategies (or other data collection methods) based on their combined performance in coverage (enumeration) and address accuracy. Traditionally, these metrics are measured separately. Furthermore, existing quality assessment tools, such as the Dual System Estimator (DSE) used in Post-Enumeration Surveys (PES), primarily address coverage errors and often fail to account for geographical misclassification (assigning a correctly counted person to the wrong location). Additionally, standard estimation methods often assume that non-response in validation surveys is Missing At Random (MAR), whereas in reality, non-response may be Non-Ignorable (NMAR), particularly when persons with incorrect imputations are harder to locate for follow-up.

Methodology
The authors propose a Total Statistical Error (TSE) framework that unifies coverage and measurement accuracy into a single unit-level metric.

Unit-Level Correctness Indicator ( $Z_{ki}$ ):
For a person $i$ under method $k$ , the framework defines a binary indicator:
$Z_{ki} = I_{ki}(1 - D_{ki})$
Where:
- $I_{ki} = 1$ if the person is captured (enumerated) and $0$ otherwise.
- $D_{ki} = 1$ if the assigned address is incorrect (measurement error) and $0$ if correct.
- $Z_{ki} = 1$ only if the person is both captured and assigned to the correct reference address.
  The complement, $E_{ki} = 1 - Z_{ki}$ , represents the unit-level Total Statistical Error, decomposable into "missed-person" ( $1-I_{ki}$ ) and "wrong-unit" ( $I_{ki}D_{ki}$ ) components.
Population Correctness Rate ( $Q_k$ ):
The primary metric for comparison is the population correctness rate, $Q_k = \frac{1}{N} \sum Z_{ki}$ . Method $B$ is preferred over $A$ if $Q_B > Q_A$ (or equivalently, if the error rate $E_B < E_A$ ).
Two Assessment Paradigms:
- Paradigm I (Truth Unknown): Used when reference values ( $T_i$ ) are unknown for the full population. A validation survey (e.g., PES) provides a probability sample. $Q_k$ is estimated using survey weights. The paper extends the Chipperfield–Brown–Bell (CBB) estimator to this context. It addresses NMAR PES non-response by augmenting the response propensity model with the census non-response indicator ( $I_{ki}$ ) and imputed values, creating a calibrated weight ( $\tilde{w}_i$ ) that yields an approximately unbiased estimator ( $\hat{Q}^{CBB}_k$ ) even when PES response depends on the correctness of the census record.
- Paradigm II (Truth Known): Used when a complete, reliable benchmark (e.g., a traditional census) exists for the entire population. $Q_k$ is computed exactly without sampling variance, allowing for direct arithmetic comparison and exact error decomposition.
Subgroup and Overcount Handling:
The framework extends to subgroup analysis (e.g., by mobility status) and includes mechanisms to handle overcoverage (duplicate records) by distinguishing between missed persons, wrong addresses, and phantom records.

Empirical Illustration and Results
The authors illustrate the framework using 2021 Australian Census microdata ( $N=840,402$ ), simulating a Not-Missing-At-Random (NMAR) mechanism where higher-income records are more likely to be deleted (non-respondents). They compare three imputation methods: $k$ -nearest-neighbour ( $k=1, 5$ ) and Random Forest (RF).

Metric Sensitivity: The overall correctness rate ( $Q_k$ ) was dominated by the deletion rate ( $\rho$ ) and failed to discriminate between methods (all $\approx 0.95$ ). However, the conditional correctness rate ( $q^{imp}_k$ ), measuring accuracy only among deleted records, revealed significant differences.
Imputation Performance:
- Per-Variable: Random Forest outperformed $k$ -NN on individual variables (e.g., +37% accuracy on Income), driven by its ability to model non-linear interactions.
- Joint Correctness: $k$ -NN (specifically 1-NN) achieved higher joint correctness (all four variables correct simultaneously) than RF. This is attributed to the coherence of single-donor imputation, which preserves the joint distribution of variables, whereas RF fits independent models that may yield correct answers for different subsets of records.
Bias Correction (Paradigm I Simulation):
- Simulating a PES with NMAR non-response (where persons with incorrect imputations were less likely to respond) caused standard estimators to overstate correctness by ~2.5 percentage points.
- Standard calibration using only auxiliary covariates (CBB-setB) reduced bias by less than 1%.
- CBB-NR (Census Non-Response Indicator): Augmenting the response model with a binary indicator for whether a record was imputed (census non-response) reduced bias by approximately 90%.
- Key Finding: The inclusion of the imputed values themselves contributed negligible additional bias reduction; the binary non-response indicator was the operative instrument driving the correction.

Key Contributions

Unified Framework: The paper provides the first formal framework for comparing census methods based on a combined metric of coverage and address accuracy, moving beyond the traditional separation of these errors.
Paradigm Distinction: It clearly delineates between estimation scenarios where truth is known (Paradigm II) versus unknown (Paradigm I), offering specific methodologies for each.
Bias Correction for NMAR: It demonstrates that standard PES weighting fails under dependent non-response and validates the extension of the CBB estimator to the correctness rate. Crucially, it identifies that including the census non-response indicator in the PES response model is sufficient to correct severe biases in subgroup quality estimates.
Operational Insights: The empirical study highlights that while complex models (RF) may improve per-variable accuracy, simpler coherent imputation (1-NN) may better preserve joint data integrity, and that "overall" accuracy metrics can mask critical failures in imputation quality.

Significance and Claims
The authors claim this framework fills a critical gap in census methodology, enabling agencies to make evidence-based decisions between competing data collection strategies (e.g., imputation methods, Place-of-Enumeration vs. Usual-Residence, or Field vs. Administrative data).

The paper emphasizes that the framework is operationally feasible with standard census microdata. Its primary practical significance lies in the demonstration that dependent non-response between a census and a validation survey can severely distort quality assessments (inflating subgroup correctness rates by factors of 15–69 in extreme cases) and that this bias can be effectively mitigated (by ~90%) using a specific, low-cost augmentation to the weighting model (the census non-response indicator). The authors position this as a necessary tool for the transition toward register-based and hybrid censuses, where address accuracy and linkage errors are paramount.