From surveillance maturity to analytical readiness: an estimand-first framework for real-time outbreak analysis under imperfect data
This paper proposes an estimand-first framework for assessing analytical readiness in real-time outbreak analysis, demonstrating through Ebola data that the suitability of data for specific inferences depends on the interplay between surveillance maturity, epidemic informativeness, and decision-matched validation rather than simply on data volume or model convergence.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a new disease outbreak begins, public health teams face a race against time that is as much about information as it is about medicine. They must decide where to send doctors, how many tests to order, and which communities to warn, often before the full picture of the virus has emerged. The data they rely on is rarely perfect; it arrives in fits and starts, delayed by laboratory backlogs, interrupted by road closures, or altered as officials correct earlier mistakes. For years, analysts have tried to make sense of this messy stream of numbers, hoping to predict how many people will get sick next week or where the disease will spread. The challenge has always been knowing when the data is good enough to trust. If the numbers are too shaky, a prediction might lead to wasted resources or a false sense of security. If they wait too long for perfect data, the outbreak may have already moved past the point where intervention could stop it. The core question is not just whether the data exists, but whether it is ready to answer the specific question being asked.
A team of researchers has developed a new way to think about this problem, shifting the focus from the quality of the data itself to the specific question being asked. Instead of treating an outbreak dataset as simply "good" or "bad," they argue that readiness depends entirely on what you are trying to learn. A set of numbers might be strong enough to tell you which neighborhood is likely to see the next case, but completely useless for predicting how many people will die from the disease. They call this concept "analytical readiness," a measure of whether the available information, combined with the methods used to study it, can responsibly support a specific decision. Their work suggests that in the chaos of an emerging epidemic, the most honest and useful answer is often not a precise number, but a clear statement about what can and cannot be known at that moment.
The researchers tested this idea by looking back at real outbreaks, including the 2018 to 2020 Ebola response in the Democratic Republic of the Congo and a 2014 epidemic in Sierra Leone. They examined how the reliability of the data changed day by day as the surveillance system struggled with violence, security disruptions, and laboratory delays. In one specific test, they watched how well short-term predictions held up as the reporting system broke down and then recovered. Before the disruption, a simple prediction of the next week's cases was fairly accurate, missing the actual count by just one case. But as the system faltered, the errors grew. During the early stages of recovery, when officials were just starting to get reports back, the prediction missed the mark by eighteen cases. This showed that even when the numbers started coming in again, the data was not yet stable enough to trust a forecast. The system had recovered its ability to count, but not yet its ability to predict.
The study also looked at how different questions require different levels of data maturity. The researchers found that you might be able to rank neighborhoods by which ones are likely to be hit next, even if you cannot yet say exactly how many cases will occur there. In one instance, they tried to use past forecasts to decide which three health zones needed the most attention. While the mathematical scores for the forecasts looked good, they only improved the decision-making process in one specific moment in time. This highlighted a crucial point: a model that looks mathematically perfect on paper does not automatically mean it is ready to guide real-world choices. The researchers discovered that the same data could support a ranking of locations while failing to provide a reliable estimate of the total number of infections.
Perhaps the most striking finding concerned the concept of fatality rates. In many outbreaks, officials calculate the death rate by dividing the number of reported deaths by the number of reported cases. The researchers showed that this number can look very precise, with a narrow range of uncertainty, while actually hiding a massive amount of unknown information. In their simulations, a reported death rate of forty percent was mathematically compatible with a true death rate anywhere between ten percent and eighty percent. The precision of the number was an illusion created by the way the data was collected, not a reflection of reality. Increasing the number of reported cases made the statistical range look tighter, but it did not solve the underlying problem of not knowing who was actually getting sick or dying. The data was simply not designed to answer that specific question.
The researchers argue that the solution is not to wait for perfect data or to force complex models to produce a single answer. Instead, they propose a framework where analysts first define exactly what question they are trying to answer, and then check if the current data can support that answer. If the data is too immature, the correct output might be a scenario rather than a prediction, or a ranking of risks rather than a specific number. They emphasize that admitting what cannot be known is more valuable than presenting a false sense of certainty. In the heat of an outbreak, where decisions must be made with incomplete information, the goal is to match the strength of the claim to the strength of the evidence. This approach does not stop the analysis; it simply changes the product, ensuring that public health leaders are not misled by numbers that look solid but are actually built on shaky ground.
By treating readiness as a task-specific judgment rather than a global label, the researchers offer a practical guide for navigating the uncertainty of real-time epidemics. Their work demonstrates that more data, better computers, or tighter statistical intervals cannot replace the need to understand how the data was collected and what it can truly reveal. Whether the system is stable or chaotic, the answer to "what can we know?" depends on the question being asked. This shift in perspective allows analysts to provide useful, honest guidance even when the picture is far from complete, ensuring that decisions are based on what the evidence can actually support rather than on what the models wish it could say.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.