🧬 biology

Auditing model-based antiviral recommendations: a decision-relevance diagnostic for seasonal influenza surveillance

This paper introduces a variance-weighted decision-relevance score (DRS) to audit seasonal influenza surveillance models, demonstrating that standard calibration checks often fail to detect when antiviral stockpile recommendations are untrustworthy due to parameter non-identifiability, whereas the proposed metric effectively flags these high-risk scenarios using existing posterior data.

Original authors: Tian-Yu Wang

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tian-Yu Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Every winter, health agencies face a difficult choice: how much antiviral medicine to release to the public. They cannot wait for the flu to finish its work; they must act while the outbreak is still unfolding, relying on computer models that predict how the virus will spread. These models are tuned to match the number of reported cases, but there is a hidden trap in the data. Health officials only see the people who visit a doctor or get tested, not the vast number of people who are infected but never seek care. Because of this, the models cannot easily tell the difference between a scenario where a few people are infected but almost everyone is seen by a doctor, and a scenario where a massive number of people are infected but only a tiny fraction are reported. The math treats these two very different situations as identical, creating a blind spot where the model's internal numbers can shift wildly without changing the final prediction.

For years, scientists have checked their models to ensure the internal numbers are stable, but a new study suggests this is not enough. The researchers asked a more practical question: even if the model's internal numbers are shaky, does the final recommendation for medicine stockpiles actually change? They found that a model can pass all its standard tests and still give a recommendation that is essentially a guess. In their simulations, the uncertainty about how many people are truly infected versus how many are reported caused the recommended amount of medicine to flip to a different level in more than a quarter of the cases. This means that for many flu seasons, the advice to release a small, medium, or large amount of medicine is not supported by the data, even though the model looks perfect on paper.

The study, led by Tian-Yu Wang, introduces a new way to audit these recommendations before they are used. Instead of just checking if the model fits the past data, the new method checks if the decision itself is stable. The researchers built a computer simulation of a flu season and tested two different ways to measure uncertainty. The first method, which many scientists currently use, looks at the shape of the model's uncertainty and assumes that if the decision depends on the shaky parts, it is unsafe. The researchers found this method fails completely. It often tells them the decision is safe when it is actually fragile, because the decision depends on a different part of the math than the shaky numbers. The second method, which the team developed, measures how much the final recommendation would wobble if the uncertain numbers were allowed to vary. This new score works correctly, separating the seasons where the advice is trustworthy from those where it is not.

When the researchers applied this new audit to ten years of real flu data from the United States, the results were striking. The score successfully identified specific seasons where the recommendation for medicine was untrustworthy. Most of these failures happened during the 2022-2023 flu season, which was the first major flu season after the pandemic disruptions and had the highest number of reported cases. It is a counterintuitive finding: the season with the most data was also the one where the model was least able to give a clear answer. The study shows that having more data does not always mean having a clearer decision, because the way the data is collected can hide the true scale of the outbreak.

The researchers also discovered that the computer programs used to build these models often produce results that look mathematically correct but are actually impossible. The programs sometimes suggest that the number of infected people could be outside the range of what was originally thought possible, simply because the software was not built to respect those limits. By fixing the software to stay within the realistic boundaries, the team found that the recommendations became much more reliable. The old, uncorrected software would have suggested releasing the maximum amount of medicine in almost every case, a bias that disappeared once the boundaries were enforced.

This work does not solve the problem of predicting the flu, but it provides a crucial safety check for the people who have to make the decisions. The new audit tool is simple to run and requires no new data; it can be calculated from the results a team already has. The author suggests that health agencies should run this check alongside their standard tests. If the audit shows that the decision is unstable, it serves as a warning that the recommendation is not ready to be trusted. In a field where over-committing wastes resources and under-committing leaves people sick, knowing when a model is too uncertain to guide action is just as important as knowing when it is right.

Technical Summary: Auditing Model-Based Antiviral Recommendations

Problem Statement
Public health agencies rely on transmission models calibrated to seasonal influenza surveillance data to determine antiviral stockpile commitments (e.g., coverage tiers). However, surveillance data typically records reported cases rather than total infections, creating a structural identifiability issue. Specifically, the reporting rate ( $\rho$ ) and the initial infection size ( $i_0$ ) enter the expected case counts only through their product. This creates an exact, analytically transparent "ridge" in the parameter space where the likelihood is flat along the direction $(\rho, i_0)$ .

The core problem addressed is that standard calibration checks (e.g., posterior coverage) can pass while the recommended intervention remains unidentified. A model may be well-calibrated in terms of parameter recovery, yet the posterior uncertainty along the unidentifiable ridge may be sufficient to flip the optimal decision (the antiviral tier). The paper argues that parameter-level identifiability and decision-level identifiability are distinct properties; a model can have poorly identified parameters but a stable decision, or vice versa. Existing geometric diagnostics often fail to distinguish between these cases.

Methodology
The author developed a decision-relevance diagnostic framework centered on a Decision-Relevance Score (DRS).

Model Architecture:
- Transmission: A six-parameter SEIRS model (Susceptible-Exposed-Infectious-Recovered with waning immunity) integrated over a 20-week horizon.
- Observation: Reported counts follow a negative-binomial distribution where the mean depends on the product $\rho \cdot i_0$ .
- Decision Problem: Selecting an antiviral commitment level ( $\tau \in \{0.10, 0.30, 0.60\}$ ) to minimize a loss function balancing cumulative infections and intervention costs.
- Inference: A neural posterior estimator (conditional masked autoregressive flow) was trained via simulation-based inference. Crucially, the author implemented a bounded support mechanism using a sigmoid bijector to ensure the posterior remains within the prior box, correcting a previous unbounded estimator that produced over-dispersed, invalid results.
The Diagnostic Score (DRS):
- The author defines the decision margin $m$ as the difference in loss between the two best actions.
- They propose a variance-weighted DRS: the share of the decision margin's posterior variance that originates from "weakly identified" (weakly identified) directions. A direction is considered weakly identified if its posterior variance is at least half the prior variance.
- Control Arm: A scale-invariant loss function (relative reduction in transmission) was constructed that is mathematically independent of the $\rho \cdot i_0$ ridge. This serves as a positive control; the DRS for this control should be near zero.
Evaluation Strategy:
- Synthetic Grid: A 192-cell grid varying observation window length, over-dispersion, and reporting rates.
- Real-World Audit: Application to 204 CDC FluView region-seasons (2015–2025).
- Comparisons: The proposed variance-weighted score was compared against a purely geometric index (projection of the decision gradient onto the weakly identified subspace) and evaluated under both Fisher/Laplace approximations and neural posterior samples.

Key Results

Failure of Geometric Indices: The standard geometric index (projecting the gradient onto the weakly identified subspace) failed its own control arm. Because the decision gradient is nearly orthogonal to the unidentifiable ridge (due to the burden's elasticity to $R_0$ rather than $i_0$ ), the geometric index incorrectly concluded the decision was stable, even when the posterior was wide enough to flip the action.
Success of Variance-Weighted DRS: The proposed variance-weighted score successfully separated the main arm (sensitive to the ridge) from the control arm (insensitive). In synthetic tests, the main arm scores were significantly higher than the control (e.g., 0.25 vs. 0.05 under neural posterior), correctly flagging decision instability.
Decision Instability: On the synthetic grid, the bounded posterior flipped the "plug-in" (point estimate) action in 27% of cells, resulting in a 12% loss of value compared to an oracle. Parameter-level non-identifiability did not predict decision instability (Spearman correlation +0.107, $p=0.14$ ).
Real-World Findings:
- The score separated sensitive from insensitive decisions by up to several hundredfold in real data.
- 3.4% of region-seasons (7 out of 204) had a decision margin below its own posterior standard deviation.
- 6 of these 7 failures occurred in the 2022–2023 season, the highest-volume season post-pandemic. Contrary to the hypothesis that low counts cause instability, the data showed a negative correlation between case counts and decision identifiability (higher counts associated with less identifiable decisions), though the mechanism for this inversion remains unexplained.
Calibration Necessity: The study demonstrated that unbounded neural flows produce over-dispersed posteriors that invalidate decision-theoretic conclusions. Enforcing bounded support via bijectors was essential for the diagnostic to function, reducing the flip rate from ~52% (unbounded) to ~27% (bounded) and eliminating systematic bias toward aggressive tiers.

Significance and Claims
The paper claims that a calibrated posterior is not sufficient evidence that a model-based recommendation is identified. Standard parameter-level diagnostics do not reveal when a decision is unstable.

The primary contribution is a practical, decision-level audit (the variance-weighted DRS) that:

Is computed from the posterior a modeling team already holds.
Requires no new data or additional inference runs.
Runs efficiently on a CPU.
Flags specific seasons or scenarios where the recommended intervention is untrustworthy due to structural unidentifiability.

The author suggests this score should be run as a routine audit alongside standard calibration checks. They explicitly state that while the specific numbers are model-dependent, the diagnostic approach and the warning against relying on geometric shortcuts for decision identifiability are generalizable. The paper does not propose new inference methods but rather a validation step for existing ones.

Technical Summary: Auditing Model-Based Antiviral Recommendations

More like this