PRE-CISE: A PRE-calibration Coverage, Identifiability, and SEnsitivity analysis workflow to streamline model calibration

The paper introduces PRE-CISE, a pre-calibration workflow that integrates coverage, sensitivity, and collinearity analyses to refine prior distributions and calibration targets, thereby streamlining model calibration and enhancing the reliability of health policy models.

The Gist

Imagine you are a chef trying to recreate a famous, complex dish (like a perfect soufflé) based only on a description of how it should taste and look. You have a recipe, but some ingredients are missing, and you don't know the exact measurements. You have to guess the amounts, bake the cake, taste it, and adjust your guesses until it matches the description.

In the world of health policy, models are those recipes. They are complex computer simulations used by governments and doctors to predict how diseases spread or how treatments work. Calibration is the process of tweaking the "ingredients" (the model's numbers) until the computer's predictions match real-world data (like how many people actually got sick).

The problem? Sometimes, you can tweak the ingredients in a thousand different ways, and they all taste "good enough" to match the data. But if you pick the wrong combination, your prediction for next year could be wildly wrong. This is called non-identifiability (a fancy way of saying "we can't tell which ingredients are actually doing the work").

This paper introduces PRE-CISE, a new "pre-cooking checklist" that helps chefs (modelers) avoid wasting time on bad recipes before they even turn on the oven.

Here is how PRE-CISE works, using simple analogies:

1. The "Range Check" (Coverage Analysis)

The Analogy: Imagine you are trying to hit a bullseye on a dartboard, but you are throwing darts from a mile away. Before you start throwing, you check your map. If your map says you are standing in a field a mile away, but the bullseye is in a room across the street, you know immediately that you need to move closer. You don't need to throw 1,000 darts to know you're going to miss.

In the Paper: Before doing the heavy math, PRE-CISE checks if the model's "guesses" (based on current knowledge) are even close to the real-world data. If the model predicts 1 million sick people but the real data says 1,000, the model is "out of range." PRE-CISE tells the modeler, "Hey, your starting guesses are too far off; adjust your bounds before you waste time calculating."

2. The "Volume Knob" (Local Sensitivity)

The Analogy: Think of a sound mixing board with 100 sliders. You want the music to sound perfect. Instead of randomly moving every slider, you tap each one gently to see which ones actually change the volume. You realize that moving "Bass" slider #4 changes the sound a lot, but moving "Treble" slider #99 does almost nothing.

In the Paper: PRE-CISE tests each "ingredient" (parameter) to see which ones have the biggest impact on the final result. If a specific number (like the rate at which people get sick) has a huge effect, the modeler knows to focus their attention there. If another number barely matters, they can ignore it or give it a tight, safe range. This stops the computer from wasting energy on ingredients that don't matter.

3. The "Tangled Wire" Test (Collinearity Analysis)

The Analogy: Imagine you are trying to figure out how much salt and how much pepper are in a soup. If you add more salt and the soup tastes the same, but you also add more pepper, you can't tell which one is doing the work. The salt and pepper are "tangled" together. You can't solve for both. But if you have a second clue—like "the soup is also too spicy"—suddenly you can separate the salt from the pepper.

In the Paper: Sometimes, two different numbers in the model can swap places and still produce the same result. This is a "tangled wire." PRE-CISE uses math to detect these tangles before the final calculation. It asks: "Do we have enough different types of data (like daily numbers vs. weekly numbers) to untangle these wires?"

• Key Finding: The paper showed that using daily data (high resolution) untangled the wires, but using weekly data (low resolution) left them knotted, making the model unreliable.

The Result: A Better Recipe

By using this three-step checklist (Range Check, Volume Knob, Tangled Wire Test) before the main event, modelers can:

• Save Time: They don't waste computer power on impossible guesses.
• Be More Honest: They can admit, "We can't figure out this specific number with the data we have," rather than pretending they know.
• Make Better Decisions: Policymakers get predictions that are based on solid, identifiable math, not just lucky guesses.

In short: PRE-CISE is like a smart sous-chef who checks the pantry, tests the spices, and untangles the wires before the head chef starts cooking. It ensures that when the final dish is served, it's not just a guess—it's a reliable recipe for saving lives.

Technical Summary

1. Problem Statement

Health policy models often require calibration to align model outputs with empirical data (calibration targets). However, this process faces significant challenges:

• Computational Intensity: Calibration often involves searching high-dimensional parameter spaces, requiring repeated simulations that are computationally expensive.
• Nonidentifiability: Multiple distinct parameter sets may fit the calibration targets equally well. If not addressed, this leads to divergent projections and unreliable policy recommendations.
• Poor Prior Specification: Initial prior distributions may be too wide (wasting computation on infeasible regions) or misaligned with targets, leading to calibration failure.
• Lack of Structured Pre-calibration: While techniques like sensitivity analysis and collinearity diagnostics exist in applied mathematics, they are rarely integrated into a structured, transparent workflow specifically for health decision modeling prior to intensive calibration.

2. Methodology: The PRE-CISE Workflow

The authors propose PRE-CISE, a three-step pre-calibration workflow designed to refine parameter spaces and diagnose identifiability before running full Bayesian calibration.

Step 1: Coverage Analysis

• Goal: Verify that the model, when run with parameters drawn from their prior distributions, can generate outputs that span the calibration targets.
• Process:
  1. Define prior distributions (uniform or parametric) based on domain knowledge.
  2. Sample $N$ parameter sets and simulate model outputs.
  3. Calculate the interquantile range (e.g., 95%) of the predicted outputs.
  4. Compare these ranges to the calibration targets.
• Outcome: If targets fall outside the predicted ranges, the prior bounds are identified as inadequate. This prevents wasting computational resources on infeasible search regions.

Step 2: Local Sensitivity Analysis

• Goal: Quantify the influence of each parameter on model outputs to guide the resizing of prior bounds.
• Process:
  1. Perform a "one-at-a-time" perturbation around the baseline (mean/midpoint of priors).
  2. Calculate elasticities (proportional change in output per proportional change in parameter) to create a sensitivity matrix ($S$).
  3. Compute norms ($L_1$ or $L_2$) to rank parameter influence.
• Application:
  • If a target is under/over-predicted, the elasticity signs and magnitudes indicate which parameters to adjust and in which direction.
  • Prior bounds are resized (shifted or tightened) based on the required percentage change to cover targets, ensuring biological plausibility.
  • Low-influence parameters are given tighter bounds to reduce the search space.

Step 3: Collinearity Analysis

• Goal: Assess practical identifiability (whether parameters can be uniquely recovered from the data).
• Process:
  1. Use the normalized sensitivity matrix to compute a collinearity index ($\gamma$) based on the smallest eigenvalue of $S^T S$.
  2. Threshold: Indices $> 15$ indicate practical nonidentifiability (parameters are linearly dependent regarding the targets).
• Application:
  • Determines the maximum number of identifiable parameters given a specific set of targets.
  • Evaluates the impact of data resolution (e.g., daily vs. weekly data) on identifiability.
  • Guides the selection of additional targets or the fixing of low-influence parameters.

Final Calibration

After PRE-CISE, the refined priors and target selection are used in a standard Bayesian calibration (using the Incremental Mixture Importance Sampling [IMIS] algorithm) to generate posterior distributions.

3. Key Contributions

• Operational Workflow: Integrates coverage, sensitivity, and collinearity diagnostics into a single, repeatable pre-calibration protocol for health policy models.
• Decision Rules: Provides explicit rules for resizing priors based on elasticity signs and magnitudes, and for selecting calibration targets based on collinearity indices.
• Data Resolution Insight: Demonstrates mathematically how higher-resolution data (daily vs. weekly) significantly improves parameter identifiability.
• Implementation: The authors provide a fully documented implementation (Appendix C) and apply it to two distinct model types (Markov cohort and SEIR transmission).

4. Results

The workflow was tested on two models:

A. Testbed: Four-State Sick-Sicker Markov Model

• Initial Issue: The initial prior distribution failed to cover the calibration targets (overestimated prevalence/survival).
• Sensitivity Findings: The transition probability from "Sick" to "Sicker" had the highest influence.
• Adjustment: Prior bounds were resized based on sensitivity elasticities.
• Collinearity Findings: A single target could only identify two parameters. To identify all three parameters, at least two distinct calibration targets (e.g., combining survival and prevalence) were required.

B. Case Study: Age-Structured COVID-19 SEIR Model (Mexico City)

• Efficiency Gains: PRE-CISE reduced the computational burden of the subsequent calibration by 18% by narrowing the search space to plausible regions.
• Sensitivity Findings: Time-varying detection rates and NPI effectiveness (contact reduction) were the most influential parameters.
• Identifiability Findings:
  • Daily Data: All parameter combinations yielded collinearity indices $< 15$, indicating full identifiability.
  • Weekly Data: Collinearity indices approached or exceeded the threshold of 15, indicating near nonidentifiability. This highlights that aggregating data to a weekly level obscures critical information needed to distinguish between parameters.

5. Significance and Implications

• Improved Reliability: By ensuring priors cover targets and diagnosing nonidentifiability before calibration, the workflow produces more robust and reliable policy recommendations.
• Transparency: It forces modelers to explicitly report if a unique solution is impossible, allowing decision-makers to understand the range of plausible scenarios rather than a false sense of precision.
• Computational Efficiency: Reduces wasted computation on infeasible parameter spaces and accelerates convergence in Bayesian algorithms.
• Data Strategy: Provides a quantitative basis for arguing for higher-resolution data collection (e.g., daily vs. weekly reporting) to support model calibration.
• Broad Applicability: The workflow is model-agnostic and applicable to deterministic compartmental models, microsimulations, agent-based models, and hybrid frameworks.

In conclusion, PRE-CISE bridges the gap between inverse modeling theory and practical health policy modeling, offering a structured path to reduce uncertainty, improve efficiency, and enhance the credibility of model-based decision support.