Technical Summary: Variance Deltas for Visualizing and Explaining Posterior Uncertainty

Problem Statement

In observational settings where the data generating process is uncontrolled and complex, statistical uncertainty regarding quantities of interest is often too high to permit substantive inference. Common causes include insufficient data, incorrect data types, or non-identified nuisance parameters. While remedies such as collecting more data or incorporating prior information exist, identifying the specific missing information required to reduce uncertainty is difficult. Existing visualization tools (e.g., histograms, scatter plots) often fail to capture the multiple levels of variation and uncertainty inherent in Bayesian inferences. Furthermore, discovering explanations for uncertainty typically requires manual model analysis and substantial guesswork, as current tools do not automatically identify which subsets of unobserved model quantities or parameters are most responsible for posterior uncertainty.

Methodology

The paper proposes variance deltas, an interactive visualization system designed to structure and partially automate the search for explanations of posterior uncertainty within a Bayesian framework. The system is implemented as a software package, variance-deltas.

Core Concepts

1. Uncertainty Index ($U^2_I$): The system quantifies the explanatory power of a subset of unknowns $\psi_I$ (where $I \subseteq I^* \setminus \{i_\theta\}$) regarding a quantity of interest $\theta$. The index is defined as the expected posterior variance of $\theta$ given $\psi_I$, normalized by the total posterior variance:
   $$U^2_I = \frac{E[\text{Var}(\theta \mid y, \psi_I) \mid y]}{\text{Var}(\theta \mid y)}$$
   A low $U^2_I$ indicates that observing $\psi_I$ would significantly reduce uncertainty about $\theta$. This is related to the Sobol' index, measuring the sensitivity of the conditional mean $E[\theta \mid y, \psi_I]$ to $\psi_I$.

2. Distributary Trees: The visualization takes the form of a tree rooted at the quantity of interest $\theta$. Non-root nodes represent subsets of unknowns (explanations). The tree structure encodes conditional independence relationships derived from the model's factorization. Paths from the root to the leaves represent Markov chains, reflecting a diminishing flow of information. The horizontal position of a node corresponds to its root uncertainty index (on a standard deviation scale).

3. Construction Algorithm:

   • Factorization and Graphs: The method utilizes the posterior density's factorization over a set of factors $F$. A factor graph ($G_F$) and a Markov graph ($M_F$) are constructed where vertices represent model quantities and edges represent dependencies.
   • Specificity Measure: To avoid overly broad explanations (e.g., "all data"), the system uses a "likelihood specificity" measure ($S_{lik}$). This measures the distance between a set of unknowns and the likelihood factors in the factor graph. Explanations involving quantities relevant to a small set of likelihood factors are considered more specific and useful.
   • Tree Generation: Using Algorithm 1, the system constructs Markov chains from the root ($\theta$) to leaf nodes (hypothetical data or specific parameters) by iteratively finding separating sets in the Markov graph that satisfy the Markov condition. Algorithm 2 then grafts these chains into a single distributary tree.
   • Estimation: Uncertainty indices are estimated from posterior samples using a data-splitting strategy to avoid overfitting bias. A random forest regressor is used to estimate the conditional expectation $E[\theta \mid \psi_I]$.

4. Interactive Operations: Users can interactively transform the tree to discover better explanations through:

   • Branching: Adding subsets of a node's unknowns to test if a smaller set is nearly as explanatory.
   • Subdividing: Inserting intermediate nodes to split the gap between parent and child uncertainty levels.
   • Merging: Combining nodes to test if the union of two explanations bridges a gap in uncertainty reduction.

Key Contributions

• Automated Discovery of Explanations: The system automates the generation of candidate subsets of unknowns from minimal user input (root and leaf nodes), organizing them into a tree based on their ability to explain uncertainty.
• Interactive Exploration: It provides a suite of operations (branching, subdividing, merging) allowing users to efficiently navigate the space of potential explanations without manual guesswork.
• Theoretical Foundation: The paper establishes the mathematical properties of "distributary trees," proving that they maintain valid Markov chain properties and that the construction algorithms produce valid separating trees.
• Software Implementation: The authors provide the variance-deltas R package, enabling the reproduction of figures and application to new models.

Results and Applications

The authors demonstrate the system's utility through two case studies:

1. Simulated Causal Inference (Panel Data):

   • Context: Estimating a treatment effect in a synthetic control setting with panel data.
   • Finding: The variance delta revealed that while individual units provided little information, combining hypothetical data from specific units (1, 2, and 3) significantly reduced uncertainty.
   • Insight: The analysis identified that the primary bottleneck was not the quantity of data, but uncertainty about the latent correlations (loadings) between the treated unit and the control units. Exact knowledge of these loadings reduced expected uncertainty to nearly a quarter of its original level.

2. Polling Model (2016 U.S. Presidential Election):

   • Context: Forecasting the Democratic vote share in Wisconsin using a complex dynamic Bayesian model with multiple sources of bias (pollster effects, mode effects, nonresponse, etc.).
   • Finding: The system determined that conducting additional polls one week before the election would yield minimal improvement in forecast accuracy.
   • Insight: The uncertainty was primarily driven by state-level measurement error ($\xi$) and temporal extrapolation uncertainty. The analysis showed that even with perfect knowledge of the bias parameters, a significant portion of uncertainty remained due to the inability to perfectly predict the day-of-election outcome from pre-election data. However, combining new polls with information about state-level measurement error could substantially reduce uncertainty.

Significance and Limitations

The paper claims that variance deltas offer a novel way to move beyond static posterior visualizations to actively "discover" non-obvious explanations for uncertainty. By structuring the search for missing information, the method helps analysts determine whether collecting more data is feasible or if the model itself requires rethinking (e.g., addressing non-identification).

The authors modestly note limitations:

• Graph Density: The effectiveness of the method depends on the sparsity of the underlying Markov graph. In highly dense graphs (e.g., large regressions), the resulting trees may be too shallow or the explanation sets too large to be tractable.
• Approximation: The method relies on estimating conditional expectations via regression (random forests), which introduces bias if the estimator's Mean Squared Error does not vanish, though the data-splitting approach ensures the estimator is conservative (likely to underestimate uncertainty reduction rather than overestimate it).
• Global Parameters: For "almost sparse" graphs with a few highly connected "global" vertices, the authors propose a theoretical extension (Algorithm 3) to condition on these vertices, though practical implementation challenges remain regarding the size of the resulting index sets.

The work positions itself within the broader literature on Bayesian workflow and sensitivity analysis, extending tools like shinystan and bayesplot by focusing specifically on the explanation of uncertainty rather than just its visualization or diagnostic checking.