Predictive Concordance for Parameter Optimisation and Mixture Synthesis

Technical Summary: Predictive Concordance for Parameter Optimisation and Mixture Synthesis

Problem Statement
The paper addresses the challenge of evaluating and quantifying the concordance between a defined reference probability distribution, $p(y)$ (representing "ground truth" or an empirically supported benchmark), and a parametrised family of predictive distributions, $f(y|\alpha)$. This problem arises in statistical model emulation and decision-making contexts where a decision maker (DM) seeks to approximate a reference distribution using a specific parametric class.

A primary application motivating this work is scenario forecasting in macroeconomic policy. Here, the DM possesses a set of predictive probability density functions (p.d.f.s) derived from distinct models or contextual assumptions (scenarios). The goal is to determine the optimal mixture weights, $\alpha$, that combine these scenarios to best match the reference distribution $p(y)$. Crucially, this is not a traditional parameter inference problem based on observed data; rather, the reference $p(y)$ represents a future outcome to be observed, and the objective is to select parameters $\alpha$ that maximize the agreement between the mixture and this reference.

Methodology
The authors propose Expected Misclassification Rate (EMR) as the primary measure of predictive concordance.

1. Definition of EMR:
   EMR is derived from a statistical classification framework. Consider a random draw $y$ generated with equal probability from either the reference $p(y)$ or the candidate $f(y)$. Let $H_p$ and $H_f$ denote the hypotheses that $y$ comes from $p$ or $f$, respectively. The EMR, denoted $\pi_{pf}$, is the expected posterior probability of misclassifying a draw from $p(y)$ as coming from $f(y)$ (or vice versa, due to symmetry):
   $$\pi_{pf} = \int \frac{f(y)p(y)}{f(y) + p(y)} dy$$
   This metric is bounded within $(0, 0.5]$. A value of 0.5 indicates perfect concordance ($f \equiv p$), while lower values indicate greater divergence. The authors argue EMR is a bounded utility function, making it theoretically robust compared to unbounded measures like Kullback-Leibler (KL) divergence, which can be undefined if the tails of the distributions differ significantly (e.g., finite variance vs. infinite variance).

2. Optimisation and Synthetic Likelihood:
   The core methodology involves finding the parameter vector $\alpha$ that maximizes $\pi_{pf}(\alpha)$. The authors establish a synthetic likelihood interpretation for EMR: maximizing EMR is equivalent to finding the Maximum Likelihood Estimate (MLE) for $\alpha$ in a hypothetical binary classification problem where a latent variable $z=1$ is observed.

   To address the instability of sparse solutions (where optimal weights collapse to zero for many scenarios) inherent in simplex-constrained optimisation, the authors introduce regularisation. By adopting a synthetic prior $\rho(\alpha)$ (specifically a Dirichlet distribution), the objective function becomes a log-posterior:
   $$\lambda(\alpha) = \log\{\pi_{pf}(\alpha)\} + \log\{\rho(\alpha)\}$$
   This formulation allows for the specification of regularisation penalties that are interpretable as synthetic priors, moving the solution away from boundary zeros to small, non-zero values for stability.

3. Mixture Synthesis:
   In the specific case where $f(y|\alpha)$ is a discrete mixture of $J+1$ given p.d.f.s ($f(y|\alpha) = \sum \alpha_j p_j(y)$), the optimisation problem is proven to be convex. This ensures a unique solution and allows for the use of efficient standard convex optimisation algorithms. The evaluation of the integral in the EMR definition is performed via Monte Carlo integration using samples drawn from the reference distribution $p(y)$.

Key Contributions

• Theoretical Framework: The paper establishes EMR as a decision-theoretically meaningful, bounded measure of distributional concordance. It clarifies the relationship between EMR and KL divergence, showing that EMR acts as a lower bound to the inverse logistic transform of KL and is less sensitive to tail irregularities.
• Synthetic Bayesian Interpretation: It provides a novel interpretation of EMR optimisation as a Bayesian decision analysis with a synthetic likelihood and prior. This bridges the gap between optimisation and Bayesian emulation, offering a principled way to specify regularisation penalties.
• Algorithmic Efficiency: The authors demonstrate that for mixture synthesis, the EMR optimisation is convex, enabling efficient numerical solutions. They also detail the use of Monte Carlo methods for evaluating the objective function without requiring analytic integration.
• Handling Sparsity: The paper identifies the pathology of sparsity in unconstrained EMR maximisation (where similar scenarios receive zero weight) and proposes a regularised Dirichlet prior to induce stable, non-zero weights.

Empirical Results
The methodology is applied to a macroeconomic scenario synthesis example using U.S. Federal Reserve data (December 2020 forecasts for 2021 GDP growth).

• Setup: The reference distribution $p(y)$ is a skew-t distribution derived from the New York Fed's "Outlook-at-Risk" (OaR) model. The candidate distributions include a baseline forecast and five alternative economic scenarios, plus a "backstop" scenario to account for model incompleteness.
• Findings:
  • Unconstrained EMR maximisation (MLE) resulted in a sparse solution, heavily weighting a single "Inflationary pressures" scenario ($S_4$) and assigning zero weight to the baseline and other scenarios.
  • Regularised EMR optimisation (Bayes) produced a stable mixture where $S_4$ remained dominant but the baseline and other scenarios received small, non-zero weights.
  • A constrained version (Bayes+), enforcing that the baseline weight be at least as large as any other scenario, resulted in an equal split between the baseline and $S_4$.
  • The resulting mixture distributions achieved high concordance (EMR $\approx$ 0.49) with the reference, though they struggled to fully capture the extreme heavy-tailed nature (leptokurtosis) of the reference skew-t distribution, as the component scenarios were all Student-t with 50 degrees of freedom.
  • Comparisons with KL divergence and Total Variation (TV) distance minimisation showed that EMR-based weights were similar to KL-based weights but provided a more stable optimisation landscape than the non-convex TV approach.

Significance and Claims
The paper claims that EMR offers a natural, interpretable, and practical alternative to traditional divergence measures for assessing distributional concordance. Its significance lies in:

1. Boundedness and Robustness: Unlike KL divergence, EMR remains well-defined even when distributions have incompatible tail behaviors, making it suitable for real-world forecasting where tail assumptions often differ.
2. Decision-Theoretic Foundation: By framing concordance as a misclassification rate, the optimisation is grounded in a bounded utility function, aligning directly with Bayesian decision analysis.
3. Practical Implementation: The convexity of the mixture synthesis problem and the reliance on direct Monte Carlo simulation make the approach computationally feasible for complex, high-dimensional forecasting environments.
4. Regularisation via Synthetic Priors: The synthetic likelihood interpretation provides a clear, interpretable mechanism for introducing regularisation, addressing the instability of sparse solutions in mixture models.

The authors position this methodology as a tool for evaluating the comparative relevance of scenarios in macroeconomic policy and other fields (e.g., financial risk, business forecasting) where multiple models must be synthesized against an empirical benchmark. They note that while the current work focuses on mixtures, the framework is extensible to calibrating general parametric models against reference distributions.