Post-Hoc Inference of Cross-Classified Statistics from Hierarchical Bayes Survey Weights

1. Problem Statement

The paper addresses a critical operational gap in modern survey methodology. Following the framework established in Tam [2026], National Statistical Offices (NSOs) can significantly reduce survey sample sizes by combining Bethel multivariate allocation with Hierarchical Bayes (HB) small area models. In this framework, HB models generate posterior estimates for specific target variables (e.g., employment, hours worked) across geographic domains, and the sample is reduced while maintaining precision for these targets.

However, once the reduced survey is fielded, NSOs must publish cross-classified statistics (e.g., employment by occupation, income by health status) that were not the explicit targets of the original HB calibration. These tables often involve:

1. Filters on calibration variables (e.g., employment by age group).
2. Non-calibration variables (e.g., health status or income bands).

The Core Issue: Standard point estimation is straightforward (using calibrated weights), but uncertainty quantification is problematic.

• If uncertainty is ignored, tables appear artificially precise.
• If uncertainty is propagated mechanically from HB posterior draws without accounting for the specific structure of the cross-tabulation, the resulting Credible Intervals (CrIs) often suffer from severe under-coverage (i.e., the true population value falls outside the interval far more often than the nominal 5% rate).

2. Methodology: The Post-Hoc Inference Engine (PHIE)

The paper introduces the Post-Hoc Inference Engine (PHIE) to propagate uncertainty from HB domain posterior draws to arbitrary cross-tabulations.

A. Posterior-Calibrated Replicate Weights

For each MCMC draw $b$ of the HB posterior domain totals $\hat{T}^{(b)}$, PHIE generates a new set of survey weights $w'_i(\hat{T}^{(b)})$.

• Mechanism: It solves a constrained optimization problem (minimizing the chi-square distance from original Horvitz-Thompson weights) subject to the constraint that the weighted sum of calibration variables equals the specific MCMC draw $\hat{T}^{(b)}$.
• Formula: The weights are updated linearly:
  $$w'_i(t) = w_i \left( 1 + (t - \hat{T}^{HT})^\top G^{-1} y_i \right)$$
  where $G$ is the calibration cross-product matrix, $y_i$ is the design vector, and $t$ is the target vector (the MCMC draw).
• Result: Applying these replicate weights to any cross-tabulation statistic generates an empirical posterior distribution, from which CrIs are derived.

B. Three-Tier Taxonomy of Statistics

The paper argues that not all cross-tabulations have the same inferential status. It categorizes them into three tiers, requiring different variance correction strategies:

1. Tier 1-E (Exact Calibration Cells):

   • Definition: Cells that exactly reproduce one of the $p$ HB calibration constraints (e.g., Total Employment in NSW).
   • Inference: The PHIE CrI is exact. The calibration property guarantees that the interval matches the HB posterior interval.
   • Coverage: Nominal (100% in single run, ~95% in repeated sampling).

2. Tier 2 (Filtered Calibration Variables):

   • Definition: Sums of a calibration variable filtered by an attribute (e.g., Employment by Age Group).
   • Problem: PHIE intervals only propagate uncertainty in the domain totals (Component 2) but ignore the sampling variability of the within-domain shares (Component 1). This leads to under-coverage.
   • Solution: A Calibrated Bayes Interval (CBI) is proposed. It adds the design-based variance of the cell share (Component 1) to the model-based variance of the domain total (Component 2).
   • Formula:
     $$\text{Var}(T_c) \approx \sum_d \hat{T}_{v,d}^2 \widehat{\text{Var}}(\hat{\lambda}_{d,c}) + \sum_d \hat{\lambda}_{d,c}^2 \widehat{\text{Var}}(\hat{T}_{v,d})$$
     where $\hat{\lambda}$ is the estimated share and $\hat{T}$ is the domain total.

3. Tier 3-NCV (Non-Calibration Variables):

   • Definition: Statistics involving outcome variables not in the HB calibration system (e.g., Health Status by Occupation).
   • Problem: No direct HB posterior exists for these variables.
   • Solution: A Ratio-Estimator CBI. The non-calibration variable is treated as the numerator of a ratio, with the denominator being the HB posterior total of the most highly correlated calibration variable (the "linking variable").
   • Inference: The CBI combines the design-based variance of the ratio with the HB posterior variance of the linking variable.

3. Key Contributions

• PHIE Framework: A unified engine to transform HB posterior draws into replicate survey weights, enabling uncertainty propagation for any derived statistic.
• Taxonomy of Inference: The identification that cross-tabulations fall into distinct tiers with different variance structures, necessitating different interval types (Exact CrI vs. CBI).
• CBI Correction: The development of a Calibrated Bayes Interval that bridges the gap between Bayesian model uncertainty and frequentist design-based sampling variability. This ensures near-nominal coverage even when the statistic is not a direct calibration target.
• Diagnostics: Introduction of diagnostic metrics ($\|a_c\|$ and $\cos \theta_c$) to detect when a cell lacks a distinctive calibration profile, which would cause the model-based component of the variance to collapse.

4. Empirical Results

The methods were validated using Australian 2021 Census microdata (5% sample, $N \approx 840k$, $n \approx 42k$) with 200 Monte Carlo replications.

• Tier 1-E (Hours by Domain): Achieved 99.2% Monte Carlo coverage (near 100% in single run), confirming the exactness of the calibration property.
• Tier 2 (Employment by Hours/Occupation):
  • Raw PHIE: Severe under-coverage (ranging from 28% to 79.5%).
  • CBI Corrected: Coverage restored to 92–97.5%, close to the nominal 95%.
  • Finding: For Tier 2, uncertainty is driven primarily by compositional sampling variability (Component 1), not HB model uncertainty.
• Tier 3-NCV (Income/Health by Occupation):
  • Raw PHIE: Extreme under-coverage (as low as 13.5% for health conditions).
  • CBI Corrected: Coverage restored to 92–98.5%.
  • Finding: Even with weak correlation between the outcome and the linking variable, the CBI (dominated by Component 1) provides reliable coverage.
• Precision: The Coefficient of Variation (CV) for all CBI-based intervals remained within standard publication thresholds (typically <5%), with the highest being 4.9%.

5. Significance and Implications

• Operational Viability: The paper demonstrates that NSOs can safely reduce survey sample sizes using HB models without sacrificing the validity of derived cross-tabulations, provided the PHIE and CBI framework is applied.
• Statistical Rigor: It resolves the tension between "model-based" (HB) and "design-based" inference. The CBI approach acknowledges that while the totals are model-driven, the composition of those totals in cross-tabs retains design-based sampling error.
• Publication Standards: The results show that without the CBI correction, published tables would be misleadingly precise. With the correction, the uncertainty is accurately reflected, and the intervals remain narrow enough for publication.
• Recommendation: NSOs should not treat weighted unit record files from HB surveys as standard data. They must apply the PHIE/CBI engine to generate valid confidence intervals for any cross-classified output, particularly those involving non-calibration variables.

In summary, this paper provides the necessary "inference machinery" to make the sample-reduction benefits of Hierarchical Bayes surveys operationally safe for the production of complex, multi-dimensional statistical tables.