Introducing the b-value: combining unbiased and biased estimators from a sensitivity analysis perspective

1. Problem Statement

In empirical research, analysts often possess multiple estimators for the same parameter of interest ($\tau$). These estimators typically fall into two categories:

1. Unbiased but less precise: Derived from methods like Randomized Controlled Trials (RCTs) or Instrumental Variables (IV), which are robust to confounding but suffer from high variance (large sample sizes are often costly or infeasible).
2. Biased but more precise: Derived from observational data or Ordinary Least Squares (OLS), which have lower variance but are subject to unknown bias ($\Delta$) due to unmeasured confounding or model misspecification.

The Core Challenge: While point estimation methods exist to combine these estimators (e.g., shrinkage estimators), valid statistical inference (confidence intervals and hypothesis testing) remains difficult when the bias $\Delta$ is unknown. Standard confidence intervals based on the unbiased estimator are often too wide to be useful, while intervals based on the biased estimator are invalid if the bias is large.

The Goal: The authors aim to construct a sequence of valid confidence intervals indexed by the magnitude of the unknown bias and introduce a metric to quantify the robustness of statistical conclusions against this bias.

2. Methodology and Framework

2.1 Problem Setup

The authors formalize the problem in a finite-sample Gaussian setting (justified by Le Cam's asymptotic theory):

• Unbiased Estimator: $\hat{\tau}_0 \sim N(\tau, \sigma_0^2)$
• Biased Estimator: $\hat{\tau}_1 \sim N(\tau + \Delta, \sigma_1^2)$
• Assumption: $\sigma_0^2$ and $\sigma_1^2$ are known; $\Delta$ is unknown.
• Bias Constraint: The analysis assumes the relative bias is bounded: $|\Delta/\sigma_0| \leq b$, where $b \geq 0$ is a sensitivity parameter.

2.2 The Sequence of Confidence Intervals

Instead of a single confidence interval, the authors propose a sequence of intervals $\{I(b, \zeta)\}_{b \geq 0}$ for a given significance level $\zeta$.

• The interval is constructed to have uniform coverage probability $1-\zeta$ over all $\Delta$ satisfying $|\Delta/\sigma_0| \leq b$.
• As the assumed bias bound $b$ increases, the confidence interval widens to maintain validity.
• This sequence acts as a sensitivity curve, showing how inference changes as the researcher relaxes the assumption of "no bias."

2.3 The $b$-value

A central contribution is the definition of the $b$-value ($b^*$).

• Definition: For a hypothesis test $H_0: \tau = 0$ vs. $H_1: \tau \neq 0$, the $b$-value is the critical threshold of the relative bias bound $b$ at which the conclusion of the test changes from "reject $H_0$" to "fail to reject $H_0$."
• Interpretation: It represents the maximum relative bias the data can tolerate before the statistical significance of the result vanishes. A larger $b$-value indicates a more robust inference.
• Analogy: The $b$-value is analogous to the $E$-value (VanderWeele & Ding, 2017) or design sensitivity (Rosenbaum, 2004) but is specifically tailored for combining estimators.

2.4 Combined Estimators Analyzed

The paper evaluates three canonical estimators to construct these intervals:

1. Precision-Weighted Estimator ($\hat{\tau}_{PW}$): A linear combination of $\hat{\tau}_0$ and $\hat{\tau}_1$.
   • Issue: While efficient when bias is zero, its risk and confidence interval width grow unbounded as bias increases. It is not robust.
2. Pretest Estimator ($\hat{\tau}_{PT}$): Uses a pre-test to decide between the precision-weighted estimator (if bias seems small) and the unbiased estimator (if bias seems large).
   • Issue: Discontinuous at the threshold; the coverage probability is not monotonic in bias, making the worst-case coverage difficult to compute and the intervals conservative.
3. Soft-Thresholding Estimator ($\hat{\tau}_{ST}$): A continuous version of the pretest estimator that shrinks the biased estimator toward the unbiased one rather than discarding it abruptly.
   • Advantage: It ensures continuity and exhibits a monotonicity property: the coverage probability decreases monotonically as $|\Delta|$ increases. This allows for efficient computation of the worst-case coverage at the boundary of the bias constraint.

3. Key Results and Theoretical Contributions

3.1 Analytical Characterization

The authors derive explicit formulas for the confidence intervals and $b$-values for all three estimators:

• For $\hat{\tau}_{PW}$: The interval width depends on the folded normal distribution. The $b$-value is finite only if the initial estimate is significant; otherwise, it is 0.
• For $\hat{\tau}_{PT}$: The coverage probability involves a mixture of normal and truncated normal distributions. The worst-case coverage must be found numerically over the interval $[0, b]$ because it is not monotonic.
• For $\hat{\tau}_{ST}$: The coverage probability is symmetric and monotonically decreasing in $|\Delta|$. Consequently, the worst-case coverage occurs exactly at the boundary $|\Delta|/\sigma_0 = b$. This allows for highly efficient computation of the confidence interval and $b$-value using root-finding algorithms.

3.2 Comparison and Recommendation

• Robustness: $\hat{\tau}_{ST}$ has bounded worst-case risk, whereas $\hat{\tau}_{PW}$ does not.
• Efficiency: When bias is small, $\hat{\tau}_{ST}$ achieves confidence intervals nearly as short as $\hat{\tau}_{PW}$.
• Inference Quality: $\hat{\tau}_{ST}$ yields larger $b$-values (more robust conclusions) compared to $\hat{\tau}_{PT}$ and $\hat{\tau}_{PW}$.
• Recommendation: The authors recommend reporting the $b$-value based on the soft-thresholding estimator along with its associated sequence of confidence intervals.

3.3 Generalizations

The framework is extended to:

• Dependent Estimators: Using a reparametrization to transform dependent estimators into independent ones.
• Multivariate Parameters: Extending to vector-valued parameters ($\tau \in \mathbb{R}^d$) using ellipsoidal confidence regions and defining the $b$-value as a boundary surface in the bias space.
• Multiple Biased Estimators: Combining one unbiased estimator with $K$ potentially biased estimators, defining the $b$-value as a $(K-1)$-dimensional surface.

4. Empirical Application

The authors apply the framework to the classic Angrist and Krueger (1991) study on the returns to schooling:

• Data: OLS (biased, precise) vs. IV (unbiased, imprecise) estimates.
• Findings:
  • In the full sample, the IV estimator alone is precise enough to reject the null hypothesis of zero returns.
  • In the subsample of Black men, the IV estimator is too noisy (wide confidence interval) to reject the null.
  • By combining with the precise OLS estimator using the soft-thresholding method, the confidence interval narrows significantly under small bias assumptions, allowing the null to be rejected.
  • The $b$-value quantifies exactly how large the OLS bias must be to invalidate this conclusion, providing a transparent measure of robustness.

5. Significance and Impact

• Bridging Estimation and Inference: The paper moves beyond point estimation (which has been well-studied) to provide a rigorous framework for valid statistical inference in the presence of unknown bias.
• Sensitivity Analysis Tool: It offers a concrete, data-driven alternative to subjective sensitivity analysis. Instead of asking "what if the bias is large?", researchers can report "the conclusion holds as long as the relative bias is less than $b^*$."
• Practical Utility: The introduction of the $b$-value provides a single, interpretable metric for comparing different data fusion strategies, similar to how $p$-values are used for hypothesis testing.
• Software: The authors provide a Python package (b-value) to facilitate the computation of these intervals and values for practitioners.

In summary, this paper provides a rigorous, computationally efficient, and interpretable methodology for combining high-precision biased data with low-precision unbiased data, ensuring that statistical conclusions are robust to the inevitable uncertainty regarding unmeasured confounding.