Technical Summary: Causal Judge Evaluation (CJE) for Calibrated Surrogate Metrics in LLM Systems

1. Problem Statement

Evaluating Large Language Model (LLM) systems against long-run outcomes (e.g., user satisfaction, expert judgment, downstream KPIs) is prohibitively expensive. Consequently, teams rely on cheap "LLM-as-judge" proxies. However, uncalibrated proxies suffer from systematic biases (e.g., verbosity, style, sycophancy) that can invert policy rankings entirely—a phenomenon the authors term the "surrogate paradox."

Standard approaches face three critical failures:

1. Ranking Inversion: Uncalibrated judge scores ($S$) are not on the same scale as true rewards ($Y$). Without calibration, higher scores can predict lower true value, leading to the deployment of inferior models.
2. Catastrophic Confidence Intervals: Naive confidence intervals (CIs) on uncalibrated scores achieve near-zero coverage (0% in the authors' experiments), failing to capture the true value even at nominal 95% levels.
3. Failure of Off-Policy Evaluation (OPE): Importance-weighted estimators (IPS/SNIPS) fail to provide accurate rankings even when weight stabilization boosts the Effective Sample Size (ESS) to >80%. This is due to a "Coverage-Limited Efficiency" (CLE) problem: the logging policy rarely visits regions where the target policy concentrates, creating a hard precision floor that no amount of weight stabilization can overcome.

Existing solutions often assume the correlation between the judge and the oracle holds across different policies or deployment contexts. This paper argues that such assumptions are uncheckable and brittle.

2. Methodology: Causal Judge Evaluation (CJE)

CJE is a framework that treats surrogate validity as an auditable property rather than an identification assumption. It unifies reward calibration, weight stabilization, and uncertainty quantification into a single protocol.

Core Components

1. Reward Calibration (Two-Stage Isotonic):

   • CJE learns a mapping $f(S, X) \approx E[Y | S, X]$ from a small "oracle slice" (expensive labels) to the judge scores.
   • It employs mean-preserving isotonic regression to ensure the calibration is monotone and preserves the mean of the oracle slice.
   • A two-stage approach is recommended: first, a smooth index $Z(S, X)$ (e.g., using splines and covariates like response length) is learned; second, an isotonic map is fitted on the ranks of $Z$. This captures non-monotonic patterns while enforcing monotonicity and mean preservation.

2. Weight Stabilization:

   • For off-policy evaluation (using logged data), CJE stabilizes importance weights ($W_{\pi'}$) by projecting them onto the cone of monotone functions of the judge score $S$.
   • It uses out-of-fold (OOF) stacking of monotone candidates (increasing, decreasing, and identity) to minimize the variance of the influence function, thereby improving ESS without introducing bias in the mean.

3. Calibration-Aware Inference:

   • Standard CIs treat the calibration function $\hat{f}$ as fixed, ignoring the uncertainty from the small oracle slice. CJE propagates this uncertainty.
   • It uses a delete-one-fold jackknife to estimate calibration variance and adds it to the evaluation variance.
   • For point estimation, it employs a bias-corrected augmented estimator ($\hat{\theta}_{aug}$) derived from the efficient influence function (EIF) for missing-outcome estimation. This corrects for the covariance between the calibration function and the evaluation samples.

4. Auditable Transport Diagnostics:

   • The core innovation is replacing the untestable assumption of global surrogacy with a policy-wise mean transport test.
   • For each target policy $\pi'$, a small oracle sample is used to test the null hypothesis $H_0: E_{\pi'}[Y - f(S, X)] = 0$.
   • Pass: The calibration is reused.
   • Fail: The system either recalibrates with policy-specific data or refuses level claims (reporting only rankings or refusing absolute value estimates) for that policy.

Theoretical Foundations

• Efficiency via Model Restriction: CJE formalizes that encoding justified knowledge (monotonicity, mean-preservation) as model restrictions shrinks the tangent space, thereby reducing the efficiency bound (variance) without changing the estimand.
• Coverage-Limited Efficiency (CLE): The paper derives a lower bound on the standard error of IPS-style estimators that depends on the Target-Typicality Coverage (TTC). If the logger rarely visits target-typical regions (low TTC), the precision floor is prohibitive regardless of weight stabilization. This explains why OPE fails in open-ended generation.

3. Key Results

The authors validate CJE on a large-scale benchmark using 4,961 Chatbot Arena prompts, 5 LLM policies (including an adversarial "unhelpful" policy), and a 16x cost ratio between the oracle (GPT-5) and the judge (GPT-4.1-nano).

• Ranking Accuracy: At a 5% oracle fraction, CJE (Direct method with two-stage calibration) achieves 99% pairwise ranking accuracy, compared to 38% for standard SNIPS. Across all configurations, average accuracy is 94%.
• Confidence Interval Coverage: Naive CIs on uncalibrated scores achieve 0% coverage. CJE's bootstrap inference with bias correction achieves ~95% coverage (near-nominal).
• OPE Failure Confirmation: Despite weight stabilization boosting ESS from <1% to >80%, calibrated IPS ranking accuracy remained near-random (47%). The CLE bound and low TTC (19–49% for non-clone policies) correctly predicted this failure.
• Direct vs. DR: Under low overlap, Doubly Robust (DR) estimators did not outperform the Direct method; in fact, Direct slightly outperformed DR (94.4% vs. 94.1%) because the IPS component added noise rather than information.
• Adversarial Detection: The transport audit correctly flagged the adversarial "unhelpful" policy, which failed the mean transport test (mean residual = -0.31), indicating the surrogate systematically overestimated its quality.
• Cost Efficiency: CJE achieves near-oracle accuracy at 14x lower cost than pure oracle labeling for a set of 5 policies (amortizing the calibration cost).

4. Significance and Claims

The paper claims that CJE transforms LLM evaluation from a brittle, assumption-heavy practice into a rigorous, auditable protocol.

• From Assumption to Audit: CJE replaces the "hope that correlation holds" with a falsifiable diagnostic (the mean transport test). This allows teams to detect when a calibration fails to transfer to a new policy or context.
• Unified Framework: It unifies three previously disjoint problems: calibration (mapping $S \to Y$), variance control (weight stabilization), and uncertainty quantification (calibration-aware inference).
• Operational Guidance: The paper provides concrete "gates" for practitioners:
  • Default to Direct + two-stage calibration for open-ended generation.
  • Audit transport for high-stakes decisions.
  • Gate on TTC (Target-Typicality Coverage); if TTC < 0.70, logged-data OPE is likely to fail.
  • Always use bootstrap inference with bias correction to avoid catastrophic under-coverage.

The authors emphasize that while CJE significantly reduces the cost of evaluation, it does not eliminate the need for oracle labels; rather, it optimizes their allocation and ensures that the cheap proxies used for scaling are statistically valid for the specific policies being evaluated. The framework is implemented in an open-source library (cje-eval) and validated across preference ranking and code generation tasks.