Technical Summary: The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice

Problem Statement

Offline evaluation of language models (LLMs) increasingly relies on usage logs, preference data, and judge annotations collected in real-world deployments. However, when model choice is user-driven, these logs suffer from confounding: the same unobserved user-side factors ($U$) that influence which model a user selects also influence how that model's output is judged. Consequently, raw comparisons of logged scores mix self-selected subpopulations rather than estimating a common causal quantity of interest.

While randomized experiments (EXP) can break this bias by overriding model choice, they are often scarce, costly, and disruptive in practice. Existing literature on combining large biased observational samples with small randomized ones typically assumes the unit-level outcome is observed in both. In LLM evaluation, the "outcome" (the generated text) must first be regenerated for candidate models before it can be scored, adding a layer of complexity where the target outcome is not directly observed even in the randomized sample.

Methodology: A Three-Source Design

The paper proposes a three-source design to address this challenge, separating the problem into identification (recovering causal values) and estimation (reducing error).

Data Sources

1. Observational Log (OBS): A large dataset of self-selected model choices, contexts, outputs, and outcomes. It provides scale and auxiliary supervision but is confounded.
2. Randomized Experiment (EXP): A small dataset where model choice is randomized, providing unconfounded outcome labels for realized outputs.
3. Offline Simulator (SIM): A mechanism that replays candidate models on cached contexts to generate counterfactual outputs (mediators) for any model, regardless of the original choice.

Causal Structure and Assumptions

The framework models the system using a causal graph with observed context ($X$), latent user state ($U$), chosen model ($A$), generated output ($M$), and outcome ($Y$). Key assumptions include:

• A1 (Randomization): In EXP, $A \perp U | X$.
• A2 (SIM Validity): The simulator accurately matches the interventional mediator distribution $P(M | X, do(A=a))$.
• A3 (Outcome Consistency): The conditional outcome law $P(Y | X, M, U)$ is invariant across OBS, EXP, and the interventional world.
• A4 (No Latent Mediator Confounding): Conditional on $(X, A)$, $U \perp M$. This restricts the framework to single-round, stateless generation.
• A5 (Context Alignment): The target context distribution matches the EXP context distribution.

Identification Theorem

The paper's central theoretical result (Theorem 1) establishes that SIM and EXP are jointly sufficient to identify causal model values, rendering OBS unnecessary for identification.

• The randomized experiment identifies the unbiased scoring rule $r^*(x, m) = E[Y | X=x, M=m, EXP]$.
• The simulator provides the interventional mediator distribution $p_{sim}(m | x, a)$.
• The causal value is recovered by integrating the scoring rule over the simulated mediator distribution:
  $$q(x, a) = E_{M \sim p_{sim}(\cdot|x,a)}[r^*(x, M)]$$
  $$\mu(a) = E_{X}[q(X, a)]$$
  The observational log (OBS) enters only after identification to improve statistical efficiency (reduce variance), not to validate the causal comparison.

Estimator Families

Once the target is identified, the paper evaluates six estimator families that differ in how they utilize OBS to reduce estimation error:

1. EXP-Only: Fits a reward model using only randomized outcomes. Unbiased but high variance when EXP is small.
2. OBS-Only: Fits a reward model on logged outcomes. Low variance but inherits confounding bias.
3. Representation-EXP: Uses OBS auxiliary labels to learn a lower-dimensional proxy representation $\psi$, then fits the reward head on $\psi$ using EXP data.
4. Grounded: Starts with an OBS-trained baseline predictor $f_{OBS}$ and uses EXP to learn a correction term in the proxy space.
5. CVCI (Cross-Validated Causal Inference): Directly pools OBS and EXP regression problems, selecting the pooling weight $\lambda$ via causal cross-validation on EXP.
6. CVCI-Residual: Residualizes the target around the OBS baseline ($Y - f_{OBS}$) and pools the residual fit in proxy space.

Empirical Results

The authors evaluate these estimators in a controlled semi-synthetic validation and two real-task cached benchmarks (summarization on CNN/DailyMail and coding on SWE-bench Verified).

• No Universal Dominance: No single estimator family dominates all regimes. Performance depends heavily on the amount of unbiased EXP supervision and the alignment between the observational structure and the target reward.
• Summarization Benchmark:
  • CVCI achieved the best average rank and lowest excess regret across 48 settings.
  • Increasing the EXP budget ($n_{EXP}$) significantly improved EXP-based estimators, while increasing OBS budget ($n_{OBS}$) had diminishing returns under smooth reward maps.
  • When the reward map was "sharpened" (non-linear), the alignment between proxy representations and the target degraded, causing performance drops for representation-based methods.
• Coding Benchmark:
  • Rankings shifted based on the reward definition. When the reward prioritized "patch quality" (auxiliary features), Representation-EXP performed competitively.
  • When the reward prioritized "true fix success" (a binary outcome not well captured by auxiliary patch statistics), OBS-based families (OBS-Only, CVCI) overtook Representation-EXP.
  • This highlights that the utility of OBS-derived proxies depends on whether they retain the signal relevant to the specific target reward.

Significance and Claims

The paper claims to provide a rigorous framework for offline LLM evaluation that acknowledges the limitations of observational data without discarding its scale.

• Identification vs. Estimation: The primary contribution is the theoretical separation of identification (solved by SIM+EXP) from estimation (where OBS helps). This clarifies that OBS cannot fix causal bias; it can only reduce variance once the bias is removed via randomization and simulation.
• Practical Guidance: The empirical results suggest that the choice of estimator should be guided by the "target-alignment" diagnostic. If the observational data's structure (e.g., auxiliary labels) aligns well with the target reward, hybrid methods (like CVCI or Grounded) are effective. If the target relies on signals not present in the observational proxies (e.g., true fix success vs. patch quality), OBS-assisted methods may fail to outperform pure randomized baselines.
• Scope: The authors modestly note that these conclusions apply to one-shot cached-evaluation settings. The framework assumes single-round generation (Assumption A4) and does not yet address multi-round interactive settings or broader population-level uncertainties beyond the cached targets.

In summary, the paper argues that while large logs are valuable for scale, they are insufficient for causal evaluation on their own. A small randomized experiment combined with a simulator is the minimal requirement for valid identification, with observational data serving as a variance-reduction tool contingent on the specific alignment between the data and the evaluation goal.