Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

Technical Summary: Evidence Lock Before Commitment

Problem Statement
The paper addresses a specific workflow assumption prevalent in LLM-as-a-judge evaluation: that extracting criteria and evidence in an intermediate step before rendering a final verdict preserves sufficient information for a later, separate decision call. While structured evaluation methods (e.g., asking for plans or rationales before a score) are motivated by the need for auditability and robustness, they often assume that a natural-language record of evidence is a sufficient interface to reproduce a judgment. The author questions whether persisting this evidence record and making it the exclusive input to a final decision call—thereby removing access to the original source answers—degrades evaluation quality. This is particularly relevant for reasoning-capable models where visible output order does not necessarily reflect internal decision order.

Methodology
The study employs a controlled, repeated-measures evaluation across 24,000 judgments using two reasoning-capable proprietary judges (Claude Sonnet 4.5 and GPT-5) and three datasets (HelpSteer3, FeedbackQA, and CoVal). The researchers compare four distinct judging protocols:

1. Standard Pairwise Judging: The judge sees the context and both answers (A and B) and outputs a verdict directly.
2. Structured One-Call Judging: The judge sees the context and both answers but is prompted to explicitly generate criteria and evidence before selecting a winner within the same call.
3. Evidence Lock (Two-Call):
   • Call 1: The judge inspects the context and both answers, writes evidence/criteria, and persists the record without selecting a winner.
   • Call 2: The judge sees only the persisted evidence record (no original context or answers) and selects A, B, or a tie.
4. Pointwise Lock (Three-Call):
   • Calls 1 & 2: Answers A and B are evaluated independently to generate frozen assessments.
   • Call 3: The judge compares the two frozen assessments to determine the winner, without seeing the original answers.

The study measures two primary outcomes:

• Preference Agreement: The accuracy of the judge's verdict against released strict human-preferred labels (A/B), averaged over both answer presentation orders (AB and BA).
• Answer-Order Inconsistency: The frequency with which the normalized verdict changes when the presentation order of candidates is swapped.

Key Contributions

1. Observable Intervention: The paper introduces "Evidence Lock," an observable intervention that isolates the effect of freezing an intermediate record and removing source access, without relying on unverifiable claims about hidden chain-of-thought ordering.
2. Large-Scale Evaluation: A comprehensive evaluation involving 24,000 judgments across two models and three diverse datasets, utilizing non-parametric bootstrap resampling for statistical confidence.
3. Negative Result on Locked Interfaces: The study provides consistent evidence that while structured elicitation (asking for evidence in one call) remains effective, converting that evidence into a frozen, exclusive interface for a subsequent decision call significantly degrades performance.

Results
The experimental results demonstrate that "locking" evidence leads to measurable degradation in evaluation quality compared to structured one-call judging:

• Reduced Preference Agreement: Evidence locking reduced agreement with human preferences by 4 to 6 percentage points relative to structured one-call judging.
  • Claude Sonnet 4.5: Dropped from 76.0% (structured) to 71.7% (evidence lock).
  • GPT-5: Dropped from 75.6% (structured) to 70.0% (evidence lock).
  • Pointwise locking showed even lower agreement (67.3% and 69.2%).
• Increased Order Inconsistency: Locked protocols significantly increased sensitivity to answer order.
  • Evidence locking increased inconsistency by 8 to 10 percentage points relative to structured judging.
  • For Claude Sonnet 4.5, inconsistency rose from 10.9% (structured) to 18.9% (evidence lock).
• Computational Cost: Locked protocols require more API calls and tokens (e.g., 4,183 tokens for evidence lock vs. 1,864 for structured judging for Claude) while producing worse results.
• Generalizability: The negative effects were consistent across all three datasets (HelpSteer3, FeedbackQA, CoVal) and both judge models.

Significance and Claims
The paper concludes that persisted evidence records should not replace the source answers at decision time. The degradation in performance occurs specifically when the evidence record becomes the exclusive interface to the final verdict, not merely when evidence is requested.

• Information Flow: The author argues this is an "information-flow effect." Natural-language records may omit distinctions, compress trade-offs, or preserve framing dependent on the original presentation order. Once the final decision call loses access to the source answers, it cannot correct for these omissions or framing choices.
• Auditability vs. Decision Making: While persisted evidence supports auditability, logging, and modularity, the practical lesson is that the record should assist the decision rather than replace the original pair. A final judge should ideally inspect both the artifact and the source answers.
• Modesty of Claims: The author explicitly states that the "Evidence Lock" is a bundled intervention (combining stage separation, an additional call, record freezing, and source removal). Therefore, while the workflow causes degradation, the specific component responsible (e.g., the freezing vs. the removal of source access) is not uniquely identified, though the "information-bottleneck" account is presented as a plausible interpretation. The study does not claim to prove hidden reasoning mechanisms but rather demonstrates the observable consequences of restricting input data.