Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

Technical Summary: Who Drifted? Anytime-Valid Attribution in LLM Evaluation Pipelines

1. The Problem

Continuous evaluation of Large Language Model (LLM) products increasingly relies on "LLM-as-judge" pipelines, where a strong model scores interactions to trigger alerts when performance degrades. The fundamental assumption in these systems is that the judge acts as a fixed yardstick; thus, a drop in scores implies a regression in the product. However, the judge itself is an API-based model subject to silent version bumps, prompt updates, or policy changes.

When a monitoring system triggers an alarm, it faces an ambiguity: did the product quality degrade, or did the judge's scoring behavior change? Acting on the wrong attribution is costly: teams may roll back a healthy product or ship a regression by dismissing the alarm as a "judge drift." Classical statistical methods, such as rolling z-tests, fail in this continuous monitoring context because they lack anytime-validity; re-running a fixed-$\alpha$ test at every step inflates the false-alarm rate, leading to spurious detections even on drift-free streams.

2. Methodology

The paper proposes a framework to disentangle system drift from judge drift using anytime-valid e-processes and a human-anchored attribution rule.

A. The Budgeted Main Monitor (System Drift Detection)
The system employs a stratified, prediction-powered monitor (following Csillag et al., 2025b; Angelopoulos et al., 2023) to detect quality regressions:

• Architecture: A cheap judge ($\mu$) scores every item, while an expensive strong judge ($y$) scores a sampled subset under a fixed budget.
• Mechanism: For each (rubric $\times$ stratum) cell, a prediction-powered e-process accumulates "wealth" to test the null hypothesis that the mean score has not fallen below a healthy baseline.
• Cost Efficiency: The system uses an e-wealth escalation trigger. As evidence of drift accumulates, the sampling probability for the strong judge increases. This allows the system to detect localized blind-spot regressions at approximately one-third the cost of full evaluation while maintaining anytime-validity (guaranteeing the false-alarm rate remains bounded at any stopping time).

B. The Anchor Construction (Judge Drift Detection)
To resolve the attribution ambiguity, the authors introduce a second, independent e-process anchored to fixed human labels:

• Anchor Set ($A$): A fixed, held-out collection of items with frozen human labels ($h$), drawn before monitoring begins.
• Interleave: At a steady rate (e.g., every $1/\rho$ items), the current strong judge re-scores a randomly selected anchor item.
• The E-Process: A betting e-process tracks the gap between the judge's current score on the anchor and the frozen human label.
• Asymmetry: Because the anchor items and human labels are fixed, only a change in the judge can alter the gap distribution. System drift affects incoming items but leaves the anchor gap unchanged. This creates a "one-way mirror" that detects judge drift exclusively.

C. The Attribution Rule (Guard Window)
The system combines the stopping times of the main process ($\tau_{sys}$) and the anchor process ($\tau_{anc}$) using a guard window ($W$) to issue a verdict in $\{none, system, judge\}$:

• Judge Verdict: If the anchor process fires first ($\tau_{anc} < \infty$) and the main process fires within the guard window ($\tau_{sys} \le \tau_{anc} + W$), the verdict is judge. The logic is that a contaminated judge invalidates the main alarm.
• System Verdict: If the main process fires first and the anchor process does not fire within the guard window, the verdict is system.
• Race Condition: The design relies on an "attribution race." To prevent misattribution, the anchor process must be provisioned to detect judge drift faster than the contaminated main process can falsely trigger a system alarm.

3. Key Contributions

1. Anchor Construction & Attribution Rule: A novel method using a human-anchored, anytime-valid e-process to explicitly separate judge drift from system drift, emitting a three-state verdict rather than a binary alarm.
2. Theoretical Guarantees:
   • Anytime-Validity: The anchor family maintains a bounded false-alarm rate ($\alpha_{judge}$) at any data-dependent stopping time.
   • One-Way Identification: The anchor process is mathematically guaranteed to be insensitive to system drift; it can only be triggered by a change in the judge.
   • Attribution Race: The paper establishes a design law where the anchor process must be provisioned to "out-run" the main process. Misattribution (false system alarms during judge drift) is a function of the guard width, anchor rate, and anchor set size.
   • Orthogonality: The anchor budget and main process configuration are independent, allowing for separate tuning of judge-latency and system-power.
3. Empirical Validation on Real Drifts: The method was tested on two real-world judge changes (a silent version bump and a strict-prompt policy update) across two domains (HelpSteer2 and TL;DR summarization) without re-tuning.
4. Operational Superiority: The proposed e-process significantly outperforms classical baselines (naive rolling z-tests and calibrated Page–Hinkley detectors) in terms of false-alarm control and detection of subtle drifts.

4. Results

• Detection Accuracy:
  • Silent Version Bump: Detected as judge drift in 60/60 runs with zero judge-to-system misattribution.
  • Strict-Prompt Change: Correctly attributed in 110/120 runs (at guard width $W=300$) on HelpSteer2. On the TL;DR domain, where the prompt shift was more severe, attribution was perfect (240/240).
  • Misattribution: The rate of false system alarms during judge drift decreased monotonically as the guard window ($W$) increased, confirming the "attribution race" theory.
• Baseline Comparison:
  • A naive rolling z-test (industry default) generated false alarms on 75% of drift-free streams.
  • A calibrated Page–Hinkley detector held its false-alarm budget but failed to detect the subtle real version bump (only 8% detection on HelpSteer2) and required held-out calibration data.
• Cost Efficiency: The monitor operates at approximately 0.64 of the cost of strong-evaluating every item (or 0.21 in a "cheaper-but-deafer" regime), while maintaining valid detection guarantees.

5. Significance and Claims

The paper claims to solve a critical, unaddressed failure mode in LLM operations: the inability to distinguish between product degradation and judge instability in continuous monitoring.

• Practical Impact: The construction provides a sharp guarantee that judge drift is never silently charged to the system beyond a defined error budget ($\alpha_{judge}$). This prevents teams from making expensive, incorrect operational decisions (e.g., rolling back healthy models).
• Theoretical Contribution: It demonstrates that anytime-validity is not just a theoretical nicety but a practical necessity for continuous monitoring, as standard fixed-horizon methods fail catastrophically under sequential testing.
• Design Law: The paper establishes that attribution is a race condition. It provides a concrete "co-provisioning" recipe: the anchor budget must be sized to ensure the anchor process detects judge drift faster than the contaminated main process can trigger a false system alarm.
• Limitations: The authors modestly note that the method assumes human labels remain a valid reference (requiring periodic refresh if standards evolve) and that it does not claim optimality in betting strategies or minimal misattribution at fixed budgets, but rather validity and identification.

In summary, the paper presents a robust, cost-aware framework that transforms the ambiguous "drift alarm" into a verifiable attribution, ensuring that the "yardstick" is monitored as rigorously as the "product" it measures.