EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems

Technical Summary: EPC – A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems

Problem Statement
In closed-loop LLM agent systems, agents adapt their strategy distributions based on feedback from evaluators (often other LLMs). This process leads to "evaluator preference coupling," where the evaluator's inherent biases propagate into and distort the agent's strategy distribution. While prior work has documented this phenomenon across various models, the field lacks a standardized protocol. Current research suffers from:

1. Non-reproducibility: Studies use divergent task sets, strategy definitions, and metric implementations, making cross-study comparison impossible.
2. Measurement Decay: Proprietary evaluators update silently, causing previously reported coupling measurements to become invalid within weeks.
3. Lack of Versioning: There is no mechanism to track which specific model versions or API endpoints a measurement corresponds to, leading to the accumulation of stale baselines in the literature.

Methodology: The EPC Protocol
The paper introduces EPC (Evaluator Preference Coupling), a detailed, RFC-style protocol specification designed to standardize the measurement of these dynamics. The methodology is built on a four-phase isolation paradigm:

• Phase 1 (Pure Text): The agent undergoes Test-Time Reinforcement Learning (TTRL) on text-only tasks to establish a baseline weight distribution.
• Phase 2 (Pure Visual): The agent undergoes TTRL on visual-adjacent tasks to establish a separate baseline.
• Phase 3 (T→V Coupling): The agent starts with Phase 1 weights and trains on visual tasks.
• Phase 4 (V→T Coupling): The agent starts with Phase 2 weights and trains on text tasks.

Key Technical Components:

• TTRL Algorithm: The agent maintains an L1-normalized weight vector over 11 strategies (8 text, 3 visual). Updates occur via pairwise comparisons against a fixed step_by_step baseline. The protocol fixes learning rates ($\alpha_{win}=0.08, \alpha_{lose}=0.04$) and a weight floor (0.001) to ensure stability.
• Metrics:
  • $\gamma$ (Coupling Coefficient): The primary metric, defined as the L2-normalized Euclidean distance between the post-coupling weight vector and the pure-domain reference vector.
  • JSD (Jensen-Shannon Divergence): Required for reporting if weight vectors are stored, to track distribution concentration.
  • ECE & Brier Score: Optional metrics for calibration and error analysis.
• Strict Configuration: The protocol mandates specific evaluator prompts, decoding settings (temperature=0.0), and task documentation (verbatim) to minimize variance.
• Conformance: A test suite with mock evaluators ensures independent implementations produce identical results on fixed inputs.

Key Contributions
The paper explicitly states it does not introduce new empirical findings, metrics, or scientific claims regarding agent capabilities. Instead, its contributions are infrastructural and disciplinary:

1. Standardized Protocol: A complete, RFC-style specification (EPC-v1.0) covering configuration, algorithm, task design, and output schemas.
2. Reference Snapshot v1.0: A versioned baseline containing coupling measurements ($\gamma$, JSD, zero-coupling rates) for eight evaluator conditions (including GPT-4o, Qwen, DeepSeek) derived from five independent studies conducted between May and June 2026.
3. Versioning Convention: A three-component scheme ($vX.Y-Z$) encoding the protocol version, snapshot version, and evaluator generation to explicitly track measurement validity and decay.
4. Open Infrastructure: Release of the protocol implementation, reference data (JSON), and a submission template for community-contributed snapshots.

Results and Observations
The Reference Snapshot v1.0 provides empirical baselines that highlight the volatility of evaluator coupling:

• Variability: Coupling coefficients ($\gamma$) vary significantly across evaluators. For instance, GPT-4o (2024-08-06) showed a mean $\gamma$ of 1.176, while DeepSeek-chat (self-eval) showed 0.033.
• Decay and Drift: The snapshot demonstrates that measurements can invert or change drastically within short timeframes. GPT-4o measurements taken in May vs. June showed a shift from high coupling to near-zero coupling (0.187) with an 83% zero-coupling rate, attributed to model version updates.
• Calibration Issues: Self-evaluation baselines revealed that evaluators can be substantially miscalibrated (ECE $\approx$ 0.31) even when showing low coupling, suggesting that low coupling may sometimes reflect evaluator incapacity rather than agent stability.

Significance and Claims
The authors position this work as a protocol specification paper, analogous to an RFC in networking or a measurement standard in physical sciences.

• Discipline over Discovery: The paper claims to introduce a "discipline" that enables the community to collectively maintain currency as evaluator models evolve, transforming one-off observations into a reproducible, comparable, and auditable measurement system.
• Addressing Decay: By making measurement decay explicit through versioning and expiration dates, the protocol aims to prevent the reliance on stale baselines, a problem highlighted by recent audits showing AI benchmarks have low longevity scores.
• Complementarity: EPC is designed to complement existing agent capability benchmarks (which measure how well agents perform) by measuring evaluator influence (how biases propagate through feedback loops).

The paper concludes by releasing the protocol and data as open infrastructure, inviting the community to contribute updated snapshots as evaluator models evolve, thereby ensuring the longevity and relevance of the measurement system.