Technical Summary: Breaking the Mirror – Activation-Based Mitigation of Self-Preference in LLM Evaluators

Problem Statement

Large Language Models (LLMs) are increasingly deployed as automated evaluators ("LLMs-as-judges") for tasks lacking ground truth, such as preference tuning and model routing. However, these evaluators suffer from self-preference bias: a tendency to disproportionately favor their own outputs over those of other models, even when authorship is hidden. This bias scales with model size and performance, undermining the fairness and reliability of evaluation pipelines. While prior work has focused on detecting this bias or mitigating it through expensive fine-tuning (e.g., Direct Preference Optimization) or style changes, there is a lack of lightweight, inference-time interventions that can correct unjustified self-preference without retraining the model.

Methodology

The authors propose a framework to disentangle self-preference bias from genuine output quality and apply steering vectors to mitigate the bias at inference time.

1. Disentangling Bias from Quality

To distinguish between illegitimate self-preference (bias) and legitimate self-preference (where the model's output is objectively better), the authors construct a curated evaluation set based on the XSUM dataset.

• Setup: A self-evaluating model $J$ and a comparison model $K$ generate summaries for source articles.
• Gold Standard: An ensemble of "gold judges" ($G$) from diverse model families (Phi-4, DeepSeek V3, Claude 3.5-Sonnet) determines the objectively better summary for each pair.
• Categorization: Each evaluation instance is classified into three outcomes:
  1. Illegitimate Self-Preference: Model $J$ prefers its own output, but gold judges prefer $K$'s.
  2. Legitimate Self-Preference: Both $J$ and gold judges prefer $J$'s output.
  3. Unbiased Agreement: Both $J$ and gold judges prefer $K$'s output.
• Validation: Human validation confirms high agreement between gold judges and humans for legitimate cases and unbiased agreement, though agreement drops for the bias subset, highlighting the difficulty of the task.

2. Steering Vector Construction

The authors construct lightweight steering vectors to modulate model activations during inference using two methods:

• Contrastive Activation Addition (CAA): Computes the difference in hidden-state activations between prompts leading to unbiased completions ($X^+$) and biased completions ($X^-$). The vector is the average difference: $v_{CAA} = \text{mean}(h_L(X^+)) - \text{mean}(h_L(X^-))$.
• Optimization-Based Steering: Uses gradient descent to learn an additive vector that minimizes a contrastive loss function, promoting the desired completion ($Y^+$) and suppressing the undesired one ($Y^-$). This is optimized on layers 14–16 of Llama-3.1-8B-Instruct.

3. Evaluation Metrics

The effectiveness of the steering vectors is measured by:

• Effectiveness: The "flip rate" of illegitimate self-preference (how often the steered model corrects a biased decision).
• Stability: The preservation of correct decisions in legitimate self-preference and unbiased agreement cases (low flip rates in these categories are desired).

Key Results

The study evaluates the steering vectors on Llama-3.1-8B-Instruct against baselines including prompting and Direct Preference Optimization (DPO).

• High Effectiveness on Bias: Steering vectors successfully reduced unjustified self-preference by up to 97%. Both CAA and optimization-based methods outperformed prompting (0% flip rate) and DPO (49% flip rate).
• Surprising Generalization: Vectors constructed in an "unaware" setting (hiding authorship) performed comparably to or better than those in an "aware" setting, suggesting the representation of self-preference has a partially linear structure in activation space.
• Instability on Legitimate Cases: A critical limitation is the lack of stability. The same vectors that successfully flip illegitimate bias also frequently flip legitimate self-preference and unbiased agreement. For instance, CAA vectors flipped 93% of legitimate self-preference cases and 20% of unbiased agreement cases.
• Implication: This instability suggests that while illegitimate self-preference may be encoded linearly, legitimate self-preference and unbiased agreement likely span multiple or nonlinear directions in the activation space, making them difficult to isolate with a single linear vector.

Significance and Claims

The paper claims to demonstrate that lightweight steering vectors offer a promising, low-cost alternative to fine-tuning for mitigating self-preference bias in LLM evaluators. The primary contribution is the empirical demonstration that inference-time activation edits can drastically reduce unjustified bias (up to 97% correction).

However, the authors maintain a modest stance regarding the universality of this solution. They explicitly state that the current approach is not robust for preserving legitimate judgments. The results underscore that self-preference is a complex phenomenon where bias and quality are entangled in the model's representation. The paper concludes that while steering vectors are a powerful tool for "breaking the mirror" of self-preference, their current linear limitations necessitate more robust, potentially nonlinear interventions to safeguard LLM-as-judge systems without compromising valid evaluations.

The authors emphasize that their work provides a curated dataset and a reproducible framework for future research into mechanistic interpretability and bias mitigation, rather than a final, perfect solution for all evaluation scenarios.