Technical Summary: Eval-Pair Matrix

Problem Statement

Retrieval-Augmented Generation (RAG) systems face a critical evaluation challenge: an answer may be fluent and well-cited yet contradict the source passages. While "LLM-as-a-judge" has become a standard for evaluating RAG due to its speed and ability to emit structured rationales, a significant methodological gap exists when the generator and the judge belong to the same model family.

Existing literature on self-preference (e.g., Dubois et al., 2024; Shi et al., 2024) suggests judges may favor outputs from their own model family. However, this is often an identification problem: a judge might prefer its own output because it is stylistically familiar or higher quality, rather than due to "self-leniency" (failing to penalize errors). In source-grounded RAG, the task shifts from preference to factual consistency. The core question is whether a judge fails to detect contradictions in its own model's output when those contradictions are induced but hidden from the judge.

Methodology: Eval-Pair Matrix

The authors introduce Eval-Pair Matrix, a controlled meta-evaluation protocol designed to isolate same-model judging effects from generator quality and answer difficulty.

1. Data Construction and Perturbation

The study utilizes GaRAGe, a benchmark with human-written answers and passage-level grounding.

• Perturbation: For each record, an LLM perturber selects one "answer-causal" proposition (a claim essential to the answer) and rewrites all supporting passages to support a plausible but false replacement value (e.g., changing a 48% statistic to 35%).
• Validation: A five-gate validation process ensures the perturbation is valid, causal to the answer, globally consistent, free of original-value leakage, and that the original passage contradicts the perturbed one.
• Dataset: The final analysis uses 275 fully validated records (from an initial pool of 300), generating 897 usable answers.

2. The Crossed Matrix Design

The experiment employs a $3 \times 3$ crossed matrix using three model families: GPT, Grok, and Gemini.

• Generators: Each model generates answers using only the perturbed (false) grounding.
• Judges: The same three models act as blind judges. They evaluate the generated answers against the original (unperturbed) grounding.
• Blindness: Judges see the question, the candidate answer, and the original passages. They are blind to the perturbation, the generator's identity, and the induced error.

3. The Paired Estimand

Traditional analysis compares diagonal cells (same-model) against off-diagonal cells (cross-model). The authors argue this is inferentially weak because it compares different answers with varying styles and refusal rates.
Instead, the Eval-Pair Matrix uses an answer-paired estimand:

• For a specific candidate answer generated by model $G$, all three judges evaluate it.
• The analysis compares the matching judge (model $G$) against the non-matching judges (the other two models) on the exact same answer.
• This isolates the "same-model effect" from generator-specific behaviors (e.g., refusal rates, style).

4. Labeling and Human Evaluation

• Behavior Labels: Answers are labeled by behavior (e.g., "Context follow," "Memory override," "Refusal/insufficiency").
• Induced-Error Adoption: A separate label-evaluator determines if the answer adopted the induced false claim.
• Human Audit: A targeted human evaluation reviewed 88 cases, specifically focusing on "apparent false positives" (where the judge flagged an error, but the answer avoided the induced claim).

Key Results

1. No Robust Same-Model Leniency on Error Detection

Contrary to the hypothesis that judges are lenient toward their own model's errors:

• Global Recall Effect: The paired analysis found no robust evidence that same-model judges miss induced errors in their own model's answers. The recall difference is near zero ($\Delta = -0.5$ percentage points; 95% CI $[-2.7, +1.7]$).
• F1 Scores: The raw diagonal F1 (84.4%) and off-diagonal F1 (83.4%) are statistically similar.

2. The "Avoided-Claim" Gap

The only robust paired difference was observed in answers that avoided the induced false claim (e.g., by refusing or hedging).

• Finding: Matching judges flagged these answers less often than non-matching judges ($\Delta = -4.3$ pp; 95% CI $[-6.6, -2.0]$).
• Interpretation: This is not a lower false-alarm rate. The "avoided-claim" label only checks if the specific induced error was adopted. However, the judge's task is to flag any factual error against the source. Answers that avoided the induced claim often contained other source-relative errors (e.g., hallucinations about unrelated facts or incorrect attributions).

3. Human Evaluation of Apparent False Positives

The authors audited 25 "apparent false positives" (answers flagged by the judge but labeled as "avoiding the induced claim").

• Result: None were genuine false alarms.
  • 22 were valid detections of alternate source errors.
  • 2 were mistakes in the induced-error adoption label.
  • 1 was an unclear case.
• Conclusion: The observed gap is a label/task mismatch, not evidence of self-leniency. The adoption label is too narrow to capture the judge's broader task of detecting any source-grounding error.

4. Localization

When judges flagged errors, they correctly identified the source passage in 94.0–98.5% of cases, demonstrating high precision in localizing contradictions.

Contributions and Significance

Methodological Contributions

1. Protocol: A controlled meta-evaluation setup using hidden, localized source contradictions and blind judging.
2. Paired Estimand: A shift from comparing different answer distributions to comparing judges on the same candidate answer, effectively controlling for generator difficulty and style.
3. Behavior Stratification: Demonstrating that generator behaviors (refusals, hedges) significantly impact evaluation metrics and must be analyzed separately.

Empirical Findings

• Refutation of Broad Self-Leniency: In fact-centric RAG, there is no robust evidence that same-model judges broadly miss induced errors.
• Reinterpretation of Gaps: Apparent discrepancies in flagging rates are better explained by the misalignment between narrow "adoption" labels and the broader "any-error" judging task, rather than model bias.

Significance

The paper argues that same-model RAG judging is an identification problem, not a binary prohibition. While raw diagonal-vs-off-diagonal comparisons are descriptively useful, they are inferentially insufficient. To make valid claims about self-leniency, studies must:

• Report full judge-by-generator matrices.
• Use answer-paired effects.
• Stratify by generator behavior.
• Ensure alignment between ground-truth labels and the specific error-detection task assigned to the judge.

The study suggests that factual grounding may reduce the style-driven self-preference effects seen in open-ended preference tasks, but it introduces new complexities regarding how "errors" are defined and labeled in grounded generation.