Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

Technical Summary: Benchmarking Rubric LLMs for Deep-Research Source Attribution

Problem Statement
As Reinforcement Learning with Verifiable Rewards (RLVR) becomes the dominant post-training paradigm for tasks lacking programmatic verifiers (e.g., medical advice, scientific writing), the community relies on LLM judges to score prompt-specific rubrics. These judges effectively act as reward models, meaning the design of the training rubric equates to designing the operationalized goal. However, a critical calibration question remains: how capable must a judge be to provide a trustworthy reward signal, and how does its inherent bias affect downstream training?

While citation quality in deep-research systems is a primary target for RLVR, existing work has not systematically benchmarked off-the-shelf (general-purpose) LLMs as judges on citation-specific rubric tasks. Prior systems often assume a single large or dedicated model is necessary, or they focus on retrieval quality rather than the judge's ability to distinguish between relevant and factually supporting sources. There is a gap in understanding whether cheaper, smaller models can reliably close the loop as frontier models, and specifically, how their directional biases (e.g., false positives vs. false negatives) might shape the training signal in ways that scalar accuracy metrics obscure.

Methodology
The authors introduce the Deep-Research Citation Benchmark, an adversarial, long-form dataset designed to stress-test citation quality evaluation.

• Dataset Construction: The benchmark spans 25 diverse domains (e.g., quantum computing, history, biology). It begins with clean, attributed long-form summaries, of which approximately 60% are adversarially edited using 19 strategies to introduce factual errors, irrelevant sources, or plausible but unsupporting citations. This yields 624 attribution-citation pairs.
• Gold Labels: A council of 6 LLM judges independently evaluated the pairs on two dimensions: Source Relevance (topical fit) and Factual Support (truthfulness of the claim relative to the source). A human reviewer validated all 1,248 decisions (624 pairs × 2 dimensions), adjudicating 378 cases where the council disagreed to create a gold-standard dataset.
• Judges Evaluated: The study benchmarks 8 off-the-shelf LLMs from three families (Anthropic, Google, OpenAI), ranging from low-cost models (e.g., GPT-OSS-120B, Gemini 3.1 Flash Lite) to frontier models (e.g., Claude Opus 4.6, GPT-5-mini).
• Metrics: Beyond standard accuracy (Pass-class F1 and Cohen's $\kappa$), the authors measure directional bias metrics critical for RLVR: Pass-rate drift, False Positive Rate (FPR), and False Negative Rate (FNR). These metrics determine whether a judge systematically over-rewards bad citations or under-rewards good ones.

Key Results

1. Cost vs. Performance: Cheaper judges are competitive with frontier models.
   • Source Relevance: GPT-5-mini (the third-cheapest model) achieved the highest F1 (0.908), outperforming more expensive models like Claude Opus 4.6 (F1=0.866).
   • Factual Support: No single model statistically dominated. While Claude Opus 4.6 had the highest point estimate (F1=0.750), its 95% confidence interval overlapped with all other models, including the lowest-cost GPT-OSS-120B (F1=0.649).
2. Directional Bias Obscures Scalar Metrics: Models with similar F1 scores exhibited substantial differences in bias.
   • Most judges were stricter than the gold labels on source relevance, leading to high precision but moderate recall (systematic under-rewarding).
   • On factual support, false negative rates varied widely (from 0.183 to 0.470), meaning some judges rejected a large fraction of genuinely supported claims.
   • The authors note that scalar F1 masks these asymmetries, which are critical because a high FPR trains models to exploit permissive judges, while a high FNR trains models to over-hedge or under-cite.
3. Disagreement and Hard Cases: On the 378 "hard" cases where the initial council disagreed, rankings shifted significantly. No single judge reliably matched the human-adjudicated label on this subset, suggesting that full-set F1 is an incomplete proxy for reliability on ambiguous citations.

Significance and Claims
The paper argues that calibrating the judge is a prerequisite for using citation rubrics as reward signals in RLVR. The authors claim that:

• Frontier models are not strictly necessary: The most expensive available model does not dominate performance, and cheaper models can achieve comparable or superior results on specific dimensions (e.g., source relevance).
• Bias matters more than raw accuracy: The choice of judge should be driven by the specific directional bias required for the training loop (e.g., minimizing false negatives vs. false positives) rather than just maximizing scalar F1.
• Human adjudication is essential: Because judge rankings shift on ambiguous cases and no single model is perfectly reliable on hard instances, human review of disagreements is necessary to establish robust gold labels.

The study concludes that while lower-cost judges can close the loop effectively, the selection of a judge must be a deliberate design choice based on per-dimension rubric fit and bias characteristics, rather than a default assumption that larger models are superior.