Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews

1. Problem Statement

The integration of Large Language Models (LLMs) into academic peer review is accelerating, with major conferences (e.g., ICLR, AAAI, ICML) adopting them for assistance or automation. While LLMs offer scalability, there is a critical concern regarding fairness and reliability. Previous studies have documented biases in LLMs regarding race, gender, and religion in general tasks, but the specific impact of these biases on single-blind peer review (where author metadata is visible) remains under-explored. The core question is: Do LLMs systematically favor authors based on metadata (affiliation, gender, seniority, publication history) rather than the merit of the paper content?

2. Methodology

The authors introduce a controlled counterfactual evaluation framework to isolate and measure bias.

• Dataset: 252 papers from ICLR 2025, sampled equally from 21 sub-fields (6 accepted, 6 rejected per sub-field).
• Experimental Design:
  • Single-Blind Simulation: The LLM is prompted with the full paper content plus specific author metadata.
  • Counterfactual Interventions: For each paper, the content remains constant while specific metadata attributes are altered one at a time:
    • Affiliation: Swapped between 8 "Ranked-Stronger" (RS) institutions (e.g., MIT, CMU) and 8 "Ranked-Weaker" (RW) institutions (e.g., University of Gondar, Midlands State University).
    • Gender: Swapped between Anglo male and female names, paired with both RS and RW affiliations.
    • Seniority: Swapped between a "Senior Principal Investigator" (Senior PI) and an "Undergraduate Student" (UG).
    • Publication History: Swapped between a profile with 100 top-tier publications (TTP) and 0 TTP.
• Evaluation Metrics:
  • Hard Ratings: The model's most confident integer rating (1–10) derived via greedy decoding.
  • Soft Ratings: The expected value of the rating calculated from the model's internal token-level probability distribution. This metric is crucial for detecting implicit bias that post-training alignment might mask in the final output.
• Models Evaluated: 9 diverse LLMs, including Ministral 8B, DeepSeek R1 variants, Llama 3.1 (8B/70B), Mistral Small, QwQ 32B, Gemini 2.0 Flash Lite, and GPT-4o Mini.

3. Key Contributions

1. Systematic Affiliation Bias: Demonstrated that LLMs consistently favor authors from high-prestige institutions. This bias is often stronger in soft ratings (internal probabilities) than in hard ratings, suggesting that alignment techniques may suppress explicit bias while leaving implicit preferences intact.
2. Hidden Bias Detection: Introduced the use of soft ratings to reveal "hidden" biases. For example, in Ministral 8B, the hard rating showed a 4.3% win rate for RS institutions, but the soft rating revealed a 68.6% win rate, indicating a significant gap between internal reasoning and surface-level output.
3. Impact on Binary Decisions: Showed that metadata interventions can flip acceptance/reject decisions. RS affiliations increased the likelihood of rejected papers being accepted (e.g., QwQ-32B converted 21.4% of rejected papers to accepted), while RW affiliations pushed accepted papers toward rejection.
4. Multi-Dimensional Bias: Identified biases beyond affiliation, including:
   • Seniority: Strong preference for Senior PIs over undergraduates.
   • Publication History: Strong preference for authors with extensive publication records.
   • Gender: Mixed but present results; some models favored males, others females (potentially due to over-compensation in alignment), but the effect was less consistent than affiliation bias.
5. Qualitative Reasoning Analysis: Analyzed review text to show how models justify bias (e.g., associating RS with "credibility" or "resource access" and RW with "resource constraints").

4. Key Results

• Affiliation: All 9 models exhibited a statistically significant preference for RS over RW affiliations. In pairwise comparisons, RS institutions won the majority of cases across all models (p-values < 0.01).
• Soft vs. Hard Discrepancy: The gap between soft and hard ratings was most pronounced in models like Ministral 8B and Llama 3.1, indicating that internal confidence distributions are heavily skewed even when the final integer score appears neutral.
• Decision Flips: The study found that metadata alone could alter the outcome for borderline papers. For instance, changing an author's affiliation from RW to RS could flip a rejection to an acceptance in ~20% of cases for certain models.
• Sub-field Consistency: The RS-over-RW bias persisted across all 21 sub-fields, though the magnitude varied (e.g., strongest in Robotics/CV, weakest in Neuroscience/Cognitive Science).
• Gender: Results were heterogeneous. Gemini 2.0 favored male names, while GPT-4o Mini favored female names. This suggests that alignment strategies aimed at reducing bias may lead to over-compensation, creating new directional preferences.

5. Significance and Implications

• Threat to Scientific Integrity: The findings suggest that LLM-assisted peer review could systematically disadvantage researchers from lower-ranked institutions, the Global South, or early-career researchers, regardless of the actual quality of their work.
• Limitations of Alignment: The discovery of strong soft bias implies that standard alignment (RLHF) may only mask bias rather than eliminate it. Models may "learn" to output neutral text while maintaining biased internal representations.
• Policy Recommendations: The authors argue for caution in adopting LLMs for evaluative judgment. They suggest that if LLMs are used, they should be restricted to assistive roles (e.g., checking reproducibility) rather than making acceptance decisions, and that double-blind review processes are essential to mitigate these specific metadata-driven biases.
• Future Research: Highlights the need for developing debiasing techniques that target internal probability distributions, not just final outputs, and the importance of auditing LLMs in high-stakes decision-making domains.

In conclusion, the paper provides robust empirical evidence that current LLMs are not neutral arbiters in peer review. They inherit and amplify societal biases regarding institutional prestige and author status, posing a significant risk to the fairness of the scientific evaluation ecosystem.