Traditional statistical representations outperform generative AI in identifying expert peer reviewers

Technical Summary: Traditional Statistical Representations Outperform Generative AI in Identifying Expert Peer Reviewers

Problem Statement
The exponential growth of scientific submissions has strained the global peer review system, rendering manual expert identification unfeasible. While institutions have increasingly adopted Large Language Models (LLMs) and automated systems to match reviewers to submissions, the reliability of these generative models in accurately identifying domain experts lacks rigorous, large-scale evaluation. Current methodologies often rely on proxy labels that introduce bias (e.g., self-assessments or external human annotation) and lack a standardized ground truth for scientific expertise. Furthermore, the field suffers from a fragmented landscape of algorithms without a consensus on which representation methods best retrieve experts in operational environments.

Methodology
The authors frame expert identification as an Information Retrieval (IR) problem, utilizing the Distributed Peer Review (DPR) system of the European Southern Observatory (ESO) during Period 110 (P110) as a testbed. This framework provides a unique "closed-loop" environment where proposal authors simultaneously act as the reviewer pool, allowing for rigorous evaluation.

• Data: The study analyzes 435 proposals and 379 unique proposal-designated reviewers. The primary data inputs are proposal abstracts and reviewer expertise profiles derived from publication abstracts (retrieved via NASA/ADS) and self-reported keywords.
• Ground Truth: The study employs a dual validation strategy:
  1. Proxy Gold Labels: Treating the "proposal-designated reviewer" (the author nominated to review their own proposal's peer group) as the ground truth expert for that specific proposal.
  2. Self-Reported Labels: Comparing retrieval scores against reviewers' self-categorized expertise levels (Expert, Intermediate, Non-expert).
• Evaluated Methods: Six distinct methodologies were benchmarked:
  1. Keywords: ESO's baseline keyword matching system.
  2. LDA (Latent Dirichlet Allocation): Topic modeling.
  3. TF-IDF (Term Frequency-Inverse Document Frequency): Statistical lexical representation.
  4. SPECTER2: A Transformer-based model fine-tuned on scientific citations.
  5. SentenceTransformer: A general-purpose semantic embedding model.
  6. GPT-4o mini: A generative LLM used to score semantic alignment.
• Evaluation Metrics: The study utilizes rank-based IR metrics including Median Rank, Mean Reciprocal Rank (MRR), Hit@25 (fraction of proposals where the designated expert is in the top 25), and Z-scores. Additionally, Normalized Discounted Cumulative Gain (NDCG) is used to evaluate alignment with self-reported expertise hierarchies. Statistical significance is determined via bootstrap resampling and the Wilcoxon signed-rank test.

Key Results
The empirical evaluation demonstrates that traditional statistical representations significantly outperform generative AI and semantic neural architectures in this specialized domain:

• Retrieval Performance: TF-IDF achieved the highest performance, successfully identifying the designated expert within the top 25 recommendations 79.5% of the time (Hit@25) with a Median Rank of 4.0. In contrast, GPT-4o mini achieved a Hit@25 of only 51.5% with a Median Rank of 24.0, performing comparably to LDA (50.3% Hit@25) and significantly worse than the TF-IDF baseline.
• Alignment with Human Judgment: When evaluating alignment with self-reported expertise labels, TF-IDF again led with an NDCG of 0.832, followed closely by SentenceTransformer (0.827). Both significantly outperformed the self-selected Keywords baseline (0.789). Notably, GPT-4o mini (0.811) and SPECTER2 (0.815) also outperformed the keyword baseline but did not surpass TF-IDF.
• Robustness: TF-IDF and SentenceTransformer demonstrated robustness in data-constrained scenarios (e.g., limited publication histories), whereas LDA and SPECTER2 performance degraded significantly with fewer publications.
• Semantic Smoothing: The authors observe that while neural embeddings capture broad semantic relationships, they apply a "smoothing effect" that obscures the fine-grained, domain-specific vocabulary (e.g., "Type Ia," "Brown Dwarf") required to distinguish highly specialized subfields. Generative models, in particular, failed to leverage these precise lexical signals effectively.

Significance and Claims
The paper claims that for specialized scientific tasks, transparent and reproducible statistical representations remain superior to computationally expensive and probabilistic LLMs. The authors argue that the success of TF-IDF highlights that in specialized domains, the matching of precise technical vocabulary is more discriminative than the broad semantic clustering offered by modern neural architectures.

The study establishes a rigorous evaluation framework for automated peer review, demonstrating that:

1. Precision over Semantics: In scientific peer review, exact matches of domain-specific terminology are critical, a strength of statistical methods like TF-IDF that is often diluted by the semantic smoothing of generative models.
2. Reproducibility: Deterministic statistical methods offer the reproducibility required for formal peer review decisions, whereas generative LLMs introduce non-determinism and high computational costs that undermine these requirements.
3. Operational Reality: The findings challenge recent literature suggesting deep learning models require full-text manuscripts to compete with TF-IDF; this study shows TF-IDF outperforms semantic models even when constrained to publication abstracts, provided the evaluation mirrors the operational reality of global expert retrieval.

The authors conclude that as funding agencies and observatories automate peer review, transparent, computationally efficient lexical methods should remain the robust standard for scientific expertise retrieval.