💬 NLP

Large Language Models Threaten Double-blind Review

This paper demonstrates that large language models can effectively de-anonymize authors in double-blind peer reviews by identifying stable conceptual signatures in titles and abstracts, thereby threatening the integrity of the current review system.

Original authors: Bulambo Mwendelwa Gloire, Prasenjit Mitra

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bulambo Mwendelwa Gloire, Prasenjit Mitra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the scientific world as a giant, bustling marketplace where researchers bring their newest ideas to be judged. To keep things fair, many of these markets use a special rule called "double-blind review." It's like a game of mystery where the person judging a paper (the reviewer) doesn't know who wrote it, and the writer doesn't know who is judging. The goal is to make sure the idea is judged purely on its own merits, not on whether the writer is a famous professor from a top university or a student just starting out. For this game to work, the paper has to be a perfect disguise. It's supposed to be a "ghost story" where the author's identity is completely erased, leaving only the science behind. But what if the ghosts left behind a secret fingerprint that only a super-smart machine could see? That's the big question scientists are asking today, especially with the rise of Artificial Intelligence (AI) that can read and understand language better than ever before.

This paper is a detective story about whether that "ghost disguise" is actually working anymore. The researchers set up a test to see if modern AI models, specifically Large Language Models (LLMs), could figure out who wrote a scientific paper just by reading its title and abstract (the short summary at the beginning). They didn't use any fancy tricks like looking at who the author cited or how they used commas; they only gave the AI the bare minimum of text. They then asked the AI to guess the author from a lineup of five possible suspects: the real author and four other experts who work on similar topics.

The results were a bit spooky for the old rules of fairness. When human experts tried to play this guessing game, they were terrible at it. Out of 100 tries, a human could only correctly guess the real author first time just 11 times. They were mostly just guessing in the dark, spreading their confidence across many suspects. But when the AI played the same game, it was a different story. The AI managed to guess the correct author first time 42 times out of 100. That's nearly four times better than the humans!

The paper suggests that the AI isn't just guessing; it's finding "latent conceptual signatures." Think of it like this: even if two people wear the same uniform and speak the same language, they might still have a unique way of holding their coffee cup or telling a joke. In science, researchers have unique ways of framing their problems or describing their goals. The AI is so good at reading between the lines that it can spot these subtle patterns in the title and abstract, narrowing down the list of suspects to just a few people with high confidence.

The study also checked if this only happened with famous, well-known scientists. It didn't. The AI was just as good at guessing the authors of junior researchers as it was for senior ones. It also found that the AI got even better when the "suspects" were less similar to each other, but even when the suspects were all experts in the exact same field, the AI still did much better than humans.

However, the paper is careful not to say that double-blind review is completely broken or that the AI can always name the author with 100% certainty. Instead, it suggests that the system is becoming "leaky." The AI can't always point to the one true author, but it can shrink the pool of possibilities so much that the anonymity isn't as safe as we thought. It turns the "perfect disguise" into a "blurry disguise" that a smart machine can still see through. The authors conclude that while the AI isn't a magic wand that solves the mystery every time, it has changed the game enough that we might need to rethink how we keep our scientific reviews fair in an age where computers can read our minds better than we can.

Technical Summary: Large Language Models Threaten Double-blind Review

Problem Statement
Double-blind peer review is a foundational mechanism in scientific publishing designed to mitigate bias related to author reputation, institutional affiliation, and prior visibility. Its efficacy relies on the assumption that anonymized manuscripts (stripped of names, affiliations, and explicit citations) do not contain sufficient information to reliably reveal their authors. While previous research has demonstrated that authorship can be recovered using stylistic markers, lexical patterns, or bibliographic structures, this study investigates a deeper vulnerability: whether Large Language Models (LLMs) can deanonymize papers using only minimal scientific text (titles and abstracts) by inferring latent conceptual signatures. The core question is whether the assumption of anonymity holds when inference is limited to small pools of closely related domain experts and surface-level cues are removed.

Methodology
The authors conducted a controlled authorship attribution experiment using a corpus of over 12,000 scientific papers published between 2024 and 2025, sourced from Semantic Scholar. To ensure the models were not relying on memorized training data, the evaluation set was strictly temporally isolated from the training cutoffs of the models used (Qwen2.5-72B and LLaMA3-70B, both with 2023 cutoffs).

Input Constraints: Models were provided only with paper titles and abstracts. All explicit identifiers, affiliations, and bibliographic metadata were excluded.
Candidate Pools: For each paper, the model was presented with a closed set of five candidates: one true author and four distractors. The study varied the construction of these distractors:
- Random: Drawn from unrelated domains.
- Domain Experts (Semantic Scholar): Authors recommended as working on similar topics.
- Domain Experts (OpenAlex): Authors retrieved via OpenAlex, representing a more strictly aligned, conceptually grounded pool.
Ground Truth Definitions: The "true author" was defined in three ways to test robustness across disciplines: the last author, the most senior author (highest publication count), and the most junior author (lowest publication count).
Evaluation Framework: Instead of requiring a single prediction, the study employed a confidence-aware ranking framework. Models ranked candidates and assigned confidence scores. A "suspect set" was constructed by accumulating candidates until their cumulative probability exceeded a threshold $\tau$ (e.g., 0.5 or 0.9). Metrics included Top-k accuracy, Mean Reciprocal Rank (MRR), Suspect Set Recall, and Expected Calibration Error (ECE).
Human Baseline: A control group of twenty graduate researchers with machine learning expertise performed the same task to establish a human baseline.
Advanced Inference Checks: The study tested model aggregation (combining predictions from Qwen and LLaMA) and multi-round deliberation (iterative reasoning) to see if complexity improved attribution.

Key Results

Superior LLM Performance: LLMs significantly outperformed human researchers in identifying authors from titles and abstracts alone.
- Top-1 Accuracy: LLMs achieved ~42% accuracy (Qwen2.5-72B), compared to 11% for human researchers.
- Ranking Quality: LLMs achieved an MRR of 0.623 versus 0.36 for humans (statistically significant, $p < 0.001$ ).
- Suspect Set Efficiency: At a moderate confidence threshold ( $\tau = 0.5$ ), LLMs retained the true author in 61.6% of cases while narrowing the suspect set to an average of 1.95 candidates. Humans retained the true author in only 22% of cases with a similar suspect set size.
Latent Conceptual Signatures: The study found that anonymity collapse is driven by "latent regularities" in how researchers frame problems, scope contributions, and position their work, rather than stylistic quirks.
- When distractors were drawn from unrelated domains, recall jumped to 84.1%, indicating models rely heavily on topical alignment.
- Even with highly confusable distractors (OpenAlex experts with overlapping research profiles), recall remained at 51.5%, significantly above the human baseline.
Disciplinary and Seniority Variations:
- Discipline: Anonymity erosion was more pronounced in Medicine (66% recall at $\tau=0.5$ ) than in Computer Science (58%), suggesting field-specific research conventions amplify identifiable signals.
- Seniority: The effect was consistent across senior and junior authors, indicating that anonymity risks are not limited to prominent researchers but apply to early-career scientists as well.
Limits of Complexity:
- Aggregation: Combining predictions from multiple models yielded only marginal improvements over the best single model.
- Deliberation: Multi-round reasoning actually degraded performance, increasing the suspect set size and reducing recall. This suggests that additional reasoning steps introduce uncertainty rather than new authorship signals when input is restricted to short abstracts.

Significance and Claims
The paper argues that double-blind review faces a structural limitation in the era of AI-augmented research. The findings challenge the foundational assumption that removing explicit identifiers is sufficient to preserve anonymity. The authors claim that:

Anonymity is Probabilistic, Not Absolute: Double-blind review does not fail in an absolute sense (i.e., perfect identification is not guaranteed), but it suffers from "graded vulnerability." LLMs can systematically narrow the set of plausible authors to a small, high-confidence subset, which may be sufficient to influence reviewer expectations and behavior.
Inference Beyond Style: The threat arises not from stylistic mimicry or bibliographic artifacts, but from the semantic content itself—specifically, the unique ways researchers frame problems and define contributions.
Policy Implications: Current anonymization guidelines, which focus on surface-level features (names, citations, affiliations), may be insufficient. The authors suggest the research community must re-evaluate how anonymity and fairness are maintained, acknowledging that identity can be inferred from the scientific framing of the work itself.

The paper concludes modestly, noting that while the models reduce uncertainty, they do not always identify the correct author with certainty. However, the consistent reduction of the candidate pool represents a significant shift in the risk profile of double-blind review, necessitating a re-examination of review policies in an AI-integrated ecosystem.

Technical Summary: Large Language Models Threaten Double-blind Review

More like this