FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes

Technical Summary: FirstPass – Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes

Problem Statement
Current AI systems for scientific peer review suffer from three fundamental failures:

1. Domain Narrowness: Existing datasets (e.g., PeerRead, ReviewMT, MARG) are exclusively derived from Computer Science and Machine Learning conferences. Models trained on these fail to capture domain-specific scientific norms in fields like biology, chemistry, or physics (e.g., contamination controls or NMR spectral assignments).
2. Static Treatment: Peer review is an iterative dialogue, yet current systems generate reviews in a single shot, ignoring the author-response iterations where scientific claims are stress-tested.
3. Circular Evaluation: Systems are often evaluated on stylistic mimicry (how well they look like a review) rather than alignment with actual editorial decisions and outcomes.

Consequently, these systems treat peer review as a text generation task rather than an exercise in scientific judgment, failing to predict whether a manuscript will require further revision cycles or to identify which methodological concerns will survive rebuttal.

Methodology
The authors introduce FIRSTPASS, a dataset and fine-tuned model designed to address these gaps through three core components:

• Dataset Construction: The authors curated 3,668 complete multi-round peer-review dialogues from Nature Communications (January 2023 – December 2025) across five scientific domains: biology, chemistry, neuroscience, physics, and earth science.

  • Integrity: Leveraging the journal's mandatory transparent peer review (since Nov 2022), the dataset ensures 100% content integrity. Automated audits and manual verification of a random sample confirmed negligible error rates in the extracted text.
  • Labels: Editorial outcomes are derived from the number of revision rounds: two rounds are labeled STANDARD, and three or more are labeled EXTENDED. This serves as a proxy for unresolved methodological concerns.
  • Tasks: The dataset supports three training tasks: (1) Review generation (Paper $\to$ Round 1), (2) Reviewer updating (Paper + Round 1 + Author Response $\to$ Round 2), and (3) Outcome prediction (Full dialogue $\to$ Label).

• Model Architecture and Training:

  • Base Model: Qwen2.5-7B-Instruct was selected for its strong scientific reasoning capabilities and 32k-token context window, necessary for handling full papers and multi-round dialogues.
  • Fine-tuning: Low-Rank Adaptation (LoRA) was applied with rank $r=32$ and $\alpha=64$. Two separate adapters were trained: one for classification (CLS) and one for generation (SFT).
  • Critical Design Decision (Response-Only Loss Masking): The authors identified that standard instruction fine-tuning computes loss over the entire sequence, including thousands of input tokens. For long-input/short-output tasks, this causes the classification signal to be drowned out by the prediction of input text. The authors implemented response-only loss masking (excluding input tokens from loss calculation).
  • Truncation Strategy: Inputs exceeding the sequence limit are truncated symmetrically (retaining the first 55% and last 45% of tokens) to preserve both the paper's scientific content and the most recent reviewer exchanges.

Key Findings and Results

• The Necessity of Response-Only Masking: The most significant finding is that response-only loss masking is an architectural prerequisite, not merely an optimization.

  • Without masking, the model's accuracy on outcome prediction drops to 62.0%, performing worse than the majority baseline (65.4%).
  • With masking, accuracy rises to 80.5% with an F1-macro of 78.2%. This 18.5 percentage point swing demonstrates that omitting masking is catastrophic for this task regime.

• Revision-Cycle Prediction:

  • FIRSTPASS achieved 80.5% accuracy on the test set (318 examples), outperforming the majority baseline by 15.1 percentage points and the best zero-shot baseline (Gemini-3.1-flash-lite-preview) by 10.4 percentage points.
  • Statistical significance was confirmed via McNemar's exact test ($p < 0.001$) against all baselines except Qwen2.5-7B zero-shot ($p=0.185$).
  • Performance was consistent across all five domains (ranging from 76.9% to 83.8%), indicating the model captures domain-general signals of unresolved concerns rather than overfitting to specific vocabularies.

• Review Generation:

  • FIRSTPASS generated reviews averaging 1,187 words, significantly closer to human references (2,155 words) than other baselines.
  • It achieved a ROUGE-L score of 0.154, significantly outperforming zero-shot Qwen, DeepSeek, and Gemini ($p < 0.001$).
  • While Llama-3-8B achieved a slightly higher ROUGE-L (0.164), the authors attribute this to "length artifacts" where the model generated shorter, formulaic text (1,006 words) that optimized n-gram overlap without depth. FIRSTPASS produced longer, content-rich critiques.

Significance and Claims
The paper positions FIRSTPASS as a step toward AI systems acting as "anticipatory scientific co-authors" rather than mere tools.

• Judgment vs. Mimicry: By training on complete dialogues and evaluating against real editorial outcomes, FIRSTPASS demonstrates that AI can exercise scientific judgment—predicting whether a manuscript will require extended revision cycles—rather than just mimicking review prose.
• Generalizability: The consistent cross-domain performance suggests that the signals of unresolved reviewer concern are transferable across disciplines, a prerequisite for a trusted co-author in a multidisciplinary setting.
• Methodological Contribution: The paper provides the first controlled empirical demonstration that response-only loss masking is critical for long-input/short-output classification tasks, offering a generalizable lesson for training LLMs on scientific documents.

Limitations Acknowledged by the Authors
The authors maintain a modest stance on the current deployment status:

• Data Scope: The dataset includes only published papers; rejected manuscripts (which might offer stronger signals about publishability boundaries) are inaccessible.
• Modality: The pipeline is text-only, excluding figures and supplementary data which often drive reviewer concerns.
• Label Validity: The "STANDARD/EXTENDED" label is a proxy based on round count and may be influenced by communication styles or editorial logistics rather than purely scientific merit.
• Validation: The "co-author" use case is a deployment hypothesis. The current evaluation measures prediction accuracy on completed dialogues. Prospective validation (where authors use the tool pre-submission and outcomes are tracked) is identified as the necessary next step to confirm the co-authorship claim.
• Generalizability: Performance beyond Nature Communications (e.g., journals with different review cultures or formats) remains untested.