GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

The paper introduces GoodPoint, a training framework that leverages author responses to curate a dataset of valid and actionable feedback, enabling a fine-tuned LLM to significantly outperform existing models in generating constructive scientific paper feedback that authors find practically valuable.

The Gist

The Big Idea: A Coach, Not a Robot

Imagine you are writing a masterpiece novel. You send it to a publisher, and they send it back with a stack of notes. Some notes are helpful ("The plot twist in Chapter 3 is confusing, try clarifying it"), while others are useless ("I didn't like the font" or "This book is bad because I hate your genre").

Currently, AI tools trying to act as editors often give you a lot of useless noise. They might sound confident but give generic advice that doesn't actually help you improve your story.

The authors of this paper say: "Let's not replace human editors with robots. Let's give authors a super-smart AI coach that knows exactly what kind of feedback actually works."

They built a system called GOODPOINT. Its goal isn't to write the paper for you; it's to give you feedback that you will actually listen to and act on.

---

How They Taught the AI: The "Author's Reply" Secret Sauce

How do you teach an AI to give good advice? You can't just ask it to "be nice." You need to know what "good" looks like.

The researchers realized that the best signal for good feedback is what the author does next.

• The Analogy: Imagine a student turning in a homework assignment.
  • Scenario A: The teacher writes, "Your essay is boring." The student ignores it and submits the same essay next time. (Bad feedback).
  • Scenario B: The teacher writes, "Your second paragraph contradicts your thesis; try adding a transition sentence here." The student fixes it and gets an A. (Good feedback).

The researchers looked at thousands of real scientific papers and the conversations between authors and reviewers. They only kept the feedback where the author said, "You're right, and I will fix this" or "That's a good point, I'll add that experiment."

They called this "Valid and Actionable" feedback.

• Valid: The author agrees the problem is real.
• Actionable: The author knows exactly what to do to fix it.

They used these "winning" comments to train their AI model (based on a model called Qwen3-8B).

---

The Training Recipe: SFT and DPO

The team used a two-step cooking recipe to make their AI chef (GOODPOINT) perfect:

1. SFT (Supervised Fine-Tuning) - "Learning the Recipe":
   They fed the AI thousands of examples of "winning" feedback. It's like showing a cooking student the best recipes from a Michelin-star chef so they learn the basic techniques. The AI learned, "Okay, when I see a paper, I should sound like these successful reviewers."

2. DPO (Direct Preference Optimization) - "The Taste Test":
   This is the secret sauce. They took the AI's output and created "bad" versions of it on purpose.

   • Good Version: "The data in Figure 2 is unclear; please add error bars."
   • Bad Version (Corrupted): "Your data is bad." (Too vague) OR "Your data is wrong because I hate math." (Rude/Invalid).

   They showed the AI both versions and said, "Pick the one that is actually helpful." This taught the AI to avoid being vague, rude, or factually wrong. It learned to be precise and constructive.

---

The Results: Small Model, Big Impact

Usually, to get the best AI, you need a massive, expensive super-computer brain (like the giant models from Google or OpenAI).

But GOODPOINT proved that a smaller, smarter brain can beat a bigger, dumber one.

• The Test: They tested their AI on 1,200 real scientific papers.
• The Result:
  • The base AI (Qwen3-8B) gave feedback that authors ignored 92% of the time.
  • The GOODPOINT AI gave feedback that authors accepted and acted on 83% more often than the base model.
  • Even better: GOODPOINT was more precise than massive, expensive models like Gemini-3-flash. It didn't waste time giving generic advice; it gave specific, high-value critiques.

Why This Matters

In the past, people worried that AI would take over science, replacing human judgment with robotic, soulless reviews.

This paper argues for a different future: AI as a Power-Up.

• It helps junior researchers or non-native English speakers get the same high-quality feedback as experts.
• It doesn't replace the human reviewer; it acts as a "pre-reviewer" that filters out the noise and highlights the constructive points.

The Bottom Line

GOODPOINT is like a personal writing coach that has studied thousands of successful editor-author conversations. It doesn't just tell you what is wrong; it tells you how to fix it in a way that makes you want to listen. By focusing on feedback that authors actually use, the team created a tool that makes science better, faster, and more collaborative.

Technical Summary

1. Problem Statement

The paper addresses the challenge of using Large Language Models (LLMs) to generate constructive feedback for scientific papers. While LLMs show potential in automating research tasks, existing automated peer reviews often suffer from:

• Lack of Specificity: Feedback is often generic or boilerplate.
• Limited Actionability: Suggestions do not lead to concrete revisions.
• Poor Alignment: Feedback priorities often diverge from human reviewers and authors.
• Hallucinations: Models may invent facts or miss existing content.

The authors argue that AI should augment rather than replace human researchers. The core problem is defining and operationalizing "constructive feedback" in a way that is measurable and trainable, moving beyond simple mimicry of reviewer behavior to generating feedback that authors actually find useful and act upon.

2. Methodology

The proposed framework, GOODPOINT, consists of three main components: a definition of success, a curated dataset, and a novel training recipe.

A. Definition of Constructive Feedback

The authors define feedback quality along two author-centric axes, derived from author responses in peer review discussions:

1. Validity: The author agrees with the concern raised (i.e., the critique is factually correct and relevant).
2. Author Action: The author commits to a concrete follow-up action (e.g., "will revise," "defer to future work").

• Success Criterion: Feedback is considered "successful" (constructive) only if it is both valid and actionable.

B. Dataset: GOODPOINT-ICLR

• Source: 19,534 ICLR papers (2020–2026) with reviewer feedback and author responses.
• Annotation: Each feedback item is parsed and labeled for Validity (Agreed/Rebutted) and Actionability (6 classes, with the first two considered actionable).
• Split: ICLR 2020–2025 used for training/validation; ICLR 2026 held out as a temporally separate test set to prevent data contamination.
• Scale: Contains ~14.5K rejected and ~4.4K accepted papers.

C. Training Recipe (GOODPOINT)

The training strategy uses Qwen3-8B as the base model and involves two stages:

1. Supervised Fine-Tuning (SFT):

   • The model is fine-tuned on "successful" feedback units (those labeled as both valid and actionable) from the GOODPOINT-ICLR dataset.
   • This grounds the model in patterns of feedback that authors acknowledge and act upon.

2. Direct Preference Optimization (DPO):

   • To refine content and style, the SFT model is further aligned using two types of preference pairs:
     • Validity Contrast: Pairs contrasting valid vs. invalid feedback items.
     • Quality Corruption: Successful feedback is synthetically corrupted in five dimensions (Specificity, Clarity, Accuracy, Prioritization, Supportive Tone). The model is trained to prefer the original (uncorrupted) version over the corrupted one.
   • This step ensures the model avoids hallucinations, generic statements, and unhelpful tones.

D. Evaluation Framework

The authors propose two complementary evaluation methods:

1. Author Response Prediction: An automatic metric where a separate model predicts whether an author would agree with the feedback (validity) and what action they would take.
2. Human Consensus Matching: The model's feedback is compared against a "golden set" of feedback that was raised by multiple human reviewers and successfully acted upon. Metrics include Precision, Recall, and F1.
3. Human Study: A user study with 13 authors evaluating feedback on their own papers across validity, actionability, specificity, and helpfulness.

3. Key Contributions

1. GOODPOINT-ICLR Dataset: A large-scale, annotated dataset of 19K papers linking reviewer feedback to author responses, explicitly labeled for validity and actionability.
2. Constructive Feedback Definition: A rigorous operationalization of feedback quality based on author responses rather than just reviewer scores or text similarity.
3. Training Strategy: A novel recipe combining SFT on successful feedback with DPO using synthetic quality corruptions to improve specificity and reduce hallucinations.
4. Evaluation Protocols: New automatic metrics (Author Response Prediction, Consensus Matching) that better reflect practical utility than standard NLP metrics.

4. Results

The experiments were conducted on a test set of 1,198 ICLR papers, comparing GOODPOINT-trained models against baselines (Llama 3.1-8B, Qwen3-8B base, GPT-5.2, Gemini-3-flash).

• Author Response Prediction (Success Rate):

  • GOODPOINT-DPO achieved a 14.7% combined success rate (valid + actionable), a 6.7% absolute improvement over the base Qwen3-8B (8.0%).
  • It significantly outperformed other open-source models (e.g., Llama 3.1-8B at 1.8%).

• Human Consensus Matching:

  • GOODPOINT-SFT achieved the best performance among size-comparable models, with an F1 score of 0.108 (vs. 0.068 for base Qwen3-8B), representing a 58.8% improvement.
  • Precision: GOODPOINT-SFT achieved higher precision (0.138) than larger proprietary models like Gemini-3-flash (0.128) and GPT-5.2 (0.130), indicating it generates more selective, high-fidelity critiques with fewer low-quality comments.

• Human Evaluation:

  • In a study with real authors, GOODPOINT-DPO ranked second overall (behind Gemini-3-flash) but significantly outperformed the base Qwen3-8B in all dimensions:
    • Validity: 58.1% vs. 41.5%
    • Actionability: 40.3% vs. 32.3%
    • Specificity: 3.50 vs. 2.89
    • Helpfulness: 2.77 vs. 2.25
  • This confirms that the automatic improvements translate to perceived practical value.

• Failure Mode Analysis:

  • Untrained LLMs tend to be verbose and generic. GOODPOINT models reduced the number of feedback units to be closer to human averages (4.15–5.71 units vs. 20+ for untrained models) while increasing specificity and reducing hallucinations (e.g., missing existing content).

5. Significance

• Paradigm Shift: The paper moves the field from "mimicking reviewer scores" to "optimizing for author utility." It demonstrates that author responses are a viable, natural signal for training high-quality feedback models.
• Efficiency: It proves that smaller, open-weight models (8B parameters) trained with the right data and strategy can rival or exceed much larger proprietary models (e.g., GPT-5, Gemini) in generating specific, actionable scientific feedback.
• Human-AI Collaboration: The work supports a future where AI acts as a "co-pilot" for researchers, particularly aiding junior or non-native English-speaking authors who lack access to timely expert feedback, without replacing human scientific judgment.
• Reproducibility: The authors plan to release the dataset, code, and trained models, fostering further research in constructive AI feedback.