IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR

This paper introduces IntelliAsk, a question-generation model trained via RLVR with a novel reward model (IntelliReward) and DAPO optimization to produce high-quality, evidence-based research questions that outperform human reviewers and strong baselines in expert evaluations while also enhancing broader reasoning and writing capabilities.

The Gist

Imagine you are a chef who just finished cooking a magnificent, complex dish. You invite a food critic to taste it.

The Problem:
In the world of academic research, the "critics" are peer reviewers. Their job is to taste the "dish" (the research paper) and ask tough, thoughtful questions to make the chef (the author) improve the recipe.

However, there's a crisis. Because there are too many papers and not enough time, many reviewers are using AI (Large Language Models) to write their questions. The problem? These AI questions are like a food critic saying, "This soup is salty." It's true, but it's shallow. It doesn't ask why the salt was added, if the recipe called for it, or if the saltiness hides a bad ingredient. The AI is just repeating what it sees on the first page of the menu, rather than actually tasting the whole meal.

The Solution: IntelliAsk
The authors of this paper, "IntelliAsk," decided to build a new kind of AI critic. They didn't just want an AI that sounds polite; they wanted an AI that asks deep, evidence-based questions that actually help the author improve.

Here is how they did it, broken down into simple steps:

1. The "Taste Test" (The Data)

First, the team went to a giant library of past reviews (from a conference called ICLR). They didn't just grab any question; they hired expert human "tasters" to grade thousands of questions based on three specific flavors:

• Effort: Did the critic actually think hard about this, or did they just copy-paste a generic complaint?
• Evidence: Did the critic point to a specific ingredient (a specific chart, equation, or paragraph) in the paper?
• Grounding: Is the question actually about this specific dish, or is it a generic question that could apply to any soup in the world?

2. The "Smart Judge" (IntelliReward)

The team realized that asking humans to grade every single AI question is too slow and expensive. So, they built a "Smart Judge" robot called IntelliReward.

Think of IntelliReward as a taste-test robot that was trained by the human experts. It can look at a question and a paper and instantly say, "This question is shallow and lazy (Low Score)" or "This question digs deep into the chemistry of the recipe (High Score)." It learned to spot the difference between a question that sounds smart and one that is smart.

3. The "Training Camp" (Reinforcement Learning)

This is the most important part.

• Old Way (SFT): Usually, you teach an AI by showing it examples of good questions and saying, "Copy this style." This is like teaching a student to memorize a textbook. The student learns to sound like a critic, but they don't actually understand the food. They just mimic the tone.
• New Way (IntelliAsk): The authors used a technique called Reinforcement Learning. Imagine a dog trainer.
  1. The AI (the dog) tries to ask a question.
  2. The Smart Judge (IntelliReward) gives it a treat (a high score) if the question is deep and evidence-based.
  3. If the question is shallow, the judge gives a "no treat" (a low score).
  4. The AI learns through trial and error: "Oh, if I look at the data tables and ask about the numbers, I get a treat! If I just look at the title, I don't."

Over time, the AI stops just mimicking the style of a critic and starts actually thinking like one.

The Results: A New Super-Critic

When they tested their new AI, IntelliAsk, against other top AI models and even human reviewers:

• Depth: IntelliAsk asked questions that required real thinking, not just surface-level observations.
• Focus: It didn't just look at the first page of the paper (where the summary is); it read the whole thing, including the complex data in the back.
• Bonus: Surprisingly, by learning to ask better questions, the AI also got better at writing and reasoning in general. It's like a chef who, by learning to critique food deeply, also becomes a better cook themselves.

The Big Picture

This paper is a wake-up call. It shows that simply telling an AI to "write like a human" isn't enough. If you want an AI to be truly helpful, you have to teach it to think critically, look for evidence, and put in the effort.

IntelliAsk is the first AI that doesn't just pretend to be a smart reviewer; it actually becomes one, helping researchers improve their work with questions that matter.

Technical Summary

Here is a detailed technical summary of the paper "IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR".

1. Problem Statement

The paper addresses a critical gap in the application of Large Language Models (LLMs) to academic peer review. While LLMs can generate fluent text, they currently struggle to produce high-quality, substantive research questions required for effective peer review.

• Current Limitations: Existing LLMs tend to generate surface-level queries that lack technical depth, critical reasoning, and proper grounding in the paper's content. They often mimic the style of a review without the substance.
• Failure of Existing Approaches:
  • Supervised Fine-Tuning (SFT): Models trained via SFT on review datasets primarily learn to mimic reviewer tone and phrasing but fail to improve the actual reasoning, effort, or evidence behind the questions.
  • Automated Evaluation: Current benchmarks often rely on "LLM-as-a-Judge" or generic QA metrics, which fail to capture the nuanced qualities of a good review question (e.g., whether it requires deep synthesis or merely restates facts).
• The Gap: There is a lack of datasets and training methodologies specifically designed to teach models to ask effortful, evidence-based, and grounded questions that help authors improve their work.

2. Methodology

The authors propose a three-stage pipeline: Data Curation, Reward Modeling, and Reinforcement Learning (RL) Optimization.

A. Data Curation and Human Preference Study

• Dataset Extraction: The team scraped 15,500 high-quality reviewer questions from 5,841 ICLR 2024 papers using the OpenReview API. They employed a multi-stage filtering process (length, semantic redundancy, technical relevance, and specificity) to remove low-quality or generic comments.
• Human Annotation Study: To define "quality," they conducted a study with expert annotators evaluating 572 question-paper pairs.
• Evaluation Rubric: Questions were scored on three binary dimensions:
  1. Effort: Does the question require synthesis and deep thought, or can it be answered by surface-level restatement?
  2. Evidence: Is the question backed by specific results, assumptions, or arguments from the paper?
  3. Grounding: Is the question anchored in the actual content (methods, experiments) rather than generic phrasing?
• Key Finding: Human-authored questions significantly outperformed strong baseline LLMs (including o3, Gemini 2.5 Pro, and Qwen2.5-32B) across all three dimensions. Baselines also exhibited a "First Page Bias," relying heavily on introductory text rather than the full paper.

B. Reward Model: IntelliReward

To scale training beyond manual annotation, the authors trained a reward model, IntelliReward.

• Architecture: Built upon a frozen autoregressive LLM (gpt-oss-20b) with trainable multi-head Transformer heads (specifically, a "Transformer Residual Head" processing the last 50 tokens). This allows the model to predict scores for the three specific objectives (Effort, Evidence, Grounding) independently.
• Performance: IntelliReward achieved ~72% accuracy in predicting human judgments, significantly outperforming off-the-shelf closed-source LLMs (e.g., GPT-5, Gemini 2.5 Flash) and SFT-tuned baselines.

C. Training IntelliAsk via RL

Instead of SFT, the authors used Reinforcement Learning to align the model with human preferences.

• Model: They trained IntelliAsk (based on Qwen2.5-7B and Qwen2.5-32B).
• Algorithm:
  • IntelliAsk-7B: Trained using DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization).
  • IntelliAsk-32B: Trained using GRPO (Group Relative Policy Optimization).
• Process: The model generates multiple candidate questions for a given paper. These are scored by IntelliReward, and the scores serve as rewards to optimize the policy. This encourages the model to explore questions that maximize effort, evidence, and grounding rather than just mimicking style.

3. Key Contributions

1. IntelliReward: A scalable, human-aligned reward model and evaluation benchmark specifically for assessing the quality of research questions (Effort, Evidence, Grounding). It outperforms standard API-based LLM judges.
2. IntelliAsk: A specialized question-generation model trained via RL. Unlike SFT models that mimic style, IntelliAsk generates technically deeper, more grounded questions.
3. Human Preference Dataset: A curated dataset of 15.5k expert-annotated question-paper pairs with detailed quality scores, released to the community.
4. Empirical Insight: Demonstration that learning to ask high-quality questions improves general reasoning and writing capabilities, suggesting a strong correlation between critique quality and broader cognitive abilities.

4. Results

The paper presents extensive evaluations comparing IntelliAsk against strong baselines (Qwen2.5-32B, Gemini 2.5 Pro, o3, and SFT-trained models).

• Human Evaluation:

  • IntelliAsk-32B achieved a total score of 0.66/3.0, outperforming Gemini 2.5 Pro (0.60) and significantly beating Qwen2.5-32B (0.05).
  • It demonstrated the lowest First Page Bias (21.37%), indicating it utilized the full paper content rather than just the introduction.
  • In pairwise preference tests, IntelliAsk-32B was preferred over GPT-4.1, Gemini 2.5 Flash, and Qwen3-32B in 81–96% of cases.

• Automatic Evaluation (IntelliReward):

  • IntelliAsk-32B scored 0.55/3.0, substantially higher than SFT-only baselines (which scored ~0.03–0.10) and open-source models like OpenReviewer (0.10).

• Generalization Benchmarks:

  • Reasoning: IntelliAsk-32B improved on MuSR (68.3 vs 64.7 Acc) and GPQA-Diamond compared to its base model (Qwen3-32B).
  • Writing: It outperformed Qwen3-32B on WritingBench (8.31 vs 8.07) and Arena Hard, confirming that the skill of asking critical questions transfers to better general writing.

5. Significance

• Beyond Style Imitation: The paper proves that SFT is insufficient for complex reasoning tasks like critical questioning. RL guided by a domain-specific reward model is necessary to move beyond stylistic mimicry to genuine critical engagement.
• New Paradigm for Peer Review: By focusing specifically on the question component of peer review, the work offers a practical tool to augment human reviewers, potentially alleviating reviewer overload and improving the quality of feedback in academic publishing.
• Proxy for Reasoning: The results suggest that the ability to ask grounded, evidence-based questions is a strong proxy for deep understanding and reasoning, offering a new direction for training LLMs to be more "thoughtful" rather than just "fluent."
• Open Science: The release of the dataset, annotations, reward model, and trained models provides a new benchmark for evaluating grounding and effort in LLM-generated critiques.