💬 NLP

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

The paper introduces DR Tulu, an open-source deep research model trained via Reinforcement Learning with Evolving Rubrics (RLER) that co-evolves evaluation criteria with the policy, enabling it to outperform existing open agents and rival proprietary systems on long-form research tasks while being significantly more cost-effective.

Original authors: Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca So

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Be a Researcher

Imagine you want to teach a robot to write a long, detailed research report on a complex topic, like "How do genetic mutations cause rare diseases?" or "What is the history of renewable energy?"

Most AI models today are like students who only study for short quizzes. They are great at answering simple questions like "What is the capital of France?" but they struggle when asked to write a 20-page report that requires digging through hundreds of websites, checking facts, and citing sources.

The authors of this paper built a new AI called DR Tulu. It's the first open-source model specifically trained to be a "Deep Researcher." It doesn't just guess; it plans, searches the web, reads papers, and writes a long, well-sourced report.

The Problem: How Do You Grade a Long Report?

To teach an AI to do better, you usually use a method called Reinforcement Learning. Think of this like training a dog:

The dog (AI) does a trick.
If it does it right, it gets a treat (a reward).
If it does it wrong, it gets no treat.

The problem with long research reports is that it's hard to know what a "good" report looks like.

Static Rubrics (The Old Way): Imagine a teacher giving the dog a fixed checklist: "Must have 5 paragraphs. Must mention the year 1990." If the dog writes a brilliant report about 1991, the teacher fails it because it didn't follow the rigid checklist. The checklist doesn't know what the dog actually found.
The "Closed-Book" Problem: If you ask an AI to write the checklist based only on what it already knows, it might miss new facts it just discovered on the internet.

The Solution: "Evolving Rubrics" (The Smart Teacher)

The paper introduces a new method called RLER (Reinforcement Learning with Evolving Rubrics).

Imagine a Smart Teacher who doesn't use a fixed checklist. Instead, the teacher watches the student (the AI) do their research in real-time.

The Student Searches: The student goes out, finds new facts, and writes a draft.
The Teacher Adapts: The Smart Teacher looks at what the student found. "Oh, I didn't know that fact existed! I should add a new rule to my checklist: 'Must explain this new fact.'"
The Feedback Loop: The teacher updates the checklist while the student is learning. If the student tries to cheat (like copying text without reading it), the teacher immediately adds a rule: "No cheating allowed."

In technical terms, the paper calls these Evolving Rubrics. They are evaluation criteria that change and get smarter as the AI explores more information. This allows the AI to learn from its own discoveries rather than being stuck with a static set of rules.

The Result: A Super-Researcher on a Budget

The authors trained DR Tulu using this "Smart Teacher" method. Here is what happened:

It's Smarter: DR Tulu outperformed other open-source research models by a huge margin. It wrote better reports, found more accurate facts, and cited sources correctly.
It Rivals the Giants: It performed just as well as (and sometimes better than) expensive, proprietary systems from big companies like OpenAI and Google.
It's Cheap: This is the biggest win. The expensive systems cost about $1.80 per question. DR Tulu costs about $0.002 per question. That's roughly 1,000 times cheaper.

The "Genetic Disease" Test

To prove it really works, the team gave DR Tulu a very hard task: analyzing genetic mutations to see if they could be treated with specific drugs. This is a task usually reserved for human medical experts.

DR Tulu successfully searched through medical databases, found relevant papers, and synthesized a report.
It was able to compete with the most expensive, closed-source AI systems on this task.
Other open-source models failed because they couldn't find the right citations or verify the facts.

The Toolkit: "dr-agent-lib"

The paper also released a "toolbox" called dr-agent-lib.

Think of this as a Swiss Army Knife for AI researchers.
It lets the AI use different tools: Google Search, reading specific webpages, and searching academic papers.
Crucially, the AI learns to pick the right tool for the job. If the question is about science, it uses the "Paper Search" tool. If it's about general news, it uses "Web Search." It doesn't just use one tool blindly.

Summary

The paper presents DR Tulu, an open-source AI that learns to do deep, long-form research. It uses a special training method (Evolving Rubrics) where the AI's "grading rules" update in real-time based on what the AI discovers. This makes the AI much smarter, more accurate, and significantly cheaper than the current state-of-the-art research tools. The authors have shared the code, data, and the model so anyone can use it.

Technical Summary: DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Problem Statement

Deep research (DR) agents are designed to produce in-depth, well-attributed answers to complex tasks by planning, searching, and synthesizing information from diverse sources. However, existing open-source DR agents face significant limitations:

Training Mismatch: Most open agents are trained via Reinforcement Learning with Verifiable Rewards (RLVR) on short-form, easily verifiable QA tasks. This approach does not generalize to realistic, long-form deep research tasks where answers are complex and under-specified.
Reward Signal Deficiency: Defining reliable reward signals for open-ended long-form generation is challenging. Static, pre-defined evaluation criteria often fail to capture the nuances of quality, and accurate assessment frequently requires access to extensive, up-to-date external information beyond a model's parametric knowledge.
Lack of Open Infrastructure: Existing open systems often rely on single tools, lack full citation support, or do not release training data and code, hindering reproducibility and further research.

Methodology: Reinforcement Learning with Evolving Rubrics (RLER)

To address these challenges, the authors introduce DR Tulu-8B, the first fully open model trained end-to-end for open-ended, long-form deep research. The core innovation is Reinforcement Learning with Evolving Rubrics (RLER), a framework where evaluation criteria (rubrics) co-evolve with the policy model during training.

Core Components of RLER

Rubric as Rewards: Instead of static criteria, the system uses sample-wise rubrics defined in natural language. A judge Large Language Model (LLM) evaluates responses against these rubrics to generate scores.
Search-Augmented Evolving Rubrics:
- Initialization: For each training prompt, initial rubrics are generated by an LLM ( $G_{rubric}$ ) using retrieved documents from an external search API. This grounds the criteria in up-to-date external knowledge.
- Co-Evolution: During training, the system samples multiple model responses (rollouts) and search traces. The rubric generator then creates new rubrics by contrasting these responses.
  - Positive Rubrics: Capture strengths or new, relevant knowledge explored by the current policy that existing rubrics missed.
  - Negative Rubrics: Identify common undesirable behaviors (e.g., reward hacking, verbatim copying) observed across responses.
- Buffer Management: To prevent linear growth in rubric count, a buffer management strategy filters and ranks rubrics based on their discriminative power (standard deviation of rewards across responses), retaining only the most informative ones.
Auxiliary Rewards: The framework includes auxiliary rewards for format correctness, effective search tool usage, and high-quality citations to support relevant claims.

Training Pipeline

Base Model: The model is initialized from Qwen3-8B.
Supervised Fine-Tuning (SFT): A "cold start" phase using 16K high-quality trajectories (long-form and short-form) generated by teacher models (including GPT-5) to distill search patterns and tool usage.
Online RL: The model undergoes RL training using a variant of GRPO (Group Relative Policy Optimization). The training loop involves:
1. Generating agentic rollouts with real tool calls (Google Search, Web Browse, Paper Search).
2. Scoring responses against the dynamically updated rubric buffer.
3. Updating the policy model.
4. Refining the rubric buffer by adding new evolving rubrics and pruning low-variance ones.
Infrastructure: The authors release dr-agent-lib, an extensible agent library supporting asynchronous tool calls, global caching, and multi-tool search integration.

Key Contributions

DR Tulu-8B: The first fully open deep research model trained end-to-end for long-form tasks, capable of producing well-attributed, long-form reports.
RLER Framework: A novel training method that generates instance-specific, search-grounded rubrics that adapt to the policy model's evolving capabilities, enabling more discriminative on-policy feedback.
Open Ecosystem: The release of the model, training data, evaluation rubrics, and the dr-agent-lib infrastructure, providing a complete, reproducible stack for deep research agents.
New Benchmark: The introduction of GeneticDiseasesQA, a challenging clinical dataset requiring models to synthesize evidence for therapeutic eligibility of genetic variants.

Experimental Results

The model was evaluated across four long-form deep research benchmarks: AstaBench-ScholarQA-CS2 (SQAv2), HealthBench, ResearchQA, and DeepResearchBench (DRB).

Performance vs. Open Models: DR Tulu-8B (RL) substantially outperforms all existing open deep research agents. It achieves an average score of 65.6, exceeding the previous state-of-the-art open model (Tongyi DR-30B) by 15.6 percentage points. Notably, it outperforms larger open models (e.g., 32B backbones) despite using an 8B parameter base.
Performance vs. Proprietary Models: DR Tulu-8B matches or exceeds proprietary systems (OpenAI Deep Research, Perplexity DR, Gemini 3 Pro + Search) on average. On SQAv2, it achieves the highest performance among all systems tested.
Cost Efficiency: DR Tulu-8B is significantly more cost-effective. On SQAv2, it costs approximately $0.0019 per query, which is roughly 1,000 times cheaper than OpenAI Deep Research ($1.80/query) and orders of magnitude cheaper than other open models.
Tool Adaptivity: The model learns to select appropriate tools based on the task. For scientific literature tasks (SQAv2), it relies on paper search 90% of the time; for general-domain tasks (DRB), it utilizes web search and browsing more frequently.
GeneticDiseasesQA: On this specialized clinical task, DR Tulu-8B competes with or exceeds proprietary agents, demonstrating the ability to produce reliable, verifiable citations where other open agents fail.

Significance and Claims

The paper positions DR Tulu and RLER as a critical step toward democratizing deep research capabilities.

Bridging the Gap: The work addresses the long-standing barrier of training deep research agents on realistic long-form tasks by providing a method (RLER) that generates reliable reward signals without requiring human experts to manually refine rubrics for every instance.
Reproducibility: By releasing the full training stack (data, code, models, infrastructure), the authors aim to enable rigorous evaluation and further research in the community, a capability previously limited to proprietary systems.
Scalability of Verifiers: The authors frame RLER as a new dimension of scaling verifier compute, where "privileged information" (external search results and contrastive model responses) is provided to the judge to improve evaluation quality, rather than simply increasing the reasoning tokens of the reward model.
Modest Claims: The authors acknowledge that while DR Tulu performs competitively with proprietary systems, it is primarily a research artifact. They emphasize that downstream use requires careful evaluation, transparency about uncertainty, and human oversight, particularly in high-stakes domains like healthcare. They also note that the training reward does not always perfectly correlate with downstream evaluation performance due to mismatches in rubric design and judge models.