💬 NLP

Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation

This paper introduces RioRAG, a framework that enhances long-form retrieval-augmented generation by defining informativeness as a verifiable objective and employing nugget-centric, cross-source verification to provide stable, dense rewards for reinforcement learning without relying on handcrafted supervision or strong teacher models.

Original authors: Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, Haifeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart but sometimes overconfident librarian (an AI) to write a long, detailed report on a complex topic, like "How does quantum tunneling work?"

The librarian doesn't just guess; they go to the library stacks, pull out ten different books, and try to write a perfect summary based on what they found. This is called Retrieval-Augmented Generation (RAG).

However, the paper points out a major problem: How do you grade the librarian's work?

The Problem: The "Exact Match" Trap

In short quizzes (like "What is 2+2?"), grading is easy: the answer is either "4" or it isn't. But for long reports, there is no single "correct" answer.

The Old Way: Current AI systems try to grade these long reports using vague rules or by asking another AI, "Is this good?" This is like asking a tired teacher to grade 50 essays at once. They get tired, their scores jump up and down randomly (instability), and they often miss the point. The AI gets confused feedback and doesn't learn how to get better.
The Result: The AI might write a huge, fluffy essay that sounds nice but misses the actual facts, or it might hallucinate (make things up) because it doesn't know what "truth" looks like in a long text.

The Solution: RioRAG (The "Fact-Check" System)

The authors propose a new system called RioRAG. Instead of asking "Is this essay good?", they change the game to: "Did you include every single important fact?"

Here is how RioRAG works, using a simple analogy:

1. The "Nugget" Hunt (Breaking it Down)

Imagine the librarian's source books contain hundreds of tiny, golden facts (nuggets).

Old Way: The grader reads the whole essay and gives a vague score like "7/10."
RioRAG Way: Before the librarian writes, the system extracts all the specific, verifiable facts from the source books and puts them on a Checklist.
- Example Checklist: "Mentions crossing barriers," "Mentions Josephson junctions," "Mentions STM devices."

2. The "Fact-Check" Grading (Verifiable Rewards)

When the librarian writes their answer, the system doesn't guess if it's "good." It simply checks the Checklist.

Did you mention the barriers? (Yes/No)
Did you mention the junctions? (Yes/No)
Did you mention the devices? (Yes/No)

If the answer covers 3 out of 3 items, it gets a perfect score. If it covers 1, it gets a lower score. This is verifiable because you can point exactly to where the fact came from in the source book. It removes the guesswork.

3. The "Length Penalty" (Don't Just Ramble)

Sometimes, an AI tries to cheat by writing a 10-page essay just to increase its chances of hitting the right facts by accident.

RioRAG has a rule: If you write too much without adding new facts, your score goes down.
It encourages the AI to be concise and dense with information, rather than long and fluffy.

4. Self-Improvement (The Gym)

The system puts the AI through a training loop:

The AI tries to write an answer.
The system checks it against the "Fact Checklist."
The AI gets a clear score (Reward).
The AI tries again, using that score to learn how to hit more facts next time.

Because the feedback is clear and based on real facts (not vague opinions), the AI learns much faster and more stably. It doesn't need a "super-teacher" to write the perfect answers for it to copy; it learns by trying to hit the checklist targets itself.

The Results

The authors tested this on two big benchmarks (LongFact and RAGChecker). They found that:

More Facts: The AI remembered and included more actual facts from the source documents.
Less Hallucination: The AI made up fewer fake facts because it was strictly rewarded for sticking to the checklist.
Better Stability: The training didn't crash or go crazy because the scoring system was reliable.

In a Nutshell

RioRAG stops trying to guess if an AI's long essay is "good" and starts checking if it is complete. By turning a vague essay into a simple "Did you check off these facts?" game, they built a system that teaches AI to be more truthful, more detailed, and more reliable when answering complex questions.

Technical Summary: Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation (RioRAG)

1. Problem Statement

Long-Form Question Answering (LFQA) requires synthesizing coherent, factually grounded content from multi-source evidence. While Retrieval-Augmented Generation (RAG) mitigates hallucinations by grounding generation in retrieved documents, current systems face significant challenges in reinforcement learning (RL) reward design.

Standard RL rewards for LFQA often rely on:

Exact-match assumptions: Suitable for short-form QA or math, but ineffective for open-ended, multi-sentence responses where multiple valid phrasings exist.
Unstable and non-verifiable signals: Rule-based Outcome Reward Models (ORMs) struggle with diverse responses, and model-based scoring often exhibits high variance and unreliability (as illustrated in Figure 1 of the paper), leading to sparse reward signals that hinder self-exploration and optimization.
Dependence on strong supervision: Existing methods often rely on handcrafted templates, powerful teacher models for distillation, or static datasets, limiting generalization to real-time web knowledge and diverse domains.

Consequently, current RAG systems lack verifiable reward mechanisms, resulting in unstable feedback and sub-optimal optimization outcomes for long-form generation.

2. Methodology: RioRAG

The authors propose RioRAG (Reinforced Informativeness Optimization), a framework designed to optimize long-form RAG through stable, verifiable reward learning. The methodology consists of two core components:

A. Reinforced Informativeness Optimization

RioRAG redefines the optimization objective from vague holistic scoring to measurable informativeness.

Definition of Informativeness: Instead of measuring response length or style, informativeness is defined as the coverage of "nuggets"—concise, reusable, and verifiable factual units (e.g., short facts or claims) distilled from retrieved sources.
Training Data Construction: The framework utilizes the ELI5 dataset but removes human-written answers, relying solely on queries. It dynamically retrieves top-K webpages for each query to form the context $D_q$ , ensuring the model learns from current, diverse online sources rather than static annotations.
Optimization Algorithm: To address gradient instability in long-form text, RioRAG employs Group-wise Relative Policy Optimization (GRPO). This normalizes rewards within sampled groups to mitigate variance and stabilize policy updates.
Action Activation: Unlike methods that introduce new delimiter tokens (e.g., <thought>) which hinder warm-starting from instruction-tuned models, RioRAG leverages existing Markdown headers to separate reasoning from the final answer, preserving pretrained priors.

B. Nugget-Centric Hierarchical Reward Modeling

To provide dense, action-discriminative, and verifiable feedback, RioRAG decomposes reward computation into three stages:

Factual Nugget Extraction: Concise evidence statements are extracted from retrieved documents.
Evidence Checklist Synthesis: Extracted nuggets are clustered and merged into a unified checklist $C(q, D_q)$ , removing redundancy and creating a ground-truth reference for the query.
Informativeness Assessment: The reward is calculated based on the proportion of checklist nuggets covered in the generated answer. Each match is explicitly traceable to a source document.

Length Decay Mechanism: To prevent "reward hacking" via excessively long responses, a length-decay normalization is applied only to the final answer segment (leaving reasoning tokens unaffected). This penalizes unnecessary verbosity while preserving the deliberative process.

3. Key Contributions

Verifiable Reward Formulation: RioRAG defines informativeness as an externally verifiable objective derived from evidence support rather than heuristic lexical rules or black-box model scoring.
Nugget-Centric Verification: By converting open-ended evaluation into checklist-based credit assignment, the framework enables fine-grained, reproducible feedback. This avoids long-context degradation and provides denser signals for RL.
Self-Evolution without Strong Supervision: The framework enables smaller policy models to self-evolve through interaction with real-time web knowledge, eliminating the need for handcrafted policy supervision or distillation from strong teacher models (e.g., sequential SFT).
Efficiency: Despite the multi-stage verification process, the use of compact inputs (single documents or short checklists) keeps token counts low. The system supports parallel and asynchronous execution, making it computationally comparable to generative reward baselines.

4. Experimental Results

The authors evaluated RioRAG on two benchmarks: LongFact (covering 38 domains) and RAGChecker (assessing factual grounding and retrieval quality).

Performance Gains: RioRAG achieved superior performance across all metrics compared to prompt-based methods, Supervised Fine-Tuning (SFT) baselines, and other RL-based approaches (like DPO).
- On LongFact, it showed significant improvements in Fact Recall (FR) and Information Density (ID) across diverse domains (Science, Medicine, Law, etc.).
- On RAGChecker, it excelled in knowledge point coverage, retrieval utilization, hallucination mitigation, and faithfulness.
Ablation Studies:
- Removing informativeness optimization or nugget-based rewards led to significant performance drops, confirming the necessity of the proposed reward structure.
- Removing the length-decay term resulted in lower information density, validating the mechanism's role in preventing verbose, low-quality outputs.
- Comparisons with off-policy RL and generative reward models highlighted the superiority of the on-policy GRPO approach and verifiable reward signals.
Scaling and Cold-Start:
- Scaling Law: RioRAG demonstrated consistent performance improvements as model size increased (1.5B to 14B), though the growth curve showed a sublinear relationship.
- Cold-Start: Initialization with an R1-distilled model (incorporating slow-thinking patterns) yielded the largest improvements (29.6%), suggesting that structured reasoning priors facilitate stable RL training.

5. Significance and Claims

The paper claims that RioRAG establishes verifiable reward modeling as a foundational step for trustworthy long-form RAG. By shifting from vague holistic scoring to explicit, nugget-aligned checklist verification, the framework addresses the core instability and unreliability of current reward designs.

The authors position RioRAG as a solution that allows models to self-improve in open-domain settings without relying on scarce, high-quality annotated data or powerful teacher models. They argue that future evaluation frameworks for long-form tasks should prioritize granular factual alignment over surface-level metrics.

Limitations Acknowledged:

The current focus is on English corpora.
Reliance on automatic nugget extraction may inherit biases from pre-trained models.
The reward primarily targets factual informativeness and does not explicitly capture linguistic style, coherence, or readability, which are also important for human preference.
Computational constraints limited experiments to models up to 32B parameters.