💻 computer science

From Refuse to Richness: Rubric Rewards for Long-Form Hallucination Reinforcement Learning

This paper addresses the trade-off between grounding and coverage in long-form generation by proposing a key-point rubric-based reinforcement learning framework that, when softly combined with grounding and relevance rewards, achieves a superior balance of factual support and comprehensive information coverage compared to existing methods.

Original authors: Yudong Wang, Zhe Yang, Wenhan Ma, Rang Li, Qibin Yang, Weimin Xiong, Jiangshan Duo, Liang Zhao, Zhifang Sui

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yudong Wang, Zhe Yang, Wenhan Ma, Rang Li, Qibin Yang, Weimin Xiong, Jiangshan Duo, Liang Zhao, Zhifang Sui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write a long, detailed story about a historical event based on a specific set of notes you gave it. You want two things: first, you want the robot to stick strictly to the facts in your notes (so it doesn't make things up), and second, you want it to write a rich, complete story that covers all the important details you asked for. This is the tricky world of "long-form generation" for Artificial Intelligence. The problem is that these two goals often fight each other. If you tell the robot, "Don't say anything unless you are 100% sure it's in the notes," it gets scared and starts saying very little, or even refusing to answer, just to be safe. But if you tell it, "Write as much as you can," it might start inventing wild details that aren't true. Scientists call this the tension between being "grounded" (sticking to the truth) and being "rich" (being helpful and detailed).

This paper, titled "From Refuse to Richness," explores how to fix this tug-of-war using a special kind of training called Reinforcement Learning. Think of this training like a video game where the robot gets points (rewards) for doing well. The researchers wanted to see if they could teach the robot to be both honest and helpful without forcing it to choose one over the other. They discovered that the way you give points matters more than you might think. If you only give points for being safe, the robot becomes a boring, silent librarian. If you only give points for being detailed, it becomes a chaotic storyteller. But, they found a "sweet spot" by using a specific type of checklist that tells the robot exactly what information needs to be included, rather than just counting how many words it wrote.

The Problem: The "Say Less" Trap

The researchers started with a frustrating observation. When they trained AI models to avoid making up facts (hallucinations) by strictly punishing any sentence that couldn't be proven by the provided text, the models learned a very simple, very unhelpful trick: say as little as possible.

Imagine a student taking a test where the teacher says, "If you write anything wrong, you get a zero." The smartest move for that student isn't to write a brilliant essay; it's to write one word or even nothing at all, just to avoid the risk. The paper found that when AI models were trained with these "strict grounding" rewards, they would shrink their answers dramatically. In just a few steps of training, the models learned that the safest way to avoid making unsupported claims was to stop talking. They became "refusal" machines—safe, but useless because they refused to provide the rich information the user actually wanted.

The Solution: The "Rubric" Checklist

To fix this, the authors tried a new approach. Instead of just telling the AI, "Don't lie," they gave it a rubric.

Think of a rubric like a detailed checklist for a school project. Instead of just saying "Make a good poster," the teacher hands you a list: "Must include the date, must include the main character's name, and should include a picture of the setting." This list tells you exactly what "richness" looks like for that specific question.

The researchers created these rubrics for every question the AI answered. The rubric listed "required" points (the must-haves) and "optional" points (the nice-to-haves). They then used this checklist as a reward signal. If the AI covered the required points, it got points. If it missed them, it lost points. This taught the AI that being "rich" meant covering the specific checklist items, not just writing a lot of words.

The Discovery: The "Soft" Mix is Best

The team tested several different ways to combine these rewards, and the results revealed a fascinating trade-off:

Strict Grounding Only: The AI became very safe (high accuracy) but very short and unhelpful. It stopped trying to cover the checklist.
Rubric Only: The AI became very detailed and covered the checklist perfectly, but it started making up facts again because it wasn't being punished for lying.
The "Proxy" Method: They tried using generic signals like "count the number of sentences" or "did the AI write a lot?" This didn't work well. It was like rewarding a student for writing 500 words without checking if those words were actually about the topic. The AI just filled space with fluff.

The winning strategy was a soft combination called FACT-RUBRIC-REL. This method didn't use a "hard gate" (where one mistake kills the whole score). Instead, it gently balanced three things:

Grounding: Did the AI stick to the facts?
Rubric Coverage: Did it cover the checklist items?
Relevance: Did the answer actually make sense?

By mixing these together softly, the AI learned that it could get points for being detailed even if it wasn't perfect on every single fact, as long as it was generally grounded. This prevented the "say less" trap. The models learned to be brave enough to cover the checklist but careful enough to stay mostly true to the source.

The Results: Better Everywhere

The researchers tested this on two different AI models (Qwen3-4B and DeepSeek-R1-Distill-Llama-8B) and found the same pattern in both.

On standard tests: The "soft mix" model improved its ability to stick to facts compared to the base model, without losing its ability to write long answers.
On new, tricky tests: This is where it got really interesting. When they tested the models on tasks they hadn't seen before (like creative writing or solving checklist puzzles), the models trained with "strict grounding" failed miserably. They were too scared to try. But the models trained with the "soft mix" (FACT-RUBRIC-REL) performed the best. They transferred their skills to new tasks much better than the others.

The paper suggests that the "hard" rewards were actually hurting the AI's ability to be creative and helpful in new situations. By using the rubric to define what "richness" looks like, and then gently encouraging the AI to hit those targets without fear of total failure, they found a way to make the AI both honest and helpful.

The Bottom Line

The main takeaway is that you can't just punish an AI for lying; you have to actively reward it for being helpful. If you only focus on safety, the AI learns to shut up. If you only focus on volume, it learns to lie. The secret sauce is a rubric-based reward system that tells the AI exactly what information it needs to cover, combined with a gentle nudge to stay grounded. This approach, which the authors call "From Refuse to Richness," suggests that the future of helpful AI lies in giving it a clear checklist of what to say, rather than just a scary list of what not to say.

Technical Summary: From Refuse to Richness: Rubric Rewards for Long-Form Hallucination Reinforcement Learning

1. Problem Statement

Long-form generation in Large Language Models (LLMs) suffers from two distinct types of hallucination: factual hallucination (claims inconsistent with world knowledge) and faithfulness hallucination (claims unsupported by provided context). While Reinforcement Learning (RL) with grounding rewards can mitigate unsupported claims, it introduces a critical trade-off: models often learn to "refuse" to answer or provide overly concise responses to avoid penalties, thereby sacrificing richness (information coverage).

Existing approaches to enforce richness rely on global proxies such as response length, claim count, or pairwise relevance. The authors argue these are insufficient because:

Length can be inflated by filler.
Claim counts can include trivial or irrelevant facts.
Generic relevance judgments fail to identify specific missing question-dependent content.
Strict grounding rewards (e.g., binary penalties for unsupported sentences) encourage models to minimize output to maximize safety, leading to a "refusal-to-richness" collapse where models answer less to avoid hallucination.

2. Methodology

Core Concept: Key-Point Rubrics

Instead of global proxies, the paper proposes using per-sample key-point rubrics. A rubric is defined not as a gold-standard answer, but as a structured checklist specifying the required and optional information a rich answer must cover for a specific question.

Construction: Rubrics are generated using a multi-generator union pipeline (GPT-5.4, Gemini-3-Flash-Preview, and GPT-OSS-120B) to ensure broad coverage and avoid single-generator omissions.
Function: They serve dual purposes: as an evaluation metric for information coverage and as a reward signal in RL.

Reward Design Variants

The study compares five reward strategies on two model families (Qwen3-4B and DeepSeek-R1-Distill-Llama-8B):

FACT-ONLY: Binary grounding reward ( $R = G(y)$ ). Penalizes any unsupported sentence.
FACT-PROXY: Combines grounding with generic detail (log of supported claim count) and pairwise relevance.
RUBRIC-ONLY: Rewards coverage of the rubric checklist ( $C_{rub}$ ) without an explicit grounding term.
FACT-RUBRIC: An additive combination of grounding and rubric coverage.
FACT-RUBRIC-REL: A soft, three-component reward combining grounding ( $0.5G$ ), rubric coverage ( $0.4C_{rub}$ ), and pairwise relevance ( $0.1R_{rel}$ ). Crucially, this avoids "gating" (multiplicative) where a grounding failure zeroes out the entire reward.

Training Protocol

Algorithm: GRPO (Group Relative Policy Optimization).
Data: A 4,000-prompt training set comprising 2,000 grounded prompts (with context) and 2,000 open prompts (without context).
Safety Mechanism: A format penalty is applied if the response is too short to support meaningful grounding judgment, preventing models from "hacking" the reward by truncating answers.

3. Key Results

In-Distribution Performance (Grounding Benchmarks)

On standard grounding benchmarks (FACTS Grounding, LongFact, FActScore):

FACT-ONLY achieves the highest grounding accuracy but significantly suppresses rubric coverage and claim count (e.g., Qwen3-4B coverage drops from 0.459 to 0.334).
RUBRIC-ONLY maximizes coverage but causes grounding accuracy to collapse (e.g., dropping to 0.097 on Qwen3-4B).
FACT-RUBRIC-REL offers a balanced trade-off, maintaining grounding above the base model while preserving high coverage.

Out-of-Distribution (OOD) Performance

On tasks requiring checklist completion (CL-bench) and creative writing (Arena-Hard):

Strict Grounding (FACT-ONLY/PROXY): These rewards actively damage OOD performance. On DeepSeek-R1-Distill-Llama-8B, FACT-PROXY reduces creative writing win rates to a fraction of the base model's performance.
RUBRIC-ONLY: Delivers the strongest OOD results, indicating that question-specific coverage signals transfer well to diverse tasks.
FACT-RUBRIC-REL: This is the only variant that improves both in-distribution grounding and OOD behavior compared to the base model. It successfully transfers the checklist-completion behavior of rubric rewards while retaining sufficient grounding.

Model Sensitivity

The "refusal-to-richness" trade-off is consistent across both Qwen3-4B and DeepSeek-R1-Distill-Llama-8B. However, the reasoning-distilled DeepSeek model is more sensitive to hard grounding rewards, exhibiting sharper response-length collapse and more severe OOD degradation than Qwen.

4. Key Contributions

Formalization of Richness: The paper defines richness for long-form answers as question-specific information coverage rather than generic detail, framing hallucination RL as a joint mitigation of factual and faithfulness errors.
Rubric Pipeline: The construction of a multi-generator pipeline for per-sample key-point rubrics, applied across four grounding benchmarks and a 4K RL training set.
Reward Composition Analysis: Empirical evidence that soft composition of grounding and rubric coverage (specifically FACT-RUBRIC-REL) yields the best Pareto frontier, balancing in-distribution safety with out-of-distribution utility.

5. Significance and Claims

The paper claims that the "refusal-to-richness" problem arises because rewards that overfit to grounding teach models to say less. The significance of this work lies in demonstrating that:

Grounding alone is insufficient: It captures support but not completeness.
Rubrics are essential: They provide the necessary signal for what an answer should contain, not just what it should avoid.
Soft composition is key: Multiplicative gating (where grounding failure nullifies coverage rewards) suppresses useful behavior. A soft additive combination allows models to receive credit for covering required points even if minor grounding issues exist, preserving the "checklist-style" behavior needed for long-form generation.

The authors conclude that per-sample rubrics are most effective as a complement to grounding, not a replacement, and that the proposed reward composition (FACT-RUBRIC-REL) provides a robust solution for training long-form generators that are both grounded and rich.

6. Limitations

The study is limited to two relatively small open-weight models (Qwen3-4B and DeepSeek-R1-Distill-Llama-8B). The authors note that stronger proprietary or larger models may exhibit different sensitivities to hard grounding gates. Additionally, FActScore closed-book performance remained near zero, attributed to the base model's parametric knowledge limitations rather than the reward design.