Technical Summary: LongR – Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

1. Problem Formulation

Reinforcement Learning (RL) has proven effective for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in domains like mathematics. However, applying RL to long-context scenarios (e.g., long-dialogue understanding, structured data analysis) presents unique challenges. In these settings, the primary difficulty is not merely token consumption but performing rigorous deduction over extensive content.

Existing approaches face two main limitations:

1. Reward Sparsity: Standard RL relies on sparse, outcome-only rewards (e.g., final answer correctness). In long-context tasks, these signals are often too coarse to effectively guide the complex reasoning process, leading to vanishing gradients for contextual grounding.
2. Structural Constraints: Recent works attempting to mitigate sparsity (e.g., LongRLVR) often require pre-segmenting documents into discrete chunks and training models to output specific chunk identifiers. This rigid segmentation risks disrupting information integrity, introduces artificial data construction overhead, and forces models into non-natural output formats that decouple "looking up" from "reasoning."

The core problem addressed by this paper is how to enhance long-context reasoning in standard autoregressive generation without rigid segmentation, while providing dense, informative feedback to guide the model toward relevant evidence.

2. Methodology: The LongR Framework

LongR proposes a unified framework that operates strictly within standard autoregressive generation, integrating three key mechanisms:

A. Dynamic "Think-and-Read" Mechanism

Instead of pre-segmenting documents, LongR encourages the model to interleave reasoning steps with document consultation directly within the chain of thought.

• Process: The model generates a trajectory where reasoning steps (internal thought) alternate with grounded evidence extraction (citations from the document).
• Benefit: This mirrors human cognitive processes, preserves the integrity of the original information, and eliminates the need for discrete chunk identifiers or specialized output formats.

B. Contextual Density Reward via Relative Information Gain

To address reward sparsity, LongR introduces a dense reward signal based on Relative Information Gain.

• Concept: The framework quantifies the utility of retrieved document segments by measuring how much the context reduces uncertainty about the extracted evidence.
• Implementation:
  1. A frozen verifier model (e.g., Qwen3-4B) estimates the probability of a generated segment $s_k$ both with the context ($P(s_k|c)$) and without it ($P(s_k)$).
  2. The utility is calculated using the Negative Log-Likelihood (NLL) reduction: $Score(s_k, c) = L(s_k) - L(s_k|c)$.
  3. Normalization: To ensure scale invariance and prevent the model from gaming the reward by generating verbose or esoteric text, the score is normalized by the prior uncertainty, resulting in Relative Information Gain:
     $$r(s_k, c) = 1 - \frac{L(s_k|c)}{L(s_k)}$$
  4. The final trajectory reward is the average of these scores across all unique quotes. If no quotes are generated, the reward is zero.

C. Progressive Curriculum Learning

To avoid costly, long-context-specific data engineering (such as constructing "consult-then-reason" demonstrations), LongR employs a curriculum learning strategy:

• The model is progressively exposed to documents of increasing length (e.g., starting at 16K tokens and scaling to 32K+).
• This allows the model to learn the "Think-and-Read" pattern purely through RL, starting from generic data and gradually mastering longer contexts.

3. Key Contributions

1. LongR Framework: A unified approach for long-context reasoning that integrates a dynamic "Think-and-Read" mechanism with a contextual dense reward, operating entirely within standard autoregressive generation.
2. Dense Utility Supervision: The proposal of a Relative Information Gain metric as a reward signal. This white-box, information-theoretic approach provides fine-grained feedback on the utility of retrieved context without relying on learned black-box reward models or rigid document segmentation.
3. Self-Learned Reasoning Patterns: Demonstration that the "Think-and-Read" pattern can be acquired from generic, readily available data via a progressive curriculum, eliminating the need for expensive, long-context-specific supervised fine-tuning (SFT) data.

4. Experimental Results

The authors evaluated LongR using the Qwen3 family (4B and 8B parameters) across three major benchmarks: LongBench v2, RULER, and InfiniteBench.

• LongBench v2: LongR achieved a 9% overall performance gain compared to the SFT baseline and significant improvements over outcome-only RL (DAPO). For the Qwen3-8B model, the score rose from 26.64 (SFT) to 36.23 (LongR). Notably, gains were most pronounced on "Hard" questions and "Long" context splits.
• RULER: LongR consistently outperformed baselines at both 64K and 128K context lengths. For the 8B model, it achieved an average score of 71.98 on the 64K split, surpassing the DAPO baseline by +16.96.
• InfiniteBench: LongR attained the highest average scores (50.18 for 8B, 47.60 for 4B), with substantial improvements on specific subtasks like En.MC and En.QA.
• Robustness to RL Algorithms: The framework demonstrated consistent improvements across diverse RL algorithms, including DAPO, CISPO, and GSPO, indicating the method's algorithm-agnostic effectiveness.
• Ablation Studies:
  • Reward Design: The "Relative Density" reward significantly outperformed "Outcome-only," "Absolute Density," and "Threshold Density" baselines, confirming that scale-invariant relative gain is crucial for preventing reward hacking.
  • Verifier Size: Using a 4B verifier provided an optimal balance between performance gains and computational efficiency, with diminishing returns observed when scaling to 32B.
  • Efficiency: Analysis of reasoning chain lengths showed that LongR does not induce "length inflation" or reward hacking (blindly copying text). Instead, it encourages "surgical extraction" of relevant evidence, maintaining token counts comparable to or lower than outcome-only baselines.

5. Significance and Claims

The paper claims that LongR addresses the critical bottleneck of reward sparsity in long-context RL without imposing the structural constraints of previous methods. By leveraging Relative Information Gain, the framework provides a robust, intrinsic signal that incentivizes the model to actively seek high-value information rather than relying on superficial query matching.

The authors emphasize that LongR enables models to:

• Navigate extensive contexts with robust efficiency.
• Learn complex reasoning patterns without costly, domain-specific data engineering.
• Maintain information integrity by avoiding rigid document segmentation.

Ultimately, LongR demonstrates that integrating dense utility rewards with a progressive curriculum can significantly unlock the reasoning potential of LLMs in long-context scenarios, offering a scalable path toward more capable autonomous agents and reasoning systems.