Technical Summary: Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

Problem Statement

While Reinforcement Learning (RL) has proven effective for enhancing the reasoning capabilities of Large Language Models (LLMs), existing efforts predominantly focus on short-context tasks. Extending these gains to long-context reasoning remains underexplored. Current approaches often rely on complex reward engineering, supplementing outcome-based rewards with auxiliary signals (e.g., evidence grounding scores or intermediate reasoning steps) to guide models over lengthy inputs. However, these methods face challenges: outcome-only rewards provide sparse signals for locating evidence in long contexts, leading to plateaued recall and shortcut reasoning. Furthermore, high-quality, diverse training data for long-context RL remains scarce, with existing synthetic datasets often being narrow in task coverage or closed-source.

Methodology

The authors propose a data-centric perspective, hypothesizing that long-context reasoning decomposes into a small set of complementary core abilities that must be exercised jointly. Instead of complex reward engineering, they demonstrate that a simple yet effective data recipe paired with a minimal outcome-based RL setup suffices to substantially improve performance.

1. The Data Recipe

The training mixture targets three complementary task families, totaling ~14,000 examples constructed across eight datasets:

• Retrieval: Focuses on locating relevant evidence from long inputs where information is scattered or semantically indirect.
  • FuzzyNeedle: Removes lexical shortcuts by paraphrasing "needles" (target information) using Wikidata IS-A relations, requiring models to resolve instance-of relations rather than keyword matching.
  • MultiNeedle: Requires distinguishing among many near-duplicate needles and retrieving a specific one based on its order of occurrence.
• Multi-evidence Synthesis: Requires integrating information scattered across different parts of the input to derive an answer no single piece can provide.
  • CrossEntity: Generates questions requiring aggregation of derived attributes from multiple entities (e.g., calculating "winning age" from birth and championship years).
  • WebSearch: Simulates search-agent tasks where models must trace multi-hop relation chains while distinguishing gold evidence from topically similar distractors.
  • MultiQuery: Embeds multiple queries in a document; missing any piece of evidence leads to an incorrect response, stressing complete coverage.
  • KeyChain: Hides questions behind chains of UUID pointers, requiring models to trace chains to recover the actual query.
  • LongDocQA: Extends existing datasets (HotpotQA, MuSiQue, Qasper) with full-length documents or irrelevant padding to increase context length.
• Reasoning: Focuses on performing complex multi-step problem solving (e.g., mathematics) over long inputs to preserve deep thinking abilities.
  • LongMath: Adapts difficult short-context math problems into narrative scenarios where variables and conditions are distributed across at least five story fragments scattered throughout an irrelevant long document.

Data Curation: The authors apply a two-stage pipeline: discarding samples exceeding 64K tokens and filtering for "moderate difficulty" by retaining only samples where a base model (Qwen3-30B-A3B) gets some but not all rollouts correct.

2. Reinforcement Learning Setup

The authors employ Group Relative Policy Optimization (GRPO) with a minimal reward design:

• Reward: Primarily a word-level recall reward ($r = |\hat{y} \cap y^*| / |y^*|$) between the model's final answer and the label. For complex tasks (CrossEntity, LongMath), an LLM-as-judge (DeepSeek-V3.2) is used to verify correctness.
• Stabilization: To handle the heterogeneous reward distributions of the eight datasets, the authors use task-balanced sampling (fixing per-dataset ratios in batches) and task-level advantage normalization (rescaling advantages by per-dataset standard deviation).
• No Process Rewards: Unlike prior work, this approach does not use auxiliary signals for intermediate reasoning steps or evidence grounding, relying solely on the outcome reward.

Key Contributions

1. Data-Centric Paradigm: The paper challenges the prevailing focus on reward engineering, showing that a diverse data recipe alone is sufficient to drive significant improvements in long-context reasoning.
2. Unified Task Taxonomy: The authors define and curate a training mixture covering three complementary abilities (Retrieval, Multi-evidence Synthesis, Reasoning), arguing that all three are necessary for optimal performance.
3. Open Datasets: The authors construct and release eight datasets totaling ~14K examples to facilitate future research.
4. Transferability: The study demonstrates that improvements in long-context reasoning transfer to agentic tasks, suggesting that strengthening core reasoning abilities benefits long-horizon agent capabilities.

Experimental Results

The method was evaluated on three models (Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B) across seven long-context benchmarks.

• Performance Gains: The data recipe yielded consistent average improvements of +7.2, +3.2, and +6.4 points across the three models, respectively.
• Comparison: The approach outperformed two prior long-context RL training sets (DocQA-RL-1.6K and KeyChain-15K) across all models and benchmarks.
• Generalization: Despite training on inputs capped at 64K tokens, the gains transferred to substantially longer contexts (up to 512K+ tokens), indicating the models learned length-generic reasoning skills rather than overfitting to the training distribution.
• Ablation Studies:
  • Removing any single task category or pairwise combination resulted in lower performance than the full mixture, confirming the complementarity of the three task families.
  • Removing task-balancing techniques caused a moderate performance drop, highlighting the need to mitigate inter-task competition.
  • Adding a "process reward" (LLM-as-judge scoring reasoning coherence) did not improve performance over the outcome-only setup, suggesting the diverse data already provides informative signals.
• Agentic Transfer: Continuing RL training on an agent-tuned model (AgentCPM-Explore) with this data recipe improved GAIA by +4.8 points and BrowseComp by +7.0 points, demonstrating the transferability of long-context skills to autonomous agents.

Significance and Claims

The paper claims that diverse training data is the primary bottleneck for long-context RL, not the reward design. By constructing a balanced mixture of retrieval, synthesis, and reasoning tasks, the authors achieve state-of-the-art results without complex reward engineering. The work suggests that strengthening core long-context abilities is an effective pathway toward enhancing agentic capabilities, as agents naturally accumulate massive context during tool use. The authors modestly note limitations, including the use of synthetic data which may not fully match real-world distributions, and the current constraint to models up to 30B parameters and 64K context lengths.