Technical Summary: GoLongRL

Problem Statement

While large language models (LLMs) have rapidly expanded context windows through pretraining and mid-training, effectively utilizing these extended contexts during post-training remains a bottleneck. Existing reinforcement learning (RL) approaches for long-context reasoning suffer from two primary limitations:

1. Narrow Data Coverage: Current methods often construct data around complex retrieval paths (e.g., UUID chains, chunk-based QA), leading to homogeneous task coverage. This results in inadequate supervision for critical capabilities such as summarization, ranking, aggregation, and structured reasoning.
2. Optimization Instability in Multitask Settings: Standard Group Relative Policy Optimization (GRPO) faces challenges when applied to heterogeneous tasks with different reward metrics (e.g., EM, F1, NDCG, ROUGE-L).
   • Difficulty Bias: Per-prompt normalization (dividing by standard deviation $\sigma_u$) can inflate advantages for easy and hard prompts while suppressing medium-difficulty samples.
   • Cross-Task Scale Inconsistency: Different reward metrics exhibit varying variance profiles. Without proper normalization, high-variance tasks can dominate the gradient updates, distorting the optimization trajectory.

Methodology

1. Capability-Oriented Data Construction

The authors propose a data construction framework guided by a taxonomy of 9 core long-context capabilities, rather than simply injecting length into prompts.

• Dataset Composition: A fully open-source dataset of 22,965 RLVR (Reinforcement Learning with Verifiable Rewards) samples.
  • Sources: Roughly 14K curated open-source samples from established corpora (e.g., LongBench Pro, CLongEval) and 9K synthetic samples where QA pairs are generated from real documents (books, academic papers, legal filings).
  • Task Taxonomy: The dataset spans 9 task types (T1–T9), including Precise Retrieval, Comprehension, Exhaustive Retrieval, Numerical Reasoning, Structured Extraction, Structured Matching, Graded Ranking, Sequence Ordering, and Summarization.
  • Reward Alignment: Each task is paired with its natural evaluation metric (e.g., NDCG for ranking, F1 for exhaustive retrieval, ROUGE-L for summarization) rather than collapsing them into a single binary signal.
• Construction Pipeline: A four-phase process involving source collection, task-oriented filtering, sample construction with multi-stage quality control (using models like Gemini-2.5-Pro and DeepSeek-V3.2), and iterative refinement based on benchmark diagnostics.

2. TMN-Reweight Algorithm

To address optimization challenges arising from heterogeneous rewards, the paper introduces TMN-Reweight (Task-level Mean Normalization with difficulty-adaptive Reweighting).

• Step 1: Task-Level Mean Normalization: Instead of normalizing by the per-prompt standard deviation ($\sigma_u$), which conflates scale correction with difficulty amplification, the method normalizes by a task-level aggregated standard deviation ($\sigma_{task}$). This is calculated as the root mean square of per-prompt standard deviations within a specific task. This reduces cross-task scale differences while preserving relative advantage magnitudes within each task.
• Step 2: Difficulty-Adaptive Reweighting: To correct for difficulty bias within tasks, the method estimates prompt difficulty using a smoothed pass rate (interpolating between per-prompt and task-level baselines).
  • Four-Quadrant Gradient Reallocation: The difficulty weight ($w$) is applied asymmetrically:
    • For hard prompts ($w > 1$): Positive advantages are amplified to reinforce rare successful rollouts; negative advantages are scaled down ($1/w$) to reduce destabilizing gradients.
    • For easy prompts ($w < 1$): Positive advantages are attenuated to prevent probability concentration; negative advantages are amplified to increase learning from unexpected failures.

Key Contributions

1. Open-Source Capability-Oriented Dataset: Release of a 23K-sample dataset covering 9 diverse task types with heterogeneous reward functions. The authors demonstrate that training on this data alone (using vanilla GRPO) outperforms the closed-source QwenLong-L1.5 dataset at both 4B and 30B scales.
2. TMN-Reweight Algorithm: A novel optimization strategy that jointly mitigates cross-task reward scale inconsistency and difficulty-induced advantage bias. It yields consistent performance gains over vanilla GRPO without requiring a separate value network.
3. Generalization Preservation: The framework demonstrates that long-context RL with capability-oriented data preserves or improves general reasoning and memory capabilities, including transfer to agentic memory and long-term dialogue benchmarks not seen during training.

Results

• Benchmark Performance:
  • On the Qwen3-4B-Thinking model, training with the capability-oriented dataset and vanilla GRPO raised the long-context benchmark average from 53.0 to 62.2, surpassing QwenLong-L1.5 (59.4). Adding TMN-Reweight further improved the average to 63.0.
  • On the Qwen3-30B-A3B model, the dataset improved the average from 60.1 to 69.8 (vs. 67.2 for QwenLong-L1.5).
  • The model trained on this data achieves performance comparable to much larger closed-source models like DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507 on long-context benchmarks.
• Ablation Studies:
  • The dataset itself is identified as the primary driver of performance gains, with TMN-Reweight providing a consistent but modest additional boost (0.8 points on the 4B model), particularly on aggregation-intensive tasks like CorpusQA.
  • The hyperparameter $\alpha$ (controlling the balance between prompt-level and task-level difficulty estimation) performs best at 0.8.
• General Capabilities: Evaluations on MMLU-Pro, AIME24/25, and GPQA show that general reasoning capabilities are preserved or improved. Significant gains are observed in agentic memory (Memory-Vec, Memory-Rec_Sum) and dialogue memory (LongMemEval).
• Length Extrapolation: Models trained on 160K contexts generalize effectively to evaluation sequences up to 1M tokens, showing consistent gains across model scales.

Significance and Claims

The paper claims that data coverage and reward diversity are primary bottlenecks in long-context RL, often more critical than algorithmic refinements alone. By shifting from retrieval-centric data construction to a capability-oriented taxonomy, the authors provide richer training signals that benefit a broad spectrum of long-context tasks.

The work suggests that broader coverage and greater reward diversity substantially benefit long-context capability improvement, even with standard RL algorithms. Furthermore, the proposed TMN-Reweight offers a stable optimization mechanism for heterogeneous multitask settings, ensuring that the benefits of long-context training do not come at the cost of general reasoning or memory degradation. All resources, including the dataset, construction pipeline, and training code, are publicly released to facilitate further research.