Technical Summary: InftyThink+

1. Problem Statement

Large reasoning models (LRMs) have demonstrated superior performance by scaling inference-time chain-of-thought (CoT) reasoning. However, this paradigm faces three fundamental barriers when reasoning traces become exceptionally long:

1. Quadratic Cost: The self-attention mechanism in Transformers incurs $O(L^2)$ computational complexity, making very long reasoning traces prohibitively expensive.
2. Context Limits: Reasoning is hard-bounded by the model's maximum context window. If a problem requires a chain of thought exceeding this limit, generation terminates prematurely, leaving complex problems unsolvable.
3. Lost-in-the-Middle: As reasoning traces grow, models suffer from degraded performance where critical early information becomes inaccessible, even if the context limit is not exceeded.

Existing solutions, such as iterative reasoning with periodic summarization, attempt to mitigate these issues. However, current methods rely on supervised fine-tuning (SFT) or fixed heuristics to determine when to summarize, what to preserve, and how to resume reasoning. These approaches fail to optimize these sequential decisions, which have long-horizon consequences. A poor summary can doom subsequent reasoning, while unnecessary iterations waste compute. The paper argues that these trade-offs require trajectory-level optimization that SFT cannot provide.

2. Methodology: InftyThink+

The authors propose InftyThink+, an end-to-end reinforcement learning (RL) framework that optimizes the entire iterative reasoning trajectory. The method builds upon the InftyThink paradigm, which decouples reasoning depth from context length by decomposing reasoning into multiple iterative rounds connected by explicit, model-generated summaries.

2.1. The InftyThink Paradigm

Unlike the "Vanilla" paradigm where a model generates a single continuous long CoT, InftyThink operates in rounds:

• Iteration $i$: The model conditions on the original query $q$ and the summary $s_{i-1}$ from the previous iteration.
• Generation: It generates a reasoning segment $r_i$ and an updated summary $s_i$.
• Termination: The process repeats until the model autonomously decides to generate a final conclusion $c$ instead of a summary.
• Bounded Context: Each iteration operates within a bounded context window (query + latest summary), preventing context overflow and reducing per-step computational cost.

2.2. Two-Stage Training Scheme

InftyThink+ employs a two-stage training process to separate format acquisition from strategy optimization:

1. Cold-Start (Supervised Fine-Tuning):

   • Goal: Teach the model the basic InftyThink format (generating reasoning segments and summaries).
   • Data Transformation: Existing vanilla reasoning data $(q, r, c)$ is transformed into iterative instances. The reasoning trace $r$ is partitioned into segments $\{r_1, \dots, r_n\}$ based on a token limit $\eta$. A general-purpose LLM generates summaries $\{s_1, \dots, s_{n-1}\}$ conditioned on the current segment and the previous summary.
   • Training: The model is fine-tuned on these transformed instances to produce syntactically valid iterative outputs. This stage does not optimize when to summarize or what to preserve; it only establishes the procedural format.

2. Reinforcement Learning (Trajectory-Level Optimization):

   • Goal: Optimize strategic decisions (timing of compression, content of summaries, continuation strategies) via trajectory-level feedback.
   • Rollout: For a query $q$, the model generates a trajectory $O_i = \{o_1, \dots, o_{n_i}\}$ consisting of multiple reasoning rounds. The rollout terminates if the model outputs a conclusion, fails to produce valid format, or reaches a maximum iteration cap $\phi$.
   • Reward Design:
     • Task Reward ($R_{task}$): Binary reward based on the correctness of the final answer.
     • Efficiency Reward ($R_{eff}$): A quadratic decay function $1 - (\frac{n_i-1}{\phi})^2$ that penalizes excessive iterations, encouraging the model to solve problems in fewer steps when possible.
     • Total Reward: $R(O_i) = R_{task}(O_i) \cdot R_{eff}(O_i)$. This multiplicative formulation ensures efficiency is only rewarded for correct solutions.
   • Policy Optimization: The authors use Group Relative Policy Optimization (GRPO). Crucially, advantages are computed at the trajectory level and shared across all tokens in a trajectory. This ensures that early iterations (e.g., a high-quality first summary) receive positive gradient signals if they contribute to a correct final answer, even if the summary itself does not contain the answer.
   • Stability: The framework employs IcePop, a token-level gradient masking technique, to mitigate training instability caused by mismatches between the inference and training engines.

3. Key Contributions

1. End-to-End RL for Iterative Reasoning: The paper introduces the first framework to apply reinforcement learning directly to the iterative reasoning paradigm, enabling the model to learn optimal strategies for summarization and continuation rather than relying on fixed heuristics or imitation.
2. InftyThink+ Framework: A complete training recipe comprising:
   • A cold-start phase to establish the iterative format.
   • Trajectory-level optimization with shared advantages to credit early decisions.
   • Efficiency-aware reward shaping to balance accuracy and inference cost.
3. Empirical Validation: Extensive experiments demonstrate that InftyThink+ outperforms both SFT-based iterative reasoning and standard long-context RL baselines.

4. Experimental Results

Experiments were conducted on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B-Base.

• Accuracy Improvements:
  • On AIME24, InftyThink+ improved accuracy by 21% compared to the baseline, and outperformed conventional long-CoT RL by a clear margin (an additional 9% gain over standard RL).
  • On out-of-distribution benchmarks like GPQA Diamond, it achieved a 5% accuracy improvement over the baseline and a 4% gain over vanilla approaches.
• Efficiency Gains:
  • Inference Latency: On AIME25, InftyThink+ reduced reasoning latency by 32.8% compared to the vanilla approach.
  • Training Speed: The framework yielded a 18.2% speedup in RL training time due to reduced computational overhead per step.
• Generalization: The method generalized well to larger models (Qwen3-4B) and out-of-distribution tasks, including code and scientific reasoning.

5. Significance and Claims

The paper claims that InftyThink+ demonstrates that trajectory-level optimization is essential for effective iterative reasoning. The key insight is that the effectiveness of iterative reasoning stems not from the format itself, but from the model's ability to make optimal strategic decisions (when to compress, what to preserve, how to continue) at each step.

By separating format learning (via SFT) from strategy optimization (via RL), InftyThink+ enables models to learn adaptive behaviors that:

• Decouple reasoning depth from context length: Allowing indefinite reasoning while maintaining bounded computational cost.
• Optimize the accuracy-efficiency trade-off: Through efficiency-aware rewards, the model learns to terminate reasoning earlier without sacrificing accuracy.
• Improve training stability and speed: By reducing the quadratic complexity of attention during training rollouts.

The authors position InftyThink+ as a significant step toward scalable, efficient, and effective long-horizon reasoning in large language models, moving beyond the limitations of static context windows and heuristic-based compression.