1. Problem Statement

Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have demonstrated exceptional performance on complex tasks (e.g., mathematical Olympiads, competitive programming) by employing "System-2" style reasoning—generating long chains of thought, self-verification, and exploring alternative strategies. However, this capability comes with significant drawbacks:

• Overthinking: LRMs frequently generate redundant reasoning steps, consuming excessive computational resources and inference time even after sufficient information has been gathered to solve a problem.
• Training Bias in RL: Existing Reinforcement Learning (RL) methods struggle to teach models to stop thinking efficiently. Because LRMs naturally favor verbose outputs, standard RL rollouts rarely produce short reasoning paths. This creates a distribution mismatch where the model is not exposed to concise, correct trajectories during training, leading to flawed reward signals that fail to encourage brevity.
• Distribution Mismatch: Simply truncating long answers or providing external short answers for training introduces a distribution shift, destabilizing gradient updates and impairing the model's ability to learn its own generative process.

2. Core Insight

The authors draw inspiration from Evidence Accumulation Models in cognitive science, which posit that human decision-making integrates information until a threshold is reached, after which further evidence is redundant.

• Hypothesis: LRMs accumulate the necessary information to determine the correct answer early in the reasoning process. Subsequent steps often constitute "overthinking" that adds noise or redundancy without improving accuracy.
• Pilot Experiments: Experiments on the MATH500 dataset with DeepSeek-Distill-Qwen-7B revealed that:
  • Truncating reasoning chains to the first 25–50% of tokens retained >75% of the original accuracy.
  • Truncation could actually improve accuracy for some initially incorrect answers by preventing the model from over-complicating the solution (Error Correction Rate increased).
  • This confirms that short, distributionally consistent reasoning paths exist within the model's natural generation.

3. Methodology: Just-Enough Thinking (JET)

JET is a reinforcement learning framework designed to train models to proactively terminate unnecessary reasoning. It consists of two main components:

A. Two-Stage Rollout Construction

To expose the model to diverse reasoning lengths while maintaining distributional consistency, JET employs a two-stage strategy during the RL rollout phase:

1. Stage 1 (Full Reasoning): The model generates a complete reasoning trajectory ($S$) and the final answer using standard autoregressive generation.
2. Stage 2 (Progressive Early-Stopping - PES):
   • Instead of searching for a single optimal truncation point (which is computationally expensive), JET uses Progressive Early-Stopping.
   • It generates a sequence of truncated variants along the full trajectory at intervals (e.g., $t_0, t_0+\Delta t, \dots$).
   • At each truncation point, an explicit stop-thinking cue (e.g., "Wait, I have enough information...") is inserted, forcing the model to immediately output a final answer based on the partial context.
   • Significance: This ensures that the training data (short paths) remains consistent with the model's natural generation distribution, avoiding the distribution mismatch issues of artificial truncation.

B. Quality-Controlled Length Reward

JET introduces a novel reward mechanism to balance accuracy and efficiency:

• Base Reward: Includes format rewards (e.g., \boxed{}) and correctness rewards (exact match with ground truth).
• Length Reward ($r_\ell$):
  • Condition: Applied only to correct responses. Incorrect answers receive zero length reward to prevent the model from learning to be concise but wrong.
  • Normalization: For a set of correct responses to a specific question, the reward is normalized based on the shortest ($\ell_{min}$) and longest ($\ell_{max}$) correct trajectories.
  • Formula: Shorter correct trajectories receive higher rewards, with the reward decaying linearly (or via other functions) as length increases.
  • Objective: This encourages the model to find the "just enough" reasoning depth required for correctness, rather than the shortest possible (which might be wrong) or the longest (which is redundant).

C. Training Objective

The method utilizes the DAPO (Direct Alignment Policy Optimization) algorithm. The loss function is computed over both full trajectories and the truncated-and-completed trajectories generated in Stage 2. This allows the policy to learn the optimal stopping point dynamically based on task difficulty.

4. Key Contributions

1. JET Framework: A novel RL method that trains LRMs to stop reasoning proactively, leveraging trajectory truncation and distribution-consistent data construction.
2. Empirical Validation of Early Accumulation: Demonstrated that LRMs accumulate sufficient problem-solving information early, and that truncating rollouts provides a principled basis for training efficient reasoning.
3. Theoretical Stability: Provided an analysis showing that appending a short forced suffix (stop-thinking cue) preserves the sequence-level probability distribution, ensuring stable RL updates within the trust region.
4. Progressive Early-Stopping (PES): A computationally efficient strategy to generate diverse reasoning lengths without exhaustive search, improving training efficiency.

5. Experimental Results

The authors evaluated JET on DeepSeek-Distill-Qwen-1.5B and 7B models across various benchmarks (MATH500, AIME24, AMC, Olympiad, GPQA, etc.).

• Efficiency vs. Accuracy Trade-off:
  • On the Olympiad benchmark with the 1.5B model, JET achieved a 4.6% increase in accuracy while reducing output length by 46.3%.
  • On AIME24 (7B model), JET improved accuracy by 2.7% while significantly reducing token usage compared to baselines.
• Comparison with Baselines:
  • vs. SFT/DPO: JET outperformed Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) on length reduction without sacrificing accuracy.
  • vs. Other RL Methods: Compared to methods like AdaptThink, Laser, and LCR1, JET consistently achieved better balance. For instance, LCR1 achieved high compression but suffered significant accuracy drops (e.g., -4.4% on MATH500), whereas JET maintained or improved accuracy.
• Generalization:
  • JET generalized well to out-of-domain tasks (CommonsenseQA, GPQA-Diamond, MMLU), showing that the "stop overthinking" capability is not limited to math.
  • It performed robustly across different model scales (1.5B, 7B, 8B Llama) and hardware platforms (NVIDIA GPUs and Huawei Ascend NPUs).
• Training Efficiency: The PES strategy reduced rollout generation time by up to 5x compared to generating full chains, accelerating the overall RL training process.

6. Significance

• Solving the "Overthinking" Problem: JET addresses a critical bottleneck in deploying LRMs: the high computational cost of inference. By teaching models to recognize when they have "enough" information, it makes advanced reasoning models more practical for real-world applications.
• Distributional Consistency: The paper highlights a crucial flaw in previous attempts to shorten reasoning (artificial truncation) and offers a solution (PES) that respects the model's native generation distribution, leading to more stable and effective learning.
• Adaptive Reasoning: Unlike fixed-token-budget methods, JET enables models to dynamically adjust reasoning depth based on problem complexity, optimizing resource usage without manual intervention.

In conclusion, JET represents a significant step forward in making Large Reasoning Models not only smarter but also more efficient, proving that "less thinking" (when done correctly) can lead to "better results."