Technical Summary: CLORE – Content-Level Optimization for Reasoning Efficiency

Problem Statement

Recent advancements in Large Language Models (LLMs), particularly those utilizing Reinforcement Learning (RL) post-training (e.g., OpenAI o1, DeepSeek-R1), have demonstrated remarkable capabilities in multi-step reasoning tasks like mathematical problem solving. These systems incentivize extended Chain-of-Thought (CoT) traces. However, the paper identifies that excessively long reasoning trajectories often suffer from "overthinking," characterized by three specific patterns of low-quality content:

1. Repetitive reasoning segments: Redundant revisits to equivalent intermediate states without new progress.
2. Illegible or task-irrelevant content: Semantically incoherent, corrupted, or disconnected text that reduces interpretability.
3. Superfluous exploration: Continued reasoning after the correct solution has already been determined.

Existing efficient reasoning methods primarily address this issue through length-level control, such as enforcing explicit token budgets or designing length-aware rewards. The authors argue that these approaches rely on coarse, sequence-level signals that regulate how much reasoning is generated but fail to supervise what content is generated. Consequently, they leave intermediate reasoning content weakly supervised, allowing low-utility, illegible, or non-contributory segments to persist and potentially destabilize RL training dynamics.

Methodology: CLORE Framework

The paper proposes CLORE (Content-Level Optimization for Reasoning Efficiency), a framework that optimizes reasoning efficiency by editing the content of correct reasoning trajectories rather than merely constraining their length.

Core Workflow

1. Rollout Generation: The policy model $\pi_\theta$ samples multiple reasoning trajectories $\tau$ for a given prompt $x$.
2. Selection of Correct Trajectories: The framework restricts augmentation to trajectories where the final answer is correct ($R(x, \tau) = 1$). This ensures that the base data contains less low-quality reasoning than incorrect trajectories and keeps the augmented data closer to the policy distribution.
3. Reasoning Augmentation: An external augmentation model (an LLM acting as an editor) processes the correct trajectories. It is instructed to:
   • Delete repetitive reasoning spans.
   • Remove illegible or task-irrelevant content.
   • Cut superfluous exploration occurring after the solution is established.
   • Constraint: The editor must not rewrite, paraphrase, or reorder content; it performs deletion-only operations to preserve the original structure and solution validity.
4. Preference Pair Construction: The result is a pair of trajectories $(\tau, \tilde{\tau})$, where $\tilde{\tau}$ is the edited, more concise version. A preference is established where $\tilde{\tau} \succ \tau$ based on reasoning quality, not answer correctness (since both are correct).
5. Training Objective: The policy is trained using a hybrid objective combining:
   • Standard Policy Gradient (PG): Optimizing the base trajectories (e.g., GRPO, DAPO).
   • Reference-Free DPO: An auxiliary Direct Preference Optimization (DPO) objective applied to the $(\tau, \tilde{\tau})$ pairs. The loss is formulated as:
     $$L_{DPO} = -\log \sigma(\beta [\log \pi_\theta(\tilde{\tau} | x) - \log \pi_\theta(\tau | x)])$$
     Unlike standard DPO, this formulation does not require an explicit reference policy. The authors argue that because $\tilde{\tau}$ is derived via local deletion from $\tau$, it remains structurally close to the on-policy distribution, implicitly constraining deviation.

Compatibility

CLORE is designed to be agnostic to the underlying RL backbone. It can be integrated with various policy-gradient methods (PPO, GRPO, REINFORCE) and existing length-based efficient reasoning methods (e.g., ThinkPrune, Training Efficient) by adding the DPO term to the existing loss function.

Key Contributions

1. Content-Level Optimization Framework: CLORE shifts the focus from sequence-level length control to content-level supervision. It explicitly targets and removes low-quality reasoning components (repetition, illegibility, post-answer noise) while preserving the final answer.
2. Augmented-Original Preference Learning: The method introduces a mechanism to train policies using augmented–original pairs via a reference-free DPO objective, mitigating off-policy mismatch issues typically associated with post-hoc editing.
3. Empirical Validation: The framework is evaluated on DeepSeek-R1-Distill-Qwen-7B and Qwen2.5-Math-7B across five mathematical reasoning benchmarks (OlympiadBench, Minerva, MATH500, AMC2023, AIME2025).
4. Complementarity Demonstration: The paper demonstrates that CLORE is not a replacement for length-level control but a complementary layer that improves the accuracy–efficiency trade-off when combined with existing methods.

Experimental Results

• Accuracy-Efficiency Trade-off: CLORE consistently improves the Accuracy-Efficiency (AE) Score across all tested baselines (GRPO, DAPO, Training Efficient, ThinkPrune).
  • On DeepSeek-R1-Distill-Qwen-7B, applying CLORE improved average AE scores by approximately 0.4 across benchmarks.
  • On Qwen2.5-Math-7B, improvements were more pronounced, with average AE gains often exceeding 1.0, and a specific gain of over 6 points on Minerva when applied to GRPO.
• Length Reduction: CLORE significantly reduces output token counts (20–30% on DeepSeek, 30–50% on Qwen) without causing systematic accuracy collapse.
• Content Analysis:
  • Repetition: CLORE reduces both local n-gram repetition and longest repeated spans.
  • Illegibility: It drastically lowers the rate of illegible or task-irrelevant content (measured via LLM-as-a-judge).
  • Post-Answer Exploration: It effectively suppresses reasoning that continues after the solution is found.
• Ablation Studies:
  • DPO Weight: A moderate DPO weight ($\lambda$) provides the best balance between length reduction and accuracy retention.
  • Augmentation Model: The method remains effective even with a smaller augmentation model (Qwen3-1.7B), though a stronger model (Qwen3-4B) yields slightly better results.

Significance and Claims

The paper claims that efficient reasoning requires dual optimization: controlling the quantity of reasoning (length) and supervising the quality of reasoning (content).

• Beyond Length Control: The authors argue that simply minimizing length or enforcing budgets is insufficient because it does not distinguish between high-value reasoning and redundant noise. CLORE addresses the "weak supervision" of intermediate content inherent in standard RL training.
• Stable Optimization: By restricting augmentation to correct trajectories and using local deletion, CLORE keeps edited rollouts close to the policy distribution, mitigating the off-policy mismatch that often plagues post-hoc editing methods.
• General Applicability: The framework is presented as a general enhancement to existing RL-based reasoning training pipelines, capable of improving both training efficiency (by reducing noisy gradients from low-quality tokens) and inference efficiency (by generating more concise, dense reasoning traces).

The authors conclude that while length-level control regulates how much reasoning is generated, content-level supervision (as provided by CLORE) is necessary to determine which reasoning content is informative and worth retaining.