🤖 machine learning

HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression

HMPO is a cost-effective, single-stage reinforcement learning framework that efficiently compresses chain-of-thought reasoning by 19%–46% with negligible accuracy loss across diverse tasks and model scales, overcoming the limitations of manual tuning and multi-stage training in existing methods.

Original authors: Minghui Zheng, Hongxu Chen, Huimin Ren, Hongsheng Xin, Xiaoyang Qu, Ze Wang, Shuling Yang, Ziyu Peng, Kaike Zhang, Pan Zhou, Kun Zhan

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghui Zheng, Hongxu Chen, Huimin Ren, Hongsheng Xin, Xiaoyang Qu, Ze Wang, Shuling Yang, Ziyu Peng, Kaike Zhang, Pan Zhou, Kun Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but overly chatty student. When you ask them a math problem, they don't just give you the answer; they write a 50-page novel about every single thought that crossed their mind, including dead ends, "what ifs," and long-winded explanations. While this "thinking process" (called Chain-of-Thought) helps them get the right answer, it's incredibly slow, expensive to run, and wastes a lot of energy.

The paper introduces a new method called HMPO (Hybrid Median-length Policy Optimization) to teach this student how to be concise without losing their smarts.

Here is how HMPO works, using simple analogies:

1. The Problem: The "Over-Thinking" Student

Current AI models are great at reasoning, but they "overthink." They generate thousands of words (tokens) for a single question. This is like asking someone for the time and getting a lecture on the history of clocks. It costs a lot of money (in computing power) and takes a long time.

Existing methods to fix this are clunky:

They use rigid rules (e.g., "Stop talking after 1,000 words"), which might cut off a necessary explanation for a hard problem.
They require expensive, multi-step training (like hiring a tutor, then a coach, then a manager), which takes forever and costs a fortune.
They often break when applied to very large, complex models.

2. The Solution: HMPO (The Smart Coach)

HMPO is a single, efficient training session that teaches the AI to be brief but accurate. It uses three clever tricks:

A. The "Goldilocks" Budget (Adaptive Median)

Instead of setting a fixed word limit (like "max 500 words"), HMPO looks at the student's recent successful answers.

The Analogy: Imagine a coach watching a group of runners. Instead of saying "Run exactly 10 minutes," the coach looks at the median (the middle) time of the runners who actually finished the race successfully.
How it helps: If the problems are hard, the "successful" answers are naturally longer, so the budget relaxes. If the problems are easy, the successful answers are short, so the budget tightens. The AI learns to aim for the "just right" length automatically, without a human needing to guess the number.

B. The "Smooth Landing" Reward (Cosine Decay)

Usually, if you tell an AI "be shorter," it might cheat by giving a short, wrong answer just to get the reward. HMPO prevents this.

The Analogy: Imagine a video game where you get points for finishing a level quickly. But if you finish the level wrong, you get zero points, no matter how fast you were.
How it helps: HMPO uses a special math formula (multiplicative reward) that says: "Correctness is the only thing that matters first. If you are wrong, your length doesn't matter; you get zero." If you are correct, then you get extra points for being concise. This stops the AI from cheating by being lazy or wrong.

C. The "One-Stop Shop" (Single-Stage Training)

Old methods required a long, complicated process: first teach the AI to be short (Step 1), then teach it to be right (Step 2), then mix them.

The Analogy: Instead of building a house brick by brick over three years, HMPO is like a 3D printer that builds the whole house in one go.
How it helps: It cuts the training time and cost by 1.5 to 2.5 times compared to older methods.

3. The Results: Smarter, Faster, Cheaper

The researchers tested this on AI models ranging from small (9 billion parameters) to massive (122 billion parameters).

The Magic: They trained the AI only on math problems.
The Surprise: Even though it only learned to be concise on math, it became concise on coding, science, and following instructions too. It's like teaching a student to write a short essay on math, and suddenly they start writing short, clear emails and code without being told to.
The Numbers: The AI reduced its "thinking" length by 19% to 46% (cutting the word count significantly) while keeping its accuracy almost exactly the same.
The Comparison: A compressed 122-billion-parameter model trained with HMPO performed as well as much larger, unoptimized models, but it did it with far fewer words and less computing power.

Summary

HMPO is a new way to train AI to stop "overthinking." It uses a smart, self-adjusting system to find the perfect balance between being brief and being right. It's cheaper to train, works on huge models, and surprisingly, teaches the AI to be concise in many different subjects just by practicing on math.

What the paper does NOT claim:

It does not claim this works for multi-turn conversations (like a long chat where the AI remembers previous turns).
It does not claim this is ready for medical diagnosis or legal advice.
It does not claim this works for AI agents that use tools (like browsing the web or using a calculator) in complex loops.

Technical Summary: HMPO – Hybrid Median-length Policy Optimization for Chain-of-Thought Compression

1. Problem Statement

Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through extended Chain-of-Thought (CoT) processes, as seen in models like OpenAI o1, DeepSeek-R1, and Qwen-3. However, this "thinking" paradigm incurs prohibitive inference costs, characterized by high latency and substantial resource consumption due to the generation of thousands of tokens per query.

Existing CoT compression methods face three critical limitations:

Inflexible Constraints: Current approaches rely on manually tuned, static length budgets or rigid binary "think-or-skip" decisions, limiting their applicability to narrow domains (primarily mathematics) and failing to adapt to varying problem complexities.
Prohibitive Training Costs: Effective compression often requires computationally expensive multi-stage pipelines, including data distillation, Supervised Fine-Tuning (SFT) warm-ups, or progressive iterative pruning.
Fragile Scalability: Evaluations are largely restricted to small dense models (≤32B), leaving the applicability to state-of-the-art large-scale and Mixture-of-Experts (MoE) architectures unverified.

2. Methodology: HMPO

The authors propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework built upon Group Relative Policy Optimization (GRPO). HMPO addresses the limitations of prior work through three synergistic components:

A. Adaptive Median-Based Budget

Instead of using a static, manually defined token budget, HMPO dynamically derives a budget $b$ from the median length of successful (correct) rollouts within a training batch.

Mechanism: For a group of $G$ rollouts, the subset $C$ containing correct answers is identified. The budget is calculated as $b = \text{median}(\{n_i \mid i \in C\})$ , where $n_i$ is the token length.
Benefit: This approach isolates the length signal from noisy, failed attempts and creates a difficulty-aware constraint. Complex queries naturally yield longer correct rollouts (relaxing the budget), while simpler queries are subjected to tighter constraints. As the policy improves, the budget automatically decreases, forming a self-tightening curriculum without manual intervention.

B. Cosine-Decay Token Reward

To penalize verbosity without degrading accuracy, HMPO applies a length preference exclusively to successful trajectories.

Mechanism: A token reward $R_{token}$ is defined using a smooth cosine decay function for correct rollouts where the length $n < b$ :
$R_{token} = \min\left(1, \cos\left(\frac{\pi n}{2b}\right) + \lambda\right)$
Incorrect or empty responses receive $R_{token} = 0$ . The scalar $\lambda$ controls the magnitude of the length-induced gain.
Benefit: This provides a "soft landing" penalty, encouraging shorter correct responses while preserving necessary reasoning steps, avoiding the harsh penalties of binary thresholds.

C. Multiplicative Reward Formulation

To strictly prioritize answer correctness and prevent "reward hacking" (where the model generates short but incorrect answers to maximize length rewards), the final reward is formulated multiplicatively:
$R_{final} = R_{acc} \cdot R_{token}$

Mechanism: $R_{acc}$ is 1 for correct answers and 0 for incorrect ones.
Benefit: This enforces a "correctness-first, length-second" hierarchy. Mathematically, any incorrect response (where $R_{acc}=0$ ) results in a final reward of 0, regardless of its length. This eliminates efficiency gradients for incorrect responses, inoculating the training against reward hacking.

3. Key Contributions

Methodological Innovation: HMPO introduces adaptive correct-median budgeting, cosine reward shaping, and multiplicative reward factorization. These components jointly resolve the rigidity of manual budgets, the aggressiveness of static penalties, and the vulnerability to reward hacking found in prior RL-based compression methods.
Cross-Domain Generalization: The paper presents an extensive evaluation across 9B to 122B dense and MoE models. Crucially, HMPO is trained exclusively on mathematical data yet demonstrates robust generalization to code, science, and instruction-following tasks without task-specific tuning.
Unprecedented Scalability: HMPO achieves the first successful CoT compression on massive MoE architectures (up to 122B parameters). It maintains near-perfect accuracy parity while significantly reducing token usage, enabling these models to compete with frontier models of much larger activated scales (e.g., 700B+ MoEs) using far fewer tokens.

4. Experimental Results

Experiments were conducted on Qwen3.5 series models (9B, 35B-A3B, and 122B-A10B) trained on 6.5K instances from DeepMath-103K.

Compression vs. Accuracy: HMPO achieves 19%–46% token compression across different model scales with negligible accuracy degradation.
- On Qwen3.5-9B, it reduces output tokens by 46% with only a 0.02 percentage point (pp) drop in average accuracy.
- On Qwen3.5-122B, it achieves 19% compression with a 0.10 pp accuracy drop.
Comparison with Baselines:
- ThinkPrune: Achieves higher compression (64–72%) but suffers severe accuracy degradation (11–22 pp drops).
- Thinkless: Exhibits catastrophic failure on smaller models (<15% accuracy) due to SFT-induced mode collapse.
- AdaptThink: Provides marginal gains but fails to compress effectively on larger models, sometimes inflating token lengths.
Training Efficiency: By eliminating distillation and SFT phases, HMPO reduces training GPU hours by 1.5×–2.5× compared to multi-stage baselines.
Token Efficiency: On AIME 2026, HMPO-trained 122B models slash token consumption by 23% while maintaining competitive accuracy against significantly larger frontier models (e.g., Nemotron-3-Super-120B and Kimi-K2.5-1T).

5. Significance and Claims

The paper claims that HMPO represents a paradigm shift in efficient reasoning by offering a single-stage, drop-in optimization strategy. Its significance lies in:

Eliminating Manual Tuning: The adaptive median budget removes the need for tedious, domain-specific length budget tuning.
Mathematical Robustness: The multiplicative reward formulation mathematically prevents the model from learning to be concise at the expense of correctness.
Practical Deployment: By achieving high compression with negligible accuracy loss across diverse architectures and domains (despite math-only training), HMPO enables the efficient real-world deployment of advanced reasoning LLMs, drastically reducing inference costs without requiring complex, multi-stage training pipelines.

The authors acknowledge limitations regarding multi-turn reasoning and agentic tasks, noting that the current framework is designed for single-turn settings and that extending it to interactive or tool-based scenarios remains a direction for future work.