Technical Summary: Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning

1. Problem Statement

Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex problem-solving by employing detailed Chain-of-Thought (CoT) sequences. However, these models often suffer from "over-thinking," generating prolonged reasoning paths cluttered with non-essential steps, repetitive verifications, and redundant meta-talk. This redundancy creates significant inference latency, hindering deployment efficiency.

Existing solutions primarily rely on Reinforcement Learning (RL) with outcome-based rewards coupled with a global length penalty. The paper argues that this approach suffers from misaligned credit assignment: by indiscriminately penalizing the total sequence length, these methods fail to distinguish between redundant steps and critical reasoning logic. Consequently, models may sacrifice essential reasoning steps to satisfy length constraints, leading to a trade-off where efficiency gains come at the cost of accuracy. The core challenge identified is how to perform fine-grained compression—eliminating redundancy while preserving high-value logical steps—without introducing inference-time overhead.

2. Methodology: SCMA Framework

The authors propose Self-Compression via Multi-Agent Reinforcement Learning (SCMA), a framework that shifts from single-agent RL to a Multi-Agent Reinforcement Learning (MARL) paradigm. The system utilizes a shared base Large Language Model (LLM) instantiated into three specialized agents that collaborate during training but allow for the exclusive deployment of a single agent during inference.

2.1. Agent Roles

The framework decomposes the reasoning process into a collaborative game involving three agents:

1. Reasoning Agent ($\pi^{reason}_{\theta}$): Explores the solution space to generate an initial reasoning path $y$ given a prompt $x$.
2. Segmentation Agent ($\pi^{seg}_{\theta}$): Structurally parses the generated reasoning path into discrete logical chunks $\{s_1, ..., s_n\}$. This agent ensures lossless parsing, where the union of chunks reconstructs the original path.
3. Scoring Agent ($\pi^{score}_{\theta}$): Evaluates the inferential significance of each chunk, assigning an importance score $w_i \in \{1, ..., 5\}$ that quantifies the chunk's essentiality to the final solution.

2.2. Importance-Weighted Length Penalty

Unlike standard RL which applies a uniform penalty to the total length, SCMA introduces an importance-weighted length penalty. The reward function is reformulated as:
$$R(y|x) = R_{acc}(y|x) - \alpha f\left(\sum_{i=1}^{n} \phi(w_i) \cdot |s_i|\right)$$
Where:

• $R_{acc}$ is the outcome-based accuracy reward.
• $|s_i|$ is the length of the $i$-th segment.
• $\phi(w_i)$ is a weighting function (e.g., $\phi(w_i) = 5 - w_i$) that inversely maps importance to penalty weight.
• Mechanism: Redundant segments (low scores, $w_i \to 0$) receive a high penalty weight, enforcing compression. Critical segments (high scores, $w_i \to 5$) receive negligible weight, exempting them from the penalty.

2.3. Optimization Strategy

The framework employs Multi-Agent Group Relative Policy Optimization (MARL-GRPO).

• Shared Parameters: All three agents share parameters $\theta$ from a single base LLM, distinguishing them only via role-specific prompts.
• Shared Reward: The agents are optimized jointly using a shared global reward derived from the importance-weighted penalty. This ensures the co-evolution of the Reasoning, Segmentation, and Scoring agents, aligning their objectives toward generating accurate yet concise reasoning chains.
• Training vs. Inference: The Segmentation and Scoring agents serve as training-time auxiliaries. During deployment, only the Reasoning Agent is active, ensuring zero additional computational overhead.

3. Key Contributions

1. Framework Proposal (SCMA): A novel MARL framework that reformulates CoT compression as a collaborative game, enabling fine-grained structural modeling and decoupled value estimation, which single-agent paradigms lack.
2. Mechanism Design: The introduction of an importance-weighted length penalty driven by a multi-agent GRPO objective. This replaces coarse-grained length penalties with a dynamic signal that preserves critical logic while pruning redundancy.
3. Empirical Verification: Comprehensive experiments demonstrating that SCMA achieves simultaneous improvements in efficiency and accuracy across multiple model scales and benchmarks.

4. Experimental Results

The authors evaluated SCMA on datasets ranging from arithmetic (GSM8K) to competition mathematics (MATH500, AMC23, AIME24/25) using models from 1.5B to 8B parameters (DeepSeek-R1-Distill-Qwen and Qwen3 series).

• Performance Gains: SCMA reduced response lengths by 11.1% to 39.0% while simultaneously boosting accuracy by 4.33% to 10.02% compared to vanilla baselines.
• Comparison with Baselines:
  • Compared to standard GRPO, SCMA achieved higher accuracy and significantly shorter responses.
  • Compared to length-penalized baselines (LC-R1_LP, RL+LP), SCMA avoided the "accuracy collapse" often seen when aggressively truncating sequences. For instance, while RL+LP reduced length by 16% on a 7B model, it dropped accuracy below the baseline; SCMA maintained or improved accuracy.
• Generalization: Despite being trained exclusively on GSM8K, SCMA demonstrated strong out-of-distribution generalization, significantly improving accuracy and efficiency on unseen challenging benchmarks like AIME24.
• Ablation Studies: Experiments freezing the Segmentation and Scoring agents (using off-the-shelf models) showed that joint parameter optimization is critical. The full SCMA framework outperformed variants where auxiliary agents were not co-trained, even when those variants used larger models (e.g., 8B or GPT-4o) for the auxiliary roles.

5. Significance and Claims

The paper claims that the synergistic optimization within the MARL framework fosters emergent behaviors that yield more powerful LRMs compared to vanilla RL paradigms. Specifically:

• Synergistic Co-Optimization: The framework enables the model to learn "Semantic Concentration," pruning low-confidence, repetitive fragments while converging on high-density logical kernels.
• Content-Adaptive Segmentation: The Segmentation Agent evolves from uniform segmentation to fine-grained, content-adaptive partitioning, dynamically allocating text length based on cognitive load rather than fixed rules.
• Stability: The dynamic penalty mechanism prevents the "training collapse" (where models stop thinking entirely to minimize length) observed in standard length-penalized RL, ensuring a stable equilibrium between brevity and logical integrity.

The authors conclude that SCMA addresses the bottleneck of redundant reasoning by providing a mechanism to distinguish and preserve high-value reasoning steps, offering a path toward efficient, high-performance reasoning models without inference-time overhead.