💬 NLP

Short Chains, Deep Thoughts: Balancing Reasoning Efficiency and Intra-Segment Capability via Split-Merge Optimization

The paper introduces CoSMo, a framework that enhances Large Reasoning Models by using a consistency-guided split-merge algorithm and structure-aligned reinforcement learning to eliminate structural redundancy, thereby significantly improving accuracy while reducing computational overhead.

Original authors: Runquan Gui, Jie Wang, Zhihai Wang, Chi Ma, Jianye Hao, Feng Wu

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Runquan Gui, Jie Wang, Zhihai Wang, Chi Ma, Jianye Hao, Feng Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like a "Where's Waldo?" but with facts instead of a striped shirt. You have a team of detectives (the AI) trying to find the answer.

In the past, these detectives were told to "think step-by-step." They did, but they often overthought things. They would write down every single thought, even the ones that were just repeating what they already knew or checking things they didn't need to check. This resulted in a massive, messy notebook full of redundant notes. It took a long time to read, wasted a lot of paper (computing power), and sometimes, the sheer volume of notes actually confused the detective, making them miss the obvious answer.

This paper introduces a new method called CoSMo (Consistency-Guided Split-Merge Optimization) to fix this. Think of CoSMo as a smart editor that teaches the detective how to write a concise, perfect story without losing any important plot points.

Here is how CoSMo works, using simple analogies:

1. The Problem: Too Much Chatter

The paper argues that current AI models are like people who talk too much. They don't just say "The answer is X"; they say, "I am thinking about X, and I am also thinking about X again, and maybe I should check if X is true, and oh look, X is true again."

The Issue: This "chatter" creates structural redundancy. It's not that the words are too long, but that the number of distinct steps (segments) is too high.
The Analogy: Imagine trying to walk from your house to the store. A normal person walks straight there. An over-thinking AI might walk to the park, check the time, walk back, walk to the library, check the time again, and then walk to the store. The distance (token count) might be similar, but the number of stops (segments) is huge and inefficient.

2. The Solution: The "Split-Merge" Algorithm

CoSMo treats the AI's reasoning chain like a sentence that needs editing. It uses a two-step process to clean it up:

The Merge (Cutting the Fluff): If the AI writes two steps that say the same thing or are just minor variations of each other, CoSMo says, "Hey, these are the same thought!" It merges them into one strong, clear sentence.
- Analogy: Instead of saying "I am hungry. I feel a rumble in my stomach," the editor combines it into "I am hungry."
The Split (Filling the Gaps): Sometimes, the AI tries to be too efficient and skips steps, jumping from "A" to "C" without explaining "B." CoSMo spots this "logical leap" and says, "Wait, you skipped a step!" It splits that big jump into two smaller, logical steps.
- Analogy: If the AI says, "I saw a dog, so I bought a leash," CoSMo splits it: "1. I saw a dog. 2. I need a leash for the dog."

3. The Training: Learning to Be "Just Right"

The paper describes a two-phase training process to teach the AI this new way of thinking:

Phase 1: The Editor (Supervised Fine-Tuning): The researchers take the AI's messy, over-long answers and use the Split-Merge algorithm to rewrite them into perfect, concise versions. They then teach the AI to mimic these perfect versions. It's like a student studying a model essay to learn how to write clearly.
Phase 2: The Coach (Reinforcement Learning): Now, the AI tries to solve problems on its own. The "Coach" (the reward system) doesn't just count how many words the AI uses. Instead, it counts how many distinct steps the AI takes.
- The Twist: The Coach says, "You can write as many words as you need inside a step to be very detailed and accurate. But, you must not take too many steps."
- Why this matters: Previous methods tried to limit the total number of words. CoSMo realizes that sometimes you need a long sentence to explain a complex idea. So, it punishes the AI for taking too many stops (segments), not for writing long sentences.

4. The Results: Shorter Chains, Deeper Thoughts

The paper tested this on various difficult questions (like multi-step trivia or reading comprehension).

The Outcome: CoSMo produced answers that were more accurate and used fewer steps than other methods.
The Metaphor: Imagine a race. Other runners were running in circles (redundant steps) or taking giant, clumsy leaps (skipping logic). CoSMo's runner took the most direct path, stopping exactly the right number of times to check their map, but never stopping to tie their shoe twice.
The Stats: The paper claims CoSMo improved accuracy by about 3.3 points while reducing the number of reasoning steps by nearly 29%.

Summary

In short, this paper says: Don't just make AI talk less; make it think more efficiently.

By teaching the AI to merge its repetitive thoughts and split its skipped steps, CoSMo creates a "Goldilocks" reasoning style: not too short (which misses details), not too long (which wastes time), but just right for the complexity of the problem. It allows the AI to be deep and detailed where it needs to be, while cutting out the structural clutter that slows it down.

1. Problem Statement

Large Reasoning Models (LRMs) have demonstrated impressive capabilities in complex tasks (e.g., multi-hop QA, math, code) by generating long chains of thought (CoT). However, this reliance on verbose generation introduces two critical issues:

Computational Overhead: Long reasoning chains result in significant latency and token consumption.
Structural Redundancy: Simply shortening chains indiscriminately (e.g., via token limits or pruning) often disrupts logical coherence. Conversely, excessive length introduces redundant steps that can confuse the model, particularly in knowledge-intensive multi-hop Question Answering (QA).
The Core Challenge: Existing methods fail to distinguish between structural redundancy (unnecessary reasoning steps/segments) and intra-segment depth (the necessary token volume required to explain a single logical step). Current approaches often penalize total token length, inadvertently suppressing the deep reasoning capabilities needed for complex problems.

2. Methodology: CoSMo Framework

The authors propose CoSMo (Consistency-Guided Split-Merge Optimization), a framework designed to optimize the topology of the reasoning chain rather than just its length. It operates in two stages:

A. Split-Merge Optimization Algorithm (Data Curation & SFT)

Inspired by image segmentation algorithms, CoSMo treats the reasoning chain as a sequence of discrete segments ( $s_1, s_2, ..., s_N$ ). It iteratively refines the chain to align the number of segments ( $N$ ) with the ground-truth hop count ( $k^*$ ) of the problem.

Merge Phase: If $N > k^*$ (redundancy), the algorithm uses an LLM-based Consistency Judge to identify adjacent segments that are semantically redundant (e.g., paraphrases or trivial extensions). A Semantic Generator then fuses these into a single, concise logical unit.
Split Phase: If $N < k^*$ (logical leaps), the algorithm detects coarse-grained logic where multiple steps are compressed. The Generator decomposes these into finer-grained, coherent sub-segments to ensure logical continuity.
Outcome: This process creates a high-quality, logic-dense dataset ( $D_{cosmo}$ ) used for Supervised Fine-Tuning (SFT), instilling a structural prior in the model.

B. Structure-Aligned Reinforcement Learning (RL)

Following SFT, the model is further optimized using Group Relative Policy Optimization (GRPO).

Novel Reward Mechanism: Instead of penalizing total token count, CoSMo introduces a Segment-Level Budget.
- Format Reward: Ensures strict adherence to the numbered list structure.
- Correctness Reward: Rewards exact answer matches.
- Structural Efficiency Reward: Penalizes the deviation of the segment count from the ground truth ( $|N - k^*|$ ).
Key Innovation: This design allows the model to expand the token length within a segment (for deep reasoning) without penalty, as long as the number of segments remains optimal. This decouples structural efficiency from descriptive depth.

3. Key Contributions

Decoupling Structure from Depth: The paper formally defines reasoning efficiency not by token volume, but by topological alignment with problem complexity. It separates structural redundancy (segment count) from intra-segment reasoning (token granularity).
Consistency-Guided Split-Merge: A novel algorithm that dynamically curates training data by merging redundant steps and splitting logical gaps, ensuring both conciseness and logical continuity.
Segment-Level Budgeting in RL: A reinforcement learning strategy that replaces global token constraints with a segment-count penalty, preventing the suppression of necessary deep reasoning details.
Empirical Validation: Comprehensive experiments demonstrating that optimizing structure yields better performance than optimizing length alone.

4. Experimental Results

The authors evaluated CoSMo on multiple benchmarks (HotpotQA, HaluEval, NQ, CRAG) using Llama-3.1-8B and Qwen-2.5 models.

Performance Gains: CoSMo achieved an average accuracy improvement of +3.3 points across benchmarks compared to reasoning efficiency baselines.
Efficiency:
- Reduced average token consumption by 29.5%.
- Reduced average segment count by 28.7% compared to efficient reasoning baselines.
- On HotpotQA (mostly 2-hop questions), CoSMo reduced structural redundancy (segments exceeding ground truth) from ~1.6 (CoT) to 0.6.
Robustness:
- CoSMo maintained high accuracy as task difficulty increased (from 2-hop to 4-hop questions), whereas baselines like CoT and C3oT suffered significant accuracy drops.
- The model showed consistent superiority across different LLM judges (Qwen-7B/14B/72B, Llama-3.1), proving the gains are due to intrinsic logical validity rather than judge bias.
Ablation Studies:
- The SFT phase (Split-Merge) alone provided significant gains in accuracy and token reduction.
- The RL phase further enhanced intrinsic capabilities.
- The combination of both stages yielded the best results, confirming that structural priors and RL alignment are complementary.

5. Significance and Impact

Computational Sustainability: By eliminating structural redundancy without sacrificing reasoning depth, CoSMo offers a path toward "Green AI," reducing the carbon footprint and inference costs of LRMs.
Paradigm Shift: The work challenges the prevailing "token budget" approach, suggesting that the structure of reasoning is a more critical optimization target than the volume of tokens.
Interpretability: The resulting segmented reasoning structure is more transparent, allowing for better analysis of how models allocate resources during inference.
Generalizability: The method is model-agnostic and effective across different backbone architectures and difficulty levels, making it a promising approach for deploying efficient reasoning models on resource-constrained devices.