Technical Summary: Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Training

1. Problem Statement

Large Language Models (LLMs) have demonstrated the ability to solve complex problems through long Chain-of-Thought (CoT) reasoning. However, the trade-off between reasoning performance and token cost remains a central challenge for deploying LLM agents. To mitigate token costs, practitioners often employ Supervised Fine-Tuning (SFT) on "compressed" reasoning data, where CoT traces are shortened by aggregating or omitting intermediate steps.

Despite the prevalence of this practice, the paper identifies a critical gap in understanding:

1. Mechanism of Compression: It is unclear how different structural forms of CoT compression (e.g., aggregating operations vs. omitting them) impact learning dynamics.
2. Data Requirements: The relationship between compression granularity, data volume, and training steps required for SFT is poorly understood.
3. Decomposition Capability: It remains unknown whether models trained on compressed data can decompose these aggregated steps to solve tasks requiring finer-grained reasoning, and whether subsequent Reinforcement Learning with Verifiable Rewards (RLVR) can facilitate this decomposition.

2. Methodology

2.1 Taxonomy of CoT Compression

The authors propose a formal taxonomy to categorize CoT reasoning traces based on how atomic operations ($f_i$) and values ($s_i$) are presented:

• Explicit CoT: Outputs every operation and intermediate value sequentially ($s_1 = f_1(s_0), s_2 = f_2(s_1), \dots$). This corresponds to a compression granularity ($g$) of 1.
• Composed CoT: Aggregates multiple operations into a single step, explicitly listing the operations but omitting intermediate values (e.g., $s_2 = f_2(f_1(s_0))$). Here, $g > 1$.
• Implicit CoT: Omits both intermediate values and the explicit listing of intermediate operations, outputting only the final result of a chunk (e.g., $s_4 = f_4(s_3)$). This hides both operations and values, corresponding to a higher $g$.

2.2 Synthetic Task Construction

To control variables such as difficulty, compression granularity, and data size, the authors constructed a synthetic arithmetic reasoning task.

• Task Structure: Problems involve sequential dependencies between parameters (e.g., "The number of A equals B times 4").
• Operations: Dependencies involve addition, subtraction, and multiplication modulo 23 to prevent numerical explosion.
• Difficulty ($op$): Defined as the total number of atomic operations required to solve the task.
• Granularity ($g$): The number of operations grouped into a single CoT step.
• Evaluation: Models are trained on tasks with specific $op$ values (e.g., 8, 16, 24) and evaluated on Out-of-Distribution (OOD) tasks with longer sequences (e.g., 32 to 104 operations) to test generalization.

2.3 Experimental Setup

• Models: Experiments were conducted across various model families and sizes, including Qwen2.5 (0.5B to 14B) and Llama-3 (1B to 8B).
• Training Regimes:
  • SFT: Trained on Explicit, Composed, and Implicit CoT data with varying $g$ (2, 4, 8).
  • Data Scaling vs. Repetition: Compared training on large diverse datasets (384k samples, 1 epoch) versus small datasets repeated many times (6k samples, 64 epochs).
  • RLVR: Applied Group Relative Policy Optimization (GRPO) on models pre-trained with SFT to test if RL can decompose compressed steps.
• CoT Ordering: Investigated Forward (standard), Backward (reverse), and Hierarchical (chunked sub-problems) CoT orders.

3. Key Contributions and Results

3.1 Compression Granularity and Training Steps

• Finding: Coarser-grained Compressed CoT (higher $g$) requires significantly more training steps and data volume to achieve the same performance level as Explicit CoT.
• Implication: To train models on highly compressed reasoning traces, practitioners must prepare larger datasets to compensate for the increased complexity of the learning signal.

3.2 Data Scaling vs. Data Repetition

• Scaling: Compressed CoT (both Composed and Implicit) benefits more from data scaling (increasing the number of unique samples) than Explicit CoT does.
• Repetition:
  • Composed CoT: Benefits from data repetition (training on fewer samples for more epochs).
  • Implicit CoT: Suffers from performance degradation when data is repeated. The authors suggest this is due to overfitting or memorization of specific lengths, as Implicit CoT hides the operational structure required for generalization.
• Takeaway: When data is limited, Composed CoT is preferable for repetition-based training, whereas Implicit CoT should be avoided in favor of diverse data scaling.

3.3 Decomposition of Reasoning Chains

• SFT Limitation: Models trained on compressed CoT (e.g., $g=2$) fail to solve tasks requiring decomposition into atomic steps (e.g., tasks with odd $op$ values when trained on even $op$). They cannot spontaneously decompose the aggregated chunks observed during SFT.
• RLVR Capability: Subsequent RLVR successfully enables decomposition. When applied to tasks requiring decomposition (odd $op$), RLVR allows models to break down the $g=2$ chunks into $g=1$ atomic steps.
• Dynamics: During RLVR, response length initially increases (indicating the model is exploring decomposed, explicit steps) before converging to an efficient strategy that uses compressed steps for the bulk of the computation and explicit steps only where necessary.
• Takeaway: SFT learns to mimic compressed patterns, but RLVR is necessary to discover the underlying atomic skills and recombine them for unseen compositions.

3.4 CoT Ordering and Generalization

• Unidirectional vs. Hierarchical: Forward and Backward CoT orders generalize well to longer sequential tasks (OOD). In contrast, Hierarchical CoT (which splits problems into chunks and stores intermediate variables) fails to generalize to longer tasks.
• Data Efficiency: Backward CoT requires more data than Forward CoT to achieve comparable generalization, likely due to misalignment with the data distribution used in pre-training.
• Takeaway: For sequential compositional tasks, unidirectional CoT designs are more robust and data-efficient than hierarchical chunking.

4. Significance and Claims

The paper claims to provide a foundational understanding of the trade-offs involved in compressing reasoning data for LLM post-training. Its primary contributions are:

1. Guidelines for Data Design: It offers practical rules for constructing compressed datasets under resource constraints. Specifically, it advises that coarser compression requires more data, Composed CoT benefits from repetition while Implicit CoT does not, and unidirectional ordering is superior for generalization.
2. Mechanistic Insight into SFT vs. RL: The work elucidates the distinct roles of SFT and RL in post-training. It posits that SFT is effective for learning compressed representations but is limited in decomposing them, whereas RLVR acts as a mechanism to "unzip" these compressed chunks, enabling the model to generalize to novel compositions by recovering atomic operations.
3. Optimization of Token Costs: By identifying when compressed data is effective and how to pair it with RL, the paper aims to help navigate the balance between reasoning length (token cost) and performance, supporting the development of more efficient LLM agents.

The authors maintain modesty regarding their claims, noting that their findings are derived from synthetic arithmetic tasks and may not directly translate to all real-world domains or non-Transformer architectures without further investigation. They emphasize that while their results support the view that RL can discover new solutions through skill composition, the specific mechanisms in complex, real-world reasoning remain an area for future study.