Technical Summary: CIRF (Chain-of-thoughts Into Reusable Functional units)

1. Problem Statement

Large Language Models (LLMs) have demonstrated significant improvements in complex reasoning tasks through Chain-of-Thought (CoT) prompting, which generates explicit intermediate rationales. However, generating these full natural language traces incurs substantial inference latency and memory costs. While recent research has explored Implicit CoT methods to internalize reasoning into hidden states or special tokens, existing approaches face critical limitations:

• Lack of Alignment: Many methods use homogeneous placeholder tokens (e.g., repeated "pause" tokens) that fail to align with the semantic structure of explicit rationales.
• Adaptivity Issues: Current implicit methods often struggle to adapt reasoning length to the complexity of the input, leading to either underthinking or overthinking.
• Training Instability: Approaches that recursively feed hidden states as inputs (e.g., continuous latent spaces) introduce substantial training costs and instability, particularly when modeling multiple reasoning steps.
• Scalability: The difficulty in scaling the number of implicit tokens limits performance, as the optimal number of reasoning steps varies significantly across tasks.

2. Methodology: CIRF

The authors propose CIRF, a framework that tokenizes explicit CoT traces into a dynamic sequence of discrete functional tokens. The core idea is to treat each semantically coherent reasoning unit as a reusable functional operation, encoded as a single token.

2.1 Functional Token Assignment

The process begins by segmenting explicit CoT rationales into semantically coherent units (e.g., "Step 1," "Step 2").

• Semantic Embedding: Each segment is encoded using a pre-trained sentence embedding model.
• Bias Reduction: To prevent the embeddings from being biased toward the specific context of a question, the authors apply mean-centering to the segment representations within each question. This isolates the functional operation from situational bias.
• Vector Quantization (VQ-VAE): The centered embeddings are quantized into a discrete vocabulary of functional tokens using a VQ-VAE approach.
  • Codebook Initialization: A balanced clustering procedure based on the Sinkhorn-Knopp algorithm is used to initialize the codebook, ensuring an equal distribution of assignments across tokens.
  • Learning: The codebook and encoder/decoder networks are trained to minimize reconstruction error while maintaining discrete token assignments.
• Embedding Initialization: The new functional tokens are added to the LLM's vocabulary. Their initial embeddings are set by rescaling the learned codebook vectors, allowing the model to immediately exploit the geometric relationships between functional codes.

2.2 Autoregressive Generation and Alignment

The target LLM is fine-tuned to autoregressively generate a sequence consisting of:

1. Functional Tokens: Representing the reasoning steps.
2. Optional Result Tokens: Minimal textual bridges describing the output of the functional operation (e.g., "0.4" instead of a full sentence).
3. Final Answer: The solution to the problem.

The training objective is standard autoregressive language modeling. The model learns to predict the next functional token and its optional result, conditioned on the question and previous steps.

2.3 Result Compression

To further optimize the accuracy-latency trade-off, CIRF introduces a result compression mechanism. Since functional tokens alone are often sufficient for correct answers, the system iteratively prunes result tokens that do not significantly reduce the loss. This allows for a tunable trade-off: a higher pruning threshold yields faster inference with slightly lower accuracy, while a lower threshold retains more intermediate textual information.

3. Key Contributions

• Discrete Functional Tokenization: CIRF introduces a method to convert continuous reasoning traces into a discrete, reusable vocabulary of functional tokens, bridging the gap between explicit CoT and efficient latent reasoning.
• Adaptive Reasoning Length: Unlike methods with fixed reasoning depths, CIRF generates variable-length sequences of functional tokens, dynamically adjusting the reasoning depth based on the complexity of the input instance.
• Efficient Training: By utilizing discrete tokens and standard autoregressive training (with teacher forcing), CIRF avoids the instability and high computational cost associated with recursively feeding hidden states.
• Interpretability: The learned functional tokens correspond to distinct, interpretable reasoning operations (e.g., addition, subtraction, commonsense selection), rather than arbitrary identifiers.

4. Experimental Results

The authors evaluated CIRF on mathematical, symbolic, logical, and commonsense reasoning benchmarks (including GSM8K, MATH-500, BIG-Bench Hard, and CommonsenseQA) using Qwen3-8B and Llama3.1-8B backbones.

• Accuracy-Latency Trade-off: CIRF variants (CIRFFull, CIRFFast, CIRFFaster) consistently achieve a favorable position on the Pareto frontier compared to state-of-the-art baselines (including iCoT-SI, Coconut, CODI, and Pause Tokens).
  • CIRFFull achieves the highest accuracy, outperforming the best implicit baseline (CODI) by up to 45.7 percentage points on GSM8K.
  • CIRFFast and CIRFFaster significantly reduce inference time (up to 2x faster than CODI) while maintaining competitive accuracy.
• Generalizability: CIRF demonstrates robust performance on out-of-domain tasks (e.g., MATH-500, StrategyQA), where many baselines suffer from degenerate results or excessive latency.
• Adaptive Behavior: Analysis shows a positive correlation between the number of generated functional tokens and the difficulty of the problem, confirming that CIRF adapts its reasoning length to the task.
• Token Interpretability: Qualitative analysis reveals that specific functional tokens consistently map to specific operations (e.g., token ID 23 maps to multiplication/division), validating the semantic grounding of the learned vocabulary.

5. Significance and Claims

The paper claims that CIRF offers a promising direction for building language models that reason more efficiently. Its primary significance lies in:

• Preserving Functional Structure: By converting reasoning into reusable functional units, CIRF maintains the structural integrity of the reasoning process while eliminating the overhead of generating verbose natural language.
• Scalability and Stability: The framework enables single-stage training with teacher forcing, avoiding the complex curriculum learning or auxiliary models required by other implicit CoT methods.
• Efficiency: CIRF achieves an optimal balance between reasoning quality and inference cost, making it a viable alternative to explicit CoT for latency-sensitive applications.

The authors acknowledge limitations, noting that the quality of functional token supervision depends on the structure of the explicit CoT traces and that the current segmentation relies on ordinal markers. They suggest future work could explore more robust segmentation methods and the behavior of CIRF on larger models and long-context tasks.