Technical Summary: Structured Reasoning Signals Beyond Pure Code

Problem Statement

While code data is a standard component of modern foundation language model (LM) training, its specific role in enhancing general reasoning capabilities remains debated. Previous studies have suggested that incorporating code improves reasoning across logical, scientific, and mathematical domains. However, these findings often treat "code" as a monolithic category, potentially conflating executable programming logic with structured natural language traces (e.g., code-text mixtures, notebooks, or markup). This paper questions whether pure, standalone executable code acts as a general reasoning enhancer or if it competes with other capability dimensions under fixed training budgets. The core problem is to disentangle the marginal contributions of pure code versus cross-domain structured reasoning signals and to understand the competitive dynamics between data domains during pretraining.

Methodology

The authors conducted large-scale, tightly controlled pretraining experiments using a 10-trillion-token corpus and Mixture-of-Experts (MoE) models of varying sizes (1B, 5B, and up to 32 experts).

1. Corpus Construction and Domain Separation:
The study constructed a high-quality corpus spanning seven domains: Web, Code, Code-NL, Math, Wikipedia, Books, and Multilingual. Crucially, the authors enforced a strict distinction between:

• Code: Standalone executable functions, scripts, and program segments (excluding comments, explanations, and instructional text).
• Code-NL: Mixed data where code is interleaved with natural language (e.g., notebooks, Q&A pages, Markdown documents).
• Math: Formal mathematical expressions, proofs, and derivations.

2. Ablation Studies:
To isolate causal effects, the authors performed systematic ablations on a fixed 10T-token budget:

• w/o Code: Removed the pure Code corpus while retaining Code-NL.
• w/o Math: Removed the Math corpus.
• Cognitive Scaffolding: Within the Math domain, they identified a subset of "cognitive scaffolds"—highly structured samples containing explicit subgoals, symbolic manipulations, and hierarchical derivation steps—using a lightweight FastText classifier trained on code-like structural patterns. They replaced standard math data with these scaffolds while keeping the total math token budget constant.

3. Model Architecture:
Experiments utilized MoE models with a fine-grained expert strategy (increasing expert count while reducing intermediate dimensions) and a shared expert pathway. The routing mechanism included stochastic warmup to prevent expert collapse.

4. Analysis:
Performance was evaluated across five capability dimensions: General Knowledge, Programming Ability, Mathematical Ability, Comprehensive Reasoning, and Professional Knowledge. Additionally, the authors analyzed expert-routing patterns (using Jensen-Shannon divergence) to observe how data composition affects internal model dynamics.

Key Findings

1. Pure Code Competes with Knowledge-Intensive Tasks:
Contrary to the hypothesis that code broadly enhances reasoning, the study found that pure executable code competes with knowledge-intensive tasks, particularly complex mathematical reasoning.

• Removing code data improved performance on mathematical benchmarks by an average of 14.38%.
• The negative impact was most severe on complex benchmarks (e.g., Minerva-Math: -71.53%, OlympiadBench: -47.16%) but negligible or mixed on simpler tasks like GSM8K.
• Code data also showed competitive effects on comprehensive reasoning benchmarks (e.g., PIQA, HellaSwag).

2. Math Data Competes with Comprehensive Reasoning:
Conversely, math data competes with comprehensive reasoning tasks.

• Including math data significantly improved competitive programming performance (e.g., CodeForces: +37.11%) but reduced performance on code reasoning tasks (e.g., CruxEval: -17.30%) and commonsense reasoning (e.g., HellaSwag: -2.94%).
• This suggests a trade-off where math data provides algorithmic signals beneficial for coding but may interfere with the holistic reasoning patterns required for other tasks.

3. The Role of Structured Reasoning (Cognitive Scaffolds):
The discrepancy between these findings and prior work is attributed to the inclusion of Code-NL and mixed-format data in previous studies. The authors argue that the reasoning gains previously attributed to "code" are actually driven by cross-domain structured reasoning traces (Code-NL and structured Math).

• By increasing the density of cognitive scaffolds (structured math samples) within a fixed math budget, the model achieved substantial gains on difficult mathematical reasoning (e.g., College Math: +30.05%, MATH: +23.17%) with only a negligible drop (~1%) in programming performance.
• This indicates that structured reasoning signals serve as a targeted cognitive scaffold that mitigates cross-domain trade-offs.

4. Mechanism-Level Evidence:
Routing analysis revealed that data composition directly influences expert activation patterns.

• Ablating code or math caused significant redistribution of expert routing, indicating that these domains rely on distinct expert subsets.
• The "cognitive scaffold" ablation induced smaller, more diffuse deviations, suggesting that structured reasoning data enhances transferable reasoning structures without destabilizing domain-specific expert allocations.

Key Contributions

1. Re-evaluation of Code's Role: The paper challenges the consensus that code is a general reasoning enhancer, demonstrating instead that pure code competes with mathematical and knowledge-intensive reasoning under fixed budgets.
2. Granular Domain Separation: By strictly separating pure code from Code-NL, the study isolates the marginal contribution of executable code, revealing that prior reasoning gains likely stemmed from mixed-format structured data.
3. Identification of Cognitive Scaffolds: The authors define and operationalize "cognitive scaffolds" as a class of highly structured mathematical samples that improve complex reasoning without degrading other capabilities, offering a strategy to optimize data composition.
4. Mechanism-Level Insights: The work provides evidence linking data composition to internal MoE routing dynamics, showing how competitive and synergistic interactions manifest at the expert level.

Significance and Claims

The paper claims to clarify which data characteristics transfer across capability dimensions and to point toward more precise, data-centric optimization strategies. It argues that under fixed training budgets, simply adding code is not a universal solution for reasoning; rather, the structure of the data (specifically, cross-domain structured reasoning traces) is the critical factor.

The authors conclude that appropriately incorporating cross-domain structured reasoning data can mitigate cross-domain trade-offs, allowing models to preserve performance in one domain (e.g., programming) while enhancing it in another (e.g., complex mathematical reasoning). The study emphasizes that future data curation should focus on identifying and prioritizing these "cognitive scaffolds" rather than broadly increasing the proportion of code or math tokens.

The work is presented as an empirical investigation into pretraining data composition, aiming to support more transparent and efficient model development without introducing new deployed systems or claiming broad societal impacts beyond the scope of data-centric AI optimization.