Technical Summary: REASONING CORE

Problem Statement

Procedural generators have long been utilized to produce verifiable reasoning problems at scale, particularly for reinforcement learning (RL) with outcome rewards. However, their utility as data for completion-supervised fine-tuning (SFT) has received less attention. A critical gap exists in understanding how broad collections of verifiable reasoning examples should be designed and represented to function effectively as SFT data.

The authors argue that verifiable generated problems are not automatically useful supervised data. Their value depends on several design factors: whether the rendered instance uniquely determines the intended answer, how that answer is represented (e.g., compact targets vs. step-by-step traces), the calibrated difficulty range, and the task's utility within a heterogeneous mixture. Furthermore, there is a lack of rigorous validation ensuring that procedural generation, rendering, target computation, and scoring remain aligned; generation alone does not guarantee correctness.

Methodology

1. REASONING CORE Collection

The authors introduce REASONING CORE, a collection of 50 procedural generators spanning nine domains: mathematics, formal proof/symbolic manipulation, logical/probabilistic inference, planning/games, state/reference tracking, graphs/constraints/relations, formal languages/transduction, sets/structured data, and code.

Key design principles include:

• Task Interface: A unified interface mapping seeds and difficulty configurations to prompts, structured reference answers, metadata, and semantic scoring functions.
• Semantic Scoring & Canonical Targets: Unlike RL environments that may accept multiple valid trajectories, SFT requires deterministic targets. The authors design compact, canonical answer serializations (e.g., lexicographic ordering for sets) to avoid arbitrary early choices. Scoring remains broader than the supervised reference (e.g., order-invariant for sets, execution-based for code).
• Difficulty Control: Generators expose controls (e.g., derivation depth, plan length) mapped via a global difficulty knob, calibrated so that levels up to 5 remain practical.
• Audit & Validation: The development process includes a repository-scale semantic audit using model-assisted review (GPT-5.5/5.6, Claude Haiku/Opus), API-based sampling, human adjudication, and regression testing to identify subtle mismatches between generation, rendering, and scoring.

2. Experimental Protocol

The study compares REASONING CORE against three alternative procedural collections: PROCEDURAL WARMUP, REASONING GYM, and SYNLOGIC.

• Setup: Four base models (SmolLM2-135M, SmolLM2-360M, OLMo-1B, SmolLM3-3B-Base) are trained with a completion-supervised objective.
• Data Mixture: The main data stream blends FineWeb-Edu text with DOLCI instruction/conversation data. Procedural collections replace 20% of the prompt-plus-answer tokens.
• Evaluation: The primary metric is the relative reduction in held-out answer negative log-likelihood (NLL) on reasoning benchmarks (DROP, LogiQA, ARC-Challenge, BBH-test) and retention benchmarks (MMLU-other, DOLCI). Accuracy and F1 scores are reported for the 3B model.
• RL Verification: A matched, single-seed GRPO (Group Relative Policy Optimization) run is conducted to test verifier-backed RL compatibility.

Key Contributions

1. A Broad, Reproducible Resource: REASONING CORE provides 50 generators with semantic scorers, difficulty controls, and a unified interface for SFT, evaluation, and RL. The release includes generated datasets (approx. 10B tokens), versioned manifests, and audit materials.
2. Controlled Collection Comparison: The authors perform a rigorous comparison of four procedural collections across four base models and multiple training durations, isolating the marginal effect of auxiliary data.
3. Repository-Scale Semantic Audit: The study exposes subtle generator, target, and scorer failures in existing collections (REASONING GYM and SYNLOGIC) through a multi-stage audit process, challenging the assumption that procedural data is correct by construction.
4. Design Insights: The paper distinguishes between semantic validity (a correct target exists) and training utility (exposure improves downstream performance). It highlights that compact targets and calibrated difficulty are critical, while faithful step-by-step traces of algorithms can sometimes be inferior training targets compared to compact answers.

Results

Supervised Fine-Tuning (SFT)

In the primary 3B comparison (SmolLM3-3B-Base, 2,400 updates):

• REASONING CORE achieved the highest mean scores on DROP (F1: 41.7), LogiQA (Acc: 47.8), and ARC-Challenge (Acc: 51.3), outperforming the main-only baseline and all three alternative procedural collections.
• Retention: REASONING CORE maintained or slightly improved retention metrics (MMLU-other, DOLCI), indicating that procedural supervision did not come at the expense of broader capabilities.
• Comparison: While all procedural collections showed positive reasoning transfer in at least some settings, their relative ordering varied by metric and model family. A filtered subset of REASONING GYM improved LogiQA but degraded other metrics, suggesting that narrowing task mixtures can reduce aggregate transfer.

Task-Level Analysis

• Learnability: Tasks with intermediate final rewards (neither too easy nor fully saturated) showed the strongest held-out transfer.
• Format: Compact answers outperformed step-by-step rationale targets for tasks like parsing and graph pathfinding. Long-input tasks (e.g., grid/board games) often hurt retention due to prompt/answer length costs.
• Zero-Shot Solvability: Even large models (DeepSeek V4 Pro, ~1.6T parameters) did not fully solve the collection zero-shot, indicating substantial headroom for training.

Reinforcement Learning

In a matched GRPO run, REASONING CORE supported verifier-backed RL, achieving higher accuracy on BBH-test and MMLU compared to REASONING GYM, though the authors note the RL comparison used a single seed and should be viewed as a demonstration of viability rather than a precise margin estimate.

Semantic Audits

The audit revealed material defects in external collections:

• REASONING GYM: 13 of 105 tasks had default-path defects (e.g., scorers awarding full credit to any non-empty answer).
• SYNLOGIC: 9 of 18 generators had defects (e.g., displayed constraints contradicting stored solutions).
• REASONING CORE: Iterative corrections during development eliminated material defects found in initial reviews.

Significance and Claims

The paper claims that REASONING CORE demonstrates that broad procedural collections can effectively strengthen the starting point for later training when designed specifically for completion-supervised objectives.

Key takeaways include:

• Design Matters: Semantic validity alone is insufficient; compact targets, calibrated difficulty, and alignment between generation and scoring are essential for training utility.
• Validation is Critical: Procedural generation does not guarantee correctness; rigorous, repository-scale audits are necessary to detect subtle mismatches.
• Objective Alignment: While collections like REASONING GYM were designed for RL, orienting design choices explicitly toward completion objectives (as done in REASONING CORE) leads to better SFT results.
• Modesty: The authors acknowledge limitations, noting that evidence stops at 3B parameters, evaluation focuses on formal/closed-answer reasoning, and the sequential effect of SFT followed by RL on the same collection remains untested.

The work provides a public resource and a diagnostic framework to support future research in supervised fine-tuning and RL for reasoning.