🤖 machine learning

Procedural Pretraining: Warming Up Language Models with Abstract Data

This paper demonstrates that "procedural pretraining," which involves initially exposing language models to a small fraction of abstract structured data generated by formal languages and algorithms, significantly enhances their reasoning capabilities, accelerates convergence, and reduces computational costs compared to standard natural language pretraining.

Original authors: Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, Damien Teney

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, Damien Teney

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Brain Gym" Before the Real Test

Imagine you are training a brilliant but inexperienced student to become a master historian. The standard way to do this is to hand them a massive library of history books (the internet) and say, "Read everything."

The authors of this paper suggest a different approach. Before opening the history books, they give the student a few weeks of logic puzzles, math drills, and pattern games. They call this "Procedural Pretraining."

The core idea is that just as human babies learn to stack blocks and play simple games before they learn complex philosophy, AI models might learn better if they first practice "thinking" with abstract, rule-based data before they start memorizing the real world.

What is "Procedural Data"?

Think of Procedural Data as a video game level that has no story, no characters, and no pictures. It's just pure rules.

The Game: "Here is a list of numbers. Now, sort them." or "Here is a string of brackets (( )). Make sure they are balanced."
The Point: The AI isn't learning what a bracket means (it's just a symbol). It's learning how to follow a rule, keep track of a list, or find a pattern.

The paper uses data generated by simple computer algorithms (like balancing parentheses or shuffling decks of cards) to teach the AI these basic "muscle moves" for thinking.

The Main Findings (The "Magic" Results)

The researchers tested this idea and found three surprising things:

1. A Little Bit Goes a Long Way

You don't need to replace the history books with logic puzzles. You just need a tiny "warm-up."

The Analogy: Imagine the AI is a marathon runner. The researchers found that if the runner does just 0.1% to 0.3% of their total training miles on a treadmill (the abstract logic puzzles) before running on the actual track (the real internet data), they run the whole race faster and better.
The Result: Models that got this tiny "logic warm-up" learned the same amount of knowledge using 30% to 45% less of the massive internet data. It's like getting a discount on the cost of training.

2. It Fixes Specific "Brain Glitches"

The paper tested if this warm-up helped with specific hard tasks.

The "Needle in a Haystack" Test: Imagine reading a 100-page book and being asked, "What was the third word on page 42?" Most AI models struggle with this; they forget the beginning by the time they reach the end.
The Fix: When the AI was warmed up with "Dyck sequences" (a specific type of balanced bracket puzzle), its ability to find that needle jumped from 10% accuracy to 98%.
The Lesson: Different logic games fix different brain muscles. Some games help with memory; others help with math.

3. The "Brain" Learns in Specific Rooms

The researchers looked inside the AI's "brain" (its internal layers) to see where this new skill lived. They found that the AI's brain has two main types of rooms:

The "Attention" Rooms: These are like the AI's eyes. They look at different parts of a sentence to see how they connect. The paper found that for coding (which is very structured), the "Attention" rooms learned the most from the logic puzzles.
The "MLP" Rooms: These are like the AI's memory banks where facts are stored. Surprisingly, for natural language (like writing a story), the "MLP" rooms benefited the most from the logic puzzles.
The Takeaway: The logic warm-up didn't just make the whole brain "smarter" in a general way; it built specific tools in specific parts of the brain.

How They Combined the Skills

The researchers also tried mixing different types of logic puzzles.

The Analogy: Imagine you have a student who is great at math but bad at memory, and another who is great at memory but bad at math. Instead of making them study together, the researchers took the "math brain" of one and the "memory brain" of the other and stitched them together.
The Result: This "Frankenstein" model was better at everything than either of the original students. This suggests we can build better AI by mixing and matching the best "logic brains" from different training sessions.

Summary: Why Does This Matter?

The paper argues that current AI models try to learn knowledge (facts) and reasoning (how to think) at the same time, which gets messy.

By using Procedural Pretraining, they are essentially saying: "Let's teach the AI how to think first, using simple, boring rules. Once it has those thinking tools, it will be much faster and more efficient at learning the actual facts of the world."

It's a simple, lightweight way to make AI smarter, cheaper to train, and better at remembering things, without needing to change the AI's architecture or invent new algorithms. It's just a better way to start the day.

Technical Summary: Procedural Pretraining: Warming Up Language Models with Abstract Data

Problem Statement

Current large language models (LLMs) acquire knowledge and reasoning abilities simultaneously during pretraining on web-scale semantic corpora. This "entangled learning" is identified as a key limitation, often leading models to rely on surface-level heuristics rather than systematic reasoning procedures. The paper posits that this entanglement can be mitigated by decoupling the acquisition of abstract algorithmic skills from the acquisition of rich semantic knowledge. The core problem addressed is how to efficiently "warm up" language models with abstract, structured data to facilitate the subsequent acquisition of semantic knowledge, similar to how humans learn simple logic before complex reasoning.

Methodology

The authors propose Procedural Pretraining, an initial "pre-pretraining" stage where models are exposed to non-semantic data generated by simple algorithms (formal languages, cellular automata, and rule-based processes) before standard pretraining on semantic data (natural language, code, mathematics).

Experimental Setup

Models: GPT-2-type decoder-only transformers ranging from small diagnostic models (2 layers) to larger models (up to 1.3B parameters).
Procedural Data Types:
- Sequence Transformations: SET (deduplication), REVERSE, IDENTITY, UNION, SORT, DELETE.
- Memory Operations: STACK (simulating push/pop operations).
- Formal Languages: K-DYCK (nested balanced brackets) and K-DYCK SHUFFLE.
- Cellular Automata: ECA Rule 110.
Training Regimes:
- Additive Setting: Fixed semantic tokens ( $T_2$ ) with varying procedural tokens ( $T_1$ ) to measure performance gains.
- Substitutive Setting: Reducing semantic tokens while adding a small amount of procedural tokens to match baseline performance, evaluating data efficiency.
- Transfer Mechanisms:
  - Full-model transfer: Using all weights.
  - Selective transfer: Transferring only Attention layers or only MLP layers to localize where useful information resides.

Diagnostic Approach

The authors use specific algorithmic tasks as diagnostic tools to probe the capabilities gained from procedural pretraining:

Memory Recall: Needle-in-a-haystack (HAYSTACK).
Arithmetic: Addition, Reversed Addition, and Multiplication.
Logical Processing: Sorting.

Key Contributions and Results

1. Specific Algorithmic Skill Enhancement

Different forms of procedural data improve distinct algorithmic skills. For instance:

Dyck sequences significantly improve context recall (HAYSTACK accuracy jumped from 10% to 98%).
ECA Rule 110 benefits Reversed Addition.
SET and UNION operations aid in sorting and set operations.
Structural Necessity: Shuffling procedural sequences (removing structure while keeping token distribution) causes performance to drop to baseline, confirming that the structure of the data, not just the tokens, is critical.

2. Localization of Pretrained Information

The benefits of procedural pretraining are not uniformly distributed across the model architecture:

Attention Layers: Crucial for structured, language-free domains like pure code (JAVACORPUS) and specific algorithmic tasks (e.g., HAYSTACK). In some cases, transferring only attention layers outperformed full-model transfer.
MLP Layers: More beneficial for natural language domains (WIKITEXT, C4). This is notable as MLPs are often associated with storing factual knowledge, yet procedural data (which lacks facts) improves them for language tasks.
Hybrid Domains: For datasets combining language and structure (documented code, informal math), full-model transfer is optimal.

3. Data Efficiency and Performance Gains

Procedural pretraining acts as a lightweight, data-efficient complement to standard pretraining:

Minimal Overhead: Adding as little as 0.1% – 0.3% procedural tokens (e.g., 2–4 million tokens) significantly outperforms standard pretraining baselines.
Substitution: Procedural data allows models to reach the same loss values with significantly fewer semantic tokens:
- C4 (Language): 55% of original data required.
- CODEPARROT (Code): 67% of original data required.
- DEEPMIND-MATH (Math): 86% of original data required.
Scaling: These gains persist and remain consistent when scaling model sizes up to 1.3B parameters and data sizes up to 10.5B tokens.

4. Downstream Utility

The improvements in pretraining perplexity and algorithmic skills transfer to downstream tasks:

Code Generation: Models with procedural warm-ups show non-random accuracy on MBPP (code generation) where baselines fail completely.
Reasoning: Improvements persist in commonsense reasoning (ARC-Easy, HellaSwag) and grammatical understanding (BLiMP) after fine-tuning.

5. Mechanisms and Combinations

Mechanism: The benefits are encoded in precise weight structures, not merely generic attention sharpening or weight rescaling.
Combination: The paper demonstrates that benefits are additive. Mixing different procedural data types or surgically combining weights from different pretrained models (e.g., Attention from a SET model + MLPs from an ECA model) yields superior performance across multiple tasks.

Significance and Claims

The paper claims that procedural pretraining is a simple, lightweight means to improve LLM performance and accelerate pretraining. Its primary significance lies in:

Disentangling Knowledge and Reasoning: It suggests a path to inject an "algorithmic reasoning prior" into models before they acquire world knowledge, potentially addressing the entanglement of knowledge and reasoning that limits current models.
Data Efficiency: It offers a method to drastically reduce the computational cost (FLOPs) and data volume required to train high-performing models.
Modularity: It reveals that different components of the Transformer architecture (MLP vs. Attention) can be specialized for different types of inductive biases (semantic vs. structural), opening avenues for modular pretraining strategies.

The authors remain modest regarding limitations, noting that their experiments use models up to 1.3B parameters (smaller than state-of-the-art LLMs) and that combining multiple procedural data types is currently a proof-of-concept requiring further optimization. However, they assert that the findings provide a promising data-driven path for improving the acquisition of reasoning capabilities in LLMs.