🤖 machine learning

Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic

This paper reveals that transformers learn arithmetic skills through non-human, correlation-driven "shattered compositionality" rather than sequential procedural rules, leading to characteristic mixing errors and poor robustness that persist despite model scaling or scratchpad supervision.

Original authors: Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve a puzzle. In the world of artificial intelligence, these robots are called "Large Language Models" (LLMs). They are like super-enthusiastic students who have read almost every book in the library. They are famous for being able to write stories, debug code, and even solve math problems that look like they belong in a high school textbook. But here is the catch: just because a student can get an 'A' on a practice test doesn't mean they truly understand the rules of the game. If you change the test just a tiny bit—maybe by adding one extra number or rearranging the words—the student might suddenly panic and fail in weird, unpredictable ways. Scientists call this "brittleness." It's like a house of cards that looks perfect until a gentle breeze blows through.

To understand why this happens, researchers look at how these models "learn" new skills. Usually, we hope they learn like humans: step-by-step, building a strong foundation before moving to the next level. For example, when learning to add numbers, a human learns to start at the very end (the ones place), handle any "carries" (where a number gets too big for one spot), and then move to the tens, hundreds, and so on. This is a logical, hierarchical way of thinking. But what if the robot learns the rules in a completely different order? What if it learns the big, scary numbers first and the small ones later, or tries to learn everything all at once? This paper dives into that mystery, using simple math puzzles to watch how these digital brains actually grow their skills, revealing that they might be taking a shortcut that looks like learning but isn't quite the same as human understanding.

The Shattered Puzzle: How AI Learns Math Backwards

In this study, researchers at the University of Wisconsin–Madison decided to stop guessing and start watching. They trained small, simple AI models (called transformers) on synthetic arithmetic tasks—basically, making them practice adding and multiplying numbers over and over again. But instead of just checking if they got the right answer at the end, they watched the learning process in slow motion, digit by digit. They wanted to see how the models figured out the rules.

What they found was a bit like watching a child try to put on a pair of shoes by starting with the laces and then trying to figure out where the toes go. The researchers discovered that these AI models don't learn math the way humans do. Instead of starting at the bottom (the ones place) and working their way up, the models often learned the highest digits first.

Imagine you are adding four large numbers together. A human would look at the ones column, add them up, carry the extra to the tens column, and so on. The AI, however, seemed to ignore the ones column entirely at first. It focused on the thousands and hundreds places, learning those digits perfectly while the ones place remained a mess. It was as if the model was trying to guess the total size of the pile before it even knew how many single items were in it. The researchers call this phenomenon "shattered compositionality." It means the model is trying to build a house of cards, but it's stacking the roof on the ground and hoping the foundation will catch up later.

The "Correlation" Shortcut

Why would a super-smart AI do something so silly? The paper suggests the answer lies in how the AI looks at its training data. Humans learn math by understanding the cause and effect: "If I add these two ones, I get a ten, so I must carry a one." The AI, however, seems to be playing a game of "statistical matching." It looks at the data and asks, "Which numbers usually show up together?"

The researchers found that the highest digits in the answer are often strongly linked to the highest digits in the input numbers, regardless of what happens in the lower digits. It's a strong, easy-to-spot pattern. The AI latches onto this easy pattern first because it's the most obvious signal in the data. It's like a detective who sees a suspect's car and immediately assumes they are guilty because they have a red car, ignoring all the other clues. The AI learns the "red car" rule (the high digits) first because it's the easiest correlation to find. Only later, once it has mastered the high digits, does it try to figure out the messy, complicated rules for the lower digits.

To prove this, the researchers ran a "sabotage" experiment. They took the training data and scrambled the thousands-place digits, making them random. Suddenly, the AI couldn't learn the lower digits at all. It was stuck. This proved that the AI wasn't learning the math rules in a logical order; it was relying on the high digits to unlock the rest of the puzzle. Without the high digits, the whole learning process collapsed.

The Chaos of Parallel Learning

The problem gets even messier when the AI tries to learn multiple skills at the same time. In tasks like sorting numbers or comparing them, the AI tries to learn different rules in parallel. It's like a student trying to learn how to ride a bike, juggle, and play the piano all at once. Sometimes, these skills get in each other's way.

The researchers found that when the AI learns these skills simultaneously, it creates "mixing errors." Imagine you are sorting a list of names. A human would check the length of the name first, then the first letter, then the second. The AI, however, might get confused. It might take the first letter of one name and the last letter of another, stitching them together into a brand new, nonsense word. The paper calls these "mixing errors." It's as if the AI is a chef who has learned to chop onions and dice carrots separately, but when asked to make a salad, it accidentally throws the onion into the carrot bowl and the carrot into the onion bowl, creating a weird, inedible mix.

This parallel learning creates a fragile system. If you ask the AI to solve a problem that is slightly different from what it saw in training—like adding one more number to the list—it often falls apart. The skills it learned in parallel start fighting each other, and the model produces answers that are just a jumbled mess of the inputs.

Does Bigger or Smarter Fix It?

You might think, "Okay, maybe the small models are just confused. What about the giant, super-powerful AI models we see in the news? Or what if we give them a 'scratchpad' to write down their thoughts, like a human would?"

The researchers tested this, too. They trained much larger models (up to 100 million parameters) and even fine-tuned a massive pre-trained model (Pythia-1B) on these math tasks. They also tried giving the models a "scratchpad" where they could write out the steps of the math problem before giving the final answer.

The result? Nothing changed.

Even the giant models learned the digits in the wrong order. Even the models with scratchpads still prioritized the high digits first and struggled with the low ones. The scratchpads helped the models get the right answer faster, but they didn't fix the underlying learning style. The models were still taking the "correlation shortcut" rather than learning the true, step-by-step logic of math. This suggests that simply making AI bigger or giving it more tools doesn't automatically make it think like a human. The "shattered compositionality" seems to be a fundamental quirk of how these models learn from data, not just a bug in a specific small model.

The Takeaway

This paper paints a picture of AI that is incredibly capable but fundamentally different from us. It can solve problems that look like human math, but it's doing so with a different set of internal rules. It's not following a logical, step-by-step algorithm; it's following a map of statistical patterns. When those patterns are perfect, the AI shines. But when the patterns shift even a little—when the problem is slightly different or the data is messy—the AI's "shattered" learning style causes it to stumble.

The researchers aren't saying AI is broken or useless. They are saying that we need to understand how it learns to trust it better. If we want AI to be reliable in the real world, where problems are rarely perfect and data is always shifting, we need to figure out how to teach it to build its skills in a stable, human-like hierarchy, rather than letting it learn in a chaotic, parallel rush. Until then, we might have to be careful when we ask these digital students to solve a problem that's just a little bit different from the ones they practiced on.

Technical Summary: Shattered Compositionality in Transformers

Problem Statement

Large Language Models (LLMs) frequently achieve high accuracy on compositional benchmarks yet exhibit brittleness under small distribution shifts. While mechanistic studies have revealed discrepancies between LLM and human skill composition (often relying on a "bag of heuristics" rather than unified algorithms), the learning dynamics of how these non-canonical compositions emerge and how data distributions shape them remain unclear. Specifically, it is unknown how non-human compositions arise during training and how they lead to failure modes under distribution shifts.

Methodology

The authors investigate these questions by training small transformers on synthetic arithmetic and logical tasks where the canonical (human-like) decomposition is known. The study employs a black-box, model-agnostic approach to track skill acquisition throughout training.

Experimental Setup

Tasks:
- Addition: 4-operand addition ( $a+b+c+d=e$ ) with multi-digit integers.
- Multiplication: Simple multiplication ( $a \times b = c$ ) with multi-digit $a$ and single-digit $b$ .
- Comparison: Determining the relationship ( $>, <, =$ ) between two 4-digit integers.
- Sorting: Sorting a list of 3- or 4-digit integers.
Data Generation: The authors utilize specific sampling strategies, including "doubly balanced" sampling for sorting (balancing length and closeness) and controlled contrast pairs for comparison to isolate subskill performance.
Metrics:
- Digit-wise Error Rates: Tracking error rates for specific digit positions (e.g., units, tens, thousands) over training steps.
- Mutual Information (MI): Measuring the correlational signals available in the training data ($I(input, output)$ and conditional MI involving carries) and comparing them to the model's predictive distributions to identify when the model matches data statistics.
- Ablation Studies: Randomizing specific output digits (e.g., thousands place) to test dependency on higher-order latent structures.

Key Contributions

Non-Canonical Acquisition Dynamics: The paper identifies a phenomenon termed "shattered compositionality," where transformers acquire arithmetic subskills in reverse order (e.g., learning thousands-place digits before units-place digits) or in parallel, rather than following the sequential, hierarchical rules used by humans.
Correlational Matching Mechanism: The authors provide evidence that learning dynamics are driven by correlational matching to the training data rather than causal or procedural reasoning. The model prioritizes the strongest unconditional correlational signals (e.g., the relationship between the highest input digit and the highest output digit) before unlocking skills that depend on conditional signals (e.g., carry propagation).
Consequences of Parallel Acquisition: The study demonstrates that parallel acquisition leads to skill competition. When subskills are learned simultaneously, they can interfere with one another, causing transient performance degradation and characteristic "mixing errors" (e.g., swapping or repeating digits in sorting tasks) under distribution shifts.
Persistence Across Architectures: These behaviors are shown to persist across different model scales (up to 100M parameters), pre-trained models (Pythia-1B), alternative positional encodings (RoPE, T5), and even with scratchpad-based reasoning supervision.

Key Results

1. Reverse and Bidirectional Learning Orders

Addition: Contrary to human intuition (units $\to$ thousands), models consistently learn the most significant digits first. Even when the output format is reversed to incentivize a human-like order (units first), the model still learns higher digits first, suggesting an internal preference for the strongest correlational signals.
Multiplication: Models exhibit a bidirectional acquisition pattern, learning from both the most significant end and the units place in parallel, rather than a single sequential order.

2. Correlational Signals Drive Learning

MI Alignment: Sharp drops in training loss correspond precisely to the moments when the model's predictions match the mutual information structure of the training data. The model first exploits the unconditional signal $I(a_1, e_0)$ (highest digit) and only later acquires skills dependent on conditional signals like $I(a_i, e_i | c_{i-1})$ (carry).
Ablation Evidence: When higher-order digits (e.g., thousands place) are randomized in the training data, the model fails to learn lower-order digits, confirming that lower-skill acquisition is gated by the successful representation of higher-order latent structures.

3. Skill Competition and Brittleness

Comparison Task: Models exhibit non-monotonic learning curves where acquiring one subskill (e.g., predicting equality) temporarily interferes with another (e.g., comparing the thousands digit), leading to a temporary spike in error rates.
Sorting Task: Parallel learning leads to "mixing errors" where the model outputs integers constructed from fragments of different inputs (e.g., swapping the last digit of two numbers). These errors are exacerbated when input cues conflict, demonstrating that non-human parallel acquisition creates concrete brittleness under distribution shifts.

4. Robustness Checks

Scaling: Increasing model size (10M $\to$ 100M) or using pre-trained models does not eliminate the reverse learning order.
Scratchpads: While scratchpad formats (explicit decomposition) accelerate convergence, they do not restore the human-like sequential learning order; the model still prioritizes higher-order digits or learns remaining digits in parallel.
Modern LLMs: Evaluations on modern LLMs (Llama, Mistral, Qwen) using GSM8K templates with extended clauses show similar brittleness: accuracy degrades as the number of operands increases, and models frequently omit or repeat terms, mirroring the mixing errors observed in small transformers.

Significance and Claims

The paper claims to offer a model-agnostic account of a failure mode of compositionality rooted in training dynamics. It argues that the brittleness of LLMs is not merely a static architectural limitation but a consequence of how gradient-based optimization prioritizes correlational signals over procedural rules.

Mismatch with Human Reasoning: The work highlights a fundamental mismatch between training-time skill acquisition (driven by data correlations) and the human-like hierarchical compositions required for robust reasoning.
Implications for Safety and Reliability: The findings suggest that "almost" correct reasoning is fragile; models may appear to compose skills correctly on in-distribution data but fail unpredictably when the compositional demand is perturbed. This has direct implications for alignment, safety, and the reliability of LLMs in out-of-distribution settings.
Limitations: The authors acknowledge that their analysis is intentionally black-box, revealing that non-human acquisition occurs and its consequences, but not the specific internal circuits implementing them. They suggest future work should combine this training-dynamics view with white-box mechanistic analysis.