💬 NLP

AI-rithmetic

This paper investigates why frontier AI models struggle with basic arithmetic, demonstrating that their performance degrades as digit counts increase and that most errors are caused by predictable issues like operand misalignment due to tokenization or failures in carrying digits.

Original authors: Alex Bie, Travis Dick, Alex Kulesza, Prabhakar Raghavan, Vinod Raman, Sergei Vassilvitskii

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Bie, Travis Dick, Alex Kulesza, Prabhakar Raghavan, Vinod Raman, Sergei Vassilvitskii

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Genius Toddler" Paradox: Why AI Can Solve Calculus but Fails at Addition

Imagine you are watching a world-class concert pianist. They can play the most complex Rachmaninoff concerto with breathtaking precision, moving their fingers in patterns that seem almost supernatural. But then, you ask them to do something incredibly simple: "Please tap your fingers to a steady 4/4 beat."

Suddenly, they stumble. They miss a beat. They lose the rhythm. They look confused.

That is exactly what is happening with modern Artificial Intelligence. This paper, titled "AI-rithmetic," explores a bizarre phenomenon: AI models can win international math competitions and help scientists prove new theories, yet they consistently fail at adding two long numbers together.

Here is a breakdown of what the researchers found, using a few simple analogies.

1. The "Broken Ruler" Problem (Misalignment Errors)

Imagine you are trying to line up two pieces of graph paper to add up the squares. If you slide one piece of paper slightly to the left or right, your entire calculation will be wrong, even if your math is perfect.

The researchers found that many AI models suffer from "Misalignment." Instead of lining up the "ones" column with the "ones" column, the AI accidentally shifts one number over.

The Twist: This isn't just random clumsiness. It’s often caused by how the AI "reads" text, a process called tokenization. Think of it like reading a book where some words are broken into single letters, but other words are grouped into chunks of three. Because the AI sees the numbers in these weird, uneven chunks, it loses its place on the "graph paper," causing it to misalign the numbers.

2. The "Flickering Lightbulb" Problem (Close Carry Errors)

In long addition, you often have to "carry the one." If you add $9 + 1$ , you write $0$ and carry a $1$ to the next column.

The researchers found that AI models have a "flickering" problem with this. Imagine a lightbulb that is mostly working, but occasionally—just for a split second—it flickers off.

The AI doesn't fail because it doesn't understand math; it fails because it makes "close calls." It either:

Forgets to carry the one when it should.
Carries a one when it shouldn't.

The researchers discovered that these aren't "big" mistakes. They are tiny, local glitches. It’s like a hiker walking a perfect path for miles, but every few miles, they take one slightly wrong step. Eventually, they are miles away from where they intended to be, even though they were "mostly" on the right track.

3. Why "Bigger" Isn't Always "Smarter"

You might think, "If the AI is struggling, just give it a bigger brain (more data and more computing power)!"

The paper suggests this might not work. While larger models are generally better, they still hit a wall as the numbers get longer. It’s like trying to fix a leaky faucet by building a bigger house. You’ve built a magnificent mansion, but the faucet is still dripping. The problem isn't the size of the system; it's the mechanism it uses to do the work.

The Big Picture: The "Black Box" Warning

The most important takeaway from this paper isn't about math—it's about reliability.

We are currently building AI systems that act like "Black Boxes." We see them do incredible, genius-level things, so we assume they are "smart" in every way. But this research proves that AI doesn't "think" like a human. A human child learns that $1+1=2$ and understands the concept of quantity. An AI is more like a very sophisticated pattern-matcher that can sometimes lose its place or trip over a tiny detail.

The Lesson: We shouldn't mistake "advanced" for "reliable." Until we fix these fundamental "glitches in the matrix," we have to be very careful about trusting an AI to do even the simplest tasks.

Technical Summary: AI-rithmetic*

Title: AI-rithmetic*
Authors: Alex Bie, Travis Dick, Alex Kulesza, Prabhakar Raghavan, Vinod Raman, Sergei Vassilvitskii (Google)
Date: February 2026

1. Problem Statement

Despite the ability of frontier Large Language Models (LLMs) to solve advanced mathematical problems and win international competitions, they exhibit a paradoxical failure in basic integer addition. As the number of digits in the operands increases, accuracy degrades significantly. The researchers seek to move beyond the observation that "AI is bad at math" to provide a systematic, mechanistic investigation into why and how these models fail at the most fundamental level of arithmetic.

2. Methodology

The researchers conducted a large-scale empirical study using a suite of frontier models, including Claude Opus 4.1, GPT-4o, GPT-5, Gemini 2.5 (Flash and Pro), and Gemma 3 27B.

Task Design: The models were prompted with the simple instruction: "What is A + B? Write just the answer." They were tested on two-argument addition for lengths $d \in \{1, \dots, 100\}$ .
Evaluation Metric: To handle the variability of LLM outputs, the researchers used a "lenient" grading system: they extracted the span of digits with the lowest edit distance to the correct sum.
Error Classification: The authors developed a taxonomy to categorize mistakes into three primary types:
1. Misalignment Errors: One operand is incorrectly shifted (offset) relative to the other.
2. Close Carry Errors: The model fails to carry a "1" when the column sum is 10, or incorrectly carries a "1" when the column sum is 9.
3. Runaway Errors: The model produces an output significantly longer (50%+) than the correct answer.
Statistical Modeling: They applied Fourier transforms to analyze periodicity in error rates and used a stochastic geometric model to test the independence of carry errors.

3. Key Contributions & Results

A. Performance Degradation:

All frontier models (except those using code execution) show a significant drop in accuracy as the number of digits increases.
Performance is non-monotonic; models are sometimes better at adding certain lengths than others, suggesting the failure is not a simple linear scaling issue.

B. Error Taxonomy and Explanability:
The study found that the vast majority of errors are highly interpretable and fall into two categories:

Misalignment: Dominant in GPT-5 and Gemini 2.5 Pro.
Close Carry: Dominant in Claude Opus 4.1.
Coverage: These two error classes explain 87.9% of Claude Opus 4.1 errors, 62.9% of GPT-5 errors, and 92.4% of Gemini 2.5 Pro errors.

C. The Role of Tokenization (Misalignment):

The researchers discovered a link between tokenization schemes and misalignment.
GPT-5 and GPT-4o use a tokenizer that groups digits into 3-digit chunks. The researchers found a periodic spike in accuracy for GPT-5 at lengths divisible by 3 (verified via Fourier transform). This suggests that misalignment errors are exacerbated when models encounter "leftover" 1- or 2-digit tokens.
Models using single-digit tokenization (like Gemma 3) showed different error patterns.

D. Stochastic Nature of Carrying (Close Carries):

The researchers modeled "close carry" errors as independent Bernoulli trials.
They found that for many models (notably Gemma 3), the occurrence of a carry error is largely independent of previous carries. If a problem has $n$ "close carry" opportunities, the model's success follows a geometric distribution, behaving like a machine flipping a biased coin at each critical digit.

4. Significance

This paper is significant because it challenges the "scaling hypothesis"—the idea that simply adding more parameters or data will eventually solve basic algorithmic reasoning.

The findings suggest that:

Architectural Idiosyncrasies: Current LLM failures are not just "lack of knowledge" but are deeply tied to design choices like auto-regression and tokenization.
Limits of Compositionality: While LLMs attempt to compose simple behaviors to solve complex ones, they do not implement these algorithms (like long addition) reliably.
A Probe for General Failure: Arithmetic serves as a "convenient probe." The paper argues that if we cannot guarantee reliability in a finite, well-defined task like addition, we cannot fully anticipate or trust the reliability of these systems in more complex, open-ended reasoning tasks.