Counting as a minimal probe of language model reliability
This paper demonstrates that despite strong performance on standard benchmarks, large language models lack general logical competence for reliable rule-following, as evidenced by their failure to count beyond a limited set of internal states that mimics finger-counting rather than true procedural execution.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: The "Finger Counting" Test
Imagine you have a very smart robot that can write poetry, debug complex code, and answer difficult science questions. You ask it, "How many apples are in this basket?" and it answers correctly. You ask again with a bigger basket, and it's still right.
But what if you asked it to count 10,000 apples? Or 100,000?
This paper asks a very simple question: Do these AI models actually "know" how to count, or are they just guessing based on patterns they've seen before?
To find out, the researchers created a test called Stable Counting Capacity (SCC). They stripped away all the "smart" stuff (like math, history, or coding) and just asked the models to count identical items (like the letter "a") in a long list.
The Main Discovery: The "Finger" Limit
The researchers found that every single AI model they tested fails at counting long lists, even though they are advertised as being able to handle huge amounts of text.
Here is the analogy for what is happening inside the AI:
- The "Fingers" Analogy: Imagine the AI is trying to count by holding up its fingers. It has a finite number of fingers (internal states). As long as the number of items is less than the number of fingers, it counts perfectly.
- The Collapse: Once the list gets longer than its "fingers," the model doesn't just make a small mistake (like saying 101 instead of 100). Instead, it gives up entirely. It stops counting and starts guessing random, round numbers like 500, 1,000, or 2,000. It's as if the model ran out of fingers, dropped the counting stick, and started throwing darts at a number board.
Key Findings in Plain English
1. The "Magic" Context Window is a Lie
AI companies say their models can read "long contexts" (huge documents). The paper shows that while the model can read a long document, it cannot keep track of simple rules within it.
- Analogy: It's like a student who can read a 500-page book but, when asked "How many times did the word 'the' appear?", they have no idea. They can process the text, but they can't hold a simple variable in their head.
2. More Thinking Time Doesn't Help
The researchers tried giving the models more time to think (test-time compute) or asking them to "think step-by-step" (Chain of Thought).
- Result: It didn't work. If the model's internal "fingers" ran out, making it think harder just made it generate more confident-sounding nonsense. It couldn't reconstruct the count it lost.
3. The "Fingers" are Specific to the Look of the Text
The researchers changed the items being counted (e.g., from the letter "a" to the letter "b" or different symbols).
- Result: The model's counting limit changed. This proves the model isn't using a universal "math brain." Instead, it's using specific, learned pathways for specific symbols. If you change the symbol, the "fingers" get confused.
4. The "Sudden Drop"
The failure isn't gradual. The model counts perfectly up to a certain point (say, 72 items), and then instantly crashes.
- Analogy: It's not like a car running out of gas and slowing down. It's like a lightbulb that works perfectly until the exact second the filament breaks, and then it goes completely dark.
5. Why Current Tests Are Missing This
Standard tests (like math exams or coding challenges) are too complex.
- Analogy: If you want to know if a car engine is reliable, you don't just drive it on a race track (where aerodynamics and speed might hide a broken piston). You need to run a simple, boring test, like spinning the wheels on a lift. The paper argues that current AI benchmarks are the "race track," hiding the fact that the "engine" (the ability to follow simple rules) is broken.
What This Means for the Future (According to the Paper)
The paper concludes that current AI models are brittle. They can mimic following rules for a while, but they don't have a reliable, internal mechanism to track variables over long periods.
- The Problem: If you ask an AI to plan a complex project with 50 steps, it might get the first 10 steps right, but by step 50, it has likely "lost count" of the constraints and is just guessing what sounds good.
- The Solution: The authors suggest that to fix this, we can't just make the models bigger or give them more time to think. We need to change their architecture to include things like explicit memory or persistent variables (like a real notebook or a counter) that don't rely on the model's fragile internal "fingers."
Summary
The paper reveals that despite their impressive performance on complex tasks, Large Language Models are fundamentally bad at simple, exact counting. They rely on a limited number of internal "states" (like fingers) to track information. Once that limit is reached, they stop following rules and start guessing. This suggests that their current "intelligence" is a fragile illusion that breaks down when asked to maintain simple rules over long periods.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.