💬 NLP

LLMs Know More About Numbers than They Can Say

This paper reveals that while large language models often fail to verbally compare numbers in mixed notations, their hidden states actually encode accurate numerical magnitudes and rankings, and leveraging this internal knowledge through auxiliary training objectives can significantly improve their explicit numerical reasoning performance.

Original authors: Fengting Yuchi, Li Du, Jason Eisner

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fengting Yuchi, Li Du, Jason Eisner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Silent Genius" Problem

Imagine you have a brilliant student who has memorized every math textbook in the library. They can solve complex equations in their head instantly. However, if you ask them to say the answer out loud, they stutter, guess, or get it wrong.

That is exactly what this paper discovered about Large Language Models (LLMs) like GPT-4, Llama, and others.

The Problem:
When you ask an AI, "Which is bigger: 5.7 × 10² or 580?" (which is a fancy way of writing 570 vs. 580), the AI often gets it wrong. It might say 570 is bigger, even though 580 is clearly larger. This happens even though the AI can easily calculate "5.7 times 100" if asked directly.

The researchers asked: Does the AI actually know the numbers, or is it just guessing?

The Investigation: Peeking Under the Hood

To find out, the researchers didn't just ask the AI questions; they put on "X-ray glasses" and looked inside the AI's brain (its hidden layers) while it was thinking.

Think of an LLM like a giant factory assembly line. Text goes in one end, gets processed through many rooms (layers), and comes out as an answer. The researchers stopped the line at different rooms to see what the "workers" (neurons) were thinking about the numbers.

They found two surprising things:

1. The "Log-Magnitude" Secret (The Internal Calculator)

The researchers found that deep inside the AI, there is a hidden "secret code" that represents the size of a number.

The Analogy: Imagine the AI sees the number "580." Instead of just seeing the text "5-8-0," a specific part of its brain instantly translates that into a mental "volume knob" setting that says "This is loud." It translates "5.7 × 10²" into the exact same volume knob setting.
The Proof: They built a simple tool (a linear probe) that could read this secret code. When they did, the tool could predict the actual size of the number with 99% accuracy. The AI knew the numbers were almost identical in size.

2. The "Comparison" Secret (The Internal Judge)

They also looked to see if the AI knew which number was bigger before it spoke.

The Analogy: Imagine the AI has a tiny judge in its brain that holds up two apples and whispers, "Left one is bigger," to itself.
The Proof: When they asked this internal judge, it was right 95% of the time. It knew 580 was bigger than 570.

The Tragedy: The "Mute" Problem

So, the AI knows the answer. It has the calculation. It has the comparison. But when it opens its mouth to speak, it fails.

The Result: When asked to say the answer out loud, the AI only got it right 50% to 70% of the time. That's barely better than flipping a coin!
Why? The paper suggests the AI is like a person who knows the answer but is too nervous to speak, or is so focused on mimicking the style of the question that they forget the math.
- Example: If the AI sees a sample question where the answer is always the second number, it might just start guessing "the second number" without actually doing the math.

The Solution: Training the Brain, Not Just the Mouth

The researchers wanted to see if they could fix this. They tried a new training method.

The Old Way: Train the AI to just give the right answer (like teaching a student to memorize the answer key).
The New Way: While training, they also forced the AI to "check its work" internally. They added a rule: "Before you speak, make sure your internal 'volume knob' and 'judge' are correct."

The Result:
This worked like magic. By forcing the AI to pay attention to its internal understanding of numbers, its spoken accuracy jumped up significantly (from ~57% to ~90%+ for some models).

The Takeaway

LLMs are smarter than they sound: They have a deep, internal understanding of numbers that is much better than their ability to talk about them.
The bottleneck is speech, not math: The problem isn't that the AI can't do math; it's that it can't translate that math into words reliably.
We can fix it: If we train AI to care about its internal "number sense," it becomes much better at talking about numbers.

In short: The AI isn't bad at math. It's just bad at telling you the math. But with a little bit of the right training, we can help it find its voice.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in domains requiring high numeracy (math, science, finance). While LLMs can perform arithmetic operations (e.g., "What is $5.7 \times 10^2$ ?") and compare numbers in the same notation (e.g., "Which is larger, 570 or 580?"), they exhibit a significant failure rate when asked to compare numbers in mixed notations (e.g., "Which is larger, $5.7 \times 10^2$ or 580?").

The core research question is: Do LLMs actually possess an internal understanding of numerical magnitude, or do they merely manipulate textual representations of numbers? The authors hypothesize that models may encode accurate numerical information in their hidden states but fail to access or verbalize this knowledge during generation.

2. Methodology

The authors employed a multi-faceted approach involving probing (analyzing internal states) and intervention (fine-tuning) on a suite of 7B–8B open-weight models (e.g., Llama-2, Mistral, Qwen, DeepSeek) and frontier models (GPT-4.1).

A. Data Construction

Synthetic Cross-Notation Dataset: Created pairs of numbers where one is in scientific notation and the other in standard decimal form (e.g., $5.7 \times 10^2$ vs. 560). The dataset includes 11,200 comparison problems.
Real-World Dataset: Derived from the peS2o (arXiv) dataset to test performance on scientific text where numbers appear in diverse semantic contexts.

B. Probing Experiments

The authors trained linear probes on the hidden states of the transformer layers to extract numerical information without altering the model's weights.

Regression Probes: Trained to predict the log-magnitude ( $\log_2$ $lo g_{2}$ ) of a numeral from its hidden state.
- Goal: Determine if the model encodes the absolute value of numbers.
Classification Probes: Trained as binary classifiers to predict whether the first number in a pair is larger than the second ( $a > b$ $a > b$ ).
- Goal: Determine if the model encodes the relationship between numbers.
Log-Ratio Probes: Trained to predict $\log_2(a/b)$ directly from the prompt's final hidden state.

C. Verbalization Evaluation

Models were prompted with a one-shot demonstration to answer "Which is larger, $a$ or $b$ ?" The accuracy of these verbal responses was compared against the accuracy of the probes.

D. Intervention (Fine-tuning)

To test causality, the authors fine-tuned models using an auxiliary loss function. The total loss was defined as:
$L_{total} = L_{LM} + \beta L_{cls}$
Where $L_{LM}$ is the standard language modeling loss, and $L_{cls}$ is the cross-entropy loss from the classification probe. This forces the model's internal representations to better support the comparison task during generation.

3. Key Results

A. Internal Representations are Accurate

Linear Encoding: Hidden states linearly encode the log-magnitudes of numerals.
- On synthetic data, regression probes achieved a median relative error of ~2.3%.
- On real-world arXiv data, the error was higher (~19.06%) but still significant, with a correlation of $\rho = 71\%$ .
Comparison Encoding: Linear classifiers trained on hidden states achieved >90% accuracy (often >95% in deeper layers) in determining which of two numbers is larger, even when the numbers were very close in value.
Discrepancy: Despite high internal accuracy, the models' verbalized accuracy on the same tasks was only 50–70% (often near random chance for smaller models).

B. Correlation Between Internal State and Output

There is a strong positive correlation between the performance of probes on the first 3 layers and the model's final verbalization accuracy.
Models with better internal magnitude representations (as measured by early-layer probes) generally performed better at verbalizing the answer, suggesting that the "knowledge" exists but is not being effectively utilized by the generation head.

C. Impact of Fine-tuning

Standard fine-tuning on the task improved verbal accuracy significantly (e.g., from ~50% to ~90% for some models).
Crucially, adding the probe classification loss ( $\beta > 0$ ) as an auxiliary objective provided an additional 3.22% improvement in verbal accuracy over standard fine-tuning alone.
For specific models (e.g., Mistral-7B, Qwen3-8B), this approach pushed accuracy to >99%, effectively "unlocking" the internal knowledge.

4. Key Contributions

Evidence of "Silent" Numeracy: The paper demonstrates that LLMs possess sophisticated, linearly encoded representations of numerical magnitude and comparison logic that are not accessible via standard text generation.
The Verbalization Gap: It identifies a specific failure mode where models fail to translate internal numerical reasoning into natural language, particularly in cross-notation scenarios.
Causal Link via Auxiliary Loss: The authors provide empirical evidence that explicitly optimizing internal representations (via probe loss) causally improves the model's ability to verbalize numerical reasoning.
Benchmarking: They introduce a rigorous evaluation framework using both synthetic and real-world scientific data to test numeracy beyond simple arithmetic.

5. Significance and Implications

Beyond Black-Box Evaluation: The study suggests that evaluating LLMs solely on output accuracy underestimates their true capabilities. Internal probing reveals "competence" that is hidden by "performance" failures.
Training Strategy: The findings suggest that improving numerical reasoning in LLMs does not necessarily require more data or larger models, but rather targeted fine-tuning that aligns internal representations with the generation objective.
Limitations: The authors note that current findings are limited to 7B–8B models (where hidden states are accessible) and that synthetic data may not fully capture the complexity of real-world scientific text. Additionally, the probes reveal representation but not necessarily the mechanism of how that representation is used during inference.

In conclusion, the paper argues that LLMs know more about numbers than they can say, and that bridging this gap requires aligning the model's internal state optimization with its generative goals.