🤖 machine learning

Efficient numeracy in language models through single-token number embeddings

This paper introduces BitTokens, a novel single-token encoding strategy based on IEEE 754 binary floating-point representation that enables language models to learn arithmetic algorithms nearly perfectly and solve complex numerical problems more efficiently than current methods relying on multi-token number splitting or external tools.

Original authors: Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, Martin J. Menten

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, Martin J. Menten

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot how to do math. This robot is a Large Language Model (LLM). Currently, when you ask this robot to multiply two big numbers, it doesn't just "know" the answer. Instead, it tries to figure it out by talking to itself, writing down step-by-step instructions, and checking its work over and over again.

The paper argues that this "talking to itself" method is incredibly wasteful. It's like asking someone to calculate $4,314.97 \times 4,080,000$ by writing out a 10-page essay explaining every single step, just to get a simple number at the end. The robot uses up a massive amount of "brain space" (tokens) just to do basic arithmetic, leaving it with no room to solve harder problems.

The Problem: The "Word-by-Word" Math Trap

Right now, these robots treat numbers the same way they treat words. If you type "4,080,000," the robot sees it as a long string of separate pieces (tokens), like reading a book one letter at a time.

The Analogy: Imagine trying to do math by reading a number like "123" as three separate letters: "1", "2", and "3". To add them, you have to process each letter individually, carry over the "1" from the tens place, and so on. It's slow, clunky, and prone to errors.

Because of this, the robot has to generate thousands of words of "reasoning" just to solve a simple multiplication problem.

The Solution: The "Magic ID Card" (BitTokens)

The authors propose a new way to teach the robot numbers, called BitTokens.

Instead of breaking a number into pieces, BitTokens give every number a single, unique ID card.

The Analogy: Think of a number not as a sentence you have to read, but as a specific color on a palette. Instead of describing "Red" by saying "It's a mix of blue and yellow," you just hand the robot a single card that is Red.
How it works: They use the computer's standard way of storing numbers (called IEEE 754, which is like a universal blueprint for how computers store decimals). They turn this blueprint into a single token (a single "word" for the robot).
The Result: When the robot sees the number 4,080,000, it doesn't see a long string of digits. It sees one single symbol that instantly tells it everything it needs to know about that number: its size, its precision, and its sign.

Why This is a Big Deal

The paper tested this on small robot models (to keep things simple and controlled). Here is what they found:

Speed and Efficiency: With BitTokens, the robot didn't need to write a 10-page essay to do math. It could solve basic addition, multiplication, and division almost perfectly in a single step.
Learning Algorithms: Because the numbers are encoded in a structured, logical way (using binary bits, just like computer chips), the robot could actually "learn" the rules of math. It's like teaching a child to count by giving them a single block for "5" instead of making them count out five individual pebbles every time.
The Trade-off: The paper notes that for very complex, multi-step problems (like calculating a standard deviation of a huge list of numbers), the single-token method is still limited by how deep the robot's "thinking" can go in one go. However, for standard arithmetic, it was vastly superior to the old methods.

What the Paper Does NOT Claim

It is important to stick to what the authors actually said:

They did not say this will immediately fix medical diagnoses or financial advice.
They did not claim this works on the biggest, most expensive models in the world yet (they tested it on smaller, "nano" models).
They did not say we should stop using reasoning entirely. They suggest that by making the basic math efficient, the robot will have more "brain space" left over to use reasoning for the really hard problems.

The Bottom Line

The paper introduces a new "language" for numbers that lets AI models stop stumbling over basic math. By giving numbers a single, efficient "ID card" instead of a long, messy description, these models can calculate faster, more accurately, and with less wasted effort. It's a fundamental upgrade to how AI "sees" the world of numbers.

Technical Summary: Efficient Numeracy in Language Models through Single-token Number Encodings

Problem Statement

Large Language Models (LLMs) currently struggle to process numerical data and solve complex calculations efficiently. While frontier models can achieve high accuracy on arithmetic tasks, they often rely on two inefficient strategies:

External Tool Use: Offloading calculations to calculators or code interpreters. This introduces latency, potential errors in tool selection, and prevents the model from generating intermediate numeric representations during a forward pass.
Reasoning Chains: Generating extensive step-by-step text to break down problems. As illustrated in the paper, solving a single multiplication can require tens of thousands of tokens, limiting the complexity and length of solvable problems due to context window and cost constraints.

The root cause identified is the standard tokenization strategy, which treats numbers like text, splitting them into multiple subword tokens (e.g., digits or triplets). This fragmentation disrupts the internal logic required for arithmetic, such as carry-over propagation, and forces models to learn inefficient algorithms.

Methodology

1. Evaluation of Frontier LLMs

The authors first benchmarked eight frontier LLMs (including GPT-5, Gemini 2.5, DeepSeek v3.1, and Llama 4) across nine numerical tasks: comparison (Min/Max, Interval, Sorting), basic arithmetic (Addition, Multiplication, Division), and multi-step operations (Exponentiation, Mean, Standard Deviation).

Findings: Models without reasoning capabilities performed poorly on multiplication, division, and exponentiation (often <60% log-sMAPE). Models with "Maximal" reasoning achieved higher accuracy but at the cost of generating 5,000 to 30,000 tokens per calculation.
Metric: The paper introduces log-sMAPE, a metric that measures the fraction of significant digits correctly predicted before the first error, providing a more granular view of numerical precision than exact match accuracy.

2. Desiderata for Single-Token Encodings

To address the inefficiencies of current tokenization, the authors propose a set of nine requirements (Desiderata) for an ideal single-token number encoding:

D1 (Token Efficiency): One token per number.
D2 (Uniqueness): Bijective mapping between value and encoding.
D3 (Structure): Geometry reflecting numeric order and distance.
D4 (Scale Invariance): Support for wide ranges and precisions.
D5 (Normalization): Bounded and stable under LayerNorm/RMSNorm.
D6 (Stability): Robust to low-precision activations (e.g., FP8).
D7 (Continuity): Smooth variation with underlying values.
D8 (Robustness): Reliable decoding under noise.
D9 (Arithmetic): Admit learnable algorithms for core operations.

3. Analysis of Existing Approaches

The paper analyzes two existing single-token strategies:

xVal: Scales a learned token by the numeric value. It fails D4 (limited range/precision) and D6 due to normalization constraints.
FoNE: Uses Fourier features (sinusoidal functions) to encode numbers. While it satisfies most desiderata, the authors prove via Proposition 4.3 that sinusoidal encodings lack a simple, local mapping for multiplication. Multiplication in this space requires a computationally intensive, non-local decoding, convolution, and re-encoding process, making it unsuitable for efficient learning by neural networks.

4. Proposed Solution: BitTokens

The authors introduce BitTokens, a novel encoding strategy based on the IEEE 754 double-precision binary floating-point format.

Encoding: A number is represented as a single token containing a 64-dimensional vector corresponding to its sign bit (1), exponent bits (11), and significand bits (52).
Construction: The binary vector is scaled to $[-1, 1]$ to satisfy normalization constraints (D5) and added to a learnable [NUM] token embedding.
Reciprocal Augmentation: To facilitate division learning, the binary representation of the number's reciprocal is concatenated to the encoding.
Decoding: A dedicated "number head" (linear layer + sigmoid) predicts the binary vector from the hidden state. A fixed threshold (0.5) generates the binary output.
Loss: Training uses Bit-wise Binary Cross-Entropy (BCE) rather than regression on the reconstructed value, as the latter is infeasible due to magnitude and thresholding issues.

Key Contributions

Systematic Evaluation: Demonstrated that frontier LLMs rely excessively on reasoning tokens for arithmetic, highlighting the need for intrinsic numerical capabilities.
Theoretical Framework: Established a set of nine desiderata for single-token encodings and proved theoretically that sinusoidal encodings (FoNE) are fundamentally ill-suited for multiplication due to non-locality and computational complexity.
BitTokens Implementation: Proposed and validated a deterministic, IEEE 754-based encoding that satisfies the desiderata, enabling even small models to learn arithmetic algorithms.

Results

Experiments were conducted using nanoGPT-2 models (117M parameters) trained from scratch on 30M samples per task, combined with text data (FineWeb).

Performance: BitTokens achieved near-perfect performance (log-sMAPE > 0.98) on comparison and single-step arithmetic tasks (Addition, Multiplication, Division), significantly outperforming subword, single-digit, xVal, and FoNE baselines.
Efficiency: BitTokens reduced the token cost for arithmetic tasks to a single token, compared to the thousands required by reasoning chains.
Limitations:
- Multi-step Tasks: BitTokens struggled with complex multi-step tasks like Mean, Standard Deviation, and Exponentiation in a single forward pass. The authors attribute this to the model's limited capacity to compress high-precision results into a single latent step, whereas multi-token models can iteratively refine answers over multiple forward passes.
- Comparison: On the Mean task, traditional multi-token methods (single-digit) outperformed BitTokens, likely due to their ability to use generated tokens as an external "scratchpad" for intermediate states.

Significance and Claims

The paper argues that BitTokens represent a promising step toward intrinsic numeracy in LLMs. By replacing inefficient tokenization with a structured, single-token representation, models can learn calculation algorithms internally without relying on excessive reasoning chains or external tools.

The authors modestly claim that while BitTokens solve basic arithmetic efficiently, they do not yet fully replace the need for reasoning in complex, multi-step problems. They suggest that the most effective future approach may be a hybrid strategy: using BitTokens for efficient intermediate calculations to free up token budget for logical reasoning, thereby expanding the complexity of problems LLMs can solve. The work emphasizes that current frontier models' reliance on long reasoning chains is a symptom of poor numerical representation, not a fundamental limit of the architecture itself.