A Triadic Suffix Tokenization Scheme for Numerical Reasoning

This paper introduces Triadic Suffix Tokenization (TST), a deterministic scheme that partitions numbers into three-digit triads with explicit magnitude markers to resolve the inconsistent fragmentation of standard tokenization methods and improve numerical reasoning in large language models.

The Gist

Imagine you are trying to teach a brilliant but very literal-minded robot how to count. You show it the number 1,234,567.

In the way most AI models currently work, the robot sees this number as a jumbled puzzle: 1, 2, 3, 4, 5, 6, 7. It has to guess, "Hmm, does this 1 mean one, or one million? Is this 4 just four, or four thousand?" It's like handing someone a bag of loose Lego bricks and asking them to build a castle without showing them the instruction manual. They might build something, but they often get the scale wrong.

This is the problem with how Large Language Models (LLMs) currently handle numbers. They break them up into tiny, confusing pieces and lose the "big picture" of how big the number actually is.

The Solution: The "Triadic Suffix" System

The paper proposes a new way to teach the robot numbers called Triadic Suffix Tokenization (TST). Think of this as giving the robot a set of labeled boxes instead of loose bricks.

Here is how it works, using simple analogies:

1. Grouping by "Thousands" (The Triad)

Instead of looking at every single digit, the system groups numbers into chunks of three, starting from the right.

• Old Way: 1 2 3 4 5 6 7 (Confusing!)
• New Way: 1 234 567 (Better, but still missing context).

2. The Magic Labels (The Suffixes)

This is the secret sauce. The system attaches a tiny, explicit label to each group of three to tell the robot exactly what that group represents.

• The last group (567) is just "ones."
• The middle group (234) gets a label k (for thousand).
• The first group (1) gets a label m (for million).

So, 1,234,567 becomes: 1m 234k 567.

The Analogy: Imagine you are moving house.

• Current AI: You hand the movers a pile of boxes and say, "Put these somewhere." They might put a heavy piano in a small closet because they don't know which box is heavy.
• TST: You label every box: "Piano - Heavy," "Books - Medium," "Lamp - Light." The movers (the AI) know exactly how to handle each piece immediately.

3. Handling Decimals (The "P" Markers)

What about numbers like 3.14159? The system treats the part after the decimal point similarly, but it uses a different set of labels (like p, pp, ppp) to show how deep the decimal goes.

• It ensures that 0.1, 0.10, and 0.100 are all treated as the exact same thing, preventing the robot from getting confused by extra zeros.

Why Is This Better?

The paper argues that this method fixes three major headaches for AI:

1. No More Guessing: The robot doesn't have to "learn" that 1 followed by 234 means a million. The label m tells it directly. It's like having a GPS that says "You are in the Million Zone" instead of making the driver guess based on street signs.
2. Perfect Precision: Because the labels are fixed and clear, the robot never makes silly mistakes like thinking 9.11 is bigger than 9.9 (a famous AI failure). The structure makes the size obvious.
3. Scalability: This system can handle numbers as small as a tiny fraction or as huge as the number of stars in the universe. You just add more labels (like b for billion, t for trillion) to the dictionary, and the robot can instantly understand them.

Two Ways to Build It

The authors suggest two ways to implement this, like choosing between a modular toolkit or a pre-assembled kit:

• Option A (The Toolkit): Keep the numbers and the labels separate. The robot sees 1, 2, 3, then k. It has to put them together itself. This keeps the dictionary small.
• Option B (The Pre-assembled Kit): Combine them into single blocks. The robot sees 123k as one single, unbreakable unit. This is faster for the robot to read and leaves zero room for confusion, though it requires a slightly larger dictionary.

The Bottom Line

This paper suggests that by simply changing how we "speak" numbers to AI—adding clear, labeled chunks instead of a stream of digits—we can make them much smarter at math and science without needing to rebuild their entire brains.

It's like realizing that to teach a child to read, you shouldn't just show them letters; you should show them words with clear meanings attached. With Triadic Suffix Tokenization, the AI finally gets the instruction manual for numbers.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) frequently fail at basic numerical reasoning tasks (e.g., confusing $9.11 > 9.9$) due to limitations in standard subword tokenization methods (like BPE).

• Fragmentation: Standard tokenizers split numbers into arbitrary subword units, destroying the positional and decimal structure of the number.
• Loss of Magnitude: Models must infer the order of magnitude (e.g., whether "100" represents 100, 100,000, or 100,000,000) solely from positional context, which is statistically inefficient and prone to error.
• Inconsistency: Existing solutions like right-to-left comma separation group digits but do not explicitly encode the magnitude of each group. Continuous encodings (like xVal) preserve smoothness but discard exact digit precision, making them unsuitable for tasks requiring exact arithmetic.

2. Methodology: Triadic Suffix Tokenization (TST)

The paper proposes Triadic Suffix Tokenization (TST), a deterministic scheme that partitions numbers into three-digit groups (triads) and annotates each group with an explicit magnitude marker.

Core Principles

1. Triadic Grouping: Digits are grouped in sets of three (base-1000).
2. Explicit Magnitude Annotation: Each triad is suffixed with a marker indicating its order of magnitude.
3. Exact Digit Preservation: Unlike continuous encodings, TST preserves the exact digits of the number.

Integer Part

Digits are grouped from right to left. Each triad receives a suffix corresponding to its power of 10:

• Suffixes: k (thousand, $10^3$), m (million, $10^6$), b (billion, $10^9$), t (trillion, $10^{12}$), q (quadrillion, $10^{15}$).
• Example: 1234567 becomes 1m 234k 567.
• Benefit: The suffix explicitly tells the model the scale of the preceding digits, removing the need for positional inference.

Fractional Part

Fractional digits are grouped from left to right with replicated markers to denote depth.

• Normalization: To ensure a 1:1 mapping between tokens and values, fractional triads are right-padded with zeros to a fixed length of three digits.
• Markers: p (tenths/thousandths), pp, ppp, etc., representing increasing decimal depth ($10^{-3}, 10^{-6}$, etc.).
• Example: 0.0045 becomes 0. 004p 500pp.
• Benefit: This ensures that numerically equivalent values (e.g., $0.1$, $0.10$, $0.100$) map to the exact same token sequence (0. 100p), eliminating surface-form ambiguity.

Implementation Variants

The paper proposes two ways to implement TST:

1. Option A (Separate Tokens): Keeps digit groups and suffixes as separate tokens (e.g., 123 + k).
   • Vocabulary Impact: Adds only ~10 new tokens (the suffixes themselves).
   • Trade-off: Longer sequence length; model must learn to combine digits with suffixes.
2. Option B (Compound Tokens): Creates combined tokens for each triad-suffix pair (e.g., 123k, 234m).
   • Vocabulary Impact: Adds up to 10,000 tokens (1000 triads $\times$ 10 suffix types).
   • Trade-off: Shorter sequence length; provides the model with pre-computed magnitude-digit units, eliminating ambiguity.

3. Key Contributions

• Explicit Inductive Bias: TST provides a stronger inductive bias for numerical reasoning by making magnitude relationships transparent at the token level, rather than forcing the model to learn them from scratch.
• Deterministic Boundaries: Unlike prefixes (which can create boundary ambiguity), suffixes clearly mark the end of a triad, allowing the model to know exactly when a magnitude group ends.
• Scalability: The scheme is architecture-agnostic and scalable. It currently covers 33 orders of magnitude ($10^{-15}$ to $10^{18}$) but can be extended indefinitely by adding new suffix tokens without altering the core logic.
• Orthogonality to Training: TST operates at the preprocessing/tokenization level. It is orthogonal to training-level improvements like Number Token Loss (NTL), meaning TST can be combined with NTL for synergistic effects.
• Drop-in Compatibility: It requires no changes to the model architecture, only a modified tokenizer and vocabulary expansion.

4. Results and Validation

• Experimental Status: The paper is a theoretical proposal; experimental validation is deferred to future work.
• Theoretical Analysis: The authors provide a comparative analysis (Table 2) suggesting TST offers the best balance of:
  • Exact Digit Preservation: (Like digit-level tokenization).
  • Explicit Magnitude Info: (Unlike digit-level or right-to-left commas).
  • Reasonable Sequence Length: (Better than pure digit-level, comparable to right-to-left).
• Hypothesis: The authors hypothesize that TST will lead to faster, more stable convergence and reduced inference errors by providing a consistent gradient signal for numerical values.

5. Significance

This work addresses a fundamental bottleneck in LLM numerical reasoning: the disconnect between token representation and mathematical reality.

• Solving the "9.11 > 9.9" Problem: By explicitly encoding magnitude, TST prevents models from misinterpreting decimal values based on string length or arbitrary subword splits.
• Precision vs. Smoothness: It bridges the gap between the precision required for exact arithmetic (which continuous encodings like xVal lack) and the efficiency of grouped tokenization.
• Practicality: As a "drop-in" preprocessing step, TST offers a low-cost, high-potential upgrade for any LLM requiring robust numerical capabilities, from scientific computing to financial analysis.

Future Work: The authors plan to validate TST on benchmarks like NumericBench and Number Cookbook, comparing it against existing methods (digit-level, xVal, right-to-left commas, NumeroLogic) to empirically confirm the theoretical advantages.