Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

1. Problem Statement

Large Language Models (LLMs) struggle with precise mathematical reasoning and arithmetic, often producing hallucinated or inaccurate results. While prior research has identified that LLMs represent numbers using sinusoidal structures (trigonometric patterns) in their internal embeddings, existing studies suffer from three critical limitations:

1. Scope Limitation: Previous findings were largely restricted to input embeddings or synthetic mathematical contexts, lacking validation in natural language settings.
2. Lack of Quantification: While similarities were observed, there was no rigorous quantification of how "universal" these representations are across different model families, sizes, and layers.
3. Inadequate Tools: General-purpose interpretability tools (e.g., linear probes, TunedLens) fail to capture the specific sinusoidal nature of numeric data, leading to underestimations of model accuracy and incorrect mechanistic conclusions.

The paper aims to quantify the universality of these sinusoidal representations, develop specialized tools to decode them, and demonstrate how leveraging this structure improves the understanding and correction of arithmetic errors.

2. Methodology

The authors employ a multi-faceted approach combining Representational Similarity Analysis (RSA), Fourier decomposition, and the design of specialized sinusoidal probes.

A. Quantifying Universality (Section 3)

• Cross-Model Alignment: The authors performed RSA on input embeddings of number tokens across eight LLMs (OLMo, Llama 3, Phi 4) of varying sizes. They compared number tokens against random word pieces to measure alignment.
• Frequency Analysis: They applied PCA followed by Fourier transforms to identify dominant frequencies in number representations. They calculated the Intersection-over-Union (IoU) of the top $k$ frequencies across model pairs.
• Layer Interchangeability: They tested whether representations learned on one layer could decode numbers in other layers (cross-layer generalization) to determine if the sinusoidal structure is maintained throughout the residual stream.

B. Developing Specialized Probes (Section 3.2)

To accurately decode internal states, the authors designed a new probe architecture, param-sin, which strictly enforces a sinusoidal inductive bias. Unlike previous linear or generic non-linear probes, this model assigns a single sine wave to each latent feature and optimizes:

• Frequency ($w_f$)
• Amplitude ($w_a$)
• Phase-shift ($w_\phi$)

The function is defined as:
$$f_{param-sin}(x) = S^T (W_{in}x)$$
Where $S$ is a matrix of sine waves sampled at evenly spaced points.

C. Experimental Setup

• Contexts: Unlike prior work using synthetic math commands, this study uses natural language contexts (temporal, arithmetic, medical, culinary) to ensure generalizability.
• Baselines: They compare param-sin against standard linear probes, log-probes, and the TunedLens method.
• Intervention: They perform activation steering on incorrect arithmetic predictions. By interpolating model activations toward the "patch" (embedding) that maximizes the sinusoidal probe's prediction of the correct integer, they attempt to mechanically correct errors.

3. Key Contributions

1. Universal Sinusoidal Representations: The paper quantifies that LLMs from different families and sizes converge to almost perfectly universal sinusoidal representations for numbers. Different models use the same dominant frequencies and topological structures.
2. Context-Aware Probing: The authors demonstrate that probes trained on natural language data generalize significantly better across contexts than those trained on synthetic math data. Synthetic probes fail to generalize to real-world distributions.
3. Layer-Wide Consistency: They show that these sinusoidal representations are not limited to input embeddings but are maintained and interchangeable across intermediate layers of the network, allowing a single probe trained on one layer to decode numbers in others.
4. Mechanistic Error Correction: They provide mechanistic evidence that the precision of sinusoidality directly correlates with arithmetic accuracy. By steering activations to align with the sinusoidal pattern, they successfully reduce arithmetic errors.
5. Extension to Ordinal Data: The sinusoidal mechanism is shown to generalize beyond integers to other ordinal data types (e.g., days of the week, months), validating the circular encoding hypothesis for temporal concepts.

4. Key Results

• Universality:
  • RSA Scores: Number tokens yield consistently higher RSA scores across different model families compared to random tokens.
  • Frequency Alignment: For the top $k=63$ Fourier frequencies, there is perfect agreement across all models for number representations, whereas random tokens show no such alignment.
• Probe Accuracy:
  • The param-sin probe achieves >90% accuracy in decoding numeric values from internal activations across most layers and models.
  • Standard linear probes and TunedLens significantly underperform, often failing to retrieve multi-token number information beyond 2-3 time steps, whereas the sinusoidal probe recovers information with 99% accuracy for immediate predecessors.
• Error Reduction via Steering:
  • By steering activations toward the expected sinusoidal pattern for incorrect predictions, the authors reduced multiplication errors by up to 30% and division errors by up to 42% in specific layers.
  • The param-sin probe consistently outperformed the previous state-of-the-art sin probe (Kadlčík et al., 2025) in these interventions.
• Ordinal Data:
  • The probe successfully decoded days of the week and months with high accuracy even with extremely small training sets (e.g., $n=6$ or $n=7$), confirming circular encoding for temporal data.

5. Significance and Implications

• Rethinking Interpretability: The paper argues that general-purpose interpretability tools are insufficient for numeric and ordinal data. Using linear tools on sinusoidal data leads to false negatives regarding model capabilities (e.g., underestimating the ability to track multi-token numbers).
• Mechanistic Understanding: It establishes that the "black box" of arithmetic in LLMs is not random but relies on a highly systematic, universal trigonometric mechanism. This allows researchers to track numeric processing throughout the entire model depth.
• Practical Application: The findings suggest that mechanistic interventions (like activation steering) can effectively improve LLM reliability in math tasks if the intervention respects the underlying sinusoidal structure.
• Future Directions: The work highlights the need for domain-specific explainability tools. It also points to limitations in current models regarding very large numbers (multi-token superposition limits) and the need to investigate non-English number systems (e.g., Chinese/Japanese myriad-based systems).

In conclusion, the paper provides a rigorous mathematical characterization of how LLMs handle numbers, proving that they learn a universal, sinusoidal "language" for arithmetic that is robust across models and contexts, provided the right tools are used to observe it.