Numbers Already Carry Their Own Embeddings

Technical Summary: Adelic Operation-Preserved Embeddings (AOE)

Problem Statement
Despite recent milestones in mathematical reasoning by large language models (LLMs), a fundamental bottleneck persists: the "number problem." Current architectures predominantly treat numbers as sequences of characters via tokenization schemes like Byte Pair Encoding (BPE). This approach fragments numerical values into arbitrary subword units, stripping away the intrinsic additive and multiplicative relationships that define numbers. Consequently, models exhibit brittle reasoning, inconsistent generalization, and systematic errors in tasks requiring arithmetic precision. While prior attempts have explored continuous embeddings or digit-wise positional encodings, these methods often rely on the hope that models will implicitly rediscover mathematical rules during training rather than embedding them directly.

Methodology: Adelic Operation-Preserved Embeddings (AOE)
The authors propose AOE, a training-free representation grounded in algebraic number theory. Instead of tokenizing numbers, AOE constructs embeddings within an Adele space, which unifies the real line with $p$-adic completions of the rational numbers.

• Mathematical Foundation: The construction leverages the Adele Ring ($\mathbb{A}_\mathbb{Q}$), representing a rational number $q$ as a tuple containing its real value ($a_\infty$) and its $p$-adic expansions ($a_p$) for a selected set of primes.
• Structure Preservation: By design, this representation preserves algebraic structure coordinate-wise. For any rationals $q_1, q_2$:
  • $A(q_1 + q_2) = A(q_1) \oplus A(q_2)$
  • $A(q_1 \cdot q_2) = A(q_1) \otimes A(q_2)$
    Here, $\oplus$ and $\otimes$ denote coordinate-wise operations on the concatenated real and $p$-adic vectors.
• Embedding Construction: To generate the embedding for a rational $q = n/m$:
  1. Real Component: Represented as a length-$N$ vector on the real line.
  2. $p$-adic Components: For a fixed set of primes, the authors compute $N$-digit $p$-adic expansions. This involves solving the congruence $mx \equiv n \pmod{p^N}$ using Hensel's Lemma to iteratively lift solutions from modulo $p$ to modulo $p^N$.
  3. Concatenation: The real vector and the truncated $p$-adic vectors are concatenated to form a tensor of shape $(n_p + 1) \times N$, where $n_p$ is the number of chosen primes.
• Integration: AOE is architecture-agnostic and plug-and-play. It replaces standard embedding layers in Transformers with pre-computed Adelic tensors. To preserve structural information, a lightweight 2D positional encoding is added, encoding the sequence position, prime index, and digit index.

Key Contributions

1. Proposition of AOE: A training-free, algebraically grounded representation that directly respects numerical structure without requiring task-specific retraining.
2. Addressing the "Number Problem": The work frames AOE as a principled remedy for the numerical representation bottleneck in AI, offering a path to overcome the limitations of token-based approaches.
3. Empirical Validation: The authors demonstrate that Transformers equipped with AOE outperform standard baselines on algebraic combinatorics benchmarks, achieving perfect accuracy on tasks where previous models failed.

Experimental Results
The authors evaluated AOE on the Algebraic Combinatorics Dataset (ACD), a benchmark designed for research-level conjecturing in algebraic combinatorics. The experiments compared a standard Transformer baseline (using trainable nn.Embedding and sinusoidal positional encodings) against an AOE-equipped Transformer (using fixed, pre-computed Adelic tensors).

• Performance: Across six classification tasks (including lattice paths, quiver mutation classes, and Schubert polynomials), the AOE model consistently outperformed the baseline.
• Notable Achievement: The AOE model achieved 100% accuracy on the Weaving Patterns benchmark (for $n=6$ and $n=7$), a task where the baseline achieved only ~53% and which has historically resisted large-scale language models.
• Generalization: Improvements ranged from modest gains (e.g., lattice paths) to dramatic jumps (e.g., Schubert polynomials and quiver mutation classes), with lower test losses indicating better model calibration.
• Efficiency: While AOE introduces computational overhead (training is roughly 4–5 times slower due to pre-computation and 2D positional encoding), the performance gains were achieved without increasing the number of learnable parameters; in fact, the baseline had more trainable parameters.

Significance and Claims
The paper posits that numbers inherently contain the structure necessary for their own representation. By making this structure explicit through AOE, neural models can achieve levels of reliability on symbolic mathematical tasks that elude even the largest LLMs.

The authors argue that the future of AI reasoning depends on embedding the mathematical structures of objects themselves, rather than merely the language used to describe them. AOE serves as a proof-of-concept that ideas from algebraic number theory can function as plug-and-play components in neural architectures, suggesting a path toward uniting symbolic rigor with neural scalability.

Limitations Acknowledged by the Authors

• Scope: The current framework is restricted to rational numbers ($\mathbb{Q}$) due to the definition of the Adele Ring. Extending this to irrationals, complex values, or transcendental constants requires new theoretical tools.
• Hyperparameters: The selection of primes and digit precision is currently heuristic and manual; more adaptive strategies are needed for efficiency.
• Computational Cost: The pre-computation and 2D positional encoding significantly increase training time compared to standard embeddings.