🤖 machine learning

Transformers Learn the Mestre-Nagao Heuristic

This paper demonstrates that a two-layer transformer trained on Frobenius traces of rational elliptic curves not only achieves near-perfect accuracy in predicting their rank but also mechanistically learns the Mestre-Nagao heuristic from analytic number theory, revealing that the model's internal circuitry and attention mechanisms effectively encode deep mathematical properties like $\log L(E,1)$ despite the absence of explicit number-theoretic features in the input.

Original authors: Pranav Venkata Konda

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pranav Venkata Konda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, mysterious library of numbers. Inside this library are thousands of "elliptic curves," which are fancy mathematical shapes that behave like secret codes. Mathematicians have a big question about these shapes: Are they "flat" (Rank 0) or do they have a "twist" (Rank 1)?

For decades, figuring this out has been like trying to hear a whisper in a hurricane. The clues are hidden in a long list of numbers called "Frobenius traces."

This paper is about a team of researchers who taught a computer (a "Transformer," which is a type of AI) to listen to these whispers and guess the rank. But they didn't just want the computer to be right; they wanted to peek inside the computer's brain to see how it figured it out.

Here is the story of what they found, explained simply:

1. The Super-Student

The researchers trained a small, two-layer AI brain on 60,000 of these curves. They gave it the first 128 numbers from each curve's secret code.

The Result: The AI became a genius. It got over 99% accuracy.
The Surprise: It didn't just memorize the answers. When they tested it on curves it had never seen before (and that didn't look like anything in its training), it still got it right. This means it actually learned a rule, not just a trick.

2. The "Prime" Detective

When the researchers looked at which numbers the AI paid attention to, they found something magical.

The AI ignored most numbers.
It focused intensely on Prime Numbers (numbers like 2, 3, 5, 7, 11 that can't be divided by anything else).
The Analogy: Imagine a detective looking at a crime scene. Instead of looking at every single footprint, the detective only looks at the footprints of the suspects wearing specific shoes. The AI realized that the "Prime" footprints held all the clues, while the "Composite" (non-prime) footprints were just noise.

3. The "Ghost" in the Machine (The Real Discovery)

This is the most exciting part. The researchers used special tools to reverse-engineer the AI's logic. They wanted to know: What formula did the AI invent?

They found that the AI had independently rediscovered a famous, 40-year-old mathematical rule called the Mestre-Nagao Heuristic.

The Rule: This rule is a specific recipe for mixing the prime numbers together to guess the rank. It says: "Take the prime numbers, weigh them by a specific formula involving logs, and add them up."
The Miracle: The AI learned this complex, abstract math formula purely from the raw data, without anyone telling it the formula exists. It's like teaching a child to cook by giving them ingredients, and they accidentally invent a famous French recipe on their own.

4. How the Brain Works: The "Push-Pull" Team

Inside the AI, there are 512 little "neurons" (think of them as tiny workers). The researchers found that only 20 of these workers were doing the heavy lifting.

The Team: 17 of these workers were "Rank 0 Detectives." They would shout "It's Rank 0!" if the numbers looked right.
The Twist: The other 3 workers were "Rank 1 Detectives," but they were weird. They rarely shouted "It's Rank 1!"
The Analogy: Imagine a voting system.
- If the "Rank 0" team pushes hard, the result is "Rank 0."
- If the "Rank 0" team stays silent and does nothing, the result is automatically "Rank 1."
- The AI doesn't actively pull for Rank 1; it just waits for the Rank 0 team to stop pushing.

5. The "Readout" Glitch

Here is a funny flaw the researchers found.

The 20 smart workers knew the answer perfectly (99% accuracy).
But the "manager" neuron that reads their votes was a bit clumsy. It didn't listen to the smartest workers perfectly. It was like a manager who ignored the best employees and listened to the ones who were just standing around.
Because of this, the final score was slightly lower (95%) than what the smart workers could have achieved on their own. The AI knew the answer, but its "voice" was a little muffled.

6. The "Attention" vs. "Action" Mismatch

The researchers also noticed something strange about how the AI "looked" at the data.

What it looked at: The AI's "gaze" (attention) was strongest on small primes like 11 and 13.
What actually mattered: When they tested which numbers actually changed the answer, the most important numbers were different (like 31 and 13).
The Lesson: Just because the AI is staring at something doesn't mean that thing is the most important. It's like a driver staring at the rearview mirror (paying attention) while the real danger is in the side mirror (causal importance).

Summary

The paper shows that a simple AI, fed only raw numbers, figured out a deep, complex mathematical secret (the Mestre-Nagao heuristic) on its own. It built a tiny, efficient team of 20 "detectives" to solve the puzzle. While the AI's "manager" was slightly clumsy, the core logic was a perfect, independent rediscovery of a classic math theorem.

In short: The AI didn't just guess; it learned the language of numbers and spoke a theorem that mathematicians had written decades ago.

Technical Summary: Transformers Learn the Mestre-Nagao Heuristic

Problem Statement
The paper addresses the challenge of predicting the analytic rank of rational elliptic curves $E/\mathbb{Q}$ from their sequence of normalized Frobenius traces $\{a_n\}$ . According to the Birch and Swinnerton-Dyer (BSD) conjecture, the rank corresponds to the order of vanishing of the $L$ -function $L(E, s)$ at $s=1$ . While theoretically determined by the trace sequence, numerically detecting the vanishing of $L(E, 1)$ is difficult due to the slow convergence of the approximate functional equation, particularly for curves with high conductors. Previous machine learning work has demonstrated high accuracy in rank prediction but has not elucidated the internal algorithms the models discover. This study seeks to reverse-engineer the specific algorithm learned by a transformer encoder trained on this task using mechanistic interpretability.

Methodology
The authors trained a two-layer transformer encoder (with 4 attention heads and 512 neurons per layer) to classify elliptic curves of conductor $\le 10,000$ as rank 0 or rank 1. The input consisted of the first 128 normalized Frobenius traces ( $\tilde{a}_n = a_n / 2\sqrt{n}$ ). The dataset comprised 62,298 curves from the LMFDB, split into 80/20 training/testing sets. To ensure the model learned intrinsic arithmetic properties rather than dataset artifacts, the authors verified performance on test curves with no isogeny or quadratic-twist relatives in the training set.

To uncover the learned algorithm, the authors employed a suite of mechanistic interpretability techniques:

Solution Space Analysis: Comparing 50 independently trained models in weight space versus function space to determine if they converge to a single functional solution.
Attention Analysis: Examining CLS token attention weights to identify which input positions (primes vs. composites) the model prioritizes.
Activation Patching: Causally intervening on specific positions to determine which inputs are necessary for correct rank prediction, distinguishing between attention focus and causal information flow.
Linear Probing & Circuit Analysis: Identifying sparse subsets of neurons in the Layer-1 MLP that contain sufficient information for rank prediction and analyzing their input weights and firing patterns.
Logit Attribution: Decomposing the output logit to understand how different components (attention heads vs. MLP neurons) contribute to the final decision.

Key Results

Performance: The model achieved >99% accuracy on both rank classes. Crucially, performance remained stable on test curves lacking isogeny or twist relatives in the training set, ruling out "twist-redundancy" as the source of accuracy.
Function Space Convergence: Despite large barriers in weight space (models are not linearly mode-connected), the 50 trained models occupy a single cluster in function space with >98.8% pairwise agreement, indicating they learn the same underlying function.
Attention vs. Causality: While Layer-0 attention heavily concentrates on prime positions (consistent with the Euler product structure), activation patching reveals a dissociation: the most causally important primes (e.g., $p=31$ ) differ from those receiving the most attention (e.g., $p=11, 13$ ).
Sparse Circuit Discovery: A linear probe on just 20 out of 512 Layer-1 MLP neurons achieves an AUROC of 0.992. These neurons form a "push-pull" architecture: 17 act as rank-0 detectors and 3 as rank-1 detectors.
Mestre-Nagao Heuristic: The input weights of the top discriminating neurons (specifically rank-0 detectors) match the Mestre-Nagao sum weights $w_p = \frac{\log p}{p \cdot \log B}$ with a Spearman correlation of $r=0.997$ and Pearson correlation of $r=0.952$ . The model independently rediscovered this classical analytic number theory result from raw trace data.
Readout Sub-optimality: The model's internal readout weights are sub-optimal. While the 20-neuron circuit contains high-fidelity information (AUROC 0.992), the model's own readout weights extract only an AUROC of 0.956 from the same neurons, indicating a mismatch between the discriminative circuit and the readout pathway.
L-value Encoding: The CLS embedding encodes $\log L(E, 1)$ with $R^2 \approx 0.962$ (after controlling for conductor), suggesting the classification task implicitly optimizes for the $L$ -value.

Significance and Claims
The paper claims to provide the first mechanistic identification of a transformer neural network learning a named mathematical result (the Mestre-Nagao heuristic) from number-theoretic data without supervision. The study demonstrates that:

Transformers can learn complex analytic number theory heuristics directly from raw data.
The learned algorithm is implemented via a sparse, interpretable circuit of neurons that functions as a thresholded estimator of the $L$ -value.
Attention mechanisms in these models can be misleading regarding causal importance, as the most attended positions are not necessarily the most causally impactful.
Even highly accurate models may possess sub-optimal readout mechanisms, where the internal representation is more discriminative than the final classification head utilizes.

The authors conclude that the model effectively implements a "push-pull" detector where rank-1 is signaled by the absence of a rank-0 push rather than a direct negative pull, and that the ensemble of these sparse detectors explains the near-complete encoding of the $L$ -value in the model's latent space.