💬 NLP

Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs

This paper demonstrates that arithmetic computation in LLMs relies on a shared, form-invariant set of "heuristic neurons" across symbolic, natural language, and code formats, revealing that cross-format failures stem from activation states rather than distinct internal circuits.

Original authors: Sharath Naganna, Tanvir Ahmed Sijan, Uddipta Kalita

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Sharath Naganna, Tanvir Ahmed Sijan, Uddipta Kalita

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to do math. You might think that if the robot learns to add numbers using a simple equation like "4 + 5 = 9," it would automatically know how to solve the same problem if you told it a story ("Tom has 4 apples, Alice gives him 5...") or asked it to write a computer program to do it. But here's the twist: these robots, known as Large Language Models (LLMs), are surprisingly fickle. They might get the answer right in the story but fail miserably when you ask for the code, even though the math is exactly the same. This isn't just a glitch; it's a mystery about how these models "think" inside their digital brains.

To understand this, we need to peek under the hood. Think of an LLM not as a single giant brain, but as a massive city of tiny workers called "neurons." When the model solves a problem, specific groups of these workers light up to do the heavy lifting. Scientists call this "mechanistic interpretability"—basically, reverse-engineering the machine to see which specific gears are turning to make it work. The big question researchers have been asking is: When the robot switches from doing math with symbols to doing it with words or code, does it recruit a whole new team of workers? Or is it the same team of workers, just getting confused because the instructions look different?

This paper sets out to solve that mystery by treating the robot's brain like a detective case. The researchers took three different versions of the same math problem—one written as a standard equation, one as a Python computer program, and one as a natural language word problem—and watched which neurons fired up in three different Llama-3 models. They found that the robot isn't using different teams for different formats. Instead, there is a tiny, compact "core crew" of about 50 to 100 neurons per layer that does the actual math, no matter how you ask the question.

Here is the most exciting part of their discovery: when the robot fails to solve a problem in one format (like code) but succeeds in another (like a word problem), it's not because it lacks the right tools. It's simply because the "core crew" isn't getting the right signal to start working. The researchers proved this by performing a digital "brain transplant." They took the activation signals from the successful word-problem run and pasted them onto the failed code run. Suddenly, the robot that was stuck started solving the code problem correctly more than 97% of the time for addition and subtraction!

This suggests that the robot's failure isn't due to a lack of knowledge or a broken circuit; it's just a case of the right neurons being in the wrong mood. The study shows that these "arithmetic heuristic neurons" are "form-invariant," meaning they are the same reliable workers regardless of whether the math is presented as symbols, stories, or code. They consistently belong to the same "families" of strategies, like checking if numbers are in a certain range or looking for specific patterns. So, while the robot might seem confused by different ways of asking the same question, its internal math engine is actually remarkably consistent and robust, waiting only for the right spark to get the job done.

Technical Summary: Are Arithmetic Heuristic Neurons Form-Invariant?

Problem Statement
Large Language Models (LLMs) frequently succeed on one formulation of a mathematical problem while failing on an equivalent formulation (e.g., solving a symbolic equation but failing a natural language word problem with the same operands). While recent mechanistic interpretability studies suggest that arithmetic in LLMs emerges from a "bag of heuristics" encoded by sparse MLP neurons, it remains unclear whether these internal circuits are specific to a particular input format (symbolic, natural language, or code) or if they are form-invariant. Specifically, it is unknown whether cross-format performance disparities arise from distinct internal circuits or merely different activation states of a shared circuit.

Methodology
The authors conducted a mechanistic analysis across three Llama-3 models (8B, 3.2-3B, and 3.2-1B), focusing primarily on the Llama-3-8B. The study utilized three semantically equivalent prompt formats: symbolic arithmetic, Python code, and natural language word problems.

Neuron Identification: The authors employed a two-stage pipeline to identify arithmetic heuristic neurons:
- Stage 1 (Attribution Patching): Used to efficiently rank the top 2,000 candidate neurons per layer based on their indirect effect on the output probability.
- Stage 2 (Activation Patching): Used to rerank the top candidates using exact causal interventions, retaining the top 200 neurons per layer for analysis.
- Layer Selection: Analysis was restricted to layers $\ell \ge 16$ and the final (answer) token, where linear probes confirmed the reliable emergence of arithmetic computation.
Circuit Validation: To determine if the identified neurons form a functional circuit, the authors performed:
- Keep-Only Interventions: Ablating all non-shared neurons to test sufficiency.
- Knockout Interventions: Ablating only the shared neurons to test necessity.
- Faithfulness Metric: Measured the fraction of the performance gap recovered relative to the full model and a completely ablated model.
Cross-Format Activation Transfer: The authors transferred activations of shared neurons from a successful execution in one format (donor) to a failed execution in another format (target) to determine if failures stem from activation states rather than circuit differences.
Heuristic Classification: Shared neurons were classified into heuristic families (e.g., operand range, result modulus, pattern matching) to assess functional consistency across formats.

Key Results

Compact Shared Circuit: The intersection of top-ranked arithmetic neurons across symbolic, code, and word problem formats is non-monotonic, peaking at approximately 50–100 neurons per layer. This indicates a compact core circuit shared across formats, while lower-ranked neurons are increasingly format-specific.
Necessity and Sufficiency:
- Sufficiency: Retaining only the shared circuit (Keep-Only) recovered nearly all arithmetic performance for addition and subtraction (approaching the full-model ceiling) with 100–500 neurons per layer.
- Necessity: Ablating only the shared circuit (Knockout) caused a rapid degradation in faithfulness, whereas ablating an equally sized random set of neurons had negligible effect.
Activation State Recovery: Transferring shared neuron activations from a successful donor format to a failed target format recovered over 97% of incorrect predictions for addition and subtraction, and 80–90% for multiplication and division. Random neuron transfers yielded near-zero recovery. This confirms that cross-format failures arise from differences in activation states, not distinct circuits.
Functional Consistency: Shared neurons consistently belong to the same heuristic families across all three formats. The pairwise Jaccard similarity of heuristic assignments averaged 0.75, and the three-way similarity averaged 0.64, significantly exceeding chance.

Significance and Claims
The paper claims to demonstrate that arithmetic computation in LLMs is largely form-invariant at the neuron level. The findings suggest that:

Mathematically equivalent problems across symbolic, natural language, and code formats recruit the same sparse set of causal arithmetic heuristic neurons.
Performance variance across formats is primarily driven by the activation states of this shared circuit rather than the recruitment of distinct, format-specific circuits.
Mechanistic circuits identified in one representation (e.g., symbolic arithmetic) generalize across input modalities.

The authors maintain a modest scope, noting that their analysis is limited to elementary integer arithmetic in base Llama models and focuses exclusively on MLP neurons at the final token. They acknowledge that extending these findings to complex reasoning, instruction-tuned models, or attention-based routing mechanisms remains a direction for future work.

Technical Summary: Are Arithmetic Heuristic Neurons Form-Invariant?

More like this