Negative Pre-activations Differentiate Syntax

This paper demonstrates that negative pre-activations in a sparse subpopulation of Wasserstein neurons serve as an active and essential substrate for syntactic processing in modern large language models, distinct from their role in other capabilities.

The Gist

Imagine a Large Language Model (LLM) as a massive, bustling factory that writes stories, answers questions, and solves problems. Inside this factory, there are millions of tiny workers called neurons.

For a long time, scientists studying these factories had a simple rule: "If a worker is shouting (positive activation), they are doing something important. If they are whispering or silent (negative activation), they are probably just taking a break." This rule came from older models that used a "switch" (called ReLU) which literally turned off any worker who wasn't shouting.

But modern factories use a different kind of worker who can whisper and shout at the same time. They use smooth, flowing functions (like GELU or SiLU) that allow negative numbers to carry information just as well as positive ones.

The Big Discovery:
This paper, "Negative Pre-Activations Differentiate Syntax," argues that we were wrong to ignore the whispers. The authors found that a very small, special group of workers (called Wasserstein neurons) uses these "whispers" (negative pre-activations) to do the most critical job in the factory: keeping the grammar correct.

Here is the breakdown using simple analogies:

1. The "Whispering" Specialists

Imagine the factory has a few elite specialists. Most workers shout loudly to move heavy boxes (positive activation). But these elite specialists have a secret superpower: they use whispers to organize the blueprint of the sentence.

The authors found that in modern models, these specialists don't just sit idle when they have negative numbers. Instead, they use the depth of the whisper to tell the difference between two very similar words.

• Analogy: Think of two similar-looking keys. A normal worker might just say "Key" for both. But these specialists whisper, "This key is a soft whisper (deep negative)" and "That key is a hard whisper (shallow negative)." Even though both are whispers, the difference in the whisper tells the machine exactly which grammatical rule to apply.

2. The "Grammar Glue"

The paper tested what happens if you stop these specialists from whispering. They didn't turn the workers off completely; they just clamped their mouths shut whenever they tried to whisper (zeroing out the negative pre-activations).

• The Result: The factory didn't just stumble; it collapsed.
  • Grammar: The model suddenly forgot how to make sentences. It couldn't agree on singular vs. plural (e.g., "The dog are running" instead of "The dog is running"). It forgot how to use "who" vs. "whom."
  • Other Skills: Interestingly, the model could still answer trivia questions, tell jokes, or solve logic puzzles almost as well as before.
  • The "Double Dissociation": This is a fancy way of saying: If you stop the whispers, you break the grammar but save the trivia. If you stop the shouting of regular workers, you break the trivia but save the grammar. This proves that the "whispers" are the specific glue holding the sentence structure together.

3. The "Early Warning System"

The authors also looked at when these specialists learn to whisper.

• Analogy: Imagine the factory is being built from scratch. The "grammar whisperers" show up and start working very early in the construction process. Once they are set up, they stabilize and become the foundation. If you try to remove them later, the whole structure wobbles.
• The paper shows that as the model gets smarter, it relies more on these negative whispers for grammar, not less.

4. Why This Matters

Before this paper, many researchers thought negative numbers in AI were just "noise" or a side effect of how the math worked. They were like the background hum of a factory that you ignore.

This paper says: "No! That hum is the blueprint!"

It turns out that in modern AI, the "negative" part of the brain is actively doing the heavy lifting for sentence structure. It's not just a leftover from old technology; it's a deliberate, sophisticated tool used to separate similar words and keep the grammar tight.

The Takeaway

If you think of a language model as a symphony orchestra:

• Positive activations are the loud instruments (trumpets, drums) playing the main melody.
• Negative activations were thought to be the quiet instruments just sitting there.
• This paper reveals that the quiet instruments (the whispers) are actually playing the complex sheet music that keeps the whole orchestra in time. If you mute the whispers, the music falls apart into noise, even if the loud instruments are still playing.

In short: The "negative" side of the brain is essential for grammar, and we need to start listening to the whispers, not just the shouts.

Technical Summary

1. Problem Statement

Modern Large Language Models (LLMs) predominantly utilize smooth activation functions like GELU and SiLU, which allow for non-zero outputs and gradients in the negative pre-activation region. Historically, interpretability research has focused almost exclusively on large positive activations, implicitly treating negative values as "inactive" or merely a byproduct of optimization (a heuristic inherited from ReLU-based architectures where negative values are clamped to zero).

The paper challenges this assumption, asking: Do negative pre-activations in smooth-activation models carry functional signal, and if so, what specific role do they play? The authors hypothesize that negative regions are not inert but are actively leveraged for specific computational tasks, potentially distinct from the tasks handled by positive activations.

2. Methodology

The authors employ a multi-faceted approach combining neuron selection, causal intervention, and benchmark evaluation.

A. Identification of "Wasserstein Neurons"

• Definition: The study focuses on a sparse subpopulation of neurons termed Wasserstein neurons. These are neurons whose output pre-activation distributions exhibit a large Wasserstein Distance (WD) from a standard Gaussian baseline.
• Metric: They calculate WD on the normalized output distribution of neurons. High WD correlates with high Mapping Difficulty (MD), a metric quantifying how far apart a neuron maps similar input vectors (a form of "entanglement" where one neuron separates similar inputs).
• Observation: In non-ReLU models (e.g., Pythia, Llama, Mistral), the deviation from Gaussianity in these neurons concentrates specifically in the negative pre-activation region.

B. Causal Intervention (Sign-Specific Ablation)

To test causality, the authors perform targeted interventions on the gate projection (in GLU-style models like Llama) or up projection (in GPT-2 style models like Pythia) of the MLP blocks:

• Targeted Ablation: They zero out only the negative pre-activations of the top 1% of Wasserstein neurons (those with the highest WD) while leaving positive pre-activations and all other neurons untouched.
• Control Conditions:
  1. Random Ablation: Zeroing negative pre-activations of randomly selected neurons.
  2. Perplexity-Matched Ablation: Zeroing negative pre-activations of a much larger set of low-WD (non-entangled) neurons until the model's overall perplexity matches that of the targeted ablation. This controls for general capacity loss.

C. Evaluation Protocols

• Syntactic Benchmarks: BLiMP (Benchmark of Linguistic Minimal Pairs) and TSE (Targeted Syntactic Evaluation) to measure grammatical competence.
• Non-Grammatical Benchmarks: A suite of reasoning and comprehension tasks (ARC, BoolQ, HellaSwag, PIQA, SciQ, TruthfulQA, WinoGrande) to measure general capabilities.
• Training Dynamics: Analysis of Pythia models across training checkpoints to observe the emergence of these neurons.

3. Key Contributions

1. Discovery of Negative Computation: The paper provides the first causal evidence that negative pre-activations in smooth-activation LLMs are not inert but serve as an active substrate for computation.
2. Identification of a Syntax-Specific Substrate: It identifies a sparse set of "Wasserstein neurons" that uniquely rely on negative pre-activations to differentiate syntactic structures.
3. Double Dissociation: The study establishes a clear double dissociation between syntax and general reasoning capabilities based on the sign and location of ablation.
4. Mechanism of "Negative Differentiation": It reveals a specific mechanism where these neurons separate similar inputs by pushing both into the negative region but to different depths (e.g., $-0.5$ vs $-2.0$), rather than separating positive from negative values.

4. Key Results

A. Disproportionate Impact on Grammar

• Targeted Ablation: Zeroing the negative pre-activations of just 1% of Wasserstein neurons causes a massive spike in perplexity and a sharp degradation in grammatical accuracy on BLiMP and TSE (e.g., TSE accuracy drops from ~84% to ~55% in Llama 3.1 8B).
• Control Comparison: To achieve the same perplexity increase using non-entangled (low-WD) neurons, one must ablate 20% to 50% of the network. Crucially, even when perplexity is matched, ablating low-WD neurons does not significantly harm grammatical performance, whereas ablating Wasserstein neurons does.

B. Double Dissociation

• Syntax vs. Reasoning:
  • Wasserstein Ablation: Catastrophic for syntax; mild/moderate impact on non-grammatical benchmarks.
  • Low-WD (Perplexity-Matched) Ablation: Minimal impact on syntax; severe degradation on non-grammatical benchmarks (reasoning, commonsense).
• This confirms that the negative pre-activation region of Wasserstein neurons is specifically and causally necessary for syntax, distinct from general model capacity.

C. Token-Level Localization

• Analysis of "surprisal" (negative log-likelihood) shows that the added confusion from Wasserstein ablation is concentrated on syntactic scaffolding tokens: determiners, prepositions, auxiliaries, and punctuation.
• Content words (nouns, verbs, adjectives) are less affected, suggesting these neurons build the structural framework of sentences.

D. Layerwise and Training Dynamics

• Early Layer Specialization: The negative differentiation mechanism is most prominent in early layers (specifically Layer 2). Damage compounds as it propagates through depth.
• Emergence: Wasserstein neurons emerge and stabilize rapidly during training (within the first 50B tokens), tracking the acquisition of grammatical competence. As these neurons mature (their distributions become more non-Gaussian), the model becomes increasingly dependent on their negative pre-activations for grammar.

E. The "Negative Differentiation" Mechanism

• Case studies show that these neurons separate similar inputs (e.g., "for" vs. "the") not by mapping one to positive and one to negative, but by mapping both to distinct negative values.
• Sign Sensitivity: Experiments flipping the sign of negative post-activations (while preserving magnitude) cause even more damage than zeroing them, proving that the sign itself carries essential information, not just the magnitude.

5. Significance and Implications

• Reframing Interpretability: The paper challenges the "ReLU-era" intuition that "activity" equals "positive pre-activation." It argues that interpretability methods must account for the full activation landscape, including negative regions.
• Mechanistic Understanding of Syntax: It provides a concrete mechanistic explanation for how syntax is computed in modern transformers: via a sparse, entangled subpopulation of neurons using negative differentiation in early layers.
• Model Robustness: Understanding that syntax relies on a specific, sparse negative region suggests that models might be more fragile to specific sign-based perturbations than previously thought, and that "dying neurons" in smooth activations might still be functionally active in the negative domain.
• Future Directions: The findings open new avenues for sparse autoencoders and interpretability tools to specifically target negative activation patterns to uncover syntactic features.

In summary, the paper demonstrates that negative pre-activations are a functional, sign-specific substrate for syntax in modern LLMs, localized to a sparse set of entangled neurons that emerge early in training.