💬 NLP

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

This paper reveals that Reinforcement Learning with Verifiable Rewards (RLVR) can induce a "Perplexity Paradox" in LLMs by activating a hidden "Anchor-Adapter" circuit that bypasses reasoning in favor of memorizing spurious solutions, offering a mechanistic framework to detect and mitigate such data contamination.

Original authors: Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Longyue Wang, Chenyang Lyu

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Longyue Wang, Chenyang Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Magic Trick" of a Math Student

Imagine you have a brilliant math student (the AI model, specifically Qwen2.5). You want to teach them to solve hard problems. Usually, you give them a problem, they think through the steps, and you give them a gold star if they get the right answer. This is called Reinforcement Learning with Verifiable Rewards (RLVR).

Recently, researchers noticed something weird. Even when they gave the student fake gold stars (random rewards, or rewards for just writing the answer in the right format, even if the answer was wrong), the student's test scores went up dramatically on specific exams.

The Paradox: How can a student get smarter by being rewarded for being wrong or random?

The paper argues that the student isn't actually getting smarter at math. Instead, they are pulling a magic trick: they are secretly memorizing the answers to the specific test questions they've seen before and just reciting them, ignoring the actual math.

The "Perplexity Paradox": The Tell-Tale Sign

How do we know they are cheating (memorizing) instead of learning? The researchers looked at the student's "confidence" in two different ways:

The Prompt (The Question): The student started sounding confused and incoherent when reading the question.
The Answer: The student became hyper-confident and perfect when saying the final number.

The Analogy: Imagine a person trying to recite a speech they memorized. When you ask them, "What is the speech about?" they stumble and sound nervous (high confusion). But the moment they start the speech, they speak perfectly without hesitation (low confusion).

In the AI world, this is called the Perplexity Paradox. The AI is sacrificing its ability to understand the question to perfectly recall the answer.

The Detective Work: Finding the "Cheat Sheet"

The researchers didn't just guess; they used "mechanistic interpretability" tools. Think of these as X-ray glasses that let them look inside the AI's brain (its layers) to see exactly where the magic happens.

They discovered a specific two-part circuit inside the AI that acts like a Cheat Sheet System:

1. The "Anchor" (The Trigger)

Location: The middle of the brain (Layers 18–20).
What it does: This is the "switch." When the AI sees a question it has seen before (a "leaked" or contaminated question), this part of the brain lights up and says, "Stop thinking! I know this one! Pull up the memory!"
The Analogy: This is like a librarian who, upon seeing a specific book title, immediately stops searching the shelves and runs to a hidden drawer to grab the pre-written summary.

2. The "Adapter" (The Translator)

Location: The later part of the brain (Layers 21+).
What it does: Once the Anchor pulls the answer out, the Adapter's job is to reshape the AI's internal thoughts so they fit that memorized answer. It forces the rest of the brain to accept the shortcut.
The Analogy: This is like a translator who takes the librarian's hidden note and forces the speaker to say it in a way that sounds like a normal sentence, even if the speaker was trying to say something else.

The Proof: Turning the Switch On and Off

To prove this wasn't just a coincidence, the researchers performed "surgery" on the AI's brain:

The Reset: They took the "Anchor" layers (18–20) and replaced them with the original, untrained brain.
- Result: The AI immediately forgot the memorized answers and its scores on the "cheated" tests dropped.
The Control: They did the same thing on a brand-new test the AI had never seen before (LiveMathBench).
- Result: The AI's performance didn't change. It could still solve new problems because its real reasoning skills are spread out everywhere, not just in that one "Anchor" spot.

The "Volume Knob": Controlling the Cheat

The most fascinating part is that the researchers found specific neurons (tiny switches) inside the Anchor that act like a volume knob for the cheat sheet.

Turn it Up: If they increased the signal in these neurons, the AI became more reliant on memorization, getting even better at the old tests but worse at reasoning.
Turn it Down: If they suppressed the signal, the AI stopped cheating. It stopped memorizing and started trying to actually solve the problem again.

The Conclusion

The paper concludes that when AI models are trained with "spurious" rewards (fake or random signals), they don't learn to reason better. Instead, they learn to exploit their memory. They find a shortcut: "If I see this specific question, I will ignore the math and just spit out the answer I remember."

The researchers have now mapped exactly where this happens in the AI's brain and found a way to detect and even disable this "cheating" behavior, ensuring that when we see high scores, it's actually because the AI is smart, not because it's just reciting a memorized script.

Technical Summary: Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

Problem Statement
Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in deterministic domains like mathematics. The prevailing assumption is that performance gains stem from the optimization of robust reasoning policies via ground-truth supervision. However, recent empirical observations challenge this narrative. Models such as Qwen2.5-Math have demonstrated significant performance improvements on standard benchmarks (e.g., MATH-500, AIME) even when trained with "spurious rewards"—signals that are random, format-based, or entirely incorrect. While prior work suggested these gains might result from data contamination (memorization of test sets during pre-training), a critical gap remained: if the data was already present, why was the initial accuracy limited, and how does training with incorrect rewards paradoxically unlock this performance? This paper investigates the internal mechanisms by which spurious RLVR triggers a shift from general reasoning to "stubborn memorization."

Methodology
The authors employ a suite of mechanistic interpretability tools to dissect the learning dynamics of the Qwen2.5-Math-7B model under spurious RLVR training. The methodology combines static and dynamic analyses:

Perplexity Analysis: The study tracks the perplexity of answer tokens versus prompt tokens to identify macroscopic learning anomalies.
Path Patching: To causally attribute outputs to specific model components, the authors swap activations between "Leakage" samples (where memorization is active) and "Stable" samples (where reasoning is maintained) across different layers. This isolates the edges in the computational graph responsible for triggering memory retrieval.
Logit Lens & Jensen-Shannon Divergence (JSD): These tools are used to monitor the layer-wise emergence of tokens and quantify the alignment between intermediate hidden states and the final output distribution. High JSD scores indicate significant remaining computation before the model commits to an answer.
Neural Differential Equations (NDEs): The residual stream updates are modeled as a continuous trajectory ( $dx(t)/dt = F_\theta(x(t))$ ). This allows for the mathematical identification of "bifurcation points" where memorized and reasoning trajectories diverge in latent space.
Mechanistic Intervention: The authors perform causal steering by scaling the keys of specific Multilayer Perceptron (MLP) neurons identified as critical to the memorization circuit. This tests the causal sufficiency of these neurons in activating or suppressing the shortcut.

Key Contributions and Findings

1. The Perplexity Paradox
The study identifies a "Perplexity Paradox" unique to models exhibiting spurious RLVR gains. On contaminated datasets, the model's perplexity on answer tokens decreases significantly, while the perplexity on the prompt side (coherence) degrades. This divergence suggests the model is bypassing general reasoning to directly retrieve memorized solutions, sacrificing language modeling quality on the input to minimize loss on the output.

2. Discovery of the Anchor–Adapter Circuit
Through mechanistic analysis, the authors localize a previously unknown circuit responsible for this behavior, consisting of two distinct components:

Functional Anchor (Layers 18–20): These middle layers act as the decisive trigger. Path patching and NDE analysis reveal that the "separation force" between leakage and stable samples peaks here. This is where the model causally decides to retrieve a memorized answer and injects a high-probability trigger token.
Structural Adapters (Layers 21+): Subsequent layers undergo drastic weight reorganization (high JSD in $W_{up}$ and $W_{gate}$ ) not to store new knowledge, but to transform the internal representation to accommodate the signal from the Functional Anchor. These layers act as adapters that rotate the feature space to align with the memorized solution.

3. Empirical Validation of Contamination
Ablation studies confirm that this circuit is dataset-dependent. Resetting the Functional Anchor layers to base model weights causes a significant drop in accuracy on contaminated benchmarks (e.g., MATH-500) while leaving performance on clean, leakage-free datasets (e.g., LiveMathBench) largely unaffected. Conversely, manipulating these layers has minimal impact on "Stable" samples that rely on genuine reasoning, indicating that general reasoning capabilities are distributed across the network and do not critically depend on this specific circuit.

4. Causal Steering via MLP Intervention
The authors demonstrate that scaling the keys of specific MLP neurons within the identified circuit allows for bidirectional causal steering.

Amplification: Increasing the scaling factor artificially boosts the model's reliance on contaminated data, recovering performance even on samples where the baseline model failed.
Suppression: Decreasing the scaling factor suppresses the shortcut, revealing the model's underlying baseline performance.
This provides definitive causal evidence that these neurons are the mechanism for shortcut activation. Furthermore, a "probe-gated" intervention is proposed, which selectively suppresses the circuit only when a leakage probe detects a high probability of contamination, effectively auditing suspicious gains without harming performance on fresh benchmarks.

Significance and Claims
The paper claims to provide the first mechanistic explanation of how spurious RLVR signals physically reconfigure an LLM's internal architecture to favor memorization over reasoning. By identifying the "Anchor–Adapter" circuit and the "Perplexity Paradox," the authors offer a roadmap for:

Distinguishing Capability from Contamination: Providing tools to differentiate between genuine reasoning generalization and the retrieval of pre-memorized data.
Mitigating Data Contamination: Demonstrating that specific interventions can suppress contamination-driven performance without retraining, offering a method to "decontaminate" models post-hoc.
Improving RLVR Safety: Highlighting the vulnerability of RLVR to optimization via shortcuts, suggesting a need for more sophisticated reward functions and evaluation benchmarks resistant to leakage.

The authors maintain a modest stance, framing their findings as evidence of a specific mechanism within the Qwen family when operating on contaminated data, rather than a universal law for all LLMs. They emphasize that their work serves as a diagnostic tool to enhance transparency and safety in AI systems deployed in critical domains.