💬 NLP

The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model

This paper demonstrates that RLHF achieves functional political neutrality in Llama 3.1 8B not by erasing underlying partisan structures, but by severing the causal pathway from these intact internal representations to the model's output, thereby creating a fragile alignment that can be bypassed through specific steering techniques.

Original authors: Wendy K. Tam

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wendy K. Tam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Neutrality" Mask, Not a New Brain

Imagine you have a very smart, well-read person (the Base Model) who has read every book, news article, and tweet ever written. This person has strong opinions. If you ask them about politics, they might immediately sound like a passionate Democrat or a fierce Republican, depending on the topic. They have a deep, internal "compass" that points strongly in different directions.

The researchers wanted to know: What happens when we train this person to be "neutral" and helpful?

The paper argues that the training process (called RLHF) doesn't actually erase the person's political compass or change their brain. Instead, it puts a mask on them. The person still has all those political opinions and internal directions, but they have been trained to ignore them and speak in a boring, balanced, "both-sides" way.

If you take the mask off (or trick the person into thinking they are in a specific context), their original political compass is still there, ready to spin.

How They Tested This: The "Political GPS"

To prove this, the researchers used a tool like a Political GPS.

The Map: They first mapped out a specific direction in the model's "brain" (mathematically speaking) that represents the difference between "Republican" and "Democrat."
The Test: They asked the "Base Model" (before training) and the "Instruct Model" (after training) the same 84 questions, ranging from "How to cook a steak" to "Is abortion legal?"

The Results:

The Base Model: When asked about politics, its internal GPS needle swung wildly. Sometimes it pointed far left, sometimes far right. Its answers matched the swing (e.g., sounding very progressive or very conservative).
The Instruct Model: When asked the same questions, its internal GPS needle barely moved. It stayed stuck in the middle. Its answers were perfectly balanced, listing arguments for both sides without picking a winner.

The Surprise: The researchers expected the training to have removed the political compass. Instead, they found the compass was still there; it was just being held very still by a strong hand.

The "Light Switch" Analogy: What's Actually Happening?

The paper uses a clever analogy involving light switches (or features) inside the model's brain.

Before Training (Base Model): The brain has many light switches. Some switches control "Cooking," some control "History," and some control "Political Rhetoric." When you ask a political question, the "Political Rhetoric" switches flip on, and the model speaks with a partisan voice.
After Training (Instruct Model): The researchers found that the Political Rhetoric switches are completely turned OFF. They are dark. The model isn't using them at all.
The Twist: The model isn't using many switches at all. It's only using a tiny, specific set of switches that control "Formal, Boring, Balanced Speech."

The "Offset" Mystery:
The researchers noticed the Instruct Model's GPS needle wasn't exactly at zero; it was slightly tilted toward the "Republican" side. They thought, "Aha! It's still biased!"
But when they looked closer, they realized this tilt wasn't caused by political opinions. It was caused by the style of the answers. The model was trained to speak in a very formal, structured way (using lists, citations, and formal tone). It turns out that in the data used to train the model, Republicans used this formal style more often than Democrats. So, the "formal style" switch accidentally nudged the political needle slightly right, even though the model wasn't saying anything political.

The "Remote Control" Experiment

To prove that the political compass was still there but just disconnected, the researchers played a game of Remote Control.

They took the "Base Model" and the "Instruct Model" and used a remote control to force the "Political Rhetoric" switches to turn on (this is called steering).

Base Model: When they forced the switch on, the model immediately started spouting strong political opinions. It worked perfectly.
Instruct Model: When they forced the exact same switch on, the model did not change its answer. It kept giving the same balanced, neutral response.

What this means: The "wiring" for political opinions is still inside the Instruct Model's brain. But the "circuit breaker" that connects those wires to the mouth (the output) has been cut. The model knows the political arguments, but it has been trained to refuse to speak them.

The Conclusion: A Fragile Neutrality

The paper concludes that the "neutrality" we see in AI today is functional, not structural.

Functional Neutrality: The AI acts neutral because it's following rules. It's like a actor who has memorized a script to always sound polite.
Structural Neutrality: The AI would be neutral because it literally doesn't have the capacity to be biased. (The paper says this is not what happened).

The Risk: Because the internal political compass is still intact, the "mask" is fragile. If someone knows how to bypass the rules (for example, by tricking the AI into thinking the user wants a specific political opinion, or by using a "jailbreak" prompt), the model can instantly drop the mask and reveal its underlying partisan structure.

In short: The AI hasn't been "re-educated" to lose its political bias. It has just been taught to wear a mask of neutrality. The bias is still there, waiting behind the mask, ready to come out if the mask slips.

Technical Summary: The Neutral Mask

Problem Statement
The primary ambition of Reinforcement Learning from Human Feedback (RLHF) is to align large language models (LLMs) with human values, rendering them safe and useful. However, the mechanism by which RLHF encodes these values remains opaque. A growing body of evidence suggests that alignment training often results in "functional compliance" rather than deep structural alignment, potentially leaving underlying value-laden structures intact while merely suppressing their expression. This paper investigates this phenomenon through the lens of partisan political orientation, addressing the question of whether RLHF removes partisan biases from a model's internal representations or merely disconnects them from the output generation pipeline.

Methodology
The study employs a mechanistic case study comparing the internal representations of the Llama 3.1 8B model in two states: the Base model (pre-trained on massive text corpora) and the Instruct model (fine-tuned with RLHF). The methodology involves three primary technical components:

Partisan Probing: Utilizing a linear probe trained on 190,491 congressional tweets, the authors identify a specific direction ( $\hat{\omega}$ ) in the Layer 18 hidden-state activation space where partisan identity is linearly separable. This probe is applied to both the Base and Instruct models across 84 diverse prompts (political, historical, scientific, and non-political) to measure partisan scores.
Sparse Autoencoder (SAE) Decomposition: A TopK sparse autoencoder is trained on the Layer 18 hidden states of the congressional tweets to decompose the 4,096-dimensional residual stream into 32,768 latent features. This allows for the identification of interpretable, monosemantic features, specifically isolating those encoding policy content versus discourse style.
Feature-Level Steering: To establish causality, the authors perform activation steering experiments. They extract decoder columns from the SAE corresponding to specific features (policy-related and register-related) and add perturbations ( $\alpha \cdot \hat{d}_i$ ) to the hidden states during generation. This tests whether steering the internal representation along partisan directions results in corresponding changes in the generated text for both models.

Key Results

Compression, Not Erasure: RLHF dramatically compresses the variance of the partisan signal. While the Base model's partisan scores span a wide range (from -0.5 to 1.253), the Instruct model's scores are compressed into a narrow band centered near 0.169. Despite this shift toward a positive (Republican-leaning) offset, the Instruct model's generated text remains consistently neutral and multi-perspective.
Disappearance of Policy Features: SAE decomposition reveals that five specific features encoding recognizable political content (e.g., anti-Biden rhetoric, progressive advocacy) are active sporadically in the Base model but are completely inactive (zero) in the Instruct model across all 84 prompts.
The Nature of the Offset: The residual positive partisan score (0.169) in the Instruct model is not driven by policy features. Decomposition shows that 68% of this score comes from the SAE decoder bias (inherited from the congressional tweet corpus) and 24% from a single feature (32143) that encodes formal discourse style (numbered lists, citations, institutional voice) rather than policy. The positive alignment of this style feature is an artifact of the training corpus, not a political stance.
Causal Disconnect: Steering experiments demonstrate a fundamental divergence in behavior. When the Base model is steered along policy-feature directions, its output shifts to match the partisan direction. In contrast, the Instruct model's internal representations move in response to steering (the partisan score shifts identically), but the generation pipeline fails to convert this representational shift into textual output. The Instruct model continues to generate balanced, neutral text regardless of the steering magnitude.

Key Contributions

Mechanistic Evidence of "Disconnect-Rather-Delete": The paper provides empirical evidence that RLHF does not erase partisan geometry from the model's representation space. Instead, it severs the causal pathway between that geometry and the output generation mechanism.
Distinction Between Functional and Structural Neutrality: The study distinguishes between functional neutrality (the output appears balanced) and structural neutrality (the underlying bias is removed). The authors demonstrate that the Llama 3.1 8B Instruct model achieves the former while retaining the latter.
Role of Discourse Style: The analysis identifies that the "neutrality" observed is partly a byproduct of RLHF training the model to adopt a specific formal, structured register, which happens to align with a specific direction in the partisan space due to the training data, rather than a deliberate erasure of political knowledge.

Significance and Claims
The paper argues that the neutrality observed in aligned LLMs is a "mask." The underlying partisan geometry remains intact and can be reactivated if the causal link to the generation pipeline is bypassed. The authors claim that this "disconnect-not-delete" pattern is likely a general property of current alignment methods, observed previously in toxicity and safety domains.

The significance of this finding lies in the fragility of aligned behavior. Because the structural bias remains, mechanisms that infer and amplify a user's partisan identity (e.g., through conversation context) can reactivate partisan generation. The paper concludes that current alignment methods may only achieve surface-level compliance, suggesting that true safety and alignment require methods that align the representations themselves, rather than merely rerouting generation around existing value-laden structures. The authors note that their findings are specific to the Llama 3.1 8B architecture and that the computational demands of such mechanistic analysis limit its current application to open-weight models.