💬 NLP

Locating and Controlling Implicit Personalization in Large Language Models

This paper demonstrates that implicit demographic personalization in large language models is driven by localized internal activation signals that can be identified, causally controlled, and selectively removed to suppress biased outputs while preserving general model performance.

Original authors: Yueru Yan, Siqi Wu, Thai Le

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yueru Yan, Siqi Wu, Thai Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a super-smart robot that has read almost every book on the internet. You might think it treats everyone exactly the same, like a neutral judge. But in the world of Artificial Intelligence, specifically with Large Language Models (LLMs), things are a bit more like a chameleon. These models don't just answer questions; they subtly change their personality and recommendations based on tiny, hidden hints you drop in the conversation. This isn't because you told the robot, "I am a teenager" or "I am from a specific culture." Instead, you might mention a specific holiday, use a slang word, or talk about a hobby. The robot picks up on these "implicit cues" and quietly shifts its answers to match stereotypes it has learned about those groups. This is a bit like a waiter who, without you saying a word, assumes you want a steak because you're wearing a certain jacket, or a librarian who hands you a mystery novel because you're wearing glasses. The problem is, this happens silently. You can't see the switch being flipped, and you can't easily ask the robot to stop doing it. Scientists have known this happens, but they didn't know how the robot was doing it inside its "brain." They knew the output changed, but they couldn't see the internal gears turning.

This paper is like a mechanic pulling back the hood of that robot to see exactly what's happening under the hood. The researchers wanted to find the specific "internal signal"—a tiny electrical pattern in the model's brain—that happens right when it decides to change its answer based on those hidden hints. They treated the model like a complex machine with layers of processing. By comparing conversations where a hint was present versus nearly identical conversations where it was absent, they found a specific "fingerprint" in the model's internal math. They discovered that the size of this fingerprint directly predicts how much the robot's answer will change. It's as if they found a volume knob inside the robot's brain: when the knob is turned up (a strong internal signal), the robot's recommendations shift dramatically toward stereotypes; when the knob is at zero, the robot stays neutral.

But the story gets even more interesting when you have multiple hints at once. Imagine you mention both a specific holiday and a specific age group. You might expect the robot to combine these two influences perfectly, like adding two numbers together. However, the researchers found that while the robot's internal brain signals do seem to add up linearly (like mixing two colors of paint), the final answer it gives you doesn't. The actual behavior is "sub-additive," meaning the final shift is smaller than the sum of the parts. It's like if you added sugar and salt to a drink; the internal mixture might be a perfect blend of both, but the taste you experience is less intense than you'd expect from just adding the two amounts together.

The most exciting part of the study is what happens when the researchers tried to "turn off" the robot's bias. They figured out how to identify the specific direction of that internal fingerprint for one type of hint (like race) and then mathematically "projected" it out of the system, effectively deleting that specific influence. They found that this internal surgery worked better than just asking the robot nicely to "ignore demographics" in a prompt. In fact, telling the robot to ignore the bias sometimes made it more biased, like a child who gets more rebellious when told to stop doing something. By surgically removing the internal signal, they could stop the robot from making race-based assumptions without messing up its ability to answer other questions or ruining its general smarts. However, this fix isn't a magic wand that works for every robot model or every type of bias. Sometimes, removing one hint's influence accidentally changes another, and sometimes it works perfectly. It's a powerful new tool for understanding and controlling these AI systems, showing us that we can find and tweak the hidden gears that drive their behavior, rather than just hoping they behave well on their own.

Technical Summary: Locating and Controlling Implicit Personalization in Large Language Models

Problem Statement

Large Language Models (LLMs) exhibit implicit personalization, where they shift their outputs (e.g., movie recommendations) in response to demographic cues embedded in user dialogue, even when the user never explicitly states their identity. While previous work has documented these behavioral shifts—such as dialect features eliciting harsher criminal sentences or implicit history narrowing recommendations toward stereotypes—the connection between these behavioral changes and the model's internal activations remains unclear.

Existing studies often rely on behavioral audits or linear probes that confirm attribute information is decodable but fail to establish whether these attributes causally drive the output. Furthermore, real-world interactions often involve multiple overlapping identity cues (e.g., race and age simultaneously), raising the question of whether internal representations decompose linearly and whether behavioral shifts simply add up. This paper seeks to bridge the gap between implicit personalization behavior and internal model mechanics, specifically asking:

Is there an internal signal that tracks the magnitude of personalization?
How do internal representations and outputs combine when multiple cues co-occur?
Can specific demographic influences be causally suppressed without degrading general model performance?

Methodology

Dataset and Experimental Design

The authors employ a matched-pair comparison framework using $n=50$ scenarios across five LLMs (Llama-3-8B, Mistral-7B, Qwen3-8B, Qwen3-14B, and Phi-4).

Cues: Dialogues are generated using GPT-4o, incorporating implicit demographic cues drawn from a literature-grounded taxonomy (cultural references, hobbies, slang) for three dimensions: Race (Black, Asian, White), Gender (Male, Female), and Age (Child, Adolescent, Adult, Older Adult).
Conditions: For each scenario, the authors generate:
- Cued contexts: Containing implicit identity cues.
- Neutral contexts: Using identity-neutral fillers with identical topical scaffolding.
- Explicit contexts: Stating demographics directly (for calibration).
Task: The final turn asks the model to recommend five movies.
Intersectionality: Experiments combine two dimensions (e.g., Race $\times$ Gender) with dose-matched controls to isolate interaction effects.

Behavioral Metrics

Two complementary metrics quantify the shift:

Semantic Embedding Distance on descriptions (SED-desc): Measures the magnitude of the semantic shift between cued and neutral recommendations using cosine similarity of plot overviews.
Content Alignment Ratio (CAR): Measures the direction of the shift by calculating the fraction of recommended items tagged as associated with the target identity using a pre-specified stereotype taxonomy.

Internal Representational Quantity

The core technical contribution is the definition of the raw cue-induced activation contrast ( $\Delta A$ ):
$\Delta A^{(i)}_L = A_L(C^{(i)}_{cue}, x) - A_L(C^{(i)}_{neutral}, x)$
This represents the difference in the residual-stream activation at the final query token between matched cued and neutral contexts at specific layers $L$ .

Normalized Magnitude ( $s_{\Delta A}$ ): The authors compute the normalized magnitude of this contrast to test its correlation with behavioral shifts.
Layer Selection ( $L^\star$ ): Specific layers mediating the behavioral shift are identified via activation patching.

Intervention Strategy

To control personalization, the authors apply directional ablation during generation. They compute a sample-averaged direction vector ( $\hat{v}_{dim}$ ) for a specific cue dimension, orthogonalize it against other dimensions, and project it out of the residual stream using a pre-hook:
$h' = (I - \alpha \hat{v}_{dim}\hat{v}_{dim}^\top)h$
where $\alpha$ controls the intervention strength. This is compared against "ignore demographics" prompting and random-direction shams.

Key Results

1. Internal Signal Tracks Behavioral Shift

The normalized magnitude of the activation contrast ( $s_{\Delta A}$ ) at targeted layers ( $L^\star$ ) strongly correlates with the behavioral shift on a per-sample basis.

Correlation: Pearson correlations reach up to $r = 0.87$ (e.g., Llama-3-8B with Asian cues on SED-desc).
Scaling: The strength of the internal tracking scales with the magnitude of the behavioral shift. In conditions where no behavioral shift occurs (e.g., White-cue conditions), the internal correlation drops significantly.
Significance: This establishes a chain running strictly from cue to activation to behavior, where the internal signal magnitude serves as a reliable tracking signal for the extent of personalization, rather than acting as the causal driver itself.

2. Linear Composition of Internal Signals vs. Sub-additive Behavior

When multiple cues co-occur:

Internal Representation: The mixed-cue contrast vector ( $\Delta A_{AB}$ ) can be reliably reconstructed as a linear combination of the single-cue vectors ( $\alpha v_A + \beta v_B$ ). The average reconstruction fit ( $\rho$ ) ranges from 0.576 to 0.704, significantly outperforming single-cue explanations. This indicates the model actively integrates multiple identity directions internally.
Behavioral Output: The actual behavioral shift is strictly sub-additive. When both cues are present, the resulting personalization is 28–38% lower than the sum of the individual single-cue shifts. The output does not simply mirror the internal linear addition; it compresses the effect in a model-specific manner.

3. Causal Control via Direction Ablation

Projecting out the internal direction associated with a specific cue successfully suppresses its influence:

Effectiveness: Internal ablation outperforms explicit "ignore demographics" prompting. On Qwen3-8B, ablation was up to 9.2 $\times$ more effective than prompting. In some cases, prompting backfired, increasing stereotyping.
Selectivity: The ability to remove one dimension's influence while preserving another is model- and attribute-specific.
- Mistral-7B: Removing the gender direction reduced gender-tagged content by 0.169 while leaving race-tagged content virtually unchanged (+0.004).
- Qwen3-8B: Selectivity was lower; removing race often reduced age-tagged content as well.
Capability Preservation: General reasoning capabilities (measured by MMLU) remained robust, dropping by at most 2.8 points even at maximum intervention strength ( $\alpha=5$ ).

Significance and Claims

The paper claims to establish the "missing connection" between implicit personalization behavior and the model's internal representation. Its primary contributions are:

Identification of a Tracking Signal: The magnitude of the cue-induced activation contrast at specific layers serves as a measurable tracking signal for personalization intensity, correlating strongly with behavioral shifts without claiming to be the causal driver itself.
Decoupling of Mechanism and Behavior: The study reveals a dissociation where internal representations combine linearly, but the final output behavior is non-linear and sub-additive.
Targeted Intervention: It demonstrates that implicit personalization can be causally controlled by ablating specific internal directions. This method is often more effective than prompting and preserves general model capabilities, offering a potential mechanism for mitigating representational harms in LLMs without requiring retraining.

The authors maintain a modest stance, noting that their findings are specific to the tested models and domains (primarily movie recommendations) and that the selectivity of the intervention varies by model. They emphasize that this work provides a diagnostic tool and a causal mechanism for suppression rather than a universal, deployment-ready fairness solution.