🤖 AI

From Profiles to Steering Vectors: Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation

GLASS is a training-free framework that achieves personalized text generation by extracting global user-style priors and local contrastive vectors via sparse autoencoders, which are jointly injected into model layers to enable context-aware style adaptation without retrieval or parameter updates.

Original authors: Liuji Chen, Zeyu Zhang, Xinyuan Zhang, Shuai Nie, Qiang Liu, Shu Wu, Liang Wang

Published 2026-07-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Liuji Chen, Zeyu Zhang, Xinyuan Zhang, Shuai Nie, Qiang Liu, Shu Wu, Liang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to write like you. You don't want it to just sound like a generic encyclopedia; you want it to sound like your diary, your funny texts to friends, or your serious emails to a boss. This is the world of personalized text generation, a branch of artificial intelligence where machines try to mimic human writing habits.

To do this, scientists usually try three things. First, they can show the robot examples of your writing (like a teacher showing a student past essays), but this can be slow and cluttered. Second, they can tweak the robot's brain (its internal settings) specifically for you, but this takes a lot of time and memory, like giving every student a custom-made textbook. Third, they can try to nudge the robot's thoughts while it writes, pushing it in the direction of "you" without changing its brain permanently. This "nudging" is called activation steering. It's fast and light, but it has a big problem: it's hard to tell the difference between what the robot is writing (the story) and how it is writing (your style). It's like trying to separate the flavor of a spice from the food it's cooking; often, the robot ends up copying the topic instead of your unique voice.

This paper introduces a new method called GLASS (Global–Local Activation Steering with Sparse priors) to solve that messy separation problem. Think of the robot's brain as a giant, tangled ball of yarn where every thread represents a different idea or feeling. The authors realized that if you could untangle the yarn into neat, separate strands, you could grab just the "style" thread without pulling on the "topic" thread. They use a special tool called a Sparse Autoencoder (SAE) to do this untangling. It's like having a magical sieve that only lets the "style" particles pass through while blocking the "content" dust.

GLASS works in two clever steps. First, it creates a Global Style Profile. Imagine taking all your past writing and running it through that magical sieve to find your "fingerprint"—your average sentence length, your favorite words, and your general tone. This is your "Global" prior, a stable blueprint of who you are. Second, it realizes that you don't write the same way in every situation. You might be formal in an email but silly in a text message. So, GLASS groups your past writing into different Local Clusters (like "Work Mode" vs. "Party Mode") and creates a specific "nudge" for each one.

When the robot needs to write something new, GLASS doesn't need to look up old examples or retrain the robot. Instead, it simply injects the right mix of your "Global Fingerprint" and the "Local Nudge" for the current situation directly into the robot's brain as it thinks. The paper shows that this method works better than the other three approaches. It suggests that by using these "sparse" tools to clean up the robot's thoughts, it can capture your true writing style much more accurately, even when the topic changes or the length of your writing varies. The result is a robot that sounds like you, without the heavy baggage of storing thousands of your old messages or spending hours retraining its brain.

Technical Summary: GLASS – Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation

Problem Statement
Personalized text generation aims to produce outputs that reflect a user's specific writing habits, stylistic preferences, and communicative patterns while adhering to task requirements. Existing approaches face significant trade-offs:

Retrieval-based methods (e.g., In-Context Learning, RAG) often rely on semantic similarity, which can introduce noisy semantic content rather than pure stylistic signals, while incurring inference latency and storage overhead.
Parameter-Efficient Fine-Tuning (PEFT) requires training and storing user-specific parameters, hindering scalability for large user populations.
Activation Steering offers a training-free alternative but struggles to disentangle stylistic signals from semantic content. Standard steering vectors often retain "semantic residue," causing models to imitate topics from historical data rather than the user's underlying style.

Methodology: GLASS
The authors propose GLASS (Global–Local Activation Steering with Sparse priors), a training-free framework that leverages Sparse Autoencoders (SAEs) to extract robust style representations. The method operates in two phases: offline vector construction and inference-time steering.

Global Style Modeling via Sparse Priors:
- To capture invariant user-level traits (e.g., lexical choice, tone, sentence length), GLASS processes the user's entire historical interaction data.
- Hidden states from a specific layer are projected into a high-dimensional sparse feature space using a pre-trained SAE encoder.
- The sparse activations are averaged across all response tokens to form a global sparse profile, which is then decoded back into the model's residual stream space to create a global style vector ( $v_G$ ). This token-level aggregation smooths out instance-specific semantic variations.
Local Style Modeling via Contrastive Clusters:
- To address context-dependent variations (e.g., formal emails vs. casual posts), user histories are clustered based on prompt semantics (using TF-IDF and K-means).
- For each cluster, a local style vector ( $v_C$ $v_{C}$ ) is derived through a contrastive process:
  - Priming: A "primed" model injects the global vector $v_G$ during the extraction process to bias subsequent operations toward style.
  - Positive-Negative Construction: For a given anchor query, positive contexts are formed by selecting historical examples from the same cluster that yield high utility scores, while negative contexts are sampled from other users.
  - Contrastive Extraction: The model generates responses under positive and negative contexts. The local vector is computed as the difference between the mean hidden states of the positive and negative answer tokens. This subtraction aims to remove shared semantic content, isolating the stylistic residual specific to that scenario.
Adaptive Dual-Stream Steering:
- During inference, a new request is assigned to the nearest cluster.
- The system retrieves the corresponding local vector ( $v_C$ ) and the global vector ( $v_G$ ).
- These vectors are injected into different layers of the model's residual stream (global at upper layers, local at intermediate layers) with tunable strengths ( $\alpha$ and $\beta$ ), enabling context-aware personalization without parameter updates or retrieval overhead.

Key Contributions

Identification of Semantic Residue: The paper identifies that the inability to separate stylistic signals from semantic content is a primary limitation of existing activation steering methods.
SAE-Based Disentanglement: The authors demonstrate that SAE-based representations are more robust to distribution shifts (specifically length and topic perturbations) than dense activation-based methods, suggesting superior disentanglement of style from semantics.
GLASS Framework: A novel, training-free framework that models user style at multiple granularities (global and local) within the SAE latent space.
Empirical Validation: Extensive experiments on LaMP and LongLaMP benchmarks show GLASS outperforms retrieval, fine-tuning, and other steering baselines across ROUGE metrics and LLM-as-judge evaluations.

Results

Performance: On LaMP-4 and LaMP-5, GLASS achieves the best results across all metrics (ROUGE-1, ROUGE-L, and Judge scores), improving upon the strongest baselines (e.g., OPPU, Fints). On LongLaMP datasets, it consistently ranks first or second across 11 out of 12 evaluation metrics.
Robustness: Perturbation experiments (Figure 1) show that SAE-based steering features degrade significantly less than contrastive or averaging-based methods when user history length or topic distributions are shifted.
Efficiency: GLASS avoids the prompt expansion and retrieval latency of RAG/ICL methods and requires minimal per-user storage (<0.01 MB) compared to PEFT methods (e.g., LoRA), with inference overhead comparable to base models.
Ablation: Removing either the global prior or the local contrastive component leads to performance drops, confirming their complementary nature. Contrastive extraction is shown to be essential for suppressing content-correlated signals.

Significance and Claims
The paper claims that GLASS establishes a scalable and effective paradigm for personalized text generation by addressing the core challenge of semantic contamination in activation steering. By utilizing sparse representations, the method successfully disentangles stylistic information from semantic content, allowing for high-fidelity personalization without the computational cost of fine-tuning or the latency of retrieval. The authors position GLASS as a solution that balances efficiency, scalability, and performance, making it suitable for real-world deployment where user-specific training is impractical. The work suggests that sparse representation-level steering is a promising direction for future research in model interpretability and personalization.

More like this