💬 NLP

How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning

This paper investigates how different attention masking strategies affect the quality of user embeddings in decoder-only LLMs and proposes "Gradient-Guided Soft Masking" to facilitate a stable transition from causal to bidirectional attention during user representation learning.

Original authors: Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Yang Chen, Xiaotong Lin, Wuliang Huang, Ziyi Gao, Xing Fu, Yu Cheng, Weiqiang Wang

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Yang Chen, Xiaotong Lin, Wuliang Huang, Ziyi Gao, Xing Fu, Yu Cheng, Weiqiang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a person’s personality just by watching them walk through a giant shopping mall.

If you only look at them through a rear-view mirror (seeing only what they’ve already done), you might miss the "why" behind their actions. If you try to look at their entire life story all at once (seeing the past, present, and future simultaneously), your brain might get overwhelmed and confused by too much information at once.

This paper, written by researchers at Ant Group, solves this exact problem for Artificial Intelligence.

The Problem: The "One-Way Street" Dilemma

Most modern AI models (like the ones behind ChatGPT) are "Decoder-only" models. They are designed to be storytellers. Because they predict the next word in a sentence, they are trained to look only at the past. They read from left to right, like a person reading a book. This is called Causal Attention.

However, to truly understand a user (like a shopper on Alipay), you don't just need a storyteller; you need a detective. A detective needs to look at a person's behavior holistically—seeing how a purchase made on Monday connects to a search made on Friday—to understand their true intent.

The researchers found a conflict:

The Storyteller (Causal): Great at predicting the next step, but "blind" to the future context.
The Detective (Bidirectional): Great at seeing the whole picture, but hard to train because it breaks the way the AI was originally "born" (pre-trained).

The Solution: The "Gradual Spotlight" (GG-SM)

The researchers created a new method called Gradient-Guided Soft Masking (GG-SM).

Think of it like training a student to become a detective. You wouldn't throw a beginner into a complex crime scene and say, "Look at everything at once!" They would freeze up. Instead, you use a Gradual Spotlight:

Phase 1 (The Storyteller): At first, you let the AI look only at the past, just like it was trained to do.
Phase 2 (The Smart Spotlight): Instead of suddenly opening its eyes to everything, the AI uses "gradients" (which you can think of as internal intuition) to decide which future clues are actually important. It’s like a detective saying, "I'm not looking at everything yet, but I'm going to peek at that suspicious briefcase over there."
Phase 3 (The Full Detective): Slowly, the spotlight widens until the AI can see the entire scene at once, having already learned which parts of the scene matter most.

Why does this matter?

By using this "smart transition" instead of just flipping a switch, the AI becomes much better at "User Representation"—which is just a fancy way of saying "understanding what a person actually wants."

The Results:
When they tested this on real-world data from Alipay (looking at things like what movies people like, how they spend money, and how they use apps), their "Detective AI" beat out much larger, more expensive models.

In short: They proved that it’s not just about how big the AI is, but about how you teach it to look at the world. By teaching it to transition smoothly from a storyteller to a detective, they created an AI that understands human behavior with much more depth and stability.

Technical Summary: Rethinking Attention Masking for User Representation Learning

1. Problem Statement

As Large Language Models (LLMs) are increasingly adopted as behavioral encoders for user representation learning, a critical architectural conflict arises. Most state-of-the-art LLMs are decoder-only and pretrained using causal (unidirectional) attention, which is designed for autoregressive generation. However, effective user modeling—such as predicting preferences or marketing sensitivity—benefits significantly from bidirectional attention, which allows for a holistic understanding of a user's entire behavioral history.

The researchers identify a gap in understanding how different attention masking strategies (Causal, Hybrid, and Bidirectional) affect the quality of user embeddings when adapted via contrastive learning. Furthermore, they observe that abruptly switching from causal to bidirectional attention during fine-tuning can lead to unstable training dynamics and suboptimal representations.

2. Methodology

The authors propose a unified framework to study and optimize the transition from causal to bidirectional modeling.

Data Construction: They utilize large-scale, real-world Alipay data, constructing two types of alignment data:
1. Rule-based Behavioral Trajectories ( $D_{behavior}$ ): Aligning historical interaction sequences with future behaviors.
2. LLM-Synthesized Query-Answer Alignments ( $D_{qa}$ ): Using a "difficulty probing" mechanism to identify hard-to-align user intent pairs and employing post-hoc Chain-of-Thought (CoT) reasoning to synthesize high-fidelity synthetic "hard positives" for training.
Core Innovation: Gradient-Guided Soft Masking (GG-SM):
To stabilize the transition from causal to bidirectional attention, the authors propose a two-stage training mechanism:
- Warmup Phase ( $t < T_{warm}$ ): Instead of a hard mask, they use a "soft mask" where future attention weights are determined by the instantaneous gradient norm ( $\|\nabla h_j^L\|$ ). This allows the model to adaptively "peek" at future tokens that have the highest influence on the loss.
- Scheduling Phase ( $T_{warm} \leq t < T_{total}$ ): A linear scheduler gradually transitions the weights from the frozen gradient-derived soft mask to full bidirectionality (where all future tokens are visible).
Training Objective: They employ a Contrastive Learning (InfoNCE) loss within a dual-tower architecture, pulling semantically related user-answer pairs together while using a "mask factor" to mitigate the impact of false negatives in the batch.

3. Key Contributions

Systematic Empirical Study: A controlled comparison of Causal, Hybrid, and Bidirectional masking strategies for LLM-based user representation learning.
GG-SM Algorithm: A novel gradient-informed pre-warmup mechanism that facilitates a smoother and more stable transition from unidirectional to bidirectional attention.
High-Fidelity Data Synthesis: A training-free framework that uses an off-the-shelf LLM to probe alignment difficulty and generate challenging synthetic positive samples.

4. Results

The model was evaluated on 9 industrial user cognition benchmarks (covering User Prediction, Behavior Preference, and Marketing Sensitivity) using the Qwen2.5-0.5B-Instruct backbone.

Superior Performance: The GG-SM approach achieved an average AUC of 0.7745, consistently outperforming massive general-purpose embedding models (like Llama-embed-nemotron) and specialized user modeling baselines (like FOUND and InstructUE).
Efficiency vs. Scale: Despite having significantly fewer parameters than 8B+ general embedding models, GG-SM yielded higher task-specific accuracy, proving that masking design and training dynamics are more critical than raw parameter scale for domain-specific behavioral logs.
Stability: GG-SM showed smoother training loss convergence compared to standard linear schedulers (as shown in Figure 3 of the paper).
Domain Robustness: The method showed particular strength in the "Marketing Sensitivity" domain, where capturing latent, non-obvious user intent is most difficult.

5. Significance

This work shifts the focus from simply "scaling up" LLMs for user modeling to "architectural adaptation." It demonstrates that the way an LLM "perceives" a sequence (via the attention mask) is just as important as the data it is trained on. By providing a mathematically grounded way to transition from generative (causal) to discriminative (bidirectional) modes, the paper offers a practical blueprint for deploying decoder-only LLMs in real-world, high-stakes industrial recommendation and marketing systems.