💬 NLP

Query as Anchor: Scenario-Adaptive User Representation via Large Language Model

This paper introduces "Query-as-Anchor," a framework that leverages a large-scale pre-training dataset (UserU) and a dual-tower LLM architecture with cluster-based soft prompt tuning to generate dynamic, query-aware user representations, achieving state-of-the-art performance and efficient deployment across multiple industrial scenarios in Alipay.

Original authors: Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Ziyi Gao, Xiaotong Lin, Yun Liu, Xing Fu, Yu Cheng, Yongchao Liu, Weiqiang Wang, Zhongle Xie

Published 2026-02-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Ziyi Gao, Xiaotong Lin, Yun Liu, Xing Fu, Yu Cheng, Yongchao Liu, Weiqiang Wang, Zhongle Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Size-Fits-All" Suit

Imagine you work at a massive department store (like Alipay) that serves millions of customers. You want to understand every customer so well that you can:

Sell them the right shoes (Marketing).
Decide if they are a good credit risk (Risk Management).
Know when they are likely to visit the store (Engagement).

Currently, most companies use a "Static User Profile." Think of this as taking a single, giant photo of a customer and putting it in a file.

The Flaw: If you show that same photo to the shoe salesman, the credit officer, and the security guard, they all see the same thing. The shoe salesman needs to know about their fashion sense; the credit officer needs to know about their debt history. A single photo can't highlight different details for different people.
The Result: Companies end up building separate, expensive "file rooms" for every single department. It's messy, slow, and expensive to maintain.

The Solution: "Query as Anchor"

The authors propose a new way to think about user data. Instead of a static photo, imagine a smart, shape-shifting hologram of the customer.

This hologram is built from all the customer's past actions (buying bills, using mini-programs, searching for things). But here is the magic trick: The hologram changes its shape depending on who is looking at it.

The "Anchor": The customer's history is the heavy anchor at the bottom. It stays the same.
The "Query": The specific question asked (e.g., "Is this person likely to buy takeout?" or "Is this person a fraud risk?") acts as a magnet that pulls the relevant parts of the hologram to the surface.

If a Marketing Manager asks, "What does this person like to eat?", the hologram highlights their food orders and ignores their bank balances.
If a Risk Officer asks, "Is this person safe?", the hologram highlights their transaction patterns and ignores their food orders.

The Analogy: Think of the user's data as a giant, messy library.

Old Way: You have to read every single book in the library to answer any question.
New Way (Query as Anchor): You have a super-smart librarian (the AI). You just ask, "Show me all books about Italian Cooking," and the librarian instantly pulls out only those books, ignoring the rest. You don't need a different librarian for every topic; you just change the question.

How They Built It (The Secret Sauce)

To make this work, the team did three main things:

1. They Built a "Training Gym" (UserU Dataset)

Large Language Models (LLMs) are usually trained on books and articles. But user data (like "User clicked button X at 3 PM") is very different from text. It's sparse and messy.

The Fix: They created a massive dataset called UserU. They didn't just feed the AI raw data; they taught it to play "Future Prediction" and "Interview."
The Analogy: Instead of just memorizing a dictionary, they made the AI practice by looking at a person's past and guessing what they will do next, or answering questions like, "Based on their spending, are they saving for a house?" This taught the AI to understand the story behind the numbers.

2. The "Hierarchical Encoder" (The Smart Organizer)

User data comes in many forms: text, numbers, clicks, and locations.

The Fix: They built a system that organizes this data like a Russian Nesting Doll.
- Layer 1 (Events): Individual actions (e.g., "Bought coffee").
- Layer 2 (Modes): Groups of actions (e.g., "All food purchases").
- Layer 3 (The Whole Person): The big picture.
The Analogy: Imagine a detective organizing clues. They don't just dump all evidence on the table. They group them by type (fingerprints, phone records, witness statements) so the AI can easily find the right clue when asked a specific question.

3. The "Soft Prompt" (The Magic Remote Control)

This is the most clever part. Usually, to make an AI good at a new task, you have to retrain the whole brain (which is slow and expensive).

The Fix: They use Soft Prompt Tuning. Instead of retraining the whole brain, they add a tiny, adjustable "remote control" (a few extra tokens) to the query.
The Analogy: Imagine you have a high-end TV. To watch a sports game, you don't rebuild the TV; you just press the "Sports Mode" button. The picture changes instantly. Similarly, the AI keeps its core knowledge but uses a tiny "button" to shift its focus to the specific business scenario (Risk vs. Marketing) without needing a massive overhaul.

Why It's a Game Changer

Speed & Cost: Because the "user profile" (the heavy part) is calculated once and saved, the system can answer 100 different questions about that user in a split second. It's like having a library card that works for every section of the library instantly.
Better Accuracy: In real-world tests on Alipay (a massive Chinese payment app), this method beat all previous models. It was better at spotting fraud, predicting who would buy things, and identifying risky users.
One Model to Rule Them All: Instead of having 10 different AI models for 10 different departments, they now have one model that adapts to all of them.

The Bottom Line

"Query as Anchor" is like giving a company a single, super-intelligent assistant who knows everything about every customer. Instead of forcing the assistant to memorize 10 different rulebooks, you simply ask it the right question, and it instantly reorganizes its knowledge to give you the perfect answer for that specific situation. It's faster, cheaper, and smarter than anything they had before.

1. Problem Statement

Industrial-scale user representation learning faces three critical challenges:

Static vs. Dynamic Adaptability: Existing methods produce static, task-agnostic embeddings that fail to adapt to diverse downstream scenarios (e.g., credit risk vs. marketing) within a unified vector space. Industrial systems often rely on multiple task-specific models, increasing complexity and maintenance costs.
Modality and Semantic Gap: User behavior data is sparse, symbolic, and heterogeneous (e.g., transaction logs, app clicks), which diverges significantly from the dense, language-centric data used to pretrain Large Language Models (LLMs). This gap limits LLMs' ability to generate adaptive representations for industrial user data.
Heterogeneous Data Integration: Integrating massive, noisy, multi-source behavioral signals requires mechanisms to selectively attend to scenario-relevant signals without introducing prohibitive inference latency.

2. Methodology: Query-as-Anchor (Q-Anchor)

The authors propose Q-Anchor, a framework that shifts user modeling from static encoding to dynamic, query-aware synthesis. The core philosophy is to decouple the encoding of user behavior from the specific downstream task, using a natural language query as an "anchor" to adapt the representation.

A. UserU Pretraining Dataset

To bridge the gap between raw logs and LLM semantics, the authors constructed UserU, an industrial-scale pretraining dataset containing ~100M samples. It consists of two components:

Behavior-based Interaction ( $D_{future}$ ): Pairs historical user profiles with future action summaries (predicting the next period's behavior) to capture temporal regularities.
Synthetic Query-Answer ( $D_{uqa}$ ): Uses LLMs to generate synthetic user-understanding queries (e.g., "financial planning," "health management") and corresponding answers based on user profiles. This includes a "self-reflect" step where the LLM verifies the answer against the raw data to ensure faithfulness.

B. Hierarchical Coarse-to-Fine Encoder

To handle multi-modal data (PayBill, Mini Programs, Search, etc.), the framework employs a hierarchical encoder:

Event Level: Raw events are projected into embeddings and refined by modality-specific adapters.
Modality Level: Event embeddings are mean-pooled and aggregated into unified modality vectors.
User Level: A global user representation is derived by consolidating all modality vectors.
The final input to the LLM is a structured concatenation of these three levels.

C. Query-as-Anchor Architecture

The architecture utilizes a Dual-Tower design with a shared LLM backbone:

Anchor Tower: Takes the hierarchical user tokens ( $e_i$ ) and appends a natural language query ( $q_i$ ) at the sequence terminus. The LLM acts as a query-aware aggregator, projecting the user profile into a scenario-adaptive embedding ( $u_{i,q}$ ).
Semantic Tower: Encodes the target answer ( $a_i$ ) into a dense vector ( $v_{a_i}$ ) to serve as the ground truth.
Optimization: The model is trained via Joint Contrastive-Generative Optimization:
- Contrastive Loss (InfoNCE): Aligns the query-anchored user embedding with the semantic answer embedding while pushing away negatives (with margin-based masking to handle false negatives).
- Generative Loss (Next-Token Prediction): An auxiliary task where the anchor tower autoregressively reconstructs the answer, ensuring semantic richness and preventing representation collapse.

D. Efficient Inference & Adaptation

KV-Cache Acceleration: Since the user profile ( $e_i$ ) is static for a given time window, its Key-Value (KV) cache is computed once and stored. Downstream queries only process the short query suffix, enabling negligible incremental latency for multi-scenario inference.
Cluster-based Soft Prompt Tuning: To bridge general pretraining with specific business logic (e.g., fraud detection), the authors introduce learnable soft prompt tokens. These tokens modulate the latent space to enforce discriminative clustering without retraining the massive backbone model.

3. Key Contributions

UserU Dataset: A large-scale, industrial pretraining corpus aligning heterogeneous behavioral sequences with user understanding semantics via future prediction and QA synthesis.
Query-as-Anchor Framework: A novel paradigm that generates scenario-adaptive embeddings by "re-anchoring" the same behavioral profile under different decision contexts using natural language instructions.
Efficient Adaptation Mechanism: Integration of soft-prompt tuning and KV-cache optimization allows for low-latency, multi-scenario inference without retraining the backbone.
Industrial Validation: Extensive offline benchmarks and large-scale online A/B tests on Alipay's production system.

4. Experimental Results

The framework was evaluated on 10 real-world binary classification tasks across Alipay (User Engagement, Risk Control, Marketing Sensitivity) and compared against strong baselines (e.g., Qwen3-Embedding, FOUND, MSDP).

Performance: Q-Anchor (Prompt Tuned) achieved SOTA performance with an average AUC of 0.8225 and KS of 0.5267, outperforming the best general-purpose baseline (Llama-Embed-Nemotron-8B) by +9.84% in AUC and +38.4% in KS.
Scalability:
- Data vs. Model: Performance scales linearly with pretraining data but shows non-monotonic behavior with model size (0.5B performed better than 1.5B/3B due to gradient attenuation in embedding tasks).
- Prompt Tuning: Gains saturate quickly with just 6 learnable tokens and 500 training steps, demonstrating high efficiency.
Interpretability: Attention analysis showed that prompt tuning dynamically shifts focus to relevant modalities (e.g., increasing attention to "Bill" data for food delivery interest by +26%).
Online A/B Testing:
- IVR Cash-Reserve Outreach: Increased drawdown rate by 12.5% and average outstanding balance by 5.3%.
- Credit Risk: Improved the critical KS score by 1.96%.
- Deployment: Successfully deployed on a 100x L20 cluster, serving hundreds of millions of users with shared KV-cache, significantly reducing infrastructure costs compared to running separate models per scenario.

5. Significance

This work addresses a fundamental bottleneck in industrial AI: the trade-off between universality and task-specificity. By treating the query as an anchor, Q-Anchor enables a single, unified model to serve diverse, high-stakes business scenarios (risk, marketing, engagement) with high precision and low latency.

The paper demonstrates that pretraining on synthetic, behavior-aligned data combined with lightweight adaptation (soft prompts) is more effective than massive parameter scaling for embedding tasks. The KV-cache optimization strategy provides a blueprint for deploying LLM-based user representations in high-throughput, real-time industrial systems, moving beyond static embeddings to dynamic, context-aware user understanding.