💬 NLP

High Fidelity Textual User Representation over Heterogeneous Sources via Reinforcement Learning

This paper proposes a reinforcement learning framework that synthesizes unified, concise, and interpretable textual user representations from heterogeneous data sources by using implicit engagement signals and rule-based constraints as rewards, significantly improving downstream performance on a large-scale job platform.

Original authors: Rajat Arora, Ye Tao, Jianqiang Shen, Ping Liu, Muchen Wu, Qianqi Shen, Benjamin Le, Fedor Borisyuk, Jingwei Wu, Wenjing Zhang

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rajat Arora, Ye Tao, Jianqiang Shen, Ping Liu, Muchen Wu, Qianqi Shen, Benjamin Le, Fedor Borisyuk, Jingwei Wu, Wenjing Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional matchmaker at a massive, global wedding agency. You have thousands of clients, and for every single person, you have a mountain of messy paperwork: their long life stories, their old resumes, a list of every party they’ve ever attended, and a diary of every person they’ve ever "liked" on a dating app.

If you tried to read every single page for every single client every time a new potential partner walked through the door, you’d never get anything done. You’d be overwhelmed, slow, and you’d probably forget the important details because of all the "noise."

This paper describes how LinkedIn solved this exact problem using Artificial Intelligence.

The Problem: The "Information Avalanche"

LinkedIn has millions of users. Each user has a huge amount of data: their profile, their work history, and their recent job searches.

In the past, computers tried to turn this data into long lists of numbers (called "embeddings"). But numbers are like a secret code—they are efficient, but if a human looks at them, they make no sense. Also, if you change the code slightly, the whole system breaks.

The new way is to use Large Language Models (LLMs)—the same tech behind ChatGPT. LLMs love text, but they have a "short attention span." If you feed them a 50-page biography of a user, they get "distracted" by the boring parts and miss the important stuff.

The Solution: The "Smart Editor" (Reinforcement Learning)

Instead of just dumping all the data into the AI, the researchers built a "Smart Editor."

Think of this Editor as a highly trained journalist. The journalist’s job is to read the massive mountain of user data and write a "High-Fidelity Synopsis"—a tiny, punchy, 150-word summary that captures only the most important bits (like "This person is a Cloud Engineer who loves Python and is currently looking for remote work in Berlin").

To train this Editor, they used Reinforcement Learning (RL). This is like training a puppy with treats:

The Action: The Editor writes a summary.
The Reward (The Treat): The system looks at what the user actually did next. Did the user click on a job? Did they apply? If the summary was so good that it helped the system predict the user's next move perfectly, the Editor gets a "treat" (a mathematical reward).
The Penalty (The "No"): If the Editor starts writing long, rambling essays just to try and "cheat" the system, the researchers give it a penalty to force it to stay concise.

The "Secret Sauce": Different Ways to Reward

The researchers experimented with different ways to give the "treats":

The "Yes/No" Reward: "Did you predict they would apply? Yes? Here is a treat!"
The "Ranking" Reward: Instead of just one job, the Editor is shown a list of jobs. If it can correctly rank them from "Most Likely to Apply" to "Least Likely," it gets a massive reward. This is like a matchmaker saying, "Don't just tell me if they like this person; tell me who their top 5 favorites are!"

The Results: A Smarter, Faster LinkedIn

By using this "Smart Editor," LinkedIn saw huge improvements:

Better Matches: The system became much better at showing people jobs they actually wanted to apply for.
Speed & Efficiency: Because the summaries are short and "clean," the AI can process them much faster.
Human-Readable: Unlike those "secret code" numbers from the past, these summaries are written in plain English. If a human engineer wants to see why the AI recommended a job, they can actually read the summary and understand it.

In short: LinkedIn taught an AI to be a world-class editor, turning a mountain of messy data into a tiny, powerful "cheat sheet" that helps connect people to their dream jobs.

Technical Summary: High Fidelity Textual User Representation over Heterogeneous Sources via Reinforcement Learning

1. Problem Statement

Large-scale professional platforms like LinkedIn face a significant challenge in modeling users for personalized recommendations (e.g., job search, alerts). Traditionally, user representations are either dense vectors (embeddings) or sparse features. Both have critical drawbacks:

Embeddings: They are "black boxes" (not interpretable), and any change to the encoder requires expensive re-embedding of the entire corpus and retraining of downstream models.
Sparse Features: They are manually engineered, brittle, and difficult to maintain.
LLM Incompatibility: Modern Large Language Models (LLMs) operate in text space. Integrating vectors or sparse IDs into LLM-based pipelines requires complex projection layers or fine-tuning, which increases latency and coupling.

The core problem is how to synthesize massive, redundant, and heterogeneous textual data (profiles, professional history, and search logs) into a unified, concise, and interpretable textual synopsis that maximizes predictive utility for downstream tasks.

2. Methodology

The authors frame the creation of user representations as a text-generation task optimized via Reinforcement Learning (RL).

A. RL Framework:

Actor: A medium-capacity (1.7B parameter) LLM that acts as the policy $\pi_\theta$ . It takes a heterogeneous context $q$ (profile, professional content, search queries, etc.) and generates a compact textual synopsis $o$ .
Reward Oracle: Instead of human labels, they use a high-capacity (30B parameter) LLM as a "preference oracle." This oracle evaluates the generated synopsis based on how well it predicts user engagement with new job postings.
Optimization Algorithm: They utilize Group Relative Policy Optimization (GRPO), which eliminates the need for a costly critic model by using group-relative baselines. They also incorporate DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) to prevent premature entropy collapse and maintain exploration.

B. Reward Design:
The researchers developed three distinct reward formulations:

Pointwise String-based Reward: A binary reward based on whether the model correctly predicts a specific action (e.g., "apply" vs. "not apply").
Pointwise Log-probability Reward: A denser reward using the model's predictive confidence. (Note: This was found to be prone to "reward hacking").
Entropy-Weighted Listwise Reward: A more sophisticated reward that ranks a list of jobs using NDCG (Normalized Discounted Cumulative Gain). It uses Shannon Entropy to weight the reward, giving more importance to lists with diverse user actions (apply, view, skip), which provides a stronger training signal.

C. Constraints and Regularization:
To prevent "reward hacking" (where the model generates excessively long text to increase the chance of including reward-relevant tokens), they implemented:

Explicit Prompting: Instructing the model to be concise.
Length Loss: A quadratic penalty for any output exceeding 150 tokens.
Format Penalty: A penalty for generating multiple paragraphs to ensure the output remains a single, grounded synopsis.

3. Key Contributions

Labeling-free Supervision: A scalable method to train representations using implicit engagement signals (clicks, applies) rather than expensive manual labeling.
Unified Textual Space: A representation that is natively compatible with LLM-based retrieval and ranking systems, reducing the need for intermediary projection layers.
Emergent Capabilities: The discovery that RL-trained models exhibit "latent profile imputation" (inferring missing skills from sparse data) and "feature discovery" (automatically identifying the most predictive user attributes).
Practical Scalability: A deployment strategy that uses a "seeker score" to intelligently sample and update only the members whose profiles or intents have changed.

4. Results

Offline Performance: The Pointwise String-based model achieved the highest improvement in pClick AUC (+3.50%) compared to the baseline. It also improved retrieval Recall@10 by 3%.
Online A/B Testing: Deployment in LinkedIn's job ranking system showed significant gains:
- CTR (Click-Through Rate): +1.48%
- Job Applications: +1.2%
- Session Success Rate: +0.8%
Qualitative Success: LLM-as-a-judge evaluations (using GPT-o1) confirmed high accuracy, coverage of heterogeneous sources, and high conciseness.

5. Significance

This work represents a shift from vector-based to text-based user modeling. By treating user representation as a generative RL problem, the authors provide a solution that is:

Interpretable: Humans can read and audit the user summaries.
Plug-and-Play: The text can be dropped directly into any LLM prompt without retraining.
Efficient: It solves the "unbounded context" problem by distilling thousands of tokens of raw data into a highly salient, low-latency summary.

This methodology is highly applicable to any large-scale recommendation system transitioning toward LLM-centric architectures.