Technical Summary: Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering

Problem Definition

Personalized Question Answering (PQA) aims to generate user-tailored responses, a task that requires aligning answers with the user's underlying intent—the implicit "why" behind a query beyond its explicit wording. While existing approaches utilize multi-turn conversational context or rich user profiles, they often fail to explicitly model user intent during the reasoning process. This limitation is particularly acute in single-turn settings, where the model must infer a latent goal from minimal input and integrate it into the thinking process. Current methods typically rely on prompting to infer intent at inference time without incorporating it into the model's training objective or reasoning trajectory, and they often neglect the role of intent-free responses, leading to generic outputs that may be coherent but misaligned with the user's specific needs.

Methodology: IAP Framework

The authors propose IAP (Intent Aware Personalization), a Reinforcement Learning (RL) framework designed to train Large Language Models (LLMs) to infer implicit user intents from single-turn questions and incorporate them into the reasoning process before generating answers.

1. Structured Reasoning Schema

IAP utilizes a tag-based schema to structure the generation process, making latent goals explicit before the final answer is produced. For a given input question $x_i$, the model generates a structured output containing:

• <thought>: Internal reasoning about the question.
• <intent>: A numbered list of candidate latent intents (e.g., seeking emotional support vs. practical guidance).
• <thought>: A second reasoning phase explicitly conditioning on the inferred intents to plan the response.
• <answer>: The final personalized response.

2. Optimization via DAPO

The framework employs Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO). For each training instance, the policy samples a group of $G$ rollouts (latent-intent trajectories). The policy is optimized using a personalized reward function that evaluates the quality of the answer trajectory.

3. Personalized Reward Function

The core innovation lies in the reward design, which consists of three weighted components:

• Personalized Reward ($r_{pers}$): Based on the LaMP-QA evaluation metric, this scores the generated answer against user-specific rubric aspects (e.g., addressing underlying needs and goals) using a frozen LLM judge (Qwen2.5-32B-Instruct).
• Contrastive Intent-Free Reward ($r_{contrast}$): To discourage generic generation, the framework generates an intent-free response ($\bar{y}_i$) using a frozen reference model. The reward penalizes the policy if its intent-aware response is semantically similar to the intent-free response (via BERTScore) or fails to outperform it in personalization quality. This forces the model to distinguish its output from a non-contextualized baseline.
• Intent Length Regularization ($r_{length}$): A penalty term discourages verbose intent descriptions, encouraging concise representations within the <intent> tag.

The total reward is calculated as:
$$r_{i,g} = \lambda_1 r_{pers} - \lambda_2 r_{contrast} - \lambda_3 r_{length}$$

Key Contributions

1. IAP Framework: Introduction of an RL-based framework that trains LLMs to infer implicit user intents from single-turn inputs and integrate them into the reasoning process via a tag-based schema, rather than relying solely on inference-time prompting.
2. Contrastive Reward Mechanism: Development of a reward function that explicitly contrasts intent-aware responses with intent-free baselines during training. This design discourages generic generation and pushes the model toward outputs that are sensitive to user intent.
3. Comprehensive Evaluation: Extensive evaluation across six LLMs (Qwen2.5, Gemma3, Mistral-Small) on the LaMP-QA benchmark, demonstrating consistent improvements over strong baselines.

Experimental Results

The authors evaluated IAP on the LaMP-QA dataset, covering three domains: Art & Entertainment, Lifestyle & Personal Development, and Society & Culture.

• Performance Gains: IAP consistently outperformed all baselines, including Non-personalized, Implicit Intent Personalization (CoT prompting), Explicit Intent Personalization, and Supervised Fine-Tuning (SFT).
• Quantitative Improvement: Across all six models, IAP achieved an average macro-score gain of approximately 7.5% over the strongest competitor.
• Statistical Significance: Improvements were statistically significant ($p < 0.01$) across nearly all models and domains.
• Ablation Studies:
  • Intent Quality: Replacing the inferred <intent> with no intent or random intents caused substantial performance drops, confirming that the quality of the inferred intent signal is critical.
  • Contrastive Reward: Removing the intent-free contrastive reward led to consistent performance degradation, validating the necessity of penalizing generic responses.
  • Intent Count: Performance was highest when the number of inferred intents was low (1–2), suggesting that excessive intent generation can complicate response quality.
  • Group Size: A group size of 5 rollouts per instance yielded the optimal balance for policy optimization.

Significance and Claims

The paper claims that modeling implicit user intent directly within the training objective is a promising direction for PQA. The results suggest that:

• Integrating intent into the model's thinking process (via the tag-based schema) is more effective than treating intent as an external cue or prompt.
• Contrastive learning against intent-free baselines is essential for training models to produce responses that are genuinely personalized rather than just contextually coherent.
• The framework is effective even in single-turn settings where rich conversational history is unavailable, bridging the gap between intent modeling and practical, single-shot personalized QA.

The authors conclude that IAP successfully trains LLMs to explicitly model the user's implicit "why," resulting in responses that better align with underlying user goals.