Learning from Natural Language Feedback for Personalized Question Answering

Technical Summary: Learning from Natural Language Feedback for Personalized Question Answering (VAC)

1. Problem Statement

Personalization in Large Language Models (LLMs) is essential for information-seeking tasks like Question Answering (QA), where responses must align with a user's unique background, intent, and preferences. Current state-of-the-art approaches typically use Retrieval-Augmented Generation (RAG) combined with Reinforcement Learning (RL). However, these RL methods rely on scalar reward signals (single numbers indicating quality).

The authors identify three critical flaws in scalar rewards:

• Weak Supervision: They indicate if a response is good or bad but fail to explain why or how to improve it.
• Inefficiency: The model must undergo extensive exploration to infer necessary adjustments, leading to slow convergence.
• Suboptimality: Supervised training on a single ground-truth response is often insufficient because there are many valid ways to satisfy a user.

2. Methodology: The VAC Framework

The authors propose VAC (Verbal-Alignment for Customization), a framework that replaces scalar rewards with Natural Language Feedback (NLF). The core idea is to use a "Feedback Model" to provide rich, actionable, and interpretable guidance to a "Policy Model" during training.

The Iterative Training Loop:
The framework operates through an alternating optimization process over $T$ iterations:

1. Optimizing the Feedback Model ($\phi_\omega$):
   • The system generates $N$ diverse feedback candidates using the previous iteration's feedback model.
   • The policy model attempts to revise its initial response using each candidate.
   • The feedback that results in the highest improvement (measured by a task-specific metric $\mu$ based on user rubrics) is selected.
   • The feedback model is then fine-tuned via supervised learning to generate that "winning" feedback.
2. Optimizing the Policy Model ($\pi_\theta$):
   • The newly optimized feedback model generates high-quality NLF for the current policy model's outputs.
   • The policy model uses this NLF to refine its responses.
   • The policy model is then fine-tuned via Supervised Fine-Tuning (SFT) to internalize these improvements.
   • Crucially, this allows the policy model to generate the improved, personalized response directly during inference, without needing the feedback model.

Inference: At test time, the system uses a standard RAG pipeline (Retriever $\rightarrow$ Policy Model) to generate the final response in a single pass.

3. Key Contributions

• Novel Supervision Signal: The first major attempt to use NLF as a primary training signal for personalization, moving beyond coarse scalar rewards.
• Co-adaptation Mechanism: An iterative training loop where the feedback model and policy model evolve together, ensuring the feedback remains relevant to the model's current capabilities.
• Amortized Personalization: A method that shifts the computational burden of feedback to the training phase, allowing for high-quality personalized inference without the latency of iterative refinement at test time.

4. Experimental Results

The framework was evaluated on the LaMP-QA benchmark across three domains (Arts, Lifestyle, and Society).

• Quantitative Performance: VAC achieved a 13.6% relative improvement over non-personalized baselines and a 6.0% improvement over state-of-the-art RL methods using scalar rewards. It also outperformed the "PlanPers" baseline (which uses high-level response planning).
• Efficiency: VAC was 1.9× more efficient in inference time than PlanPers, despite providing higher quality.
• Human Evaluation: In side-by-side comparisons, human annotators preferred VAC in 44% of cases, tied in 33%, and found it inferior in only 23% of cases.
• Ablation Insights:
  • Model Size: Larger feedback models (e.g., 7B vs 1.5B) provide significantly better guidance.
  • Optimization: Jointly training both models is superior to keeping the feedback model frozen.
  • SFT vs. DPO: Supervised Fine-Tuning (SFT) was found to be more effective than Direct Preference Optimization (DPO) for this specific task as the model converges.

5. Significance

This work shifts the paradigm of personalized LLM training from "reward-seeking" to "instruction-following." By treating personalization as a process of learning from linguistic guidance rather than numerical scores, VAC enables models to capture the nuances of human preference more effectively. This has broad implications for any human-centered AI system where user intent is subjective, complex, and multi-faceted.