CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

1. Problem Statement

While Large Language Models (LLMs) have advanced significantly in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing evaluation paradigms suffer from three main limitations:

• Lexical Overlap Metrics: Traditional metrics (BLEU, ROUGE) measure word matching rather than adaptation to user intent.
• Heuristic Limitations: Current heuristic approaches rely on surface-level similarity to historical user content, failing to capture deeper cognitive drivers.
• Generic "LLM-as-a-Judge": Existing judge models often use generic prompts or manual rules that lack specific, data-driven criteria for rigorous personalization assessment.

There is a lack of a standardized, fine-grained benchmark that quantifies how well an LLM aligns with a user's specific cognitive preferences rather than just their topic history.

2. Methodology

The authors propose a data-driven approach to mine user preferences and construct a new benchmark, CoPA.

A. Mining Community-Individual Preference Divergence (CIPD)

The core insight is the CIPD phenomenon, observed in online Q&A platforms like StackExchange. In CIPD scenarios, the answer endorsed by the community (top-voted) differs from the answer accepted by the individual user, even when both are high-quality.

• Data Source: The authors utilized the StackExchange dataset (173 communities, >100M monthly visitors).
• Signal: The divergence between the "Accepted" answer (user choice) and the "Top-voted" answer (community consensus) serves as a natural, scalable signal for personalized preference.
• Dataset Construction: They filtered for users with at least three CIPD instances, resulting in a dataset of ~626k questions from 15,963 users.

B. Factor Distillation Pipeline

To understand why users choose specific answers, the authors employed a three-step pipeline using LLMs:

1. Rationale Extraction: LLMs analyzed user QA trajectories (question, accepted answer, rejected top-voted answers, and historical context) to infer the cognitive rationales behind the user's choice.
2. Aggregation & Summarization: These rationales were aggregated and summarized by LLMs to identify recurring themes.
3. Expert Validation: Two psychology experts validated the distilled factors, ensuring they were grounded in educational and psychological theory.

This process distilled six core cognitive factors that govern personalization:

1. Cognitive Trust (CT): Alignment with the user's threshold for credibility and authority in a specific domain.
2. Situational Anchoring (SA): Calibration to the user's immediate context and specific problem constraints.
3. Schema Consistency (SC): Integration with the user's prior knowledge and existing mental models.
4. Cognitive Load Management (CLM): Tailoring complexity to match the user's processing capacity.
5. Metacognitive Scaffolding (MS): Providing structural support to foster critical thinking and self-regulated learning.
6. Affective and Motivational Resonance (AMR): Resonating with the user's emotional state and motivational orientation.

C. The CoPA Benchmark

Based on these factors, the authors constructed CoPA, a benchmark containing 1,985 curated user profiles across four domains (Engineering, Science, Lifestyle, Leisure).

• Profile Construction: For each user, an LLM quantifies their preference strength for each of the six factors based on their interaction history.
• Evaluation Metric: A 3-point Likert scale (0: Mismatch, 1: Partial Match, 2: Full Match) is used to score the alignment between a model-generated response and the user's factor profile.

3. Key Contributions

1. CIPD Analysis: A systematic exploration of the Community-Individual Preference Divergence phenomenon to extract interpretable cognitive factors from real-world data.
2. CoPA Benchmark: A novel, factor-driven benchmark enabling quantitative evaluation of personalized QA. It moves beyond generic metrics to assess alignment with specific cognitive dimensions.
3. Empirical Validation: Extensive experiments establishing baselines and demonstrating that factor-based evaluation is more discriminative than existing methods.

4. Experimental Results

The authors evaluated four approaches (No-Personalization, Time-Personalization, RAG-Personalization, and Profile-Personalization) using Qwen3-8B and GPT-4o-mini.

• Effectiveness of Factors:
  • The proposed factor-based scoring significantly outperformed baselines (Direct LLM scoring, CoT scoring, and lexical metrics like Jaccard) in distinguishing "Accepted" answers from "Top-voted" answers.
  • Accuracy: Achieved 51.43% – 55.37% (vs. ~12-31% for baselines).
  • Tie Rate: Reduced ties to 18.24% – 26.28% (vs. >55% for baselines), proving the factors can discriminate subtle user preferences.
• Generalizability:
  • Integrating the six factors into system prompts improved performance on external benchmarks (UPGC-QA and LaMP-QA) across different models and domains, confirming the factors are not overfit to StackExchange.
• Performance of Personalization Methods:
  • Profile-Personalization (constructing structured user profiles) yielded the best results, outperforming Time-based and RAG-based approaches.
  • Qwen3-8B generally outperformed GPT-4o-mini on this benchmark.
  • Domain Disparity: Models performed well in Engineering/Tools but struggled in Science/Theory, highlighting the difficulty of personalizing for deep reasoning and specialized knowledge.
• Factor-Level Insights:
  • Personalization significantly improved scores in Cognitive Trust, Schema Consistency, and Situational Anchoring.
  • Metacognitive Scaffolding remained the most challenging dimension, with all models scoring low, indicating a gap in current LLMs' ability to induce user reflection.
  • Human Validation: Human annotators showed substantial agreement with the LLM evaluator (Cohen's Kappa = 0.643), validating the benchmark's reliability.

5. Significance

• Paradigm Shift: CoPA shifts the evaluation of personalized QA from surface-level lexical matching to cognitive alignment. It provides a theoretical framework for why a response feels personalized.
• Actionable Metrics: By breaking personalization into six distinct, measurable factors, the paper offers a roadmap for developers to debug and improve specific aspects of personalization (e.g., reducing cognitive load vs. increasing trust).
• Foundation for Future Research: The benchmark and the identified factors serve as a standard for developing next-generation personalized agents, particularly in educational and complex reasoning scenarios where "one size fits all" fails.

In conclusion, CoPA demonstrates that data-driven mining of user divergence, combined with cognitive psychology principles, provides a robust and discriminative standard for evaluating and improving personalized LLMs.