💬 NLP

Preference-Aware Rubric Learning for Personalized Evaluation

This paper introduces PARL, a framework that addresses the limitations of existing personalized evaluation methods by learning preference-aware rubrics directly from user interaction histories through a self-validating, discriminative reinforcement learning approach to ensure reliable, consistent, and fine-grained assessment of user-aligned LLM responses.

Original authors: Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yuxin Chen, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Yoko Yamakata, Tat-Seng Chua

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yuxin Chen, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Yoko Yamakata, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, helpful robot assistant. Right now, this robot is great at writing stories, emails, or reviews for everyone in the same generic way. But you want it to write exactly like you do, using your specific jokes, your unique sentence structure, and your personal taste. This is called "personalization."

The big problem the paper tackles is: How do we know if the robot is actually sounding like you and not just sounding like a generic robot trying to sound like you?

Current ways of checking this are like trying to grade a student's essay with a ruler that only measures length. They miss the soul of the writing. This paper introduces a new way to build a custom "grading rubric" for every single user, learning directly from their past writing to create a perfect checklist for what "sounding like them" actually means.

Here is the breakdown of their solution, PARL, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Ruler

Imagine you are a chef who always adds a pinch of cinnamon to your coffee.

Old Way (Automatic Metrics): A robot checks your coffee and says, "It has coffee beans and liquid." It misses the cinnamon entirely because it's looking for generic ingredients.
Old Way (Human Judges): You ask a stranger to taste your coffee. They say, "It tastes good!" but they don't know your secret rule about cinnamon. They can't tell if the robot added it or if it just tasted generally "coffee-y."
Old Way (Standard AI Judges): You ask a super-smart AI to judge. It says, "This is a high-quality coffee." But it doesn't know your specific preference for cinnamon; it just knows what "good coffee" usually tastes like.

The paper says we need a custom ruler that knows exactly how much cinnamon you like, based on every cup of coffee you've ever made.

2. The Solution: Learning Your "Secret Sauce" (PARL)

The authors propose a system called PARL (Preference-Aware Rubric Learning). Instead of guessing your preferences, PARL studies your past writing (your "history") and builds a Personalized Rubric.

Think of a Rubric as a detailed checklist. For a specific user, the checklist might say:

Rule 1: "Must use short, punchy sentences."
Rule 2: "Must avoid the word 'very'."
Rule 3: "Must mention the weather at the start."

PARL doesn't just guess these rules; it learns them. It looks at thousands of things you've written and figures out the hidden patterns that make your writing yours.

3. The Three Golden Rules of a Good Rubric

To make sure this custom checklist is actually useful, the paper says it must follow three rules:

Representativeness (The "Snapshot"): The checklist must capture the whole picture of you. It can't just look at one email you wrote when you were angry; it needs to see your happy emails, your work emails, and your weekend posts to understand your true style.
User-Consistency (The "Steady Hand"): The rules must be stable. If the checklist says "You always use emojis," but you only used them once in a year, that's a bad rule. The system checks: "Does this rule hold true for all your past writing?" If not, it throws the rule away.
Discriminativeness (The "Taste Test"): This is the most important part. The checklist must be able to tell the difference between You and a Robot trying to sound like You.
- Analogy: Imagine a robot writes a story that is grammatically perfect and very long. You might write a story that is short, messy, and full of slang. A generic judge might say the robot's story is "better." But your custom checklist knows: "No! Real you writes short and messy." The system is trained to give a high score to your messy style and a low score to the robot's perfect style, even if the robot's style looks "better" to everyone else.

4. How It Works: The "Training Camp"

The system learns this in two main steps:

Drafting the Rules: It reads your history and writes a draft checklist.
The "Self-Verification" Drill: It tests this draft against your past writing.
- Step A: "Does this rule fit your old emails?" If yes, keep it. If no, delete it.
- Step B: "Can this rule tell the difference between your writing and a generic AI's writing?" The system pits your real writing against a bunch of AI-generated writing. It learns to adjust the rules so that your writing gets a high score and the AI writing gets a low score. This is like a coach teaching a referee how to spot the difference between a pro athlete and a look-alike.

5. The Results: A Perfect Match

The paper tested this on real-world tasks like writing Amazon reviews, Reddit posts, and news headlines.

The Finding: When they used PARL's custom checklists, the system could perfectly identify when a text was written by the real user versus when it was written by a robot.
The Comparison: Standard AI judges (the "generic referees") often got confused and gave high scores to robots that didn't actually sound like the user. PARL's custom rubrics were much sharper, catching the subtle differences that others missed.
The Coverage: The system worked for almost every user it tried, successfully creating a unique "style guide" for each person.

Summary

In short, this paper says: Don't try to evaluate personalized AI with a generic ruler. Instead, build a custom measuring tape for every single person by studying their past work. This new method, PARL, learns exactly what makes a person unique, creates a strict checklist of those traits, and uses it to spot the difference between a real human and a robot pretending to be one. It turns the vague idea of "sounding like you" into a concrete, learnable set of rules.

Technical Summary: Preference-Aware Rubric Learning for Personalized Evaluation

1. Problem Definition

As Large Language Models (LLMs) transition from general-purpose assistants to user-centric agents, the ability to align model behavior with individual user preferences has become critical. However, evaluating this personalization remains a significant bottleneck. Existing evaluation paradigms fail to capture the subjective, user-specific criteria embedded in long-term interaction histories:

Automatic Metrics (e.g., ROUGE, BERTScore): Rely on surface-level lexical overlap with a single objective reference, failing to account for valid alternative responses that align with specific user preferences.
Human Evaluation: Suffers from an "inferential gap" where external annotators lack direct access to a user's latent preferences, making faithful assessment difficult.
LLM-as-a-Judge: While leveraging semantic capabilities, these approaches typically rely on static, handcrafted prompts and holistic scores. They lack adaptability to individual users and fail to provide fine-grained, interpretable grounding for their judgments.

The core challenge is to move beyond static judgment toward a Personalized Evaluation as Learning paradigm, where evaluation criteria are adaptively learned from user history to satisfy three essential principles: Representativeness (faithfully distilling latent preferences), User-Consistency (reflecting stable preferences across contexts), and Discriminativeness (distinguishing authentic user responses from high-quality generic AI outputs).

2. Methodology: PARL Framework

The authors propose PARL (Preference-Aware Rubric Learning), a framework that formulates personalized evaluation as a learning problem. PARL induces structured, multi-dimensional evaluation rubrics directly from raw user historical behaviors ( $H_u$ ) and optimizes them via reinforcement learning. The framework operates through two core modules:

2.1 Preference Induction & Consistency Validation

This module transforms raw historical contexts into a compact set of preference-aware rubrics ( $R_u$ ).

Rubric Generation: A generator model $G$ induces a candidate set of atomic, single-dimensional rubrics ( $\tilde{R}_u$ ) from a seed history ( $\tilde{H}_u$ ). Unlike holistic summaries, each rubric captures a specific stylistic, structural, or semantic invariant.
Self-Verification: To ensure User-Consistency, the candidate rubrics undergo a self-verification process. A rubric is retained only if it demonstrates universal satisfaction across the diverse seed history samples. Formally, a rubric $r_j$ is kept if:
$\frac{1}{|\tilde{H}_u|} \sum_{(x_i, y_i) \in \tilde{H}_u} I(y_i \models r_j) \geq \tau$
where $\tau$ is a high consistency threshold (e.g., 1.0). This filters out transient or task-specific artifacts.

2.2 Discriminative Optimization via RL

To ensure Discriminativeness, the framework employs a Reinforcement Learning (RL) stage to sharpen the decision boundaries between authentic user-authored responses and competitive AI-generated alternatives.

Objective: The generator $G_\theta$ is optimized to maximize the scoring margin between the ground-truth user response ( $y_T$ ) and a set of negative samples ( $Y_{neg}$ ) generated by competitive personalized methods.
Reward Formulation: The framework defines a Discriminative Margin Product $\Gamma(R_u)$ $Γ (R_{u})$ , representing the joint probability that the ground-truth response is superior to all negatives. Two variants are proposed:
- PARL-A (GT-Scaled): Integrates the absolute satisfaction of the ground truth with the relative margin ( $R_A = \lambda \cdot C \cdot S(y_T; R_u) \cdot \Gamma(R_u)$ ).
- PARL-B (Margin-Only): Relies solely on the relative margin ( $R_B = \lambda \cdot C \cdot \Gamma(R_u)$ ).
Optimization: The model is trained using Group Relative Policy Optimization (GRPO), which explores the space of evaluative criteria without requiring a separate value function.

2.3 Evaluation Mechanism

The final induced rubrics $R_u$ are used to compute a personalized alignment score $S$ for a new model output $\hat{y}$ :
$S = \frac{1}{m} \sum_{j=1}^{m} I(\hat{y} \models r_j)$
This provides a verifiable, granular assessment where the score is the normalized average of binary judgments across all learned dimensions.

3. Key Contributions

Paradigm Shift: The paper introduces Personalized Evaluation as Learning, formalizing evaluation as a learnable process grounded in Representativeness, User-Consistency, and Discriminativeness, rather than a static judgment.
PARL Framework: A novel framework that induces preference-aware rubrics from user histories and optimizes them using a discriminative reinforcement learning objective to capture precise, user-specific decision boundaries.
Empirical Validation: Extensive experiments on real-world personalized text generation tasks (Amazon Review, Reddit Topic, News Headline) demonstrate that PARL induces high-fidelity rubrics that outperform existing baselines in identifying user-aligned responses.

4. Experimental Results

Experiments were conducted on three datasets using Qwen3 models as backbones. Key findings include:

Superior Alignment: PARL-A and PARL-B variants consistently achieved the highest User-level Accuracy (the proportion of times the rubric correctly identified the ground-truth user response as satisfying all criteria) across all tasks. For instance, on Amazon Review Generation, PARL-A achieved 0.931 accuracy for Ground Truth (GT), significantly outperforming static baselines like LM-8B (0.931) and LM-235B (0.794).
Discriminative Power: The Max-Diff metric (difference between GT accuracy and the strongest non-GT baseline) was significantly positive for PARL variants (e.g., +0.026 for Amazon Review), whereas static models often showed near-zero or negative margins, indicating they could not distinguish authentic user style from AI outputs.
Robustness and Coverage: PARL variants achieved near-complete User Coverage (~99-100%), whereas static pre-trained models suffered from low coverage (e.g., 49.1% for LM-8B) due to frequent generation of unusable or hallucinated criteria.
Generalization: In out-of-domain (OOD) evaluations, PARL-A and PARL-B maintained strong discriminative capabilities, suggesting the learned rubrics capture persistent stylistic invariants rather than dataset-specific noise.
Semantic Diversity: Intrinsic analysis revealed that PARL-induced rubrics exhibit structured scattering in semantic space, capturing diverse user traits, whereas static models and PARL-0 (without RL) showed semantic collapse or narrow clustering.

5. Significance and Claims

The paper claims that PARL addresses a critical bottleneck in LLM personalization by providing a reference-free, explicit, and interpretable evaluation framework. Unlike reward modeling approaches that entangle preference representation with policy optimization, PARL decouples evaluation from generation, enabling principled benchmarking across models and tasks.

The authors emphasize that their work advocates for treating personalized evaluation as a learnable, adaptive process driven by the inference of preferences from user history, rather than a static judgment. The induced rubrics serve as high-fidelity metrics that not only evaluate alignment but also offer insights into the structure of user-specific evaluative preferences.

Limitations: The authors modestly acknowledge that the framework's effectiveness depends on the quality and quantity of user history (struggling in cold-start scenarios) and assumes preferences are static signatures at a single point in time, without explicitly modeling temporal evolution. They also note that while PARL achieves promising results, it has not yet reached a state where GT user-level accuracy is consistently 1.0 and coverage is 100% for all users, particularly those with inconsistent preferences.