💻 computer science

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

This paper investigates how persona conditioning affects LLM-based information retrieval evaluation, revealing that while specific assessor roles and model capacity significantly influence judgment strictness and local ranking stability, high-capacity models generally preserve overall system rankings, thereby establishing persona-conditioned judging as a controlled diagnostic tool for stress-testing IR evaluation pipelines.

Original authors: Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie, Gianluca Demartini

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie, Gianluca Demartini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decide which movie is the best of the year. You ask a group of friends to watch them and give a score. But here's the twist: you ask one friend to judge like a strict film critic who hates slow pacing, another to judge like a teenager looking for cool special effects, and a third to judge like a parent checking for educational value. Suddenly, the "best" movie might change depending on who you asked! This is the heart of Information Retrieval (IR) evaluation, the science of figuring out how well search engines find the right answers. For decades, humans have done this judging, but it's slow and expensive. Now, scientists are using Large Language Models (LLMs)—super-smart AI chatbots—to do the judging instead. The big question is: if we ask the AI to "act" like different types of people, does it change the results? Does the AI's answer depend on the "mask" it's wearing?

This paper dives into that exact question. The researchers wanted to see if giving an AI a specific "persona" (like "a doctor" or "a lawyer") changes how it ranks search results, or if it just gives the same answer no matter what. They treated these personas like a stress test, poking and prodding the AI to see how shaky its opinions are. They found that while the AI doesn't completely flip its entire list of winners and losers, it does get a bit fussy. The results suggest that the AI's judgment isn't a fixed truth; it shifts slightly depending on the role it's playing, and this shift matters more for smaller, less powerful AI models than for the big, brainy ones.

The Story of the Shapeshifting Judge

In the world of search engines, we need to know if a new algorithm is actually better than the old one. To find out, we run a bunch of test searches and ask an "assessor" to grade how good the results are. Traditionally, this was done by humans. But humans are tired, expensive, and sometimes disagree with each other. So, researchers started using AI models as judges. The idea was that if you prompt the AI correctly, it can act like a professional human judge.

But here's the catch: AI is sensitive. If you change the way you ask a question, the answer might change. This paper asks: What happens if we change the identity of the judge? Instead of just asking the AI to "be a judge," the researchers told it to "be a doctor," "be a lawyer," or "be a skeptical fact-checker." They wanted to see if these different "masks" would make the AI change its mind about which search results were good.

The Experiment: Dressing Up the AI

The researchers set up a massive experiment using two different sets of search questions (called datasets). One set was about factual, short questions (like "Who won the Nobel Prize?"), and the other was about complex, open-ended topics (like "How should teachers handle state testing?").

They took six different AI models—some huge and powerful, some smaller and lighter—and asked them to judge the search results. But they didn't just ask them to judge; they gave them specific roles:

The Query-Aligned Judge: Focused on what the user meant.
The Domain Expert: Focused on specific fields like health or law.
The Orthogonal Judge: A "contrarian" who looked at things from a totally different angle.
The Evidence Verifier: A strict fact-checker who demanded proof.
The Global Assessor: A standard, professional rater.

They also tried two different ways of creating these roles: one using abstract descriptions (like "a parent") and another using detailed skill lists (like "a person with skills in curriculum development").

What They Found: The AI is a Mood Ring, Not a Rock

The results were fascinating and a bit surprising. The AI didn't completely lose its mind. When they changed the persona, the AI didn't suddenly decide that a terrible result was perfect and a perfect result was terrible. Instead, the changes were more subtle, like a mood ring shifting colors.

1. The "Strictness" Shift
Most of the time, the AI's rankings stayed pretty close to the standard baseline. However, the personas did change how strict the AI was. For example, the "Evidence Verifier" persona tended to be stricter, giving lower scores to results that didn't have solid proof. The "Contrarian" persona was more likely to disagree with the standard view. But crucially, these changes usually just nudged the scores up or down a little bit; they didn't cause a total chaos where the entire list of winners was scrambled.

2. Size Matters: Big Brains vs. Small Brains
This was a major finding. The biggest, most powerful AI models (like the 70-billion-parameter ones) were very stable. No matter what "mask" they wore, they kept their rankings mostly the same. They were like a wise old judge who knows the rules and doesn't get swayed by a costume change.
However, the smaller, less powerful AI models were much more sensitive. When you put a "contrarian" mask on a small AI, it got confused and its rankings jumped around a lot. It suggests that if you use a weak AI to do the judging, the results might be unreliable just because you asked it to "act" a certain way.

3. The "Contrarian" is the Most Disruptive
Among all the roles, the "Orthogonal" (or contrarian) persona caused the biggest shifts in rankings. This makes sense: if you tell an AI to look at things from a completely different angle, it's going to change its mind more than if you tell it to be a "domain expert." The researchers found that using a "skill-grounded" contrarian (one with a specific list of skills) caused even more movement than a generic one.

4. It's Not About Who the Persona Is, But What They Do
The researchers wondered if the source of the persona mattered. Did it matter if the persona came from a database of abstract descriptions or a database of professional skills? They found that it didn't matter much. Whether the AI was told to be a "sociology professor" or a "person with skills in curriculum development," the effect was similar. What mattered most was the role (e.g., being a fact-checker vs. being a contrarian) and the size of the AI model.

Why This Matters

This paper doesn't say that AI judges are useless. In fact, it suggests they are quite good at keeping the big picture stable, especially the big, powerful models. However, it warns us that AI judges aren't perfect truth-tellers. Their opinions can be influenced by how we frame the task.

The authors suggest that we should use these "persona changes" as a diagnostic tool. Think of it like a stress test for a bridge. If you shake the bridge a little bit (change the persona) and it wobbles too much, you know the bridge (or the search system) has a weak spot. By seeing which search systems change their ranking when the AI's "mood" changes, researchers can find systems that are too sensitive to how they are evaluated.

In short, the paper teaches us that while AI can be a great judge, we need to be careful about the "costume" we put it in. If we want reliable results, we should stick to the biggest, smartest models and understand that the way we ask the AI to judge can subtly nudge the outcome. It's a reminder that even in the world of artificial intelligence, perspective matters.

Technical Summary: Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

Problem Statement

Large Language Models (LLMs) are increasingly adopted as automated relevance assessors in Information Retrieval (IR) evaluation to address the scalability and cost limitations of human judging. While prior work suggests LLMs can approximate human preferences under specific prompting, their judgments remain sensitive to prompt formulation, context, and superficial cues. A critical gap exists in understanding how systematic variations in the assessor perspective—specifically through persona conditioning—affect relevance labels and downstream system rankings. Unlike previous studies focusing on prompt optimization or criteria decomposition, this work investigates whether varying the "who" (the assessor role) rather than the "what" (the assessment criteria) introduces structured sensitivity or uniform instability in IR evaluation outcomes.

Methodology

The study employs a controlled experimental design to treat persona conditioning as a diagnostic probe for assessor sensitivity.

Datasets and Models:
Experiments were conducted on two benchmark datasets: TREC Deep Learning 2020 (DL20), featuring short factual queries, and TREC RAG24, featuring longer, open-ended queries often requiring explanatory context. Six LLM backbones were evaluated, ranging from high-capacity models (GPT-4o, LLaMA-3.1-70B, Qwen-2.5-72B) to mid-capacity (GPT-4o-mini) and smaller variants (LLaMA-3.1-8B, Qwen-2.5-7B).

Assessor Personas:
Five distinct task-oriented assessor roles were instantiated to perturb the evaluation perspective:

Query-Aligned: Focuses on semantic alignment with inferred user intent.
Domain-Expert: Emphasizes topical expertise within a specific domain (mapped via a 9-domain taxonomy).
Orthogonal: Introduces a contrastive perspective by deviating from dominant query interpretations.
Evidence-Verification: Prioritizes factual correctness and source credibility, penalizing unsubstantiated claims.
Global Assessor (GAP): Acts as a professional search-quality rater applying standard guidelines.

These roles were instantiated using two persona sources: PersonaHub (abstract, synthetic profiles) and USPersona (NVIDIA Nemotron-Personas-USA, skill-grounded professional profiles).

Experimental Setup:
To ensure scalability and control, the study utilized summary-based judging, where documents were summarized once (approx. 80 tokens) and reused across all persona conditions. This fixed the document representation, isolating the effect of the assessor perspective. The UMBRELA framework served as the standard baseline. Evaluation metrics included:

Judgment Level: Quadratic-weighted Cohen's $\kappa$ against UMBRELA and human judgments.
System Level: Kendall's $\tau$ and Rank-Biased Overlap (RBO) comparing LLM-derived system rankings to human rankings.
Sensitivity Analysis: Localized rank-displacement analysis ( $\Delta r$ ) to identify specific systems and system types most affected by persona shifts.

Key Contributions and Results

1. Structured, Model-Dependent Sensitivity
The study reveals that persona conditioning does not produce uniform evaluation instability. Instead, sensitivity is structured and heavily dependent on model capacity.

High-Capacity Models: Models like LLaMA-3.1-70B and Qwen-2.5-72B maintain strong agreement with the UMBRELA baseline and human rankings. Persona conditioning primarily shifts assessment strictness or evidential thresholds rather than causing widespread relevance reversals.
Mid-Capacity Models: GPT-4o-mini exhibits similarly stable behavior to high-capacity models, maintaining strong agreement across datasets.
Low-Capacity Models: Smaller models (e.g., LLaMA-3.1-8B, Qwen-2.5-7B) exhibit significantly higher instability, with lower agreement to baselines and greater variance in system rankings.

2. Judgment-Level Effects
Persona-conditioned judgments generally remain close to the UMBRELA baseline. Differences manifest as localized shifts in:

Assessment Strictness: Some personas (e.g., Evidence-Verification) tend to be stricter.
Interpretation Emphasis: Orthogonal personas induce larger deviations by challenging standard query interpretations.
Human Agreement: Persona conditioning does not uniformly improve agreement with human judgments; in some cases (notably GPT-4o on RAG24), it moves judgments further away from human preferences.

3. System Ranking Stability
Global system rankings remain relatively stable for high-capacity models across datasets. However, local rank displacements are significant for specific systems.

System Types: Sensitivity is concentrated in particular retrieval architectures. On DL20, transformer-based neural ranking/reranking systems (e.g., bigIR-DCT-T5-F) show high sensitivity. On RAG24, retrieval-augmented and generation-oriented pipelines (e.g., iiia_standard_) are most affected.
Rank Displacement: The USPersona Orthogonal role consistently induced the highest mean absolute rank displacement, acting as a strong sensitivity probe.

4. Persona Source vs. Role
The source of the persona (abstract PersonaHub vs. skill-grounded USPersona) has a secondary effect compared to the specific assessor role and model capacity. While USPersona sometimes amplified shifts, the fundamental stability patterns were driven by the role instruction and the model's ability to adhere to it.

Significance and Implications

The paper positions persona-conditioned judging not as a method for generating "better" relevance labels, but as a controlled diagnostic stress test for LLM-based IR evaluation pipelines.

Diagnostic Utility: By systematically varying the assessor perspective, researchers can identify retrieval systems whose evaluation outcomes are fragile and sensitive to framing. This helps distinguish between robust system performance and artifacts of a specific judging configuration.
Evaluation Robustness: The findings suggest that while high-capacity LLMs provide stable global rankings, they may still mask localized sensitivities in specific system types (e.g., RAG pipelines).
Methodological Guidance: The study cautions against relying on single-perspective LLM judging for definitive system comparisons, particularly when using lower-capacity models. Instead, it advocates for using persona variation to characterize the sensitivity of evaluation outcomes, thereby providing a more nuanced view of retrieval system effectiveness.

The authors conclude that persona conditioning reveals where evaluation outcomes depend on assessor assumptions, offering a mechanism to stress-test IR evaluation pipelines without requiring additional human annotation.