Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

This paper proposes an anonymous benchmarking method to expose the bias in current role-playing evaluations caused by character name exposure and demonstrates that incorporating self-generated personality traits effectively enhances model performance under these fairer conditions.

The Gist

Imagine you are hiring an actor to play a famous movie character, like Harry Potter, in a new play.

The Old Way (The "Name Drop" Problem)
In the past, researchers tested AI actors by simply telling them, "You are Harry Potter." The AI would then start acting like Harry. But here's the catch: The AI didn't actually learn how to be Harry from the script you gave it. It just remembered everything it had read about Harry Potter in its massive training data (like a student who memorized the entire encyclopedia).

If you asked the AI, "What is Harry's favorite color?" it didn't need to think; it just recalled a fact it already knew. This made the AI look great, but it was a bit of a cheat. It wasn't proving the AI could act; it was proving the AI had a good memory.

The New Experiment: The "Blind Audition"
The authors of this paper wanted to see if AI could really act, not just remember. So, they tried something called Anonymous Benchmarking.

Imagine the director walks into the audition room and says:

"You are an orphaned boy who discovers a magical world. You are brave, a bit clumsy, and love flying on a broomstick. But we cannot tell you your name."

Suddenly, the AI can't rely on its memory of "Harry Potter." It has to build the character from scratch using only the description provided.

The Result:
When they removed the name, the AI's performance dropped significantly. It stumbled, sounded less like the character, and made mistakes. This proved that the AI was previously "cheating" by using its memory of the name rather than truly understanding the role. The "Anonymous" test is like a fairer, stricter audition that shows what the AI can actually do.

The Solution: Giving the AI a "Personality Cheat Sheet"
Once the AI was struggling in the blind audition, the researchers asked: How do we help it act better without giving it the name?

They tried giving the AI a Personality Profile. Think of this like giving an actor a detailed character sheet that says:

• "You are an INTJ (a logical, strategic, and private type of person)."
• "You are Open to new ideas but Neurotic (easily stressed)."

They tested two ways to get this profile:

1. The Human Expert: A real person reads the character's story and writes down their personality type (like a casting director).
2. The AI Detective: The AI reads the story itself and guesses, "Hmm, this character seems like an INTJ," and writes down its own guess.

The Big Surprise:
The researchers found that giving the AI a personality profile made it act much better, even without knowing the character's name.

Even more surprisingly, the AI's own guess about the personality worked just as well as the Human Expert's notes. The AI didn't need a human to tell it how to be brave or shy; it could figure out the "vibe" of the character just by reading the description and then use that "vibe" to act more convincingly.

Why This Matters

• Fairer Tests: We can now test AI on characters it has never heard of (like a real person or a brand-new fictional character) without it cheating by using its memory.
• Better Actors: By adding a simple "personality tag" (like MBTI or Big Five traits), we can make AI actors sound much more human and consistent, even when they are playing a role they don't know well.

In a Nutshell:
The paper says, "Stop letting AI actors cheat by memorizing names. Instead, give them a personality map. And guess what? The AI can draw its own map just as well as a human can!"

Technical Summary

Here is a detailed technical summary of the paper "Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects."

1. Problem Statement

Current research on Role-Playing Agents (RPAs) using Large Language Models (LLMs) suffers from a critical evaluation bias:

• Name Dependency: Most benchmarks use famous fictional characters (e.g., Harry Potter, Sherlock Holmes). LLMs often rely on memorized prior knowledge associated with these specific names from their pretraining data rather than truly understanding the character's persona.
• Lack of Generalization: High performance on famous characters does not guarantee the ability to impersonate unseen personas, such as real individuals or obscure characters, where no prior name-based knowledge exists.
• Evaluation Gap: There is a lack of systematic study on how to evaluate RPAs fairly without the "cheat code" of character names, and how to enhance performance in these anonymous, knowledge-scarce scenarios.

2. Methodology

A. Anonymous Evaluation Framework

The authors propose a new evaluation protocol where character names are removed from the prompt to force the model to rely solely on provided descriptions.

• Anonymization Process: All character names in the prompt (profiles and dialogue history) are replaced with a generic token (e.g., <anonymous character>).
• Restoration: Original names are restored after generation but before evaluation to ensure the judge (human or LLM) can assess consistency correctly without the model having seen the name during generation.
• Benchmarks: The study utilizes two major datasets:
  • CharacterEval: A Chinese dataset covering conversational ability, attractiveness, and character consistency.
  • RoleAgentBench: A bilingual (English/Chinese) dataset focusing on general response and summarization tasks.
• Models: Experiments were conducted on gemini-2.0-flash, gpt-4o, and the open-source llama-3.1-405B-instruct.

B. Personality Augmentation

To improve role fidelity in the absence of name-based priors, the authors investigate Personality Augmentation. They compare three sources of personality data:

1. Self-Report: The model is prompted to answer personality scale questions (MBTI/Big Five) directly.
2. Interview-Based: The model answers open-ended questions, and an evaluator LLM infers the personality type.
3. Human-Annotated (PDB): Ground-truth personality data retrieved from The Personality Database (PDB).

The study tests two personality frameworks:

• MBTI (16Personalities): A 4-letter type system (e.g., INTJ).
• Big Five (OCEAN): Continuous scores for Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

3. Key Contributions

1. Proposal of Anonymous Evaluation: Introduced a protocol that masks character names to eliminate name-bias, providing a fairer assessment of an LLM's ability to role-play based strictly on prompt context.
2. Systematic Study of Personality Effects: Demonstrated that incorporating personality information (MBTI or Big Five) consistently improves RPA performance, even when the model does not know the character's name.
3. Validation of Self-Generated Personas: Showed that self-generated personality traits (derived via self-report or interview) achieve performance comparable to human-annotated traits from PDB. This suggests a scalable, cost-effective method for enhancing RPAs without external annotation resources.
4. Cross-Benchmark Consistency: Validated findings across multiple models, languages, and datasets, proving the robustness of the approach.

4. Key Results

Impact of Anonymization

• Performance Drop: Removing character names significantly degrades RPA performance.
  • In CharacterEval, the "Character Consistency" dimension dropped for all models (e.g., gpt-4o dropped from 2.884 to 2.792, with statistically significant drops in specific sub-metrics).
  • In RoleAgentBench, original settings (with names) consistently achieved higher win rates against anonymized settings in pairwise comparisons.
• Implication: Previous high scores were partially inflated by the model's memorization of character names.

Impact of Personality Augmentation

• Consistent Improvement: Adding personality information (regardless of source) consistently improved scores across Conversational Ability, Role-Playing Attractiveness, and Character Consistency.
• Self-Generated vs. Human-Annotated:
  • Self-generated personalities (Self-Report/Interview) performed on par with PDB human annotations.
  • In CharacterEval, adding self-generated MBTI traits to gpt-4o improved the "Role-Playing Attractiveness" score from 2.969 to 3.176 (statistically significant).
• Framework Agnosticism: Both MBTI and Big Five frameworks yielded similar performance gains, indicating the benefit comes from the presence of structured personality data rather than the specific psychological model used.

Qualitative Findings

• Stronger Personalities: Characters with distinct, strong personality traits (scores >60% or <40%) benefited more from augmentation than those with moderate traits.
• Human Evaluation: Human evaluators (psychology experts) confirmed that personality-augmented responses were more vivid, emotionally aligned, and consistent with the character, even without the character's name being present.

5. Significance and Future Directions

• Fairer Benchmarking: The paper establishes that current benchmarks are flawed due to name bias. The proposed anonymous setting is a necessary step for evaluating true role-playing capabilities, especially for real-world applications (e.g., impersonating real people or new characters).
• Scalable RPA Construction: The finding that models can generate their own effective personality guides eliminates the need for expensive human annotation, making robust RPA construction scalable.
• Generalization: The methodology allows for the extrapolation of benchmark results to real-world scenarios where prior knowledge is absent.

Limitations & Future Work:

• Models may still infer identities from context clues (e.g., names of other characters in the dialogue).
• The magnitude of improvement varies by model architecture; further research is needed to understand why.
• Future work should explore optimal integration of different psychological frameworks.