💬 NLP

Creating and Evaluating Personas Using Generative AI: A Scoping Review of 81 Articles

This scoping review of 81 articles (2022–2025) examines the use of generative AI in persona development, highlighting high resource sharing and the rise of conversational interfaces while identifying critical gaps such as insufficient evaluation, overreliance on GPT models, risks of circularity, and a diminished role for human developers, ultimately proposing guidelines for responsible integration.

Original authors: Danial Amin, Joni Salminen, Farhan Ahmed, Sonja M. H. Tervola, Sankalp Sethi, Bernard J. Jansen

Published 2026-04-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Danial Amin, Joni Salminen, Farhan Ahmed, Sonja M. H. Tervola, Sankalp Sethi, Bernard J. Jansen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook a meal for a very specific group of people. To do this well, you need to know exactly what they like, what they hate, and how they eat. In the world of design and technology, these "groups of people" are represented by Personas. Think of a Persona as a detailed, fictional character card (like a trading card) that summarizes a real group of users. Instead of saying "We have 500 customers who are 30-year-old teachers," you create "Sarah, the 30-year-old teacher who loves coffee but hates slow websites."

For decades, making these "Sarah" cards was a slow, manual job. Researchers had to interview real people, take notes, and write these stories by hand. It was like sculpting clay: time-consuming and requiring a skilled artist.

But recently, a new tool has arrived: Generative AI (GenAI). It's like a super-fast, super-smart robot assistant that can write these character cards in seconds.

This paper is a scoping review, which is a fancy way of saying the authors looked at 81 recent studies (from 2022 to 2025) to see how people are using this robot assistant to make Personas. They wanted to know: Is the robot doing a good job? Is it safe? And are we letting the robot take over the kitchen entirely?

Here is the breakdown of their findings, using some everyday analogies:

1. The Robot is Fast, But It's a "One-Brand" Shop

The study found that almost everyone (86%) is using the same robot assistant: OpenAI's GPT models (like the famous ChatGPT).

The Analogy: Imagine that every chef in the world suddenly decided to only use one specific brand of knife. It's efficient, sure, but if that knife has a weird flaw or a specific bias, everyone's food will have that same flaw. The authors warn that relying on just one AI company is risky because it limits the variety of perspectives we get.

2. The Robot is Getting Creative (and Interactive)

In the past, a Persona was just a static PDF document you read. Now, GenAI allows these Personas to come alive.

The Analogy: Instead of reading a biography of "Sarah the Teacher," you can now chat with her. You can ask, "Sarah, how would you feel about this new app?" and the AI, pretending to be Sarah, will answer. Some studies even use AI to create video avatars or deepfakes of these users. It's like moving from reading a script to watching a live improv show.

3. The "Circular Logic" Trap (The Robot Judging Itself)

This is one of the biggest red flags the authors found. About half of the studies didn't really check if the AI did a good job. Worse, some studies used the same AI to create the Persona and then to grade it.

The Analogy: Imagine a student writing their own homework and then grading their own test. If the student is biased, they will give themselves an A even if the work is bad. The authors call this "circularity." If the AI makes a biased character and then the AI says, "Yes, this is a perfect character," we have no idea if it's actually true.

4. The "Good, Bad, and Ugly" of GenAI Personas

The authors summarized their findings into three categories:

The Good (The Superpower):
GenAI has made it easy for anyone to create Personas, even if they aren't experts. It's like giving everyone a high-quality camera; suddenly, more people can take beautiful photos. Also, researchers are sharing their "recipes" (code and data) much more often now, which is great for science.
The Bad (The Quality Drop):
Because it's so easy to make a Persona, people are making them without checking if they are accurate. The "skill barrier" is gone, but the "quality control" is still needed. The authors found that nearly half the studies didn't have a proper way to check if the AI was lying (hallucinating) or creating stereotypes. It's like a factory churning out toys at high speed, but nobody is checking if the wheels are screwed on tight.
The Ugly (The Human Loss):
This is the scariest part. Some studies are creating Personas using only fake data generated by AI, and then evaluating them with more AI.
- The Analogy: Imagine a movie director who has never met a real human. They ask a robot to invent a story about humans, and then ask another robot to critique the story. The result is a movie about humans that looks like humans but has no soul. The authors warn that if we let AI do everything, we lose the "human in the loop." Personas are supposed to represent real people; if we stop talking to real people and only talk to robots, we lose the point.

5. The Ethical Warnings

The paper highlights that AI can accidentally make things worse regarding bias.

The Analogy: If you ask an AI to describe a "doctor," it might default to a white male because that's what it saw most often in its training data. If we use that AI to design a hospital, we might accidentally design a place that feels unwelcoming to women or people of color. The AI amplifies existing stereotypes unless we are very careful.

The Takeaway: What Should We Do?

The authors aren't saying "Stop using AI." They are saying, "Use it, but don't trust it blindly."

They propose a set of rules for using AI to make Personas:

Don't use just one robot: Use different AI models to cross-check each other.
Keep a human in the loop: A real person must check the work, especially to catch cultural biases or stereotypes.
Check against reality: Compare the AI's "Sarah" with real data from real people to make sure it's not just making things up.
Be transparent: Write down exactly how you asked the AI to do its job (the "prompts") so others can repeat your work.

In short: Generative AI is a powerful new tool for understanding users, but it's like a very fast car. If you drive it without a seatbelt (human oversight) or without checking the map (validation), you might end up in a ditch. The goal is to use the speed of the car while keeping the human driver firmly in control.

1. Problem Statement

User personas are fictitious, archetypal representations of real users used to support User-Centered Design (UCD) in Human-Computer Interaction (HCI). Traditionally, persona development relies on qualitative data collection and manual analysis, which is time-consuming and resource-intensive. While Generative AI (GenAI), particularly Large Language Models (LLMs), offers the potential to automate this process, the field lacks:

Systematic Synthesis: No comprehensive overview exists on how GenAI is currently applied across the persona lifecycle.
Standardization: There are no established best practices for prompt engineering, model selection, or evaluation.
Evaluation Gaps: Traditional validation methods (user perception) may be insufficient for GenAI outputs, which face unique risks like hallucinations, circularity (self-evaluation), and algorithmic bias.
Ethical Concerns: The automation of persona creation raises questions about human agency, the "human-in-the-loop" principle, and the potential for amplifying societal biases or creating "Potemkin personas" (superficially realistic but factually hollow).

2. Methodology

The authors conducted a scoping review following PRISMA guidelines to map the landscape of GenAI in persona development.

Data Sources: Five academic databases were searched: ACM Digital Library, IEEE Xplore, Web of Science, Scopus, and arXiv.
Search Strategy: Queries combined GenAI terms (e.g., "LLM," "GPT," "Generative AI") with persona terms (e.g., "user persona," "buyer persona"). The search covered literature from 2022 to August 2025.
Selection Process:
- Initial yield: 885 articles.
- Deduplication and screening removed non-relevant documents (e.g., non-HCI contexts, NLP agent personality configurations).
- Final Corpus: 81 articles met the inclusion criteria (English, peer-reviewed or preprint, empirical research, explicit GenAI application in persona development).
Data Extraction: A coding book with 35 variables across six categories (basic info, resources, methodology, development, GenAI usage, results) was used. Four coders performed extraction with a high inter-rater reliability (Fleiss' Kappa $\kappa=0.86$ ).
Research Questions (RQs):
1. How are GenAI technologies used in persona development?
2. How are GenAI personas evaluated?
3. What ethical considerations are associated with GenAI personas?

3. Key Findings & Results

RQ1: Usage Patterns in Persona Development

Model Dominance: 86.4% of studies rely exclusively on OpenAI's GPT models. Other models (Claude, Llama, Gemini) are used in only ~8% of cases each, creating a dependency on a single commercial provider.
Development Lifecycle: GenAI is used for:
- Creation/Enrichment: Generating data-driven profiles, narrative backstories, and multimodal representations (text + images).
- Simulation/Interaction: Creating conversational agents or multi-agent simulations for decision-making scenarios.
- Personalization: Adapting content or dialogue based on persona attributes.
- Validation: Checking consistency and realism.
Innovations: The review identified four key innovations:
1. Input Sources: Use of unstructured data (video, interviews) and expert knowledge.
2. Process: Bypassing traditional clustering/segmentation steps via direct attribute specification.
3. Prompt Engineering: Use of two-stage, multi-stage, and few-shot prompting to improve quality.
4. Output Formats: Shift from static documents to conversational interfaces (chatbots) and structured data (JSON).
Participatory Design: Only 30.9% of studies involved direct user participation; the majority (65.4%) were fully automated or researcher-driven.

RQ2: Evaluation Methodologies

Evaluation Gap: 45% of the articles lack any explicit evaluation framework.
Evaluation Targets: When evaluation occurs, it focuses on:
- Description Quality (38.6%): Credibility, consistency, and linguistic quality (often using the Persona Perception Scale).
- Behavior (36.4%): Decision-making accuracy and alignment with real human patterns.
- Outcomes/Process: Less common (15.9% and 11.4% respectively).
Methods:
- Human-Driven (58.6%): Subjective ratings by experts or users.
- Computational (20.7%): NLP metrics (lexical diversity, stereotype detection).
- Benchmark-Based (20.7%): Comparison against real-world data or ground truth.
Critical Flaw (Circularity): A significant risk identified is the use of the same LLM to both generate and evaluate personas, compromising independence and validity.

RQ3: Ethical Considerations

Awareness: Only 56.8% of articles explicitly discuss ethics.
Primary Concerns:
1. Bias & Representation: GenAI tends to reinforce stereotypes (e.g., gender/age bias, "algorithmic othering" of marginalized groups) and over-represent elite demographics.
2. Societal Harm: Risks include "synthetic persuasion paradoxes" (overestimating intervention effects) and policy misapplication based on biased data.
3. Trust & Validity: Concerns about "Potemkin personas" (superficially realistic but lacking depth) and reduced semantic diversity compared to human-created personas.
4. Reduced Human Oversight: 42.0% of studies reported no human-AI collaboration, delegating the entire process to automation. This threatens the participatory design foundation of personas.

4. Key Contributions

Comprehensive Taxonomy: The paper provides the first systematic mapping of GenAI applications in persona development, categorizing usage into creation, simulation, personalization, and evaluation.
Identification of Risks: It highlights critical methodological weaknesses, specifically the lack of evaluation (45% of studies), model homogeneity (86% GPT), and circular evaluation risks.
The "Good, Bad, and Ugly" Framework:
- Good: Democratization of persona creation, increased resource sharing (61% share code/data), and new interactive formats.
- Bad: The "accessibility-quality paradox" where ease of generation outpaces rigorous evaluation.
- Ugly: The erosion of human agency and the risk of fully automated pipelines making consequential decisions about real people without human oversight.
Actionable Guidelines (PG1–PG7): The authors propose seven practical guidelines for responsible GenAI persona development, including:
- Implementing multi-model validation.
- Using structured formats (JSON/XML) for consistency.
- Documenting prompts and parameters for reproducibility.
- Validating against real demographic benchmarks.
- Establishing quantitative acceptance thresholds.
- Testing for consistency via edge cases.
- Mandating systematic human oversight for cultural and stereotype checks.

5. Significance

This study serves as a critical intervention for the HCI community at a pivotal moment where GenAI is rapidly adopted for user research.

Theoretical Impact: It challenges the notion that automation equates to improvement, arguing that without human-in-the-loop validation, personas risk becoming "algorithmic artifacts" disconnected from real user needs.
Practical Impact: It provides a roadmap for researchers and practitioners to avoid common pitfalls (bias, hallucination, circularity) and adopt robust, reproducible workflows.
Future Direction: It calls for a shift from purely technical optimization (better prompts) to ethical and methodological rigor, emphasizing that the value of a persona lies in its connection to real human experiences, not just its generative fidelity.

In conclusion, while GenAI offers powerful tools for scaling persona development, the field must urgently address the lack of evaluation standards and the erosion of human oversight to ensure these tools serve as valid, ethical, and reliable instruments for design.