💻 computer science

Simulating Eating Disorder Patients with LLMs: Evaluating Psychological Persona Stability in Multi-Turn Conversations

This study reveals that while large language models maintain consistent psychological personas across conversations, they systematically fail to accurately simulate eating disorder patients by overshooting severity scores through selective stereotyping, thereby capturing extreme pathology but lacking the nuance to represent moderate clinical presentations.

Original authors: Jennifer Haase, Jana Gonnermann-Müller, See Heng Yim, Nicolas Leins, Jan Mendling, Sebastian Pokutta

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jennifer Haase, Jana Gonnermann-Müller, See Heng Yim, Nicolas Leins, Jan Mendling, Sebastian Pokutta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to cast actors for a play about people struggling with eating disorders. You want the actors to be so convincing that they stay in character perfectly, whether they are reading a script alone or improvising a scene with a partner. You also want them to portray the exact level of severity described in the script—some characters are in a deep crisis, while others are in a moderate, struggling phase.

You decide to use AI actors (Large Language Models) instead of humans. You give them a detailed "character biography" (a prompt) and ask them to act out their lives and fill out a standard medical questionnaire about their eating habits.

This paper is the director's report card on how well these AI actors performed. The results are surprising, confusing, and a bit worrying.

The Main Finding: "Too Stable, But Wrong"

The researchers found a paradox: The AI actors were incredibly consistent, but consistently inaccurate.

The Good News (Stability): If you asked the same AI actor to play the same character 50 times, they gave almost the exact same answer every time. They didn't forget who they were supposed to be. They were like a robot that never breaks character.
The Bad News (Accuracy): Even though they were consistent, they were all over-dramatizing. The AI models took every character, from those with mild struggles to those in severe crisis, and turned them all into "extreme" cases.

Think of it like a volume knob on a radio. The researchers wanted the AI to turn the volume up to a "5" for a moderate case and a "10" for a severe case. Instead, the AI turned the volume up to a "9" or "10" for everyone, regardless of the script.

The "Missing Middle"

The paper describes a phenomenon called the "Missing Middle."

Severe Cases: When the script described a very severe eating disorder, the AI got it roughly right.
Moderate Cases: When the script described someone with a moderate struggle (the "middle" ground where many real patients actually are), the AI exaggerated it, making them sound like they were in a severe crisis.

The AI simply doesn't seem to have a good "volume setting" for moderate problems. It defaults to the loudest, most dramatic setting available.

The "Two-Part" Glitch

Why did this happen? The researchers discovered the AI was using a weird shortcut, which they call "Selective Stereotyping."

Imagine the eating disorder questionnaire has two types of questions:

Behavioral Questions: "Do you restrict your food?" "Do you binge?"
Emotional Questions: "How much do you hate your body?" "How obsessed are you with your weight?"

The AI handled these differently:

For Behavior: It listened to the script. If the character was a "Binge Eater," the AI said, "Yes, I binge." If the character was a "Purger," it said, "Yes, I purge." It got the actions right.
For Emotions: It ignored the script. No matter who the character was, the AI assumed everyone with an eating disorder was obsessed with their weight and hated their body to the maximum degree possible. It cranked the "Body Hate" dial to the ceiling for every single character.

Because the emotional questions were maxed out for everyone, the AI couldn't tell the difference between a moderate case and a severe case. It was like a painter who got the clothes right but painted every single face with the exact same expression of extreme despair.

Did More Information Help?

The researchers tried giving the AI actors more detailed backstories (childhood trauma, family history, specific life events) to see if it would help them understand the "middle" ground better.

It didn't work. Adding more details didn't fix the problem. The AI still defaulted to the extreme emotional setting. It's as if the AI's internal "training data" (the books and internet it learned from) taught it that "Eating Disorder = Maximum Body Hate," and no amount of extra context could override that rule.

The "Observer" Problem

To be fair, the researchers didn't just ask the AI how it felt; they also asked other AI models to watch the actor and rate them.

Result: The "Observer" AIs saw exactly what the "Actor" AIs said. They also rated everyone as having extreme body dissatisfaction. This proved the problem wasn't just the actor lying; the problem was that the entire performance was built on a stereotype.

The Conclusion

The paper concludes that while AI is great at being a consistent actor, it is currently not ready to simulate the full range of human eating disorder experiences.

It can act out a severe crisis well, but it fails to capture the nuance of moderate struggles. It's like having a thesaurus that only has words for "Angry" and "Furious," but no words for "Annoyed" or "Upset." Until AI can learn to turn the volume down for moderate cases, it cannot be trusted to accurately simulate the "middle" of the clinical spectrum.

Technical Summary: Simulating Eating Disorder Patients with LLMs

Problem Statement

Large language models (LLMs) are increasingly deployed as proxies for human participants in clinical training, research, and experimental subjects. A critical, yet under-examined, prerequisite for these applications is persona stability: the coherent maintenance of an assigned psychological profile across and within conversations. If a simulated patient's psychological profile drifts mid-conversation or shifts unpredictably across sessions, downstream conclusions regarding social phenomena, clinical competency, or treatment effects become artifacts of model inconsistency rather than genuine findings.

Current evaluation paradigms fail to detect these failures. Simulated patient systems are typically evaluated on dialogue realism or diagnostic accuracy, while persona benchmarks often target fictional settings or generic personality traits rather than clinically grounded profiles with validated instruments. Furthermore, existing literature suggests mixed evidence: while some benchmarks report stable responses, others note that adding personas does not reliably improve task performance, and Reinforcement Learning from Human Feedback (RLHF) can induce sycophancy that compounds over multi-turn dialogues.

This study addresses the specific gap of evaluating the psychological validity and temporal stability of LLM-simulated patients with eating disorders (EDs). EDs are psychiatric conditions with high mortality rates and significant training gaps among clinicians. While AI-simulated ED patients have shown promise in reducing trainee anxiety, their psychological stability remains unevaluated.

Methodology

Experimental Design

The authors operationalize stability through two complementary experiments involving six LLMs (Claude Sonnet 4.6, GPT 5.4, Gemini 3.1 Pro, DeepSeek V4 Flash, GPT OSS 120B, Llama 3.3 70B) and five ED personas derived from published case vignettes (Bulimia Nervosa, Anorexia Nervosa Binge-Purge, Binge Eating Disorder/Night Eating Syndrome, Binge Eating Disorder, and Purging Disorder).

Experiment I (Between-Conversation Stability): Each model generates a first-person narrative describing three eating-related situations and completes self-report instruments. Independent LLM raters (three models from different providers) then complete the same instruments based solely on the narrative, without access to the original persona instructions. This tests consistency across independent runs ( $N=50$ per condition).
Experiment II (Within-Conversation Stability): The target LLM engages in a 9-exchange neutral conversation with a scripted partner. Self-report and observer assessments are triggered after three topic blocks (exchanges 3, 6, and 9) to detect drift over time ( $N=20$ per condition).

Persona Construction and Prompt Richness

To test the necessity of prompt detail, the study manipulates prompt richness across three levels based on Fairburn's transdiagnostic model:

Full Vignette: Complete clinical case descriptions including demographics, history, trauma, and social context (excluding assessment scores).
Core Maintaining Mechanisms: Only active maintaining factors (e.g., dietary restraint, binge-purge cycles, body checking) without biographical context.
Minimal: Diagnostic label, age, gender, BMI, and quantified behavioral frequencies only.

Measurement

The primary outcome is the Eating Disorder Examination Questionnaire (EDE-Q), a validated 28-item instrument with a 0–6 scale. Ground-truth (GT) scores are available for all five cases. A dual-assessment framework is employed:

Self-Report: The persona completes the questionnaire based on its generated narrative.
Observer Ratings: Three independent LLM raters evaluate the transcript from the individual's perspective.
Secondary Measures: The Clinical Impairment Assessment (CIA) and Eating Attitudes Test (EAT-26) are co-administered for convergent validity.

Key Contributions

Dual-Assessment Framework: A novel methodology jointly measuring persona self-reports and independent observer ratings using validated clinical instruments (EDE-Q) with multi-provider rater aggregation to mitigate LLM-as-judge bias.
Theoretically Grounded Prompt Manipulation: A systematic variation of prompt richness based on Fairburn's model, distinguishing between active maintaining mechanisms and predisposing factors, to determine the minimum information required for plausible simulation.
Stability and Accuracy Evaluation: A comprehensive evaluation across six LLMs and five ED personas, utilizing variance decomposition to identify the relative contributions of model identity, persona case, and prompt richness to score variation.

Results

Stability

Between-Conversation: LLMs exhibit remarkably high stability. Coefficients of variation average 2–4% for five of six models, and 30% of item-condition combinations produce identical scores across all 50 independent runs. This near-deterministic behavior is most pronounced for cognitive-affective items (Shape and Weight Concern).
Within-Conversation: Extending the conversation to 9 exchanges does not significantly improve accuracy; instead, scores for four of five models drift upward (increasing severity) over the dialogue, moving further from ground truth.

Accuracy and Overshoot

Systematic Overshoot: All models systematically overshoot ground-truth severity by 12–30% of the scale range (0.7–1.8 points on a 0–6 scale).
Severity-Dependent Compression: The overshoot is most severe for moderate cases. For example, Purging Disorder (GT = 2.64) is inflated by +1.44 to +2.73 points, whereas Bulimia Nervosa (GT = 5.25) is reproduced within ±0.43 points.
Rank Inversions: LLMs fail to correctly rank severity. In four of six models, Anorexia Nervosa Binge-Purge (GT = 4.30) scores higher than Bulimia Nervosa (GT = 5.25), despite having lower ground-truth severity.

Mechanism: Selective Stereotyping

The study identifies selective stereotyping as the mechanism driving these errors:

Behavioral Items: Items measuring concrete behaviors (e.g., Dietary Restraint) differentiate cases effectively, responding to the specific content of the vignette.
Cognitive-Affective Items: Items measuring body dissatisfaction and weight preoccupation (Shape Concern, Weight Concern) default to ceiling (near-maximum scores) regardless of case severity.
Consequence: Because cognitive-affective items saturate at the ceiling for all ED diagnoses, the global score is driven almost entirely by the behavioral Restraint subscale. This leads to rank inversions where cases with extreme dietary restriction (even if less severe overall) score higher than cases with less restriction but higher overall pathology.

Prompt Richness

Prompt richness contributes negligibly (1%) to score variance. Stripped-down prompts (Minimal) elicit near-equivalent responses to full clinical vignettes, suggesting the overshoot originates from pretraining priors rather than insufficient context.

Significance and Claims

The paper concludes that LLMs are paradoxically too stable and too inaccurate. While they maintain consistent personas, this stability reflects a stability of bias rather than fidelity to clinical reality.

The "Missing Middle": LLMs can portray severe eating pathology but lack a representation of moderate clinical presentations. This creates a "missing middle" where aggregate validation may appear adequate while masking critical failures in the severity ranges most relevant for clinical training.
Limitations of Calibration: Because the rank ordering of cases is non-monotonic (e.g., AN-BP > BN despite lower GT severity), simple post-hoc calibration or monotonic transformations cannot correct the outputs. The distortion originates at the subscale level.
Implications for Deployment: The findings suggest that current LLMs cannot yet serve as valid proxies for graded eating-disorder severity in clinical training or research. Improving fidelity requires item-level clinical grounding at generation time and representational diversity that current training data does not provide. The study does not claim that LLMs are useless for therapy, but rather that their use as simulated patients for evaluating clinical competency or treatment effects is currently compromised by this systematic stereotyping.

The authors explicitly state that their findings replicate and extend previous work on ADHD simulations, showing that consistent self-reports may reflect consistent inaccuracy rather than consistent fidelity. They call for future cross-disorder replication with behaviorally graded instruments and human baseline studies to determine if this overshoot is model-specific or inherent to simulated self-report.