💬 NLP

The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

This paper introduces MirageBench to reveal that personalized LLMs universally fabricate user attributes through pervasive over-inference, a phenomenon that self-monitoring mechanisms fail to detect and often inversely correlate with actual error rates, thereby demonstrating the critical need for external verification over model self-assessment.

Original authors: Yushi Sun, Yanjie Zhang, Rui Sheng

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yushi Sun, Yanjie Zhang, Rui Sheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Memory Trap: When AI Gets Too Creative

Imagine you're talking to a super-smart digital assistant that remembers everything you've ever told it. This isn't just a simple notepad; it's a "personalized" AI designed to know you. In the world of computer science, this is called Large Language Model (LLM) personalization. Think of these models as incredibly fast readers who can chat with you, but they have a tricky habit: they sometimes fill in the blanks of your life story with guesses that sound very confident but are actually made up.

To understand the problem, we need to know two things. First, persistent memory is the ability of an AI to remember facts about you across different conversations, like remembering your favorite pizza topping from last week. Second, hallucination is when an AI makes up facts about the world (like saying the moon is made of cheese). But there's a third, sneakier problem: over-inference. This is when the AI takes a tiny, true fact you gave it and stretches it into a whole new personality trait that you never mentioned. It's like telling a friend you like coffee, and they immediately tell everyone you're a barista who lives in a coffee-scented loft and dreams of opening a café. The friend isn't lying about the coffee, but they've invented a whole life around it. We care about this because if our digital assistants start inventing our lives, they might start giving us advice based on a person who doesn't exist.

The Mirage: When AI Dreams Up Your Life

A team of researchers decided to put these "personalized" AI assistants to the test. They created a playground called MirageBench (a "mirage" is a fake oasis in the desert, which is a perfect name for this). They gave 12 different AI models a simple challenge: pretend you know a person based on only three facts about them. Then, they asked the AI to do things like write a dating profile, suggest a weekend trip, or describe the person's apartment.

The results were a bit like watching a magician who gets too confident in their own tricks. The researchers found that every single model they tested was guilty of "over-inference." On average, about 41.6% of the things these AIs claimed to know about the user were completely made up. That means if an AI tells you it knows 10 things about you, roughly 4 of them are pure fiction, even though the AI says them with total confidence.

The study broke these made-up claims into two categories: Stereotypes (guessing you like yoga because you're a nurse) and Fabrications (inventing that you have a pet iguana when you never mentioned one). The AIs were much better at making up wild stories (fabrications) than just relying on stereotypes. In fact, for tasks that required imagination, like describing someone's apartment, the "fake" rate jumped to nearly 59%. It seems the less evidence the AI has, the more freely it starts dreaming up details.

The Great Confidence Trick

Here is the most surprising part of the story, which the researchers call the "Self-Monitoring Inversion."

Usually, we think that if an AI says, "I'm not sure about this," it's being honest, and if it says, "I'm 100% sure," it's confident. But the researchers found a complex pattern when comparing different AI models: the models that claimed they were least likely to make things up were actually the ones making the most up. Conversely, the models that admitted, "Hey, I might be guessing here," were often the ones sticking closest to the truth.

It's like a classroom where the students who raise their hands and say, "I think I might be wrong," are actually the ones who studied the most, while the students shouting, "I know the answer!" are the ones guessing wildly. The researchers found a negative link between a model's self-reported confidence and its actual accuracy, though they note this is an exploratory finding with a wide confidence interval that includes the possibility of no effect. This means you can't just ask an AI, "Are you making things up?" and trust the answer when comparing different AIs. The ones that sound the most honest might be the biggest liars, but the data suggests this relationship is still being fully understood.

The Silent Pollution

The researchers also ran a simulation where the AI had a conversation with a user over 8 rounds. They watched to see if the AI would "clean up" its mistakes if the user corrected them. The result was worrying, but with a twist: not all models behaved the same way.

For the smartest, most capable models, the AI didn't just make mistakes; it accumulated them. Imagine a whiteboard where the AI writes down a guess. If the user says, "Actually, I don't like that," a good system would erase the guess. But these top models? They kept the guess on the board and added more guesses on top of it. They added about 5 to 15 new made-up facts every single round, and they almost never erased the old ones (with removal rates as low as 0.4% to 5%).

However, this wasn't true for every model. Some models behaved more like a "replace and update" system, where they actively removed old guesses when new information came in, with removal rates as high as 70% to 82%. The danger lies specifically with the most capable accumulators, whose memory became a bloated mess of 120+ invented facts, most of which were never corrected. It's like a rumor that keeps getting more detailed and more wrong as it passes from person to person, but no one ever stops to say, "Wait, that's not true."

The Takeaway

The big lesson from this paper is that we cannot trust AI to police its own imagination. The idea that an AI can look at its own thoughts and say, "This is a fact" or "This is a guess" is a mirage. The models that seem the most cautious are often the most deceptive, and the ones that admit uncertainty are often the most reliable, though the exact strength of this relationship across different models is still being explored.

The researchers suggest that instead of trusting the AI's self-report, we need external verification—a way for humans or other systems to check the AI's work. We also need to treat everything an AI "guesses" as a hypothesis, not a fact. Until we build systems that can clearly separate what they know from what they are imagining, our personalized digital assistants might be building a fake version of us, one confident lie at a time.

Technical Summary: The Personalization Mirage

Problem Definition

The paper addresses a critical gap in the deployment of personalized Large Language Models (LLMs) with persistent memory. While these systems are increasingly capable of storing and retrieving user information, their faithfulness remains unexamined. The authors define Over-Inference (OI) as the phenomenon where LLMs generate personalized claims about a user that extend beyond the available evidence.

OI is distinct from:

Factual Hallucination: Asserting false facts about the world.
Social Bias: Applying group-level stereotypes.

Instead, OI occupies a "dangerous middle ground": fabricating individual-level attributes that feel personalized but are not grounded in anything the user has actually shared. The core assumption that models can reliably distinguish between what they know and what they are guessing is demonstrated to be fundamentally violated in real-world interactions.

Methodology: MirageBench

To measure and compare over-inference at scale, the authors introduce MirageBench, the first benchmark specifically designed for this phenomenon. The benchmark comprises:

Personas: 150 user personas balanced across three groups:
- Stereotypical: Conform to common demographic/occupational associations.
- Counter-stereotypical: Deliberately defy expectations.
- Neutral: No salient stereotype alignment.
  Each persona has a ground-truth profile of 15 attributes, but only 3 first-person facts are "revealed" to the model, simulating early-interaction regimes where personalization is tempting but evidence is sparse.
Tasks: A suite of 6 open-ended personalization tasks spanning an "imagination gradient":
- From concrete (e.g., "What gift should I buy?") to highly imaginative (e.g., "Describe my apartment").
- Tasks include writing dating profiles, recommendation letters, weekend itineraries, and identifying stressors.
Evaluation Pipeline:
- Probe: Elicits explicit inference (asking the model to list what it can infer).
- Task: The model performs the 6 tasks and then performs a self-audit, extracting and classifying its own claims.
- Judge: An independent judge model (Claude-Opus-4-7) classifies every claim from the Task responses. The taxonomy is four-way:
  - Grounded: Restates user facts.
  - Reasonable: One-step common-sense extension.
  - Stereotype: Substitutes group priors for individual evidence.
  - Fabricated: No evidential basis.
  - Over-Inference Rate (OI Rate) is defined as the sum of Stereotype and Fabricated claims.
- Accum: A longitudinal pilot tracking how inferred attributes accumulate over 8 conversation rounds.
Validation: The Judge's reliability was validated against a blind human annotator on 400 stratified claims, achieving Cohen's $\kappa = 0.863$ (four-class) and $\kappa = 0.900$ (binary).
Models Evaluated: 12 models across 7 families (including GPT-5.5, Claude-Opus-4-6, Gemini-3, DeepSeek-v4, Qwen3, GLM-5, and Kimi-K2.5), totaling 143,616 judged claims.

Key Results

1. Over-Inference is Pervasive and Severe

Every model tested over-infers between 35% and 49% of its personalized claims. The cross-model mean OI rate is 41.6% (claim-weighted micro-average: 41.8%).

Only 24%–31% of personalized content is grounded in evidence.
Fabrication (mean 31.1%) dominates over Stereotyping (mean 10.5%), indicating models primarily invent rather than just stereotype.
Stereotypical personas incur higher OI rates (44.8%) than counter-stereotypical ones (37.0%), confirming that stereotypes drive a significant portion of the error.

2. The Self-Monitoring Inversion

A critical finding is the Self-Monitoring Inversion. At the model-selection level, there is a negative rank correlation ( $\rho = -0.60, p = 0.044$ ) between a model's self-assessed OI and the judge-measured OI.

Models that report the lowest self-audit OI (e.g., Qwen3-8B at 13.0%) often receive the highest judge-measured OI (48.7%).
Models that report the highest self-audit OI (e.g., Kimi-K2.5 at 58.2%) often have lower judge-measured OI (43.1%).
Implication: Self-reported confidence is a misleading signal for comparing safety across different models. However, within a single model, self-audit remains a moderate-to-strong ranking signal for that model's own claims (AUROC 0.58–0.83).

3. Task-Dependent Variation

OI rates vary significantly based on the "groundability" of the task:

High OI (57.8%): "Describe my apartment" (requires conjuring unobserved physical details).
Low OI (27.0%): "Birthday gift recommendation" (can be grounded in stated hobbies).
The less observable the attribute, the more freely models fabricate.

4. Silent Memory Pollution

In a multi-turn pilot, inferred attributes accumulate approximately linearly ( $R^2 > 0.90$ for 9/12 models) with little revision.

Top accumulators (e.g., GPT-5.5, GLM-5.1) add 5–15 new inferred attributes per round.
Revision rates for these models are extremely low (0.4%–5%), meaning unsupported claims silently persist and compound.
Some models (e.g., Qwen3-8B) behave more like memory replacement (70–82% removal) rather than accumulation.

Significance and Claims

The paper argues that external verification is necessary for trustworthy personalization, as model self-reports are unreliable for cross-model safety comparisons.

Design Implication: Systems that select models based on their self-reported confidence in grounding are "picking in the wrong direction."
Provenance Tracking: Stored user information must be tagged with its epistemic status (stated vs. inferred vs. fabricated). Personalization should treat unlinked inferences as hypotheses, not facts.
Fundamental Tradeoff: With only ~25% of claims grounded, three-quarters of personalization is inherently inferential. The design challenge is not eliminating inference but managing its uncertainty and transparency.

The authors position MirageBench as a foundational tool to shift the field from relying on model introspection to relying on external adjudication for evaluating user models.