💬 NLP

Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

This paper introduces the LENS framework to quantify natural prompt distribution shifts in real-world LLM deployments, revealing that even moderate shifts in user behavior, time, or geography can cause severe performance degradation (averaging a 73% loss) and highlighting the critical need for data-driven monitoring to ensure model reliability across evolving populations.

Original authors: Parker Seegmiller, Sarah Masud Preum

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Parker Seegmiller, Sarah Masud Preum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Moving Target" Problem

Imagine you hire a personal chef to cook for you. You spend months training them on your specific taste: you love spicy food, you hate cilantro, and you always ask for "medium-rare." The chef gets perfect at making exactly what you want.

Now, imagine you move to a new city, or your family grows, or your taste changes because you've been watching too many cooking shows. You start asking for different things: "Make me a vegan stew," or "Can you cook this in a French style?"

If the chef keeps cooking exactly how they were trained (spicy, cilantro-free, medium-rare), they will fail to satisfy you. They aren't "bad" chefs; they just haven't updated their menu to match your new reality.

This paper is about that exact problem, but with AI Chatbots (LLMs) instead of chefs. The researchers found that as real people use AI over time, their questions (prompts) change naturally. If the AI isn't retrained to match these new habits, it gets significantly worse at following instructions.

The Detective Work: Introducing "LENS"

To study this, the researchers built a framework they call LENS (LLM Evaluation under Natural prompt Shift). Think of LENS as a time-traveling camera that takes snapshots of how people talk to AI at different times, places, and with different types of people.

They didn't just guess that things change; they measured it using real data from millions of real user chats. They looked at three main ways the "conversation landscape" shifts:

Time (The "Seasons" of Conversation):
- The Analogy: In the winter, people wear coats. In the summer, they wear shorts. The weather changes, so people's needs change.
- The Finding: Over time, users get smarter and more specific. Early on, people asked simple questions like "Write a sentence." Later, they started asking for complex, structured formats like "Write a prompt for an image generator with these specific rules." The AI trained on the "winter" questions struggled to handle the "summer" requests.
User Groups (The "New Neighbors"):
- The Analogy: Imagine a coffee shop that usually serves office workers who want quick, black coffee. Then, a group of students moves in who want elaborate, sugary lattes with extra foam. If the barista only knows how to make black coffee, the new customers will be unhappy.
- The Finding: Different groups of people use AI differently. "Power users" (who ask many questions) tend to use templates and scripts. "Casual users" ask more random, creative questions. When an AI trained on casual users tried to serve power users, it crashed hard.
Geography (The "Local Dialects"):
- The Analogy: A chef trained in Italy knows how to make perfect pasta. If you ask them to make a specific type of American BBQ, they might struggle because the "flavor profile" of the request is different.
- The Finding: People in different parts of the world (even when speaking English) ask questions with different cultural nuances, sentence structures, and slang. An AI trained on users in California performed poorly when asked questions by users in Tokyo or Paris, even though everyone was speaking English.

The Shocking Result: The "73% Drop"

The researchers ran a massive experiment. They trained AI models on "old" data and then tested them on "new" data from the future or different groups.

The result was dramatic:

When the AI faced a Time Shift (future users), its performance dropped by about 44%.
When the AI faced a User Group Shift (new types of people), its performance dropped by 88%.
When the AI faced a Geography Shift (different locations), its performance dropped by 88%.

The Takeaway: Even a "moderate" change in how people talk to the AI causes the AI to lose its ability to follow instructions. It's like a student who studied hard for a math test but then showed up to a physics exam. They know the material, but the context has shifted so much that they can't apply it.

Why This Matters (The "So What?")

Currently, companies train AI on data from last year and then deploy it forever. This paper proves that this is a bad idea.

The "Stale Bread" Effect: Just like bread goes stale, AI models go "stale" as user behavior evolves.
The Need for Monitoring: We can't just train an AI once and forget it. We need to constantly watch (monitor) how people are talking to it. If we see the "flavor" of the questions changing, we need to retrain the AI to match the new reality.

Summary in One Sentence

AI models are like chefs who get stuck in the past; if we don't constantly update their training to match how real people actually talk today, they will quickly stop being helpful.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in dynamic, real-world environments where the distribution of user prompts naturally evolves over time, across different user groups, and across geographic regions. This phenomenon, termed Natural Prompt Distribution Shift, poses a critical threat to model reliability.

While Out-of-Distribution (OOD) robustness is a well-studied topic in machine learning, prior work has largely failed to:

Systematically measure natural prompt shifts in "in-the-wild" deployment settings (as opposed to synthetic or task-specific shifts).
Quantify the direct correlation between these natural shifts and the degradation of LLM performance (specifically instruction-following ability) post-deployment.
Address the challenge that natural language lacks a known ground-truth distribution, making shift measurement non-trivial.

The authors argue that models fine-tuned on historical user data often fail when encountering new user behaviors, leading to significant performance drops and user dissatisfaction.

2. Methodology: The LENS Framework

To address these gaps, the authors introduce LENS (LLM Evaluation under Natural prompt Shift), a data-centric framework designed to quantify natural prompt distribution shift and evaluate its impact on deployed LLMs.

A. Dataset Curation (WildChat)

The study utilizes WildChat, a dataset of 1 million real-world user queries submitted to OpenAI models, which includes metadata such as timestamps, anonymized IP addresses (for user grouping), and geographic locations.

Data Splitting: The authors constructed 192 unique {In-Distribution (ID), Out-of-Distribution (OOD)} dataset splits across three axes:
1. Time: Simulating training on past data (e.g., months 0–M) and evaluating on future data (Month N).
2. User Groups: Grouping users by adoption stage (early/late) and query volume (low/high). ID models are trained on one group, evaluated on another.
3. Geography: Training on data from one region (e.g., California) and evaluating on another (e.g., Tokyo).
Oracle Models: For every OOD evaluation setting, an "Oracle" model is trained on data from the same bucket as the evaluation data. This serves as the upper-bound performance baseline to measure how much the ID model degrades due to the shift.

B. Experimental Setup

Models: Three base models were used to ensure generalizability: Llama3-8B, Qwen2.5-7B, and Qwen2.5-14B.
Training: 81 models were trained in total (3 models × 27 unique ID datasets).
Evaluation: Models were evaluated on 57,600 prompts (100 prompts per split × 192 splits).
Judging: Performance was assessed using GPT-4o as a judge in a head-to-head comparison between the ID model and the Oracle model. The primary metric is the ID Model Loss Rate (percentage of comparisons where the Oracle model wins).

C. Measuring Distribution Shift

Since ground-truth distributions are unknown, LENS employs four unsupervised distribution shift measures to quantify the divergence between ID training data and OOD evaluation data:

MAUVE: Embedding-based measure estimating the overlap in relative frequency of prompt characteristics (lower score = higher shift).
$\Delta$ -score: A statistical depth measure estimating how representative OOD prompts are of the ID distribution (lower score = higher shift).
Perplexity: Measures how "surprising" OOD prompts are to a model trained on ID data (higher perplexity = higher shift).
BLEU: Measures n-gram overlap to capture lexical divergence (lower score = higher shift).

3. Key Results

A. Magnitude of Natural Shift (RQ1)

The study confirms that natural prompt distributions shift significantly across all three axes:

Geography: Exhibited the strongest shift (Average MAUVE: 0.05 vs. Random Baseline: 0.96), indicating distinct behavioral patterns across regions.
Time: Showed strong temporal drift (Average MAUVE: 0.54).
User Groups: Showed measurable but slightly lower shift (Average MAUVE: 0.61), yet still significant compared to random baselines.
Correlation: There is a strong positive correlation between the magnitude of the distribution shift (measured by MAUVE) and the degree of performance degradation.

B. Impact on LLM Performance (RQ2)

The performance degradation of LLMs under natural shift is severe and consistent across model sizes and families:

Time Shift: ID models lost 44% of comparisons against Oracle models on future data.
User Group Shift: ID models lost 88% of comparisons when evaluated on different user groups.
Geographic Shift: ID models lost 88% of comparisons when evaluated on different geographic regions.
Overall: The average performance loss across all settings was 73%.
Conclusion: Instruction-following abilities learned from real-world prompts do not naturally transfer to new user populations or future time periods.

C. Qualitative Analysis

Qualitative analysis of prompt pairs revealed specific trends driving the shift:

Time: Prompts evolved from exploratory/casual to more structured, complex, and instruction-heavy (e.g., specific formatting constraints).
User Groups: High-volume users utilized templated, repetitive prompts (workflow integration), while low-volume users used eclectic, diverse prompts.
Geography: Prompts reflected cultural nuances, code-mixing, and linguistic variations even within English-speaking regions.

4. Key Contributions

LENS Framework: A novel, systematic framework for evaluating LLMs under realistic, natural prompt distribution shifts, moving beyond synthetic OOD benchmarks.
Large-Scale Empirical Evidence: A massive evaluation involving 192 settings, 81 trained models, and 57.6k evaluation prompts, providing the first large-scale quantification of natural shift impacts.
Quantification of Degradation: Demonstrated that even moderate natural shifts lead to drastic performance drops (up to 88% loss), challenging the assumption that pre-training or instruction tuning provides sufficient robustness for dynamic deployment.
Unsupervised Shift Metrics: Validated the use of unsupervised metrics (MAUVE, $\Delta$ -score) to effectively measure and predict performance degradation in natural language settings.

5. Significance and Implications

Reliability Crisis: The findings highlight a critical vulnerability in current LLM deployment strategies. Models trained on static historical data are not robust to the natural evolution of user behavior.
Need for Monitoring: The paper argues for the implementation of data-driven monitoring systems to detect distribution shifts in real-time and trigger retraining or adaptation.
Future Research: The results suggest that future instruction tuning must account for dynamic user populations, potentially requiring continual learning strategies or domain-adaptive fine-tuning to maintain performance across diverse and evolving user bases.
Benchmarking: LENS provides a new standard for evaluating OOD robustness in NLP, shifting the focus from task-specific benchmarks to realistic deployment dynamics.

In summary, the paper establishes that natural prompt distribution shift is a pervasive and severe threat to LLM reliability, causing massive performance degradation that is not mitigated by current pre-training or standard fine-tuning practices.