Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

1. Problem Statement

While Large Language Models (LLMs) show promise as general-purpose user simulators, existing benchmarks suffer from critical limitations that prevent them from accurately modeling authentic human behavior:

• Isolated Scenarios: Current datasets focus on single domains (e.g., only video browsing or only e-commerce), failing to capture the holistic nature of user decision-making which spans multiple contexts.
• Short Horizons: Most benchmarks rely on short interaction sessions, ignoring long-term causal chains where a decision today (e.g., a purchase) is the result of interests accumulated over days or weeks.
• Synthetic Data: Many existing benchmarks use synthetic or simplified data, which lacks the noise, heterogeneity, and stochasticity of real-world user logs.
• Evaluation Gap: There is a lack of rigorous evaluation to determine if LLMs can faithfully simulate complex, cross-scenario, and heterogeneous behavioral traces found in industrial settings.

2. Methodology: The OmniBehavior Benchmark

The authors introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world industrial data.

A. Data Construction

• Source: Data was collected from Kuaishou, a major video platform with over 400 million daily active users.
• Scope: Logs were aggregated from September 1 to November 30, 2025 (3 months).
• Scenarios & Actions: The benchmark covers 5 major scenarios (Video Browsing, Live Streaming, Advertisement, E-commerce, Search) and 22 distinct user actions (e.g., watch, like, share, purchase, comment, search).
• Pipeline:
  1. Collection: Raw logs were interleaved chronologically to form unified behavioral sequences.
  2. Cleaning: A two-level cleaning process removed noise:
     • Behavior-level: Truncated spurious actions (e.g., accidental clicks) based on viewing duration percentiles.
     • Text-level: Used Regex and LLMs (Qwen2.5-72B) to clean OCR/ASR text and correct semantic errors.
  3. Sampling: A clustering-based strategy (K-Means) selected 200 representative users based on demographics, activity levels, interest distribution, and scenario preferences to ensure diversity.
  4. Anonymization: Sensitive entities (names, phones) were replaced with placeholders, and toxic content was pruned.
• Dataset Scale: The final dataset contains sequences ranging from 50 to over 100,000 actions, with an average length of ~8,143 actions per user (approx. 32k tokens).

B. Task Definition

The benchmark formulates user behavior simulation as a user-conditioned prediction task:

• Input: User profile ($p_u$), historical behavior sequence ($H_u$), and current scenario context ($c_t$).
• Output: Prediction of user behavior ($y_t$) in the current scenario.
• Behavior Types:
  • Binary: Click/Like/Share (Classification, measured by F1-score).
  • Continuous: Watch duration (Regression, measured by Normalized Mean Absolute Error - NMAE).
  • Textual: Customer service dialogue responses (Measured by LLM-as-a-judge on intent, persona, knowledge, and alignment).

3. Key Contributions

1. OmniBehavior Benchmark: A high-fidelity, real-world dataset integrating long-horizon, cross-scenario, and heterogeneous behavior traces.
2. Empirical Analysis of Real Users:
   • Cross-Scenario Necessity: Single-scenario data suffers from "tunnel vision," missing 20-30% of user interests. Multi-scenario data is essential for a holistic view.
   • Long-Horizon Causality: Over 80% of conversion paths span multiple scenarios and extend over several days, proving that short-session data causes "causal amputation."
   • Synthetic vs. Real: Synthetic data fails to capture the stochastic, smooth evolution of real user interests, often showing rigid, task-driven spikes.
3. Comprehensive LLM Evaluation: A systematic benchmarking of 12 state-of-the-art LLMs (including Claude-4.5, GPT-5.2, Gemini-3, and open-source models like Qwen3 and DeepSeek-V3).
4. Discovery of Structural Bias: Identification of a fundamental "Positivity-and-Average" bias in current LLM simulators.

4. Experimental Results

A. Performance Gaps

• Overall Scores: Even the best-performing model, Claude-4.5-Opus, achieved only an overall score of 44.55. Most models clustered between 32 and 41.
• Binary Prediction: F1 scores for binary actions (like, share) rarely exceeded 40% for most models.
• Context Window Limitations: Extending context windows from 16k to 128k tokens did not consistently improve performance, indicating current models struggle with long-context reasoning and memory management.
• Memory Management: Standard strategies like RAG (Retrieval-Augmented Generation) and Summarization provided only marginal gains. RAG often disrupted causal dependencies, while summarization lost fine-grained details.

B. Structural Bias Analysis

The paper identifies three critical biases in LLM-generated behaviors compared to real humans:

1. Hyper-activity Bias: LLMs significantly overestimate positive interaction rates (predicting 40-60% more "likes" or "clicks" than humans). They fail to model negative feedback or churn signals.
2. Utopian Bias (Politeness): In customer service dialogues, LLMs default to overly polite, conflict-avoiding language. Real users frequently express anger, frustration, and direct blame, which LLMs suppress due to alignment mechanisms.
3. Persona Homogenization: While real users exhibit distinct behavioral clusters (Inter-user distance $\gg$ Intra-user distance), LLM-simulated users converge toward an "average person," showing high overlap in behavioral vectors and losing individual long-tail characteristics.

5. Significance and Future Directions

• Real-World Applicability: The paper demonstrates that current LLMs are not yet ready for high-fidelity simulation in industrial settings (e.g., churn prediction, crisis management) due to their inability to model negative, long-tail, and cross-scenario behaviors.
• Research Direction: Future work must move beyond simple instruction tuning and focus on:
  • Developing structure-aware memory mechanisms that preserve causal and temporal dependencies.
  • Addressing the positivity bias to allow models to simulate adversarial and negative interactions.
  • Creating training objectives that capture heterogeneity rather than converging to an average.
• Social Science Impact: The findings serve as a cautionary framework, suggesting that without correcting these intrinsic distortions, LLMs will remain "mirrors of our ideals" rather than accurate maps of human reality.

In conclusion, OmniBehavior establishes a new standard for evaluating user simulators, revealing that while LLMs have advanced, they fundamentally struggle with the complexity, noise, and long-term causal structure of authentic human behavior.