Technical Summary: SocialPersona

Problem Statement

Current benchmarks for personalized language assistants predominantly evaluate a model's ability to recall preferences explicitly stated within dialogue histories. This "memory lens" fails to capture the more complex, real-world capability required of personalized assistants: inferring implicit user interests from the multimodal, longitudinal traces users naturally leave behind on social media. In practice, preferences are revealed indirectly through what users create, share, photograph, and repeatedly engage with. Recovering these "revealed preferences" presents a significant challenge for Multimodal Large Language Models (MLLMs), as it requires aggregating sparse, noisy, and unstructured evidence across text, images, and timestamps over long time horizons, while distinguishing between stable, recurring interests and recent, emerging fixations.

Methodology: The SOCIALPERSONA Benchmark

To address this gap, the authors introduce SOCIALPERSONA, a benchmark designed to evaluate MLLMs on inferring user profiles from multimodal social-media timelines and utilizing those profiles in dialogue.

Data Construction

• Source: The benchmark is built from the real, longitudinal timelines of 171 everyday, non-promotional social-media users. It excludes celebrities, brand accounts, and highly curated profiles to ensure natural, self-expressive preference traces.
• Scale: The dataset comprises an average of 176.81 posts and 130.38 images per user, spanning up to two years.
• Content: Each post includes text, visual content (images/video frames), and timestamps. All text is standardized to English.
• Gold Profiles: A semi-automated pipeline followed by rigorous human verification (350 annotator-hours) generated 2,597 preference tags. These tags are grounded in textual, visual, and temporal evidence across seven interest domains:
  1. Sports and outdoor activities
  2. Entertainment
  3. Gaming
  4. Food and drink
  5. Travel and city exploration
  6. Photography and creation
  7. Pets
• Temporal Granularity: Profiles explicitly separate stable interests (recurring patterns across the timeline) from recent interests (signals concentrated in the last 90 days).

Evaluation Tasks

SOCIALPERSONA evaluates MLLMs on two primary tasks:

1. Profile Construction: Models must infer active interest domains and fine-grained interest tags (distinguishing stable vs. recent) directly from raw multimodal timelines.
2. Personalized Dialogue Generation: Models receive a user's social-media context (either raw timeline or a constructed profile) and a natural user request. They are evaluated on whether their responses align with the user's stable or recent interests.

The evaluation employs both proprietary (e.g., GPT-4o-mini, GPT-5.4, Gemini-2.5-Flash) and open-weight (e.g., Qwen series) MLLMs. Dialogue quality is judged on interest coverage, concreteness, and fluency.

Key Contributions

1. Task Formulation: A new formulation challenging MLLMs to infer preferences from longitudinal, multimodal behavior and apply these in downstream dialogue, moving beyond simple memory retrieval.
2. Benchmark Release: The construction and public release of SOCIALPERSONA, featuring real-user data, long-horizon timelines, multimodal evidence, and human-validated profiles across seven domains.
3. Empirical Analysis: A comprehensive evaluation revealing specific capability gaps in current MLLMs regarding cross-modal evidence aggregation and temporal reasoning.

Key Results and Findings

Experiments across various models and settings (Direct, Hierarchical, and Extractive–Abstractive) reveal three central limitations in current MLLMs:

1. Systematic Over-Generalization: The dominant failure mode is the collapse of specific, fine-grained interests into broad, generic categories. For example, models often reduce 4–7 specific gold tags (e.g., "dining out," "birthday cake," "confectionery") into just 1–2 broad labels (e.g., "restaurants," "sweets"). This gap persists even when evaluation prompts are adjusted to be less conservative, indicating a genuine model limitation in fine-grained preference elicitation.
2. Modality Asymmetry: Models exhibit a strong bias toward visual evidence. Domains with visually concentrated signals (e.g., pets, gaming screenshots) are detected with high accuracy. Conversely, domains relying on dispersed textual evidence (e.g., travel, food/drink habits) are frequently missed or incorrectly marked as inactive. This bias propagates into dialogue, where models prioritize visually dominant interests over text-supported ones.
3. Temporal Reasoning Deficits: Models struggle to distinguish between stable and recent interests. They tend to assign nearly all predicted tags to the "stable" bucket, regardless of the temporal distribution of evidence. Even models with high overall performance fail to compute temporal dispersion patterns, treating timelines as flattened sequences rather than time-ordered evidence.

Profile-Dialogue Correlation: The study confirms a positive correlation between profile quality (Interest F1) and dialogue quality. However, because current models generate sparse or over-generalized profiles, these errors compound downstream, leading to reduced coverage and concreteness in personalized responses.

Significance

The paper posits that robust cross-modal, long-horizon user modeling remains a key unsolved challenge for personalized assistants. SOCIALPERSONA serves as a critical tool to measure progress in this area. The results indicate that while current models can identify broad interest domains, they lack the nuance to recover revealed preferences from noisy, multimodal traces and apply them effectively in dialogue. The benchmark highlights that future advancements must address fine-grained interest recovery, modality balancing, and temporal reasoning to move beyond simple memory-based personalization toward truly insight-driven assistants.