Technical Summary: Group Preference Collapse in Personalized Multimodal Large Language Models

1. Problem Definition: Group Preference Collapse

The paper identifies a critical failure mode in multi-user personalized Multimodal Large Language Models (MLLMs) termed Group Preference Collapse. While existing personalized MLLMs aim to generate user-specific responses, they currently rely heavily on stable profile information (e.g., demographics, portraits) and often overlook diverse, context-dependent user preferences.

When trained on multiple users with imbalanced preference distributions, these models exhibit two key failures:

1. Suppressed Representation: During user representation learning, sparse and low-frequency individual preferences are absorbed by dominant population-level patterns.
2. Unreliable Generation: During inference, even when preference information is provided as a condition, the model fails to select the relevant preference cue for the specific image-question pair. Instead, it defaults to generic or population-dominant reasoning patterns.

Consequently, the model drifts toward broadly plausible group-level choices (e.g., predicting "Yoga" for all users regardless of individual preference) rather than reflecting fine-grained, user-specific choices.

2. Methodology: PrefMoE

To address these failures, the authors propose PrefMoE, a preference-centric framework designed to separate stable profile information from preference-related representations and ensure their effective use during generation. The method consists of three core components:

A. Factorized User Representation

PrefMoE decomposes user information into distinct factors rather than compressing them into a single embedding:

• Profile Factors: Learned from stable user data (images and textual descriptions) using contrastive learning to align modalities and separate users.
• Preference Factors: Represented as a decomposition of shared prototypes (population-level tendencies) and personalized residuals (user-specific deviations).
  • Formula: $z^{pref}_{i,f} = \bar{z}_f + \Delta_{i,f}$, where $\bar{z}_f$ is the shared prototype for facet $f$, and $\Delta_{i,f}$ is the user-specific residual.

B. Residual-Based Preference Learning

To prevent minority preferences from being absorbed by shared patterns, the framework applies three regularization strategies to the personalized residuals ($\Delta_{i,f}$):

1. Imbalance-Aware Residual Contrast: A contrastive loss that pulls together users with the same facet-level preference under a specific visual-question context. It employs a reweighting mechanism (inspired by focal loss) to assign higher weights to low-frequency preference groups, ensuring they receive sufficient supervision.
2. Counterfactual Pseudo-User Augmentation: To break spurious correlations between user groups and dominant preferences, the method constructs "pseudo-users" by recombining preference entries from different real users across facets. These pseudo-users are used in the contrastive objective to encourage residuals to align with preference semantics rather than original user-group identities.
3. Preference Residual Decorrelation: Soft constraints are imposed to ensure residuals across different facets are not redundant and that residuals capture deviations beyond the shared prototype, preventing entanglement.

C. Factorized Hierarchical MoE Routing

To ensure these learned factors are utilized during generation, PrefMoE introduces a hierarchical Mixture-of-Experts (MoE) router based on LoRA (Low-Rank Adaptation):

• Separate Experts: Profile factors (image and description) and preference factors (facet-specific) are routed to distinct LoRA experts.
• Query-Dependent Routing:
  • A Preference Router selects relevant preference facets based on the current query.
  • A Profile Router selects the most useful profile source (image vs. description).
  • A Branch Router adaptively fuses the profile-aware and preference-aware updates based on the query representation.
    This structure ensures that the model dynamically selects the appropriate user factors for the specific visual question, rather than relying on a single mixed condition.

3. Key Contributions

The paper makes three primary contributions:

1. Identification of Failure Mode: It formally defines "group preference collapse" as a critical issue where dominant population preferences suppress individual signals, even when explicit user information is available.
2. PrefMoE Framework: It proposes a novel framework that combines residual-based preference factor learning with factorized hierarchical routing. This approach preserves individualized preference offsets and activates profile- and preference-aware adaptation paths dynamically.
3. Empirical Validation: Extensive experiments across multiple MLLM backbones (LLaVA, DeepSeek-VL, Qwen) demonstrate that PrefMoE consistently improves preference-sensitive personalization and significantly reduces preference collapse compared to full fine-tuning and other personalized baselines.

4. Experimental Results

Experiments were conducted on MMPB-Clean, a curated split of the MMPB benchmark containing multimodal queries over 50 users and six preference facets (entertainment, travel, lifestyle, shopping, fashion).

• Performance Gains: On the LLaVA-1.5-7B backbone, PrefMoE improved preference accuracy from 44.13% (Full Fine-Tuning) to 67.33% and reduced the preference collapse rate from 34.25% to 12.33%.
• Scalability: The method achieved the best overall accuracy and lowest collapse rates across various model scales, including LLaVA-OV-72B (reaching 78.93% preference accuracy and 10.96% collapse).
• Ablation Studies:
  • Removing the hierarchical MoE router resulted in significant performance drops, confirming the necessity of factorized computation paths.
  • Imbalance-aware contrast and counterfactual augmentation were crucial for reducing collapse and improving accuracy on low-frequency preferences.
  • The method remained robust even without explicit facet labels, suggesting it learns structured preference relations from underlying descriptions.
• Qualitative Analysis: Visualizations (t-SNE) showed that PrefMoE maintains clearer boundaries between preference populations and facets compared to standard fine-tuning, which tends to mix them. Case studies demonstrated that PrefMoE successfully grounds answers in specific user preferences (e.g., bohemian fashion, solo travel) where baselines defaulted to generic or dominant choices.

5. Significance and Limitations

The paper argues that effective personalized reasoning requires more than just scaling models or standard fine-tuning; it necessitates factorized adaptation that explicitly separates and preserves user-specific preference signals. PrefMoE demonstrates that by decoupling stable profiles from dynamic preferences and routing them through specialized adaptation paths, models can better reflect individual user choices.

Modest Claims and Limitations:
The authors acknowledge that while PrefMoE significantly mitigates preference collapse, it still relies on explicit and well-specified preference descriptions. The method may struggle when:

• User preferences are underspecified.
• Preferences are weakly grounded in the visual scene.
• The question does not clearly indicate the relevant preference facet.

The work does not claim to solve all personalization challenges but offers a structured approach to addressing the specific failure mode of group preference collapse in multi-user settings.