Technical Summary: Towards Customized Multimodal Role-Play

Problem Definition

Current personalized virtual character systems operate primarily in single modalities. Text-based models can simulate persona and dialogue but lack visual generation capabilities, while image personalization methods (e.g., DreamBooth) can reproduce visual identities but cannot engage in contextual conversations or react to user inputs. Although recent unified multimodal foundation models have demonstrated strong cross-modal understanding and generation, they have not been effectively adapted for persona-driven interaction that requires simultaneous consistency in language style, emotional expression, and visual identity.

To address this gap, the authors introduce Customized Multimodal Role-Play (CMRP). This task aims to adapt a general-purpose multimodal model into a specific virtual character using minimal data (a textual profile, a few reference images, and example dialogues) to generate coherent, in-character text responses and appearance-consistent images.

Methodology: UniCharacter

The paper proposes UniCharacter, a two-stage training framework designed to achieve few-shot vision-language alignment while maintaining cross-modal consistency.

1. The RoleScape-20 Dataset

To facilitate CMRP, the authors constructed RoleScape-20, the first multimodal role-play dataset. It comprises 20 diverse characters (real-world figures, anime/game characters, and animals). Unlike existing datasets that focus on simple image descriptions or text-only dialogues, RoleScape-20 provides:

• Multimodal Role-Play Data: Paired visual-textual episodes where images align with dialogue, emotion, and personality.
• Fine-grained Annotations: Including explicit "Thinking Processes" (reasoning for image generation), generation instructions, and paired Visual Question Answering (VQA) and Knowledge QA samples.
• Scale: 150–250 role-playing dialogues per character, 5–15 reference images, and ~200 VQA samples per image.

2. Training Framework

UniCharacter employs a two-stage pipeline to adapt unified multimodal models (specifically based on the BAGEL architecture):

• Stage 1: Unified Supervised Fine-Tuning (Unified-SFT)
  This stage treats the problem as multi-task learning. The model is fine-tuned on four sub-tasks using Cross-Entropy (CE) loss for text and Mean Squared Error (MSE) loss for images:

  1. Role-Play Chatting: Generating in-character text.
  2. Thinking Task: Generating a reasoning process for image creation.
  3. Visual Question Answering (VQA): Answering questions based on character images.
  4. Knowledge QA: Answering questions based on character background profiles.
  5. Text-to-Image (T2I) Generation: Generating images from text instructions.

• Stage 2: Character Group Relative Policy Optimization (Character-GRPO)
  To mitigate the visual overfitting and low diversity often seen in SFT-based image generation (which relies on fixed ground-truth images), the authors introduce a reinforcement learning stage specifically for the T2I branch.

  • Mechanism: Instead of a single ground-truth image, the model generates a group of $G$ samples for a given prompt.
  • Reward Function: A comprehensive reward signal ($R_{total}$) guides the policy ($\pi_\theta$), comprising:
    • Text-Image Alignment: Measured via CLIP similarity ($r_{CLIP}$) and VQA consistency ($r_{VQA}$).
    • Diversity: A perceptual diversity reward ($r_{div}$) using LPIPS to encourage variance within the generated group.
    • Similarity Penalty: A penalty ($p_{sim}$) based on DINO feature similarity to the training set to prevent memorization while preserving character identity.
  • Optimization: The model optimizes the advantage function across the group, allowing it to explore diverse visual representations without requiring ground-truth images for every step.

Key Contributions

1. Task Definition: Introduction of CMRP, a new task integrating textual role-playing with multimodal personalization.
2. Dataset: Release of RoleScape-20, the first dataset providing comprehensive multimodal annotations (profiles, dialogues, images, thinking processes, and QA pairs) for deep character customization.
3. Framework: Proposal of UniCharacter, a two-stage framework (Unified-SFT + Character-GRPO) that effectively aligns persona, dialogue style, and visual identity in a few-shot setting.
4. Performance: Demonstration that the proposed method outperforms existing baselines in role consistency, dialogue authenticity, image fidelity, and cross-modal alignment.

Experimental Results

Experiments were conducted on the RoleScape-20 benchmark, comparing UniCharacter against baselines including DreamBooth, UniCTokens, and Qwen2.5-VL.

• Quantitative Performance: UniCharacter achieved superior results across all metrics.
  • Image Generation: Outperformed DreamBooth in CLIP-I, CLIP-T, and DINO scores, indicating better alignment with prompts and character identity.
  • Text Role-Play: Surpassed Qwen2.5-VL (with text prompts) in Memorization, Personality, and Diversity scores (evaluated via LLM-as-Judge).
  • Multimodal Role-Play: Demonstrated higher identity fidelity and better text-image alignment than UniCTokens.
  • QA Tasks: Maintained strong performance in Knowledge QA and VQA, confirming that generative capabilities did not compromise comprehension.
• Ablation Studies:
  • Removing the GRPO stage resulted in lower image diversity and higher similarity to the training set (overfitting).
  • The inclusion of specific reward components (CLIP-T, VQA, Diversity, Similarity Penalty) was shown to be critical for balancing image quality, alignment, and variety.
  • User studies confirmed that UniCharacter was preferred over baselines in T2I generation, multimodal role-play, and text role-play tasks.

Significance and Claims

The paper argues that CMRP, coupled with unified modeling, provides a foundational basis for next-generation characterful and immersive interactive agents. By successfully unifying persona, dialogue style, and visual identity within a single framework, UniCharacter enables the creation of virtual characters that are not only visually consistent but also linguistically expressive and emotionally coherent.

The authors note that their approach allows for character customization with only 10 images and corresponding interaction examples, requiring approximately 100 GPU hours per character. They position this work as a step toward solving the "Holy Grail" of multimodal role-play, where agents can maintain a stable persona across both text and visual modalities without the fragmentation seen in current single-modality systems.

Limitations and Future Directions:
The authors acknowledge that the current task is limited to text and images. Future work is needed to extend this to customized video generation (ensuring temporal consistency) and multi-turn/long-turn dialogues (requiring robust long-term memory to prevent role-drifting). Additionally, real-time deployment, safety controllability, and user-in-the-loop customization are identified as areas for further exploration.