Technical Summary: AffectVerse: Emotional World Models for Multimodal Affective Computing

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) typically treat emotion recognition as a static pattern-matching task over complete audiovisual-text inputs. This paradigm fails to capture the temporal dynamics of affect, such as escalation, recovery, or cross-modal conflicts, reducing complex emotional processes to static snapshots. While humans infer emotions by integrating observed cues with predictions about how affective states may unfold, existing MLLMs lack a mechanism to model these "predictive belief states" under partial observations. The paper identifies a gap between static multimodal recognition and the temporally unfolding nature of human affective inference.

2. Methodology

The authors propose AffectVerse, a model built upon Qwen2.5-Omni that integrates an Emotion World Module (EWM). Unlike traditional world models used in reinforcement learning which predict future states based on actions and rewards, AffectVerse employs an action-free formulation designed specifically for affective inference. The EWM operates on the principle of predictive processing, where the model predicts latent future audiovisual representations from observed prefixes to update the LLM's belief state.

The EWM consists of three core components:

2.1 Cross-Modal Temporal Imagination

Grounded in predictive processing and multisensory integration theories, this module predicts future video and audio representations from past tokens.

• Mechanism: It utilizes a multi-step rollout process ($S=3$) with learnable future query tokens ($N_s=8$).
• Cross-Modal Context: Each modality's imagination branch attends to both its own past tokens and the other modality's past tokens, distinguished by learnable tag embeddings ($e_{self}, e_{cross}$).
• Training Signal: The module is trained via a self-supervised objective where the predicted future tokens are compared against the actual future tokens (detached from the main gradient flow) using a combination of MSE and Cosine similarity losses. This forces the model to learn transition regularities without requiring future data at inference time.

2.2 MAMA (Modality-Aware Multi-step Attention) Belief Aggregation

Inspired by Scherer's Component Process Model, this module aggregates the multi-step imagined tokens into compact "belief tokens."

• Process: It assembles imagined video and audio tokens into a memory bank, distinguished by type embeddings.
• Dynamic Capacity: The number of belief query tokens ($N_q$) adapts to the available modalities (8 for dual-modality, 4 for single-modality).
• Boundary Injection: To preserve context at the temporal split point, the aggregated belief tokens are augmented with a residual injection of the last observed "boundary tokens" ($v_b, a_b$) from the past.

2.3 Belief Injection

The resulting compact belief tokens are projected back into the LLM's hidden dimension and interleaved into the token sequence at two specific positions:

1. Immediately after the last audiovisual token (to enrich text encoding with affective context).
2. Immediately before the answer region (to provide an "emotional prior" for response generation).
   This strategy ensures the LLM can attend to both observed evidence and imagined affective continuations during reasoning.

2.4 Training Strategy

• Temporal Splitting: During training, a random keep ratio $\kappa \sim \text{Uniform}[0.7, 1.0)$ is applied to truncate future tokens, simulating partial observations.
• Modality Dropout: A modality is randomly dropped with probability $p=0.15$ to improve robustness to missing sensory evidence.
• Objectives: The model is trained end-to-end with two losses: Language Modeling Loss ($L_{lm}$) for task performance and Imagination Loss ($L_{imagine}$) for predictive representation learning, balanced with $\lambda_{img}=1.0$.

3. Key Contributions

1. Action-Free World Model for Affect: A novel adaptation of world-model-style predictive representation learning to affective computing, removing the need for actions, rewards, or control policies.
2. Emotion World Module (EWM): A specific architecture comprising Cross-Modal Temporal Imagination, MAMA Belief Aggregation, and Belief Injection that enables MLLMs to reason over latent affective dynamics.
3. Predictive Belief-State Modeling: Demonstrating that formulating emotion understanding as predictive belief-state modeling under partial observations yields superior performance compared to static fusion approaches.
4. Comprehensive Evaluation: Extensive experiments across nine benchmarks (MER2023/2024, MELD, IEMOCAP, MOSI, MOSEI, SIMS, OV-MERD+) showing consistent improvements.

4. Experimental Results

AffectVerse was evaluated on nine benchmarks under three modality settings (Audio+Text, Video+Text, Audio+Video+Text).

• Main Results: In the full Audio+Video+Text setting, AffectVerse achieved a mean score of 77.34%, outperforming the strong Qwen2.5-Omni baseline (74.76%) and AffectGPT (74.77%) by +2.57%. It ranked first on 8 out of 9 benchmarks.
• Robustness: The model demonstrated graceful degradation under temporal truncation (varying $\kappa$) and missing modalities. Notably, moderate temporal truncation during training acted as a regularizer, sometimes improving performance over full-observation training on specific datasets (e.g., CMU-MOSI).
• Ablation Studies:
  • Temporal Prediction: Adding temporal splitting alone improved the baseline by +0.87%.
  • Cross-Modal Imagination: This provided the largest single gain (+1.00% over temporal splitting), highlighting the importance of leveraging complementary cross-modal cues.
  • Belief Construction: The MAMA aggregation, interleaved injection, and residual boundary injection contributed additive gains, confirming the necessity of the full EWM pipeline.
  • Hyperparameters: Optimal performance was found at $S=3$ rollout steps and a modality dropout probability of $p=0.15$.

5. Significance and Claims

The paper claims that predictive belief-state modeling is a practical and effective alternative for affective computing. By treating emotion not as a fixed perceptual category but as a temporally unfolding process shaped by predictive inference, AffectVerse allows models to update their understanding as new evidence arrives, separating transient cues from final affective states.

The authors position this work as a bridge between affective science (specifically predictive processing and appraisal theories) and multimodal deep learning. They assert that their "action-free" world module successfully encodes transition cues predictive of subsequent affective change, enabling more robust and dynamic emotion recognition without requiring unseen signals at inference time.

Limitations Acknowledged:
The authors note that the current evaluation relies on fixed rollout and keep-ratio hyperparameters. The robustness analysis focuses on modality and temporal partial observations rather than broader domain shifts (e.g., noise, accent, topic changes). Future work is suggested to extend predictive belief modeling to stronger alignment, domain-shift evaluation, and dialogue-level emotion tracking.