💬 NLP

Large Emotional World Model

This paper introduces the Large Emotional World Model (LEWM), a novel framework that integrates human emotion as a core state variable to predict future emotional and physical states, demonstrating significant performance improvements in world-state prediction, emotion understanding, and general reasoning tasks.

Original authors: Changhao Song, Yazhou Zhang, Hui Gao, Chang Yang, Peng Zhang

Published 2026-07-30

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Changhao Song, Yazhou Zhang, Hui Gao, Chang Yang, Peng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. If you only show it the laws of physics, it learns that if it hits a brick wall, the car stops. That's the "physical world." But if you want that robot to drive in a city full of people, physics isn't enough. It needs to understand that if a driver cuts them off, the other driver might get angry, slam the brakes, or scream. If the robot doesn't predict that emotion, it can't predict what happens next. This is the big question in a field called "World Models." Scientists are trying to build digital brains that don't just see what is happening right now, but can simulate how the future will unfold. For a long time, these digital brains were like physics engines: great at moving blocks and balls, but terrible at understanding why humans do what they do. They missed the secret ingredient: feelings.

This paper introduces a new kind of digital brain called the Large Emotional World Model (LEWM). The researchers, Changhao Song and his team from Tianjin University, realized that to predict the future in a human world, you have to predict feelings first. They built a massive new dataset called EWH (Emotion-Why-How), which is like a library of 10,850 short video stories. Each story shows a person, what they were feeling before, what they did, how their feelings changed, and what happened next. Think of it as a "Choose Your Own Adventure" book where the plot twists depend entirely on whether the character is happy, angry, or sad.

Using this library, they trained LEWM to think in two steps, like a detective solving a mystery. First, the model asks, "Given what just happened, how will this person feel next?" It predicts the emotion. Second, it asks, "Now that I know how they feel, what will they do next?" By putting the emotion in the middle, the model can see that the same traffic accident could lead to a calm conversation if the driver is happy, or a shouting match if they are angry. The results are impressive: on their new emotional dataset, LEWM got up to 45.72% more accurate than previous models. It also got better at understanding emotions in general (improving by 17.47% on one test) and even got smarter at answering tricky questions about politics and science without losing its ability to understand the physical world. The paper suggests that by teaching AI to care about feelings, we aren't just making it nicer; we are making it much better at predicting how the real human world actually works.

Technical Summary: Large Emotional World Model (LEWM)

Problem Statement

Current world models are primarily designed to learn physical laws governing state transitions (e.g., object motion, spatial continuity). While effective for simulating the physical world, they fail to capture the "human world," where actions, decisions, and interactions are deeply shaped by internal affective dynamics. Existing models rarely treat emotion as an explicit causal factor in state transitions. Consequently, they cannot accurately predict future states in human-centered environments where the same physical event (e.g., a traffic accident) can lead to divergent trajectories depending on the agent's emotional state (e.g., cooperative resolution vs. hostile confrontation).

Methodology

1. Dataset Construction: Emotion-Why-How (EWH)

To address the lack of data for emotional state transitions, the authors constructed EWH, the first world model dataset centered on emotional dynamics.

Data Sources: 2,258 first-person videos from TikTok/Douyin and 1,583 episodes from eight TV series (e.g., Friends, The Office), covering diverse settings like domestic, workplace, and travel.
Structure: The dataset contains 10,850 emotion-aware transition tuples. Each tuple is formalized as $T = \langle(S_0, E_0), H, (S_1, E_1)\rangle$ $T = ⟨(S_{0}, E_{0}), H, (S_{1}, E_{1})⟩$ , where:
- $S_0, S_1$ : Observable states (synchronized video, audio, image).
- $E_0, E_1$ : Emotional states (labeled as neutral, happiness, sadness, anger, fear, surprise, disgust).
- $H$ : The intervening behavior trajectory.
Why-How Logic: The structure explicitly links the "Why" (affective reason for behavior) and "How" (effect of emotional transition on future state evolution).
Pipeline: A four-stage pipeline involving video segmentation, emotion-aware transition mining (using Qwen3.5-VL and GPT-5), sample construction, and rigorous quality control (automatic filtering + human verification).

2. Model Architecture: Large Emotional World Model (LEWM)

LEWM is a multimodal encoder–LLM–multimodal decoder framework that factorizes future prediction into two coupled steps:

Multimodal World-State Encoding: Observable states ( $V, A, I$ ) and current emotion ( $E_t$ ) are encoded via modality-specific encoders and projected into the LLM embedding space. Behavior trajectories ( $H_t$ ) are tokenized as text.
Emotion-Aware Transition Modeling (Factorized Prediction):
- Step 1 (Emotion Prediction): The model first predicts the future emotional state ( $\hat{E}_{t+1}$ ) given the current context: $p(E_{t+1} | S_t, E_t, H_t)$ .
- Step 2 (State Prediction): The predicted emotion is used as a conditioning signal to guide the prediction of the future world state: $p(S_{t+1} | S_t, E_t, H_t, \hat{E}_{t+1})$ .
- This design assumes future states are mediated by affective transitions, not just external observations.
Autoregressive Generation: The model uses dedicated generation tokens ([IMG], [VID], [AUD]) to autoregressively generate future multimodal states.
Optimization: A multi-task loss function jointly supervises future emotion classification, autoregressive token generation, and multimodal future-state prediction (via embedding-space distance).

Key Contributions

Formulation of Affective World Modeling: The paper introduces the concept of integrating emotion as a key state variable in world models to capture both physical state evolution and affective causes.
EWH Dataset: The creation of the first world model dataset explicitly centered on emotional state transitions, bridging the gap between static emotion recognition and dynamic world modeling.
LEWM Architecture: The proposal of a factorized prediction framework that conditions world-state prediction on predicted emotional states, demonstrating that this approach improves performance without sacrificing general reasoning capabilities.

Experimental Results

World-State Prediction

EWH (Emotional World): LEWM achieved the best performance on 10 of 12 tasks. Compared to the baseline WorldGPT, it improved video prediction by up to 45.73% and audio prediction by up to 24.75%.
WorldNet (Physical World): LEWM showed consistent gains (up to 3.94%) even in physical-world settings, particularly in audio prediction, suggesting emotional dynamics provide complementary cues for physical transitions.
Ablation: Removing emotion conditioning ("No Emotion") consistently degraded performance. Using "Oracle Future" (ground-truth emotion) yielded the highest gains, indicating significant potential for further improvement with more accurate emotion prediction.

Emotion Understanding

MELD: LEWM achieved a 17.47% improvement in Weighted-F1 on sentiment analysis and significant gains on fine-grained emotion recognition.
Generalization: The model outperformed its backbone on the challenging 28-class GoEmotions benchmark, indicating that learning emotional transitions yields transferable emotional representations.

General Reasoning

Benchmarks (MMLU, HellaSwag, VQAv2, NExT-QA): LEWM remained comparable to the baseline WorldGPT across general reasoning tasks. Notably, it achieved a 6.10% gain in Political Science (MMLU) and 5.95% in Animal recognition (VQAv2), demonstrating that emotional modeling does not harm, and may even enhance, general reasoning.

Counterfactual Analysis

Intervention experiments (replacing the input emotion while holding state and behavior constant) showed that LEWM produces structured, non-zero changes in future state predictions across modalities. High-arousal emotions induced larger divergences, confirming the model treats emotion as an active variable in future-state prediction.

Significance and Claims

The paper claims that incorporating emotion into world models enables a more realistic simulation of human-centered environments. By moving beyond physical state prediction to model affective dynamics, LEWM expands the predictive understanding of intelligent agents. The work establishes a foundation for "socially grounded agents" capable of better modeling human behavior, interaction, and decision-making. The authors emphasize that their approach does not merely classify emotions statically but models the dynamic evolution of emotion as a causal driver of future world states.